Cardano-db-sync: Running db sync with Shelley network

Shelley support is currently a work-in-progress. The code has not been finalized, nor has the database schema.

So yes, it is bsaically possible to connect it to the Shelley Haskell testnet, but it is completely undocumented. Basically I do this in two screen sessions:

# For cardano-node
git checkout 1.14.2 -b tag-1.14.2
nix-build -A scripts.shelley_testnet.node -o shelley-testnet-node
./shelley-testnet-node

and for db-sync:

export PGPASSFILE=config/pgpass-shelley-testnet
cabal run cacardano-db-sync-extended -- --config config/testnet-config.yaml --genesis-file /nix/store/hih30xck46bw2l4mlz6b36dsdnd2cwh4-shelley-
testnet-genesis.json --socket-path ../cardano-node/state-node-shelley_testnet/node.socket --schema-dir schema/

Obviously all of this is officially unsupported until its actually close to complete.

There may be more info here: https://github.com/cardano-community/guild-operators

what's the difference between cardano-db-sync-extended and cardano-db-sync ?

what's the difference between cardano-db-sync-extended and cardano-db-sync ?

Its in the top level README.

@refi93 Please feel free to close this if you question has been answered.

@erikd Are you sure that's the correct command? Even after correcting "cacardano-db-sync-extended", I'm getting

cabal: unrecognized 'run' option '--genesis-file'

unrecognized 'run' option '--socket-path'

unrecognized 'run' option '--schema-dir'

Sorry, insert an extra -- after cardano-db-sync-extended (instructions above updated).

Cabal is assuming that options for cardano-db-sync-extended are options for cabal.

@erikd
cabal run cardano-db-sync-extended -- --config config/testnet-config.yaml --genesis-file /nix/store/hih30xck46bw2l4mlz6b36dsdnd2cwh4-shelley- testnet-genesis.json --socket-path ../cardano-node/state-node-shelley_testnet/node.socket --schema-dir schema/

Gives

Linking <path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-db-sync-extende
d-2.0.0/x/cardano-db-sync-extended/build/cardano-db-sync-extended/cardano-db-sync-extended ...
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/buil
d/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):rrMw_info: error: undefined reference to 'crypto_vrf_proofbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/buil
d/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):rrMx_info: error: undefined reference to 'crypto_vrf_publickeybytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/buil
d/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):rrMy_info: error: undefined reference to 'crypto_vrf_secretkeybytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/buil
d/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):rrMz_info: error: undefined reference to 'crypto_vrf_seedbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/buil
d/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):rrMA_info: error: undefined reference to 'crypto_vrf_outputbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/buil
d/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_zdfVRFAlgorithmPraosVRFzuzdc
seedSizzeVRF_info: error: undefined reference to 'crypto_vrf_seedbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/buil
d/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_zdfVRFAlgorithmPraosVRFzuzdc
sizzeOutputVRF_info: error: undefined reference to 'crypto_vrf_outputbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/buil
d/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_vrfKeySizzeVRF_info: error: 
undefined reference to 'crypto_vrf_outputbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_unsafeRawSeed2_info: error: undefined reference to 'crypto_vrf_seedbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_verKeySizzeVRF_info: error: undefined reference to 'crypto_vrf_publickeybytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_signKeySizzeVRF_info: error: undefined reference to 'crypto_vrf_secretkeybytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_certSizzeVRF_info: error: undefined reference to 'crypto_vrf_proofbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_verify1_info: error: undefined reference to 'crypto_vrf_outputbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):ss0l_info: error: undefined reference to 'crypto_vrf_verify'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_zdfVRFAlgorithmPraosVRF4_info: error: undefined reference to 'crypto_vrf_seedbytes'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_skToSeed_info: error: undefined reference to 'crypto_vrf_sk_to_seed'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_skToVerKey_info: error: undefined reference to 'crypto_vrf_sk_to_pk'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):cardanozmcryptozmpraoszm2zi0zi0zminplace_CardanoziCryptoziVRFziPraos_keypairFromSeed_info: error: undefined reference to 'crypto_vrf_keypair_from_seed'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):ss5e_info: error: undefined reference to 'crypto_vrf_prove'
<path-to-dir>/cardano-db-sync/dist-newstyle/build/x86_64-linux/ghc-8.6.5/cardano-crypto-praos-2.0.0/build/libHScardano-crypto-praos-2.0.0-inplace.a(Praos.o):ss6c_info: error: undefined reference to 'crypto_vrf_proof_to_hash'
collect2: error: ld returned 1 exit status
`gcc' failed in phase `Linker'. (Exit code: 1)

Any hints?

First apt install libsodium libsodium-dev.

Then add the following tocabal.project.local:

package cardano-crypto-praos
  flags: -external-libsodium-vrf

Sorry this was not documented because this is not yet ready for release.

@erikd According to the commands you provided to run cardano-node, it no longer syncs and gives these errors:

[ip-172-3:cardano.node.DnsSubscription:Error:23963] [2020-07-15 10:07:22.94 UTC] [String "Application Exception: 3.128.147.191:3001 HeaderError (At (Block {blockPointSlot = SlotNo {unSlotNo = 0}, blockPointHash = ShelleyHash {unShelleyHash = HashHeader {unHashHeader = 600137a3495e6f2c5bde81ecd81105086c55af51c1b4587985b92d46bb26b0e4}}})) (HeaderProtocolError [[OverlayFailure (WrongGenesisColdKeyOVERLAY (KeyHash e47b777efea3146b1aa18fde95ddbd3de38f068f529997ff4141e484) (KeyHash 17d2ab1bd33faeb975ce82f531cffb4c956acf14fc5fe62a30209b55)),OverlayFailure (OcertFailure (NoCounterForKeyHashOCERT (KeyHash e47b777efea3146b1aa18fde95ddbd3de38f068f529997ff4141e484)))]]) TipGenesis (Tip (SlotNo {unSlotNo = 169612}) (ShelleyHash {unShelleyHash = HashHeader {unHashHeader = d5a2570da3b552b0761cf87f99f1d503e5e443482cb8bbb93cce9a42add0fe88}}) (BlockNo {unBlockNo = 8348}))",String "SubscriptionTrace",String "\"relays-new.shelley-testnet.dev.cardano.org\""]
[ip-172-3:cardano.node.ErrorPolicy:Warning:56] [2020-07-15 10:07:22.94 UTC] [String "ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (HeaderError (At (Block {blockPointSlot = SlotNo {unSlotNo = 0}, blockPointHash = ShelleyHash {unShelleyHash = HashHeader {unHashHeader = 600137a3495e6f2c5bde81ecd81105086c55af51c1b4587985b92d46bb26b0e4}}})) (HeaderProtocolError [[OverlayFailure (WrongGenesisColdKeyOVERLAY (KeyHash e47b777efea3146b1aa18fde95ddbd3de38f068f529997ff4141e484) (KeyHash 17d2ab1bd33faeb975ce82f531cffb4c956acf14fc5fe62a30209b55)),OverlayFailure (OcertFailure (NoCounterForKeyHashOCERT (KeyHash e47b777efea3146b1aa18fde95ddbd3de38f068f529997ff4141e484)))]]) TipGenesis (Tip (SlotNo {unSlotNo = 169612}) (ShelleyHash {unShelleyHash = HashHeader {unHashHeader = d5a2570da3b552b0761cf87f99f1d503e5e443482cb8bbb93cce9a42add0fe88}}) (BlockNo {unBlockNo = 8348}))))) 200s 200s",String "ErrorPolicyTrace",String "3.128.147.191:3001"]

How can we fix this?

By the way, do you expect major changes in the db schema?

No more major changes to schema are expect, but minor changes are still likely.

As for the problem you see that is a node problem, not a db-sync problem. It is highly likely due to a mismatch between the node version and the network version. This is still a work on progress. It is still changing all the time.

~No more major changes to schema are expected~.

I just realized that at least one whole new table will need to be added.

@erikd What table would it be?

There will need to be a table to hold rewards from the ledger state before they are withdrawn.

@erikd Is there a branch that's compatible with 1.16.x of cardano node? If so, how can I start it?

We currently have too many test networks, too many changes in flight and configuration is more difficult than it should be.

It is almost certain we do not have something that supports release/1.16.x in a painless way.

However, if you look at the node configuration and it says Protocol: RealPBFT then master should work. If on the other hand it says Protocol: Cardano then tjhe erikd/hfc-5 branch should work but it will abort on the first pool forged block (because that is what I am waiting for so I can implement #195 ).