Cardano-db-sync: DB lookup fail in insertABlock (erikd/shelley-32byte-address-hash)

This is a work-in-progress. There isn't even a PR yet. Some of the commits comments are "fixup" and "wip". You are building and running a branch I am still working on.

Let me be clear. Raising tickets on branches which have "wip" or "fixup" commits is a complete waste of time.

Hey @erikd I've been hit by this problem twice today [0][1] and I think there is a related problem about this: main thread of the process freezes but won't exit >0; so for example, my docker container is not automatically respawned.

JFTR, I'm using cardano-db-sync:5.0.1 official docker image.

[0]

[db-sync-node:Info:3354] [2020-09-26 04:29:44.87 UTC] insertByronBlock: slot 3025000, block 3023467, hash 43b510c9fa0d1021d4efbfb0a07c1e90f40258c05d7d66b266aee4fb9963f678
[db-sync-node:Error:3354] [2020-09-26 04:29:59.56 UTC] DB lookup fail in insertABlock: block hash f8452c44591e3db7d5534ceaaff56922609d81deed45b953e7220601bfd4ec87
[db-sync-node:Info:3354] [2020-09-26 04:29:59.56 UTC] Shutting down DB thread
[db-sync-node:Error:3357] [2020-09-26 04:29:59.56 UTC] recvMsgRollForward: AsyncCancelled
...
FROZEN FOREVER
...

[1]

Generating PGPASS file
Connecting to network: mainnet
[db-sync-node:Info:4] [2020-09-26 09:01:38.91 UTC] NetworkMagic: 764824073
[db-sync-node:Info:4] [2020-09-26 09:02:11.59 UTC] Initial genesis distribution present and correct
[db-sync-node:Info:4] [2020-09-26 09:02:11.59 UTC] Total genesis supply of Ada: 31112484745.000000
[db-sync-node:Info:4] [2020-09-26 09:02:11.69 UTC] Inserting Shelley Genesis distribution
[db-sync-node:Info:4] [2020-09-26 09:02:11.83 UTC] epochPluginOnStartup: Checking
[db-sync-node:Info:4] [2020-09-26 09:02:11.86 UTC] localInitiatorNetworkApplication: connecting to node via "/node-ipc/node.socket"
[db-sync-node.Handshake:Info:30] [2020-09-26 09:02:11.86 UTC] [String "Send MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073)])",String "LocalHandshakeTrace",String "ConnectionId {localAddress = LocalAddress {getFilePath = \"\"}, remoteAddress = LocalAddress {getFilePath = \"/ipc/node.socket\"}}"]
[db-sync-node.Handshake:Info:30] [2020-09-26 09:02:11.87 UTC] [String "Recv MsgAcceptVersion NodeToClientV_3 (TInt 764824073)",String "LocalHandshakeTrace",String "ConnectionId {localAddress = LocalAddress {getFilePath = \"\"}, remoteAddress = LocalAddress {getFilePath = \"/ipc/node.socket\"}}"]
[db-sync-node:Info:34] [2020-09-26 09:02:11.87 UTC] Starting chainSyncClient
[db-sync-node:Info:34] [2020-09-26 09:02:13.97 UTC] Cardano.Db tip is at slot 3025325, block 3023792
[db-sync-node:Info:39] [2020-09-26 09:02:13.97 UTC] Running DB thread
[db-sync-node:Info:39] [2020-09-26 09:02:14.45 UTC] Rolling back to slot 3025325, hash f8452c44591e3db7d5534ceaaff56922609d81deed45b953e7220601bfd4ec87
[db-sync-node:Info:39] [2020-09-26 09:02:14.46 UTC] Deleting blocks numbered: []
[db-sync-node:Info:42] [2020-09-26 09:02:14.58 UTC] getHistoryInterpreter: acquired
[db-sync-node:Error:39] [2020-09-26 09:02:42.88 UTC] DB lookup fail in insertABlock: block hash ff6d511e65fb979ac511e8658c20cfe608bdfe7d3e6172f49115e52487812423
[db-sync-node:Info:39] [2020-09-26 09:02:42.88 UTC] Shutting down DB thread
[db-sync-node:Error:42] [2020-09-26 09:02:42.89 UTC] recvMsgRollForward: AsyncCancelled
...
FROZEN FOREVER
...

If you are seeing this on version 5.0.1 you should open a new ticket because this is an old issue raised against an old work-in-progress branch,