This incident has been resolved.

Root cause has been identified. If you are a node operator with db_backend = "goleveldb" and app-db-backend = "" set in app.toml, please update your config, specifying app-db-backend = "pebbledb" instead.

This issue stems from our use of the types.DBbackend compile-time variable, which was removed in Cosmos SDK v0.47. The value derived for app-db-backend on nodes that do not specify a value effectively changed between ZetaChain v18 and ZetaChain v19. Explicitly setting app-db-backened = "pebbledb" should enable goleveldb based nodes to return to service.

Block production has resumed for the Athens Testnet.

Root cause is not known, but appears to be isolated to nodes with "db_backend = goleveldb" set in config.toml.

CrossChain transactions are currently blocked, but will resume within the next few hours after testing is completed.

The network has just under the 2/3s majority needed to restart. No root cause for failed upgrades on some validators has been identified. We have made contact with several validator operators and are actively working to them with them to restore their nodes.

We are continuing to investigate the root cause of the failed upgrades.

The network has just under the 2/3s majority needed to restart. No root cause for failed upgrades on some validators has been identified. We are continuing to investigate.