The Athens network continues to function well, and we are concluding this response effort.
Posted Sep 19, 2024 - 02:08 UTC
Update
Thank you everyone who participated in restoring the Testnet with v20.0.1 and updating their mainnet nodes to version 19.1.7. We know the update process was unusual and appreciate everyone’s patience and participation.
Nodes on the Athens Testnet began failing when they encountered a bug in the Staking module. This bug was triggered when a validator unbonded their self-bonded tokens in a very specific way. We identified the source of this bug and realized the ZetaChain Mainnet network was also vulnerable to it. Because any user (or attacker) could purposely replicate this bug and stop the ZetaChain Mainnet we decided to engage in our private patching process.
In our private patching process we distribute a patched version of the binaries and source code directly to 68% of the validators (by voting power) before sharing the source code publicly. This is the same process we would use if there were a security vulnerability that needed to be patched before it could be safely disclosed publicly.
We appreciate the efforts of all who responded during the event and we thank them for their help maintaining a reliable and secure network.
Posted Sep 18, 2024 - 20:17 UTC
Update
We are continuing to monitor for any further issues.
Posted Sep 17, 2024 - 22:56 UTC
Monitoring
Block production has resumed on the ZetaChain Athens Testnet.
We have released v20.0.1 binaries which has restored block production. If you are an impacted node operator, please apply the update as soon as possible.
If you are an NDA'd partner, you can find the source code of the binaries on the slack channel available to you.
Links to download the prebuilt binaries are available in Slack and Discord.
Posted Sep 17, 2024 - 19:49 UTC
Update
New binaries will be required to resume block production on Athens. These binaries are being compiled now.
Posted Sep 17, 2024 - 18:17 UTC
Update
We are preparing a candidate fix to resolve the underlying issue. The fix will then be validated prior to release.
Posted Sep 17, 2024 - 16:15 UTC
Identified
We've identified the likely cause, and are evaluating methods to restart the network.
Posted Sep 17, 2024 - 14:29 UTC
Investigating
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Posted Sep 17, 2024 - 13:36 UTC
This incident affected: Testnet (ZetaChain Testnet).