Notice to the Regen Network Community

At 3:33AM UTC on May 7th, the FreshREGEN validator was removed from the active validator set on the Regen Network for double signing a block. A double sign penalty, also known as tombstoning a validator, is irreversible and carries a 5% penalty on the amount staked by the validator and all its delegators.

The cause of the double sign was a sentry being brought online with the active validator signing key. As it is customary at genesis, we deploy our validator on a public IP and allow it to connect directly to other peers. This ensures more faster and more efficient peer discovery, less latency and thus less missed blocks when at network launch. Approximately 48 hours after network start, we moved the active validator to our private IP, HSM connected machine, and let it connect to the rest of the network via our 3 sentry nodes. At this time, the original validator machine from the genesis event was brought offline. About three weeks later, in the early AM hours of May 7th, we decommissioned one of our 3 sentries and used a file copy tool called rsync to move over the files from the decommissioned sentry to the machine we used at network launch (the 1st validator), which was to be used as the 3rd sentry. Since the rsync copy settings did not overwrite the validator signing key on its target (it did not overwrite any existing files), when this sentry was brought online it quickly caught up to the tip of the chain and caused the double sign event at block 300,786.

Regardless of what rsync did or didn’t do, we should have wiped the validator key from this sentry before running it, but we didn’t, and our validator got tombstoned. We take full responsibility for this incident, and it is with great regret that we communicate this event to Regen Network stakeholders. In all our history of validating on over a dozen chains, since 2018, we have never been jailed, not even temporarily for downtime.

In the spirit of owning our mistakes, in tune with our appreciation for the crucial mission of the Regen Network, it’s amazing team and community, we have decided to disburse all of our Regen holdings (approximately 8000regen) to make (as much as possible for us) whole all of the community members who have delegated to us. As soon as funds are transferable, we will disburse the funds proportionally to our delegator’s stake at the time of the jailing event. Since some delegators, including members of the RND Inc, have mentioned to us that they are willing to forego their share of the reimbursement, we hope that this amount will make up for a good part of the losses incurred by our delegators. We will be issuing another statement in the near future to coordinate the reimbursement process.

This incident hightlights an important risk that exists in the staking economy, and how crucial it is to prevent double signing. While there are a couple of (not yet open source) tools to allow for automatic failover between validators, they are used in restricted conditions by very few providers. At FreshREGEN, we’d rather ensure no doublesign at the cost of liveliness, so when we need to failover the active validator (for the reason of an upgrade or a reboot) to one of the backup machines, we follow steps that triple check (visual, scripted, then visual again) for the presence of the active validator signing key. During a failover, we might miss one or two blocks, sometimes no blocks, but the integrity of the signing proceess is preserved. To ensure that this is the first and last time this event will happen, our Standard Operating Procedures will be updated to mandate that when a key is imported into out HSM, it must be instantly wiped from the previous machine it was present in (if any), and kept only in our cold storage encrypted password safe, regardless of whether the previous machine was powered off, the service was disabled, and the blockchain data removed from disk.

On behalf of the FreshREGEN team, please accept our apologies for out mistakes. We look forward to seeing Regen Network achieve it’s goals, and we hope to continue to be a part of this amazing project.