Binance, Kerviel and NotPetya: Recovery Beyond the Bad Actor
Disaster recovery must survive compromised credentials and unavailable approvers. Binance, the Kerviel affair and Maersk’s NotPetya recovery expose the controls worth testing.
Disaster recovery must survive compromised credentials and unavailable approvers. Binance, the Kerviel affair and Maersk’s NotPetya recovery expose the controls worth testing.
A DR document states minutes and megabytes. Replication architecture delivers a convergence time, a single-copy window and a failed tail. This guide maps what actually decides your RTO and RPO.
RTO and RPO are measurable recovery objectives, not labels on an architecture diagram. Build a failure matrix and test the complete recovery path.
HEAD probes on delete markers answer 405 and replication counts that as success. 20M stuck versions turned the loop into 684M useless requests in 48 h.
During a MinIO replication incident three of the signals you reach for mislead you. The 405 on HeadObject is the success path, a retried 429 is invisible, and worker queue overflow logs nothing under the default priority. The counter that tracks dropped objects is incremented in four places and read in none.
A successful source PUT does not establish a recoverable peer copy. Separate MinIO site-replication acknowledgment, persisted retry state and catch-up capacity before committing to an RTO or RPO.
RBD mirroring in production is one of the patterns I audit and design: Ceph storage consulting. Alpha Feature in Production… Good Idea? The most promising approach, journal-based mirroring, offers near real-time replication and faster failover. However, it’s currently an alpha feature in the Ceph CSI driver and relies on rbd-nbd, Read more