MinIO site replication bug: how 684 million HTTP 405s exposed the delete-marker loop
HEAD probes on delete markers answer 405 and replication counts that as success. 20M stuck versions turned the loop into 684M useless requests in 48 h.
HEAD probes on delete markers answer 405 and replication counts that as success. 20M stuck versions turned the loop into 684M useless requests in 48 h.
During a MinIO replication incident three of the signals you reach for mislead you. The 405 on HeadObject is the success path, a retried 429 is invisible, and worker queue overflow logs nothing under the default priority. The counter that tracks dropped objects is incremented in four places and read in none.
MinIO site replication has no cross-site quorum, and its replication queue exists only inside the source cluster: lose that site and the un-replicated backlog is gone and un-enumerable. Sync mode costs latency without fixing that. The precise model, the RTO/RPO table, and the DR patterns that match real SLAs.
Alpha Feature in Production… Good Idea? The most promising approach, journal-based mirroring, offers near real-time replication and faster failover. However, it’s currently an alpha feature in the Ceph CSI driver and relies on rbd-nbd, which introduces significant risks: For documentation on Ceph RBD mirroring see IBM documentation For details, see Read more