MinIO Site Replication: Sync Mode Will Not Save Your RPO

TL;DR. MinIO site replication (mc admin replicate) mirrors an entire cluster (objects, IAM, ILM, config) across peer sites in a full mesh. Each site is autonomous. There is no cross-site quorum. In CAP terms the system is AP: a local PUT succeeds on local erasure-code quorum even if every peer is partitioned away. Conflicts reconcile by last-write-wins on object timestamp. Async is the default and keeps RPO in the seconds range under healthy lag. Sync (per peer) drives steady-state RPO toward that peer to near zero and improves cross-site read-after-write, but it is not two-phase commit: when the peer is down the source still ACKs, marks the object PENDING, and retries later. Those retries are automatic but source-side only: the replication queue is nothing but a PENDING / FAILED marker inside each object’s own metadata on the source site. No peer, no journal, no index holds a copy. An ACKed but unreplicated object therefore lives on exactly one site, the peer cannot enumerate what it is owed, and if that source is permanently lost the backlog is lost with it and cannot even be listed. My verdict on sync: do not use it. You pay inter-DC RTT on every PUT and 3× to 5× throughput, permanently, and the only RPO it improves is the one that was already acceptable. The failed replication tail inherits scanner sweep time (hours by default, about 16 minutes at scanner speed=fastest), so your RPO budget must track scan time, not erasure coding. Active-active gives RTO near zero for a single DC loss (plus DNS/GSLB). Integrity-critical buckets often want active-passive with a few minutes of RTO and zero split-brain window.

This post is a companion to Stop using MinIO as a NoSQL database and the LOSF series. It stays on one question: what site replication actually guarantees when you write RTO and RPO into a DR document.

What site replication is (and is not)

Site replication is server-side, continuous, and active-active by design. You add peers with mc admin replicate add. Once joined, the mesh covers:

• objects (PUT, DELETE, multipart),
• buckets and bucket config,
• IAM (users, groups, policies, service accounts),
• ILM rules, notifications, tags, locks, encryption config.

Each site remains a fully autonomous cluster. Multi-site replication is mirroring, not distributed consensus. Quorum lives inside one cluster only: read quorum is K data shards on the erasure stripe; write quorum is K, or K+1 when parity M equals N/2. No consistency level requires acknowledgement from N-of-M peer sites.

A few consequences follow and they are easy to miss in vendor decks:

• No minority-side fencing. A partition that splits a 3-site mesh into {A} and {B,C} lets both sides keep accepting writes.
• Conflict resolution is LWW on object timestamp. There is no TrueTime equivalent. NTP discipline is part of the correctness story, not an optional hygiene item.
• Versioning is mandatory at setup. Suspended versioning opens a permanent-loss path for LWW losers.
• Topology is always a full mesh. N sites means N(N−1)/2 links. A 3-site mesh costs roughly 2× egress per write versus two sites; five sites cost 4×. There is no native hub-and-spoke and no protocol-level read-only peer. A DR site that must not take client writes is enforced at DNS or the load balancer, not inside MinIO.

If you need hub-and-spoke, one-way archive, or per-bucket filters without IAM sync, use bucket replication (mc replicate add). It is a different tool. Do not layer per-bucket --sync rules on a site-replication-managed bucket: the combination is documented as mutually exclusive, not hard-enforced, and site replication can silently overwrite the manual rule.

Peer endpoints should be the site load balancer or multi-node DNS name. Pointing replication at a single node makes the channel a SPOF. On the LB: idle timeout at least 10 minutes, no header rewriting, no request buffering on large objects, TLS passthrough or transparent re-encrypt, health check on /minio/health/live (not /minio/health/cluster), balance leastconn, never balance source.

Sync vs async: the RTO / RPO table

Site replication is asynchronous by default. The PUT returns when the local erasure-code quorum is met. Replication runs in the background. Sync is configured per peer, never per bucket inside site replication:

mc admin replicate update <alias> --deployment-id <peer-id> --mode sync

Different peers in one mesh can mix modes. A common production shape is sync on the low-RTT DC1↔DC2 edge and async toward the distant DR peer.

Aspect	Async (default)	Sync (per peer)
PUT latency	Local only (about 5 to 20 ms)	Local plus inter-DC RTT (about 25 to 80 ms close; 70 to 130 ms at 30 ms RTT)
Steady-state RPO	Seconds of lag (typical: seconds to minutes)	Near zero toward that peer while it is up
Data-loss window on peer death	Seconds of pending objects	Near zero while peer was up; same as async during the outage
Throughput	High	About 3× to 5× fewer objects/s at the same concurrency
Behavior if remote is down	Source unaffected; PENDING builds up	Same: source unaffected; PENDING builds up
IAM / policy replication	Always async	Always async (sync applies to object data only)

The critical sync caveat

Even in sync mode, MinIO does not fail the source PUT when the remote is unreachable. The local write succeeds. The object is marked PENDING in its own metadata on the source, and workers, then the MRF, then eventually the scanner replay it when the peer returns. That marker is the only record that the write is owed to the peer, and it lives on the source alone (see the retry ladder below).

Sync buys minimal steady-state RPO and stronger read-after-write odds toward a live peer. It does not buy atomic multi-site commit. Loss of one site never takes the platform down for that reason: there is no cross-site write quorum to lose.

What sync buys when the peer is healthy:

• steady-state RPO near zero toward that peer,
• cross-site read-after-write without leaning on proxy_request,
• PENDING near zero and cleaner observability,
• a smaller in-flight backlog if the peer dies abruptly.

What sync does not buy:

• no cross-site write quorum,
• no split-brain prevention (reconnect still uses LWW, same as async),
• no better durability for the double-failure drain window,
• no higher aggregate throughput,
• no faster post-outage drain (drain is workers plus bandwidth, not the pre-outage mode).

My recommendation: do not use sync mode

Strong version, and I will defend it: do not enable synchronous site replication. You pay a latency and throughput tax you can measure, in exchange for an RPO improvement you cannot publish.

The reasoning is that the RPO written into a DR document is set by the worst case, not by the median:

• Sync only shrinks RPO while the peer is healthy, which is exactly the window where async already delivers seconds of lag. Seconds of lag is almost never the number that fails an audit.
• The moment the peer is down, sync and async follow the identical path: ACK the client, mark PENDING, drain later.
• The failed tail is scanner-bound in both modes.
• The post-outage drain is bounded by bandwidth and workers in both modes.

So the cost is certain and permanent (inter-DC RTT on every PUT, 3× to 5× fewer objects/s at the same concurrency), while the benefit exists only in the window where nothing was wrong. Latency is charged continuously; RPO is only ever tested during a failure, and during a failure sync is not there. Worse, it invites the belief it does not deserve: teams that flip the flag start writing "RPO = 0" in the DR document, and stop building the reconciliation path that would have actually saved them.

Two objections deserve an answer. If you want cross-site read-after-write, use proxy_request=enable: it targets that problem directly and charges the latency only on GETs of recently written objects, instead of taxing every PUT. If a written regulation names synchronous replication, enable it and move on. That is a compliance decision, not an engineering one, and the honest DR paragraph still has to state the peer-down and failed-tail clauses below.

Async backlog math

Under async, the in-flight backlog is roughly:

backlog ≈ write_rate × p99(replication_lag)

At 1000 PUT/s and a 5-second p99 lag, about 5000 objects exist on only one site at any moment. Sync narrows that happy-path backlog to single-digit objects, and this is the one genuine thing it buys. Judge it honestly: it shrinks the single-copy window during normal operation, and does nothing the moment anything goes wrong. If that window is the risk keeping you up at night, the cheaper fixes are lowering p99 lag (workers, bandwidth, target capacity) and holding the object in the producer until replication is confirmed.

Decision rules

• Stay on async. It is the default, it is MinIO's own recommendation, and it is the right answer at any inter-DC RTT. The case for it gets stronger as RTT rises and as the workload gets more throughput-dominated, but there is no RTT low enough to make sync a good trade on its own.
• Before enabling sync, name the failure it protects you from. Peer down? Same behavior as async. Failed tail? Scanner-bound in both modes. Split-brain? LWW in both modes. Source site destroyed with a backlog pending? Gone in both modes. The list ends at "the steady-state single-copy window", and that one has cheaper fixes.
• Async plus proxy_request=enable gives read-after-write on the peer without the sync PUT tax. The cost moves to GET latency on recently written objects (about 25 to 80 ms close; 70 to 200 ms far).
• Anti-pattern: flip a peer to sync without raising client concurrency. Existing clients see a 3× to 5× throughput drop and blame "MinIO is slow".
• Anti-pattern: layer per-bucket mc replicate add --sync on an SR-managed bucket.

The PostgreSQL analogy (and where it stops)

Teams coming from databases often map MinIO sync onto something like PostgreSQL synchronous_commit = on with a synchronous standby. The mapping is comforting and mostly wrong.

With PostgreSQL and a synchronous standby, a commit waits for the standby to flush the WAL (depending on synchronous_commit and synchronous_standby_names). If the required standby is gone and you have not reconfigured, commits block or fail. The primary does not cheerfully ACK and hope the standby catches up hours later. That blocking is exactly the durability contract people pay latency for.

MinIO sync is closer to: "try to push to the peer before returning, and if you cannot, still return 200 and leave a PENDING marker." The happy path looks synchronous. The failure path is asynchronous with an unbounded backlog.

A closer (still imperfect) image is PostgreSQL with async replication (synchronous_commit = local) plus a replication slot that is allowed to lag without bound, except that MinIO does not pause the primary when lag grows. You can sit for hours or days with a failed tail that the client already saw as durable. If your mental model is "sync means the other site has it or I get an error," MinIO will violate that model the first time a peer drops.

Use the database analogy to stress-test the DR document, not to import false guarantees. Ask: on peer loss, do we block writes, reject writes, or accept writes with growing PENDING? MinIO chooses the third option in both modes.

The retry ladder, and where the queue actually lives

"Replication retries automatically" is true and it is the most misleading sentence in the whole feature. It hides three different cadences and one architectural fact that decides your entire DR story.

Here is the full life of one object that fails to replicate:

PUT accepted on local EC quorum ──► 200 OK to the client
   │
   └─► marked PENDING in the object's own xl.meta
          │
          ├─► replication worker attempts the remote PUT
          │      (~500 workers/node by default)
          │      └─► success ──► COMPLETED
          │
          └─► failure (timeout, 503 SlowDown, 504, reset)
                 RetryCount++ , object queued into the MRF
                    │
                    ├─► MRF re-attempt, ~5-6 min cadence,
                    │   disk-persisted, survives a restart
                    │
                    └─► RetryCount > mrfRetryLimit (3)
                           ──► FAILED, dropped from the MRF
                              │
                              ├─► next scanner sweep re-queues it
                              └─► a GET/HEAD on it re-queues it

Two properties of that ladder are routinely misread:

• There is no burst of immediate retries. About four attempts total (one worker attempt plus three MRF re-attempts) spread over roughly 15 to 20 minutes. The MRF (Most Recently Failed) queue is a bounded 100k in-RAM channel flushed to disk every 5 minutes; entries that overflow it are dropped silently (counted in TotalDroppedCount) and skip straight to the slow path.
• After the last MRF attempt there is no timer at all. The object sits FAILED until the background scanner walks past it, or until a client GET/HEAD happens to touch it. Nothing else wakes it up.

The queue exists only inside the source cluster

There is no separate disk-backed replication queue, and no key/value index of pending replication anywhere: not per cluster, not globally. No sidecar database, no external journal, no list of "objects still owed to peer B". The queue is nothing but the PENDING / FAILED marker written into each object’s own xl.meta, on the same erasure stripe as the object, on the source site only. Depth is therefore bounded by object count rather than by RAM, and there is no "queue full" back-pressure on the client path.

That single design choice produces the consequences most DR documents never state:

• The peer has no idea what it is missing. It was never told. A destination site cannot enumerate what the source still owes it. Only the source can, and only by re-walking its own namespace.
• If the source cluster is permanently lost, its queue is lost with it, and cannot even be listed. The bytes never left the source. The client already got its 200. Nothing on the surviving site records that those objects ever existed, so there is no "replay the queue after the disaster" step: there is no queue left to read, and no inventory of what to go looking for. Reconstruction has to come from outside MinIO (your ingestion log, the producer, a backup), if it exists at all.
• If the source is only temporarily down, this is unavailability rather than bit-loss (its disks still hold the objects), but for the duration the client-visible effect on the survivor is identical: 404.
• Metadata loss is data loss. Lose EC quorum on a stripe and you lose the replication state of the objects on it too. mc admin replicate resync re-walks the source namespace and re-replicates what it finds; it cannot recover state for objects whose source-side metadata is gone.

The rule to carry into the DR document: an object that has been ACKed but not yet replicated exists in exactly one place. Site replication’s cross-site redundancy begins when replication completes, not when the client received 200. So read minio_bucket_replication_pending_count and _failed_count as "number of objects currently protected by a single site", not as throughput noise, and size the acceptable single-copy window deliberately.

The failed-tail RPO rule: draining can take a full cluster scan

Once an object has fallen off the MRF fast path, the only thing that will clear it from the queue is a scanner sweep (or an incidental GET/HEAD, or an explicit resync). So the worst-case time for the queue to purge itself is one full cluster scan, and that is the number your RPO has to be written against.

Default scanner speed walks a large namespace in hours. scanner speed=fastest is on the order of a full sweep every ~16 minutes under good conditions. On LOSF or multi-hundred-million-object buckets, sweeps stretch to hours or days.

Sizing rule: for the failed tail, replication RPO equals one full scanner sweep. Sync drives the successful-path RPO toward zero and does nothing here: the failed tail is a scanner problem, not a replication-mode problem.

Steady-state unreconciled backlog is roughly:

hard_failure_rate × full_scan_period

Example: 10 hard failures per day and a 7-day sweep leaves about 70 objects perpetually out of sync, each up to a week stale, even though every client already received 200.

Practical consequences:

• Couple the RPO target to scan time, not to erasure coding. Adding parity does nothing for this path. If the RPO target is R and hard failures are nonzero, keep full_scan_period ≤ R.
• Object count and retention are the same knob as replication RPO. Longer retention and more versions lengthen the sweep. Versioning is required for site replication and multiplies on-disk file count; keep a non-current-version ILM policy on replicated estates.
• If the RPO target is tighter than achievable scan time, stop waiting on the scanner. Alert on minio_bucket_replication_failed_count and MRF drop counters, schedule mc admin replicate resync start, and/or reconcile from an application journal.
• After a multi-day outage with millions of PENDING objects, do not wait for the scanner. Trigger an explicit resync and watch it drain:

mc admin replicate resync start site1 --site site2

Drain time is usually bounded by bandwidth and disk concurrency, not by worker count alone. Millions of pending objects after a multi-hour outage often take tens of minutes to hours.

DR patterns and the RTO / RPO you can honestly publish

Active-active multi-DC (default production shape)

Two or more DCs serve reads and writes with site replication between them.

• RTO ≈ 0 for a single DC loss at the storage layer: the other DC keeps serving. Quote DNS TTL and GSLB detection in the published RTO; clients do not teleport by themselves.
• RPO ≈ 0 in steady state with sync peers, or seconds with async.
• On recovery, replication catches up (workers first; active resync for large backlogs).
• Highest availability. You accept active-active split-brain risk by design.

Active-active edge plus async DR peer

One site-replication group {DC1, DC2, DC3} with sync on DC1↔DC2 and DC3 left async.

• DC3 holds a small lag. In a disaster, the last few seconds of writes may be missing on DC3. That is often acceptable for DR.
• IAM and policy still converge on DC3, which is why site replication beats standalone bucket replication for the DR leg when identity must move with data.
• Promotion is not free. If DC3 must become active, replication topology needs deliberate reconfiguration. It is not a transparent hot standby.

Active-passive for integrity-critical data

For financial records, audit logs, or regulatory artifacts, keep site replication multi-master underneath but make the secondary read-only at DNS/LB. Fail over with a runbook.

• Trade a few minutes of RTO for zero split-brain write window.
• Still keep versioning, NTP, and a post-incident version audit. The storage layer remains AP if someone bypasses the LB.

The double-failure window (T2 → T3)

This is the scenario DR slides skip.

• T1: DC2 down. DC1 ACKs writes on local EC quorum and marks PENDING to DC2.
• T2: DC2 returns. Drain starts. Without proxy_request, DC2 may 404 objects that already exist on DC1.
• T3: DC1 dies before drain finishes. Data that returned 200 at T1 is invisible on the surviving DC. This is unavailability, not bit-loss (DC1 disks still hold the bytes), but clients see 404s, and DC2 cannot tell you which objects are affected because the queue that knew about them lived only on DC1. If DC1’s loss turns out to be permanent, that backlog is permanently gone and permanently un-enumerable.
• Later: both sides drain. Same-key conflicts resolve by LWW on timestamp. The loser remains as a non-current version if versioning stayed on. Clock skew can pick the wrong winner.

Mitigations, in order of practical impact:

• Keep versioning enabled and audit for suspended buckets.
• Strict NTP: target under 50 ms drift (ideally under 10 ms); alert above 100 ms (node_timex_offset_seconds).
• Document post-incident reconciliation: wait for bilateral Queued = 0 and Errors = 0, list multi-version keys in the incident window, arbitrate in the app, promote with mc cp --version-id.
• Active-passive at the edge for integrity-critical buckets.
• proxy_request=enable where masking transient 404s during drain is worth inter-DC GET latency.
• On site recovery, start an active resync immediately; raise large-object workers for multipart-heavy backlogs.
• Model drain duration before the incident. If the estimate exceeds your acceptable second-failure window (for example one on-call shift), raise capacity or shrink the active-active scope.

Metrics that belong on the DR dashboard

Signal	Why it matters
mc admin replicate status → Queued	Growing queue means RPO is opening
mc admin replicate status → Errors	Connectivity or permission faults; target zero
minio_bucket_replication_pending_count / _pending_bytes	Async backlog in objects and bytes
minio_bucket_replication_failed_count	Failed tail; drives the scan-time RPO rule
minio_bucket_replication_latency_ms	Actual lag versus the RPO budget
Transfer rate, workers, latency in mc admin replicate status	Saturation and drain health
mc admin replicate resync status	Post-outage drain progress
NTP offset (node_timex_offset_seconds)	LWW arbiter health

Official Grafana boards worth importing: 13502 (server/cluster), 13474 (bucket), 15305 (node), 13551 (replication).

How to write the RTO / RPO section without lying

A DR paragraph that survives an audit usually looks like this:

• Scope: which buckets, which sites, sync vs async per peer.
• Steady-state RPO: seconds (async) or near zero (sync peer up), with the write-rate × lag formula and the dashboard panel that proves it.
• Degraded RPO: on peer loss, RPO becomes "time until PENDING drains after recovery," bounded by resync capacity, not by the sync flag.
• Single-copy window: state plainly that an ACKed but unreplicated object sits on one site only, that the queue tracking it exists nowhere but that site, and that a permanent loss of that site destroys the backlog with no way to enumerate what was in it. Name the external source of truth (ingestion log, producer, backup) you would reconcile against, or say there is none.
• Failed-tail RPO: one full scanner sweep at current namespace size and scanner speed; name the object-count ceiling that keeps that sweep inside the budget.
• RTO for single-site loss: storage RTO near zero in active-active, plus measured DNS/GSLB failover.
• RTO for integrity-critical failover: minutes, if you chose active-passive at the edge on purpose.
• Split-brain: acknowledged for active-active; mitigated with versioning, NTP, and a reconciliation runbook (or avoided with active-passive).
• Promotion steps for a DR peer, including replication reconfiguration, not only "point DNS at site B."

If a vendor slide says "sync replication ⇒ RPO = 0" with no peer-down clause and no scanner clause, the slide is incomplete. Fix the document before the incident fixes it for you.

Related posts

• Stop using MinIO as a NoSQL database – product fit, and the longer form of the sync-mode caveat.
• MinIO and lots of small files – why object count and the scanner dominate operations.
• When erasure coding becomes 15× replication – small-object storage math.
• MinIO on XFS: inode exhaustion and prefix design – filesystem substrate that sets scan-time reality.

Site replication is a strong multi-site mirror for object platforms that accept AP semantics. It is a weak multi-master database. Design RTO and RPO against the first sentence, not the second.

If you are writing or stress-testing a MinIO DR plan (sync vs async per peer, failed-tail RPO, active-active versus controlled failover), that is core work in a Flash Architecture Review and in a full Data Platform Performance Audit. Book a 15-min intro call to see which fits, or take one focused hour with an Expert Call.