Download this article (PDF) 515 KB

Three open hard drives side by side, the third one on fire: the failure every Kubernetes PVC backend has to plan for

Choosing a PVC backend is an architecture decision I review; the Ceph storage consulting page describes the cluster side.

Choosing a storage backend for Kubernetes PersistentVolumeClaims (PVCs) comes down to three trade-offs: performance, availability and operational cost. This guide compares the four open source backends I meet most often in production, Ceph RBD, OpenEBS Local PV ZFS, OpenEBS Replicated PV Mayastor and DRBD with LINSTOR, on premises and in the cloud. It is part of Ceph in Production: RBD, RGW and CephFS, the field guide to Ceph in production.

Scope. Only open source backends you can run without a license fee are compared here. Products that require a paid license, such as Portworx or StorPool, and CSI drivers tied to proprietary storage arrays are out of scope. Paid support for an open source backend, which LINBIT sells for DRBD, does not change that.

Updated 2026-10-26: pros and cons written out for every backend, the comparison table rebuilt as text, cloud figures refreshed (AWS raised the gp3 limits in September 2025), Ceph RBD snapshots, backups and multi-site mirroring clarified, licensing scope stated.

First question: does the application already replicate?

Before comparing backends, check whether the application replicates its own data. Kafka, Cassandra, MongoDB and Redis keep replicas across nodes, so a replicated backend underneath them stores every byte again and adds a network hop to every write. For those workloads, node-local storage is usually the right answer. A replicated backend earns its cost when the application cannot replicate itself.

Single-instance databases, stateful services with no built-in replication and workloads that share files between pods are where storage-level replication pays for itself. Everything else is a performance and cost question.

Two setups side by side: three servers with local NVMe copying data between themselves, labeled APPLICATION REPLICATES, and one server cabled to three linked storage shelves, labeled STORAGE REPLICATES
Who owns replication decides the backend: an application that replicates belongs on local NVMe, one that cannot needs a replicated storage layer underneath.

The four backends compared

Ceph RBD buys the strongest availability at the highest operational cost. OpenEBS Local PV ZFS buys the highest performance by giving up storage-level availability. Mayastor and DRBD with LINSTOR sit between the two, with replicated volumes and a lighter footprint than a Ceph cluster. The ratings below are my field assessment of the four backends relative to each other, not benchmark results.

BackendTypePerformanceAvailabilityRWXOperational complexityCloud fitBest for
Ceph RBDDistributed block storage on RADOSModestStrongThrough CephFSHighWeakMulti-node HA and scale
OpenEBS Local PV ZFSNode-localHighestNone at the storage layerNoLowStrongLocal and cloud NVMe for self-replicating applications
OpenEBS Replicated PV MayastorReplicated, SPDK and NVMe-oFHighGoodNoMediumWeakReplicated volumes on modern NVMe nodes
DRBD with LINSTORSynchronous block replicationModestGoodThrough OCFS2MediumWeakCompact HA clusters, critical databases
Relative ratings from field experience, not measurements. Cloud fit rates how much the backend gains over the provider’s managed block storage.

Ceph RBD: strongest availability, highest operating cost

Ceph RBD (RADOS Block Device) stripes each volume over objects that Ceph replicates, three times by default, across the failure domains CRUSH defines. A Ceph RBD volume survives the loss of a disk, a node or a rack with no help from the application, and the cluster scales to many nodes. The price is write latency from a networked, replicated write path and the weight of operating a full Ceph cluster.

  • Pros: survives disk, node and rack failures; snapshots, clones and backups built in (see below); the same cluster also serves S3 through RGW and a shared filesystem through CephFS; Rook runs it inside Kubernetes.
  • Cons: every write crosses the network and waits for its replicas; three replicas mean three times the raw capacity; MON, MGR and OSD daemons, CRUSH design and upgrades are a real operations workload; shared file access (RWX) goes through CephFS, since RBD filesystem volumes are RWO.
  • Best for: large multi-node clusters that need strong availability and scale.

Snapshots and backups come with RBD

Ceph RBD ships snapshots and copy-on-write clones, and the Ceph CSI driver exposes both to Kubernetes through VolumeSnapshot and volume cloning, which the ceph-csi feature matrix marks GA. Backups follow from the snapshots: rbd export-diff produces incremental exports between two snapshots, and open source backup tools such as Velero drive CSI snapshots to back up PVCs on a schedule.

Multi-site mirroring is not production-ready on RBD

Ceph RBD mirroring does not give Kubernetes a production-ready multi-site storage service today. Journal-based mirroring needs the rbd-nbd mounter, which ceph-csi still rates Alpha, and a CSI plugin restart then drops every mount on the node. Snapshot-based mirroring runs, but as a disaster recovery copy: writes since the last mirror snapshot are lost on failover, and failover is a manual procedure.

The Kubernetes API that drives RBD mirroring, the csi-addons VolumeReplication resource, is still served as replication.storage.openshift.io/v1alpha1. Two-zone mirroring also means two independent three-replica clusters, six copies of every byte. Ceph RBD mirroring for Kubernetes storage in production details both modes and the cheaper 2.5-zone topology inside one cluster.

Ceph RGW is the opposite case. RGW multi-site is a production feature: zones in a realm replicate buckets continuously and asynchronously, every zone accepts object reads and writes by default, and sync status is visible per zone. When cross-site replication is a hard requirement, data that can live behind an S3 API is better placed on RGW multi-site than on mirrored RBD volumes. Upgrading a multi-site S3 platform from Squid to Tentacle shows what running it involves.

Two panels between the same two data halls: a continuous conveyor of crates labeled RGW MULTI-SITE, and a single hand-cranked cart with crates left on the dock labeled RBD MIRRORING
RGW multi-site replicates continuously between zones; RBD mirroring ships periodic snapshots and fails over by hand.

OpenEBS Local PV ZFS: native NVMe speed, no storage-level HA

OpenEBS Local PV ZFS provisions each volume as a ZFS dataset or zvol on the node’s own disks, so I/O never leaves the machine. On NVMe it delivers device-level performance, with ZFS snapshots, clones, compression and checksums on top. The trade-off is absolute: the volume lives on one node, and when that node dies, the data is only as safe as the application’s own replicas.

  • Pros: native local NVMe performance with no network hop; ZFS snapshots, clones, compression and end-to-end checksums; one CSI driver and no storage cluster to operate; listed as stable by OpenEBS since 4.0.
  • Cons: the pod is pinned to the node that holds its volume; losing the node loses the volume unless the application replicates; no RWX; capacity is limited to the node’s own disks.
  • Best for: fast RWO volumes for applications that replicate themselves, on premises and on cloud nodes with local NVMe.

In the cloud: local NVMe against EBS

On AWS the gap between local NVMe and EBS is an order of magnitude. AWS rates the eight local NVMe drives of an i4i.32xlarge instance at 3,200,000 random read IOPS combined, while the same instance tops out at 160,000 IOPS of EBS traffic. Per volume, gp3 reaches 80,000 IOPS since AWS raised its limits on 2025-09-26, and io2 Block Express reaches 256,000 IOPS.

Instance store is ephemeral: its data does not survive a stop, a termination or a host failure. That is acceptable for Kafka, Cassandra or MongoDB only because they rebuild a lost replica from their peers. Size the rebuild time and the network traffic it consumes before relying on it, and keep EBS for anything that cannot rebuild itself.

OpenEBS Mayastor: replicated NVMe-oF with reserved resources

Replicated PV Mayastor is the replicated engine OpenEBS standardized on in version 4, after the older cStor and Jiva engines were deprecated and moved to the OpenEBS archive in 2024. Mayastor replicates each volume synchronously across nodes and serves it over NVMe over Fabrics on TCP, with an SPDK data path in user space. It is faster than classic replicated storage, at the cost of resources reserved on every storage node.

  • Pros: synchronous replicas spread across nodes and pools; a user-space SPDK data path built for NVMe; a much smaller footprint than a Ceph cluster.
  • Cons: each io-engine node reserves two CPU cores, 1 GiB of RAM and at least 2 GiB of 2 MiB huge pages; nodes need x86-64 CPUs with SSE4.2 and the nvme-tcp kernel module; the minimum is three worker nodes; volumes are RWO only.
  • Best for: replicated volumes on modern NVMe nodes when Ceph is more platform than the workload needs.

DRBD and LINSTOR: synchronous replication for compact HA clusters

DRBD replicates a block device between nodes inside the Linux kernel, and LINSTOR orchestrates those devices; in Kubernetes, the open source Piraeus operator deploys both. With synchronous replication, a write completes only once the peer holds it, so a failover loses no acknowledged write. DRBD with LINSTOR suits small, critical clusters where two or three nodes must never disagree about the data.

  • Pros: no acknowledged write lost on failover; a mature, low-overhead kernel module; reads served from the local replica when the pod runs on a node that holds one.
  • Cons: every write pays a network round trip to the peer; the DRBD kernel module must match each node’s kernel; two-node setups need a tiebreaker for quorum.
  • RWX with OCFS2: shared access works through a cluster filesystem such as OCFS2 on dual-primary DRBD, which adds fencing and split-brain handling to the runbook.
  • Best for: compact HA clusters and critical databases.

Which PVC backend should you choose?

Match the backend to who owns replication and how far the data must travel. If the application replicates, use local NVMe through OpenEBS Local PV ZFS. If the storage must replicate, pick Ceph RBD for scale and a shared platform, Mayastor for a lighter replicated layer, or DRBD for small clusters with zero tolerance for lost writes. If data must cross sites, prefer Ceph RGW multi-site over RBD mirroring.

  • Kafka, Cassandra, MongoDB or Redis, on premises or in the cloud: OpenEBS Local PV ZFS on local NVMe.
  • Scale-out resilience, one platform for block, S3 and files: Ceph RBD.
  • Replicated block storage with a lighter footprint: OpenEBS Replicated PV Mayastor.
  • Simple, robust HA for a compact cluster: DRBD with LINSTOR, plus OCFS2 when pods share files.
  • Replication across sites: Ceph RGW multi-site for data behind S3; treat RBD mirroring as a disaster recovery copy.

If you are choosing a PVC backend for a new platform, a migration or a hardware purchase, a Flash Architecture Review settles that one decision with measurements on your own nodes. The free 15-minute intro call is the place to start.

FAQ

Does Ceph RBD support snapshots and backups for Kubernetes PVCs?

Yes. Ceph RBD has native snapshots and copy-on-write clones, and the Ceph CSI driver exposes them to Kubernetes through VolumeSnapshot and volume cloning, both rated GA. Backups build on the snapshots, either as incremental exports with rbd export-diff or through an open source tool such as Velero that drives CSI snapshots.

Is Ceph RBD mirroring production-ready for multi-site Kubernetes storage?

No, not as a multi-site storage service. Journal-based RBD mirroring depends on the rbd-nbd mounter, rated Alpha in ceph-csi, and the Kubernetes VolumeReplication API is still v1alpha1. Snapshot-based mirroring works as a disaster recovery copy, with data loss since the last mirror snapshot and a manual failover. Ceph RGW multi-site, by contrast, is a production feature for data behind S3.

Do Kafka or Cassandra on Kubernetes need replicated storage?

Usually not. Kafka and Cassandra replicate data across their own nodes, so a replicated storage backend stores every byte again and adds latency to every write. Node-local storage such as OpenEBS Local PV ZFS on NVMe gives them device-level performance, provided the cluster can rebuild a lost replica from its peers in an acceptable time.

Is local NVMe faster than EBS for Kubernetes on AWS?

Yes, by an order of magnitude on the largest instances. AWS rates the local NVMe drives of an i4i.32xlarge at 3,200,000 random read IOPS, against an EBS ceiling of 160,000 IOPS for the same instance and 80,000 IOPS for a single gp3 volume. Local NVMe is ephemeral, so it fits applications that replicate their own data.

Why are Portworx and other licensed products not compared?

This comparison covers open source PVC backends that run without a license fee: Ceph RBD, OpenEBS Local PV ZFS, OpenEBS Replicated PV Mayastor and DRBD with LINSTOR. Products that require a paid license, such as Portworx or StorPool, and CSI drivers for proprietary arrays are out of scope, because their cost model and lock-in change the decision.

Sources

Related posts


0 Comments

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *