Pepitedata
  • Home
  • Services
  • Audits
  • Expert Call
  • About
  • Blog
  • Contact

lots of small files

The same payload volume split into a few large objects or many small objects, with an exaggerated pile of metadata tags
object storage

Object Storage for Data Platforms at Scale

Size an object store against bytes, object metadata, request demand and maintenance time. A practical worksheet for small files, retention, erasure coding and release-channel choices.

By Julien Laurenceau, 4 weeks2026-09-30 ago
The same abstract point-cloud terrain packaged as many small tiles or a few larger containers
object storage

Why File Count Matters as Much as File Size on Object Storage

Compression, object population and request count measure different things. Compare point-cloud packaging, range reads, update costs and cleanup with an explicit synthetic sizing example.

By Julien Laurenceau, 2 months2026-08-22 ago
Two panels. Top: a single MinIO bucket overflowing with small files, a worried Tux, and a dead directory tree on a tombstone. Bottom: the same objects split across four prefix-partitioned buckets under separate folders, with a healthy green directory tree.
MinIO

MinIO on XFS: Inode Exhaustion and Prefix Design for Lots of Small Files

MinIO has no NameNode. It puts the namespace on XFS as real directories and xl.meta files. On Lots of Small Files that means you can exhaust inodes while df -h still looks fine, and a flat leaf can stall PUT, LIST, scanner, and ILM together. Here is the on-disk model, the inode math, and a prefix recipe that keeps XFS inside a regime you can operate.

By Julien Laurenceau, 3 months2026-07-27 ago
Diagram: MinIO erasure-coding storage inflation on small objects: per-drive xl.meta metadata replicated across the erasure set
data

Stop using MinIO as a NoSQL database: why S3 object stores collapse on small-file workloads

Two MinIO platforms, same root cause: used as a NoSQL store. Field notes on LIST IOPS, XFS directory limits, scanner & heal SLAs, the erasure-coding storage-efficiency inversion on small objects, and why Apache Cassandra (or Ceph) is the right answer on-prem in 2026.

By Julien Laurenceau, 5 months2026-05-17 ago
  • Privacy Policy
  • Mentions légales
  • CGV
  • Cookies
Hestia | Developed by ThemeIsle