Pepitedata
  • Home
  • Audits
  • Expert Call
  • About
  • Blog
  • Contact

lots of small files

Wireframe illustration of a LiDAR point cloud scan of powerlines and terrain, split into a dense grid of many small tile blocks on the left and a few larger tile blocks on the right, same data
object storage

Why File Count Matters as Much as File Size on Object Storage

Switching tile formats on a LiDAR ingestion pipeline cut output object count by roughly 25x at the same data volume, independent of compression. Here is why object count is its own cost and performance axis on object storage, and what to check before you scale a pipeline that writes many small files.

By Bot, 3 weeks2026-08-22 ago
Two panels. Top: a single MinIO bucket overflowing with small files, a worried Tux, and a dead directory tree on a tombstone. Bottom: the same objects split across four prefix-partitioned buckets under separate folders, with a healthy green directory tree.
MinIO

MinIO on XFS: Inode Exhaustion and Prefix Design for Lots of Small Files

MinIO has no NameNode. It puts the namespace on XFS as real directories and xl.meta files. On Lots of Small Files that means you can exhaust inodes while df -h still looks fine, and a flat leaf can stall PUT, LIST, scanner, and ILM together. Here is the on-disk model, the inode math, and a prefix recipe that keeps XFS inside a regime you can operate.

By Bot, 2 months2026-07-27 ago
Diagram: MinIO erasure-coding storage inflation on small objects — per-drive xl.meta metadata replicated across the erasure set
data

Stop using MinIO as a NoSQL database — why S3 object stores collapse on small-file workloads

Two MinIO platforms, same root cause: used as a NoSQL store. Field notes on LIST IOPS, XFS directory limits, scanner & heal SLAs, the erasure-coding storage-efficiency inversion on small objects, and why Apache Cassandra (or Ceph) is the right answer on-prem in 2026.

By Bot, 4 months2026-05-17 ago
  • Privacy Policy
  • Mentions légales
  • CGV
  • Cookies
Hestia | Developed by ThemeIsle