Why File Count Matters as Much as File Size on Object Storage

TL;DR. On a past engagement, I profiled a point-cloud tiling pipeline that turned airborne LiDAR scans into tiled 3D layers on S3-compatible object storage, for a large US electric utility client running a vegetation-encroachment risk product over its powerline network. A single 631-million-point LiDAR file, tiled with an older open-source tiler (PotreeConverter 1.x), landed on object storage as roughly 104,000 separate objects. The same file, tiled with a newer format (PotreeConverter 2.x or Entwine EPT), landed as roughly 4,000 objects, about 25x fewer, for the same bytes and a similar compression ratio. The byte count barely moved. The object count did, and object count is its own axis of cost and performance on any S3-compatible store, independent of how well the data compresses.

The pipeline: tiling LiDAR onto object storage

The product ingests airborne LiDAR scans, several terabytes of raw LAS files per project, and converts them into a tiled, streamable 3D format that a WebGL point-cloud viewer can render without loading the whole dataset into a browser tab. That conversion step writes one small object per tile to S3-compatible storage: a handful of kilobytes to a few megabytes each, and thousands to hundreds of thousands of them per input file, depending on the tiling tool and its default tile size.

I benchmarked several open-source point-cloud tiling and storage formats side by side on two real client LiDAR datasets, tens to hundreds of millions of points each: PotreeConverter 1.x and 2.x, Entwine EPT, PDAL, LAStools, and untwine (COPC). The point was to pick a format for a project scaling toward roughly 180 billion points total. I was not benchmarking compression ratio in isolation. I was benchmarking what each tool actually put on the storage bucket, because that is what the pipeline pays for and waits on.

The number that mattered: about 104,000 objects against about 4,000

Take the single largest test file, a 631-million-point merged LiDAR dataset. Tiled with PotreeConverter 1.x, the legacy format still in production at the start of the engagement, it landed as roughly 104,000 output objects on S3. Tiled with PotreeConverter 2.x or Entwine EPT, the same file landed as roughly 4,000 objects, a difference of about 25x.

Compression ratio across the tools tested was in a comparable range, roughly 1.4x to 2.3x versus raw LAS, with the best format-and-compression combination reaching about 2.3x. That is a real difference, but it is a small one next to a 25x swing in object count. The two numbers are not the same axis. Compression ratio tells you how many bytes you store. Object count tells you how many separate storage-API calls you make to store and later retrieve those bytes, and on most S3-compatible platforms, that is a cost and performance line item of its own.

Same bytes, same rough compression, 25x more objects: where did that come from

Not from the codec. It came from partitioning strategy, the geometry of how each tool decided to slice one point cloud into tiles. The legacy tiler used a fixed, comparatively fine octree subdivision that produced a large number of small leaf nodes, each written as its own object. The newer format and Entwine EPT used a coarser, more adaptive partitioning that packed more points per tile before writing an object.

Two tools compressing the same points to roughly the same byte count made a completely different call on how many discrete objects to split those bytes across. That call is invisible in a compression benchmark. It only shows up when you count PUT requests, and it is exactly the kind of decision that gets made once, early, inside a tiling library’s defaults, and then silently multiplies every downstream cost by whatever factor the defaults picked.

Why object count is its own cost axis, not a rounding error on byte count

A handful of mechanisms turn “more objects, same bytes” into a real bill, on any S3-compatible storage, not a specific vendor’s quirk:

• Request-based pricing. Most S3-compatible platforms bill PUT, COPY, POST and LIST calls separately from per-GB storage. Write the same bytes as 25x more objects and you multiply the request-side bill by roughly 25x for zero additional bytes stored.
• Per-object overhead on the storage engine. Every object carries a metadata envelope, at minimum an ETag, a version ID and a key, regardless of how small the payload is. At high object counts that per-object envelope stops being negligible next to the payload itself. I go through exactly this mechanism, and where erasure-coded storage engines specifically fall over on it, in an earlier deep dive on MinIO and small files and its companion post: this pipeline’s tile output is precisely the kind of workload those posts describe from the storage-engine side.
• Listing and lifecycle scanning. Every background process that walks the bucket, garbage collection, lifecycle expiry, replication catch-up, integrity scanning, pays a cost roughly linear in object count. A 25x jump in object count is a 25x jump in how long those passes take, on the same data.
• Orphan cleanup becomes its own service. With enough small tile objects in flight, some inevitably become orphaned, left behind by a failed or retried tiling job with no dataset record pointing at them anymore. On this project that reached the point of needing a dedicated cleanup job just to find and delete orphaned tile objects on S3, a real engineering line item that a coarser tiling format would have made an order of magnitude cheaper to run.

None of this required a badly compressed file. It required a partitioning choice nobody was watching, multiplied by every mechanism above.

What to check before you deploy a pipeline that writes many small objects

• Count objects per unit of input, not just bytes per unit of input, at the same test scale you use for compression benchmarks. If a format comparison spreadsheet has a compression-ratio column and no object-count column, add one before picking a format.
• Check whether the tiling or chunking tool has a tunable minimum tile or chunk size. Most do. The default is rarely tuned for your object-count budget, it is tuned for read performance on the tool author’s benchmark dataset.
• Price the request side of your object storage bill separately from the storage side, and re-run that estimate at your target production scale, not your pilot scale. A format choice that looks free at a 10-million-point pilot can be a very different line item at 180 billion points.
• Ask who cleans up orphaned objects, and how long that job takes today. If nobody can answer that, the answer is probably “nobody, yet,” and object count is quietly setting the clock on when that becomes a production problem.

The throughput side is a separate, additive lever

Object count was not the only bottleneck in this pipeline. Uploading those tiles to S3 turned out to be the single largest share of total pipeline runtime, and fixing that was a distinct, additive win on top of the format migration above, not the same fix. That is its own story, with its own numbers, and it deserves its own post rather than a paragraph tacked onto this one.

Format and chunking defaults set your object count long before anyone looks at a storage bill, and by the time it shows up as cost or a scanner that never finishes, it is a pipeline redesign, not a config change. A Capacity Planning and Platform Sizing review catches this at the design stage; a Data Platform Performance Audit catches it once it is already in production. Book a 15-min intro call to see which fits, or take one focused hour on the question with an Expert Call.