Tag: Apache Iceberg

Two warehouses linked by conveyors illustrate S3A rename copying objects to new keys and deleting the originals.

Spark Output Committers on S3, MinIO and Ceph: Stop Committing by Rename

S3A committers avoid copying Spark output during commit, but they do not publish a dataset atomically. This guide shows how to configure and verify them in upstream Spark, what changes on Amazon EMR, and how to protect readers and concurrent writers.

By Julien Laurenceau, 2 weeks ago

data

Beyond Parquet and Avro: Choosing a Data File Format by Access Pattern

Lance, Vortex, Nimble, FastLanes, GeoParquet, cloud-optimized geo formats and new time-series encodings each optimize one access pattern on one storage tier. A field guide to what their published numbers measure, where tuned Parquet still wins, and how to choose.

By Julien Laurenceau, 3 weeks ago