PepiteData Julien Laurenceau
  • Home
  • Services
  • Audits
  • Expert Call
  • About
  • Blog
  • Contact

apache spark

Two warehouses linked by conveyors illustrate S3A rename copying objects to new keys and deleting the originals.
apache spark

Spark Output Committers on S3, MinIO and Ceph: Stop Committing by Rename

S3A committers avoid copying Spark output during commit, but they do not publish a dataset atomically. This guide shows how to configure and verify them in upstream Spark, what changes on Amazon EMR, and how to protect readers and concurrent writers.

By Julien Laurenceau, 3 weeks ago
Cartoon Spark engine lifted from a chassis beside a Sail engine, asking whether the results stay the same.
apache spark

Sail vs Spark: Architecture, Benchmarks, Dask and Numba

Sail changes the engine behind Spark Connect; Dask distributes Python work; Numba compiles numerical functions. I compare their roles, qualify Sail’s published benchmarks and outline the checks needed before a migration.

By Julien Laurenceau, 5 months2026-06-14 ago
Abstract grid of glowing compute cells densely packed into reserved cluster capacity
apache spark

Spark on Kubernetes Reserves CPU It Never Uses. Here’s the Overcommit Fix.

Spark sets executor CPU requests equal to limits, so a Kubernetes cluster reserves twice the CPU it uses and refuses to schedule pending pods. Kubernetes has no native overcommit. Here is the mutating-webhook operator I use to fix it.

By Julien Laurenceau, 6 months2026-05-31 ago
apache spark

Apache Spark smoke tests on kubernetes

Introduction Benchmarking remains a critical (and often underestimated) tool when designing or validating large-scale data platforms. While many teams rely on synthetic workloads or production replays, standardized benchmarks still play a key role when comparing architectures, tuning clusters, or validating infrastructure choices. TPCx-HS is one of those benchmarks: designed to Read more

By Julien Laurenceau, 10 months2026-01-12 ago
  • Privacy Policy
  • Mentions légales
  • CGV
  • Cookies
Hestia | Developed by ThemeIsle