Before a scale-up, a migration to Kubernetes or a cluster renewal, your Spark pipelines deserve better than a sizing guess. As an independent Spark consultant I measure first, then fix: partitioning, shuffle, memory, cost per pipeline. I have run Spark on Kubernetes in private cloud for production workloads, and I also step in when jobs slow down month after month with no visible cause.
Tuning
Tuning starts with measurement: job timelines, shuffle stages, partition skew, executor memory profiles. Then the fixes: file format and partitioning, parallelism, cache management, shuffle and memory configuration, source-side partition pruning. Every change is validated by a before and after measurement on the same dataset.
Benchmark
A serious Spark benchmark reproduces your real workload, not a demo dataset: volume, seasonality, job concurrency. When a reference dataset is useful, it is chosen for proximity to your transformations, and results stay reproducible: fixed environment, several iterations, reported variance. The goal is a comparison that holds up in front of engineering leadership and vendors alike.
Scale
Scaling without rewriting: executor autoscaling, Spark on Kubernetes, workload separation, queue priorities, cost per pipeline. The scale-up plan includes alert thresholds and the marginal cost of doubling volume, so growth is planned in euros rather than in incident hours.
This is not staff augmentation or development work: for a single technical decision, the flash review is enough.
FAQ
Why do my Spark jobs slow down over time?
Because three quantities grow silently: processed volume, table width and job concurrency. A pipeline calibrated on an initial scope becomes under-partitioned, shuffle grows and the cache saturates. A Spark audit measures those three quantities and recomputes the configuration that matches today’s volume.
How do you measure a Spark performance gain credibly?
By freezing the environment and the dataset, running several iterations before and after the change, and reporting median and variance rather than the best run. A measurement that omits volume, concurrency or variance does not survive a technical review.
Spark on Kubernetes or dedicated clusters?
Kubernetes brings autoscaling and sharing with other workloads, at the price of an execution layer you must master. Dedicated clusters reduce performance variance. The choice depends on job concurrency and latency sensitivity, and is settled with your measurements, not with a doctrine.