MinIO
MinIO site replication bug: how 684 million HTTP 405s exposed the delete-marker loop
HEAD probes on delete markers answer 405 and replication counts that as success. 20M stuck versions turned the loop into 684M useless requests in 48 h.
data
Benchmark Best Practices, Part 1: Amdahl’s Law Caps Your Scaling Claim
Part 1 of my benchmark best practices series: how I turn a scaling claim into an implied serial fraction with Amdahl’s law, and what the Universal Scalability Law adds. Includes a measured pipeline where assigning more vCPUs per image cut global throughput.
object storage
Why File Count Matters as Much as File Size on Object Storage
Switching tile formats on a LiDAR ingestion pipeline cut output object count by roughly 25x at the same data volume, independent of compression. Here is why object count is its own cost and performance axis on object storage, and what to check before you scale a pipeline that writes many small files.
data
Which Message Broker for IoT and Real-Time Control
Choose a message broker for real-time IoT control: start from the latency budget, test ack mode and packet loss, and when to skip the broker for UDP.
MinIO
MinIO Site Replication: Sync Mode Will Not Save Your RPO
MinIO site replication has no cross-site quorum, and its replication queue exists only inside the source cluster: lose that site and the un-replicated backlog is gone and un-enumerable. Sync mode costs latency without fixing that. The precise model, the RTO/RPO table, and the DR patterns that match real SLAs.
MinIO
MinIO on XFS: Inode Exhaustion and Prefix Design for Lots of Small Files
MinIO has no NameNode. It puts the namespace on XFS as real directories and xl.meta files. On Lots of Small Files that means you can exhaust inodes while df -h still looks fine, and a flat leaf can stall PUT, LIST, scanner, and ILM together. Here is the on-disk model, the inode math, and a prefix recipe that keeps XFS inside a regime you can operate.
MinIO
MinIO and Small Files: When Erasure Coding Becomes 15x Replication
MinIO fixed the HDFS NameNode limit, but it has no index: it writes one xl.meta per object on every drive of the erasure set. On 5 servers of 12 NVMe, MinIO picks a 15-wide set by default, so each small object is stored 15 times over. A worked sizing that looks fine for two years and dies in days.
apache spark
Sail: When Apache Spark Meets Rust (A Practitioner’s Deep Dive)
Sail is an open-source Apache Spark replacement written in Rust. It drops the JVM, speaks Spark Connect, and runs 4 to 6 times faster than Spark with native accelerators on ClickBench. A deep dive into its architecture, benchmarks, and production readiness.
AI
Why Your GPU Infrastructure Costs 40% More Than It Should
Most AI infrastructure teams spend 35-60% more on GPU compute than they need to. The cause isn’t cloud pricing. It is architecture, and it is fixable.
apache spark
Spark on Kubernetes Reserves CPU It Never Uses. Here’s the Overcommit Fix.
Spark sets executor CPU requests equal to limits, so a Kubernetes cluster reserves twice the CPU it uses and refuses to schedule pending pods. Kubernetes has no native overcommit. Here is the mutating-webhook operator I use to fix it.