What a Cache Line Costs You

I had a worker pool that got slower as I added threads. Four workers beat eight. Eight beat sixteen by a wide margin. The work was embarrassingly parallel — each worker touched its own slot in an array and nothing else.

That last sentence is the bug, and it took me an afternoon to see it.

The setup

The pool looked roughly like this:

struct Worker {
    processed: u64,
    errors: u64,
}

let mut workers: Vec<Worker> = (0..n).map(|_| Worker::default()).collect();

Each thread got &mut workers[i] and incremented its own counters in a tight loop. No locks, no atomics, no shared state. Textbook.

What actually happens

Worker is 16 bytes. A cache line on x86-64 is 64 bytes. So four workers’ counters land on the same cache line.

When thread 0 writes to workers[0].processed, the cache coherence protocol invalidates that line in every other core’s cache. Threads 1, 2 and 3 aren’t reading that memory — but their counters live on the same line, so their next write stalls while the line is re-acquired.

This is false sharing. The threads are logically independent and physically contended.

Past four threads the naive version is slower than a single thread doing all the work.

The fix

Pad each worker to a full cache line so no two share one:

#[repr(align(64))]
struct Worker {
    processed: u64,
    errors: u64,
}

That’s it. One attribute. The array grows 4x in memory and the whole thing gets 40x faster at sixteen threads.

How to see it

perf will tell you if you know to ask:

perf c2c record -- ./bench
perf c2c report

The HITM column — “hit modified”, a load that found the line dirty in another core’s cache — is the tell. In the naive build it accounted for most of the memory traffic. After padding it goes to nearly zero.

The lesson I keep relearning: logically independent is not physically independent. The hardware only sees addresses.