Your Neighbor's malloc Is Your p99: One Kernel Must Choose a Loser

Here is a task minding its own business: every five milliseconds it faults in 128 MB of its own private memory and hands it back, 56 milliseconds per cycle, 5,100 cycles every five minutes, steady as a clock. Now a batch job starts hoarding memory on the same machine. Different cores. Different memory. Not one byte shared between them. The task’s median cycle is now 14 seconds. Its p99 is 58.8 seconds. It completes 19 cycles in five minutes. Nothing it can see has changed, and it has stopped working, because the thing it shares with the hog is not visible from userspace at all: one kernel’s reclaim machinery, one set of watermarks, one swap device.

So we reached for the sanctioned fix, memory.high on the hog’s cgroup, and it worked flawlessly. The victim came back to 58 milliseconds as if the hog were gone. It effectively was: in the same five-minute window in which the unconfined hog completed 29,719 full passes over its working set, the confined hog completed 10. Not 10,000. Ten.

Protection and strangulation are the same mechanism. On a single Linux kernel, memory isolation between colocated workloads is not a setting where both run; it is a dial that chooses which one does not. cgroups can move the loss, aim it, schedule it, but not remove it, because both tenants live inside one memory-management machinery that must put the pressure somewhere.

This post measures that dial’s two endpoints with components of the kernel community’s own memory-management benchmark suite. Then it holds the hardware constant, splits the machine into two kernels, and shows the trade dissolving: both workloads running at once, the victim’s p99 bounded at 232 milliseconds, the hog at full throughput. Then it chases the residual 4x over the victim’s quiet floor to its actual home, which turns out not to be software at all. It is the tail-latency companion to our will-it-scale post, which measured what one kernel’s locks do to throughput; this is what one kernel’s reclaim does to your p99, and the failure mode is worse: throughput collapse is gradual, but the tail fails like a cliff, and the standard tool for containing it pushes the other tenant off a different cliff.

The Workload: Mel Gorman’s Stutter, Taken Apart

The antagonist pair comes from mmtests, the memory-management test harness maintained by Mel Gorman and used across the kernel MM community. Its stutterp benchmark exists because this exact failure kept reaching real users: a foreground task stuttering while background work leans on memory. It has three components, and they map perfectly onto a colocation experiment:

We used stutterp’s own C programs and its fio job template, with our own driver around them, because the stock harness runs all three components inside one kernel and we needed to move them between cgroups and kernels. Sizing follows stutterp’s formula: total working set is 120% of the memory arena, with the file share set to the dirty ratio. The arena is one NUMA node.

The machine is the same dual-socket Xeon Gold 5418Y as the previous posts: 24 cores per socket, no SMT, 64 GB per NUMA node, 8 GB of swap. Placement is identical in every configuration: the victim runs on the same four socket-1 cores with node-1 memory, and the hog mix runs on the other twenty socket-1 cores, bound to node 1, where its 54 GB of anonymous memory plus 13 GB of dirty files against a 64 GB node produces sustained hard reclaim. In the two-kernel configurations the victim’s four cores and 2 to 4 GB of node-1 memory become a multikernel instance booted with kerf, running the same kernel build; the hog stays on the host kernel, unchanged, with the same disk underneath its writeback. The only variable, ever, is where the victim lives. Each measurement is a 300-second window after a 90-second warmup.

One Kernel: the Cliff

Quiet, the victim is boring: 56 ms median, 92 ms p99, cycle after cycle. Its 128 MB burst is a deliberate amplifier, the shape of a process starting, a JIT warming, a cache resizing; each cycle asks the kernel for 32,768 fresh pages and gives them back.

Beside the unconfined hog, the victim does not degrade. It stops. The node is in sustained overcommit, so every allocation the victim makes competes with reclaim for the same watermarks: direct reclaim in the victim’s own context, allocation stalls behind kswapd, its just-touched pages eligible to be swapped out beneath it by pressure it did not create. We ran this configuration twice, and the two runs bracket how unstable this regime is: one run finished 49 cycles with a median of 824 ms and a 35.6-second p99; the other finished 19 cycles with a median of 14 seconds and a 58.8-second p99. The exact numbers are weather. The cliff is climate.

Note what this is not. The victim has its own dedicated cores; this is not CPU contention. It faults its own private memory; it shares nothing with the hog in userspace. Everything between them is inside the kernel: one set of watermarks, one reclaim machinery, one swap device, one flusher.

The Cgroup Answer Works. Watch What It Costs.

cgroup v2 gives an operator two memory limits: memory.max, a hard cap enforced by the OOM killer, and memory.high, a soft cap enforced by throttling, where any task allocating above the limit is forced to reclaim the cgroup’s own memory and then made to sleep. memory.high is the recommended tool for containing a noisy neighbor precisely because it degrades instead of kills.

So that is what we set: memory.high on the hog’s cgroup, at 48 GB. On the victim’s side of the ledger it is flawless: 58 ms median, 91 ms p99, statistically indistinguishable from an idle machine, reproduced twice.

The hog’s side: 10 completed passes, against 29,719 unconfined. Roughly a 3,000x suppression, and the mechanism explains why it cannot be tuned away. The hog’s 54 GB of anonymous memory does not fit under a 48 GB cap, and anonymous pages can only leave memory through swap, of which there are 8 GB. So the cgroup fills, swaps out what it can, and then has nowhere to go: every page a hog touches over the cap runs cgroup-local reclaim in the hog’s own context, evicting another hog page, and then serves a penalty sleep that memory.high scales up for persistently-over cgroups. The hogs spent the window asleep or grinding their own LRU lists. The victim saw a quiet machine because the machine, minus the victim, had been put to sleep.

Could a smarter limit find the middle? For this workload shape there is no middle to find. Set the cap above the hog’s total demand and it never engages; you get the 58-second column. Set it below the anonymous working set, as any safe sizing must when swap is finite, and you get the 10-passes column. The band in between is a few gigabytes wide, requires knowing the hog’s working set in advance to that precision, and vanishes the moment the hog grows. Operators know this dial by its endpoints, which is why in practice so many deployments run with limits unset and take the first column, or set them and take the third, or run an OOM daemon whose remedy, killing the hog, is the third column administered faster.

Two Kernels: Nobody Loses

Same cores, same node, same hog at full tilt on the host kernel. The victim’s four cores and a slice of node-1 memory become a separate kernel instance; the victim inside knows nothing of the hog, and more to the point, its kernel does not either. No shared watermarks, no shared LRU, no shared flusher, no shared swap.

The victim’s quiet floor in a spawn is 53 ms with a p99 of 53 ms, tighter than the host’s own quiet floor, and identical whether the instance has 2 GB or 4 GB. Beside the full-speed hog it holds 208 ms median, 232 ms p99, 240 ms worst case observed. Read the shape, not just the level: on the shared kernel the tail was three orders of magnitude above the median and the sample count collapsed; here the whole distribution sits in a 33 ms band, 1,550 cycles completed, no outliers at all. A shifted-but-tight distribution is what hardware contention looks like. A distribution smeared across three decades is what kernel entanglement looks like.

And the hog: 27,501 passes, full speed. Both columns healthy at once, which neither single-kernel configuration achieved, because the trade they were forced to make does not exist here.

Chasing the Last 4x to DRAM

208 ms against a 53 ms floor still leaves a 4x, and it deserved an explanation rather than a shrug. Two experiments decompose it.

First we kept the hog’s twenty cores on socket 1 but bound its memory to the other NUMA node, so it no longer touches the victim’s DRAM. The victim recovered to 59 ms median, 67 ms p99: 96% of the residual gone. What remains, about 6 ms, is the shared last-level cache and the all-core turbo bins, and that is the entire footprint of twenty screaming cores on the same die. The 4x was DRAM contention on the shared node.

But not bandwidth, and the second experiment proves it. The hog’s measured traffic is roughly 10 GB/s against a socket that can move an order of magnitude more; the channel is not full. The victim’s fault path suffers latency inflation: the hog’s random one-word-per-page strides, plus swap and writeback DMA, destroy DRAM row-buffer locality, so every one of the victim’s cache misses waits longer at a controller that is busy, not saturated. So we tried the resource-director knob built for this, MBA, capping the hog’s CLOS to 10% memory bandwidth. Nothing moved: the victim held at 211 ms, and the hog’s pass rate did not even drop. A latency-bound access pattern with one miss in flight slides completely under a bandwidth-credit throttle. The hardware dial for memory interference does not touch this workload.

The dial that does is placement, and placement is a thing multikernel already owns: which node a kernel instance’s memory pool comes from is a first-class parameter of creating one. A pool on the victim’s own node, or the socket-aligned layouts from the cross-socket section of the will-it-scale post, gives the victim a near-floor tail against an unthrottled antagonist, using no throttling at all.

What This Does Not Show

Honesty section, as usual. The hog is sized as a worst case: 120% of the node with thin swap is sustained hard overcommit, the failure mode of an unlimited batch job, not the steady state of a well-run fleet. The shared-kernel numbers vary wildly between runs, which is itself part of the finding, but it means the 14-second median is one draw from a bad distribution, not a constant. The victim’s 128 MB burst amplifies the effect; a steadily-serving process would show the same shape at lower magnitude. The hog’s pass counts were initially undercounted by stdio buffering; the numbers above are from a line-buffered rerun, which is why we can say 10 rather than zero. The memcg arm used memory.high alone; a memory.max cap would OOM-kill the hogs instead, which is the same trade with a shorter funeral. And the victim-side comparison is deliberately narrow: one metric, one workload shape, one node. The throughput side of the multikernel story is the previous post’s; this one is only about tails.

One more honest note: the two-kernel victim was measured against the hog running on the host kernel, so this is a two-kernel machine in the configuration that matters, victim isolated from antagonist. A fleet of hogs each in its own instance is the same argument applied more times.

Summary

Conclusion

The locks post showed a single kernel converting cores into contention. This post shows it converting neighbors into tail latency, and shows that the built-in remedy is a choice of victim: uncontained, the latency-sensitive task starves under the hog’s reclaim; contained, the hog is throttled 3,000x so the latency-sensitive task can breathe. Both outcomes come from the same root, one memory-management machinery with one set of watermarks arbitrating everything on the machine, and no cgroup boundary changes whose machinery it is.

Two kernels dissolve the choice. The victim’s tail stays bounded within a 33 ms band against an antagonist running at full speed, because there is no shared reclaim to entangle them; what interference remains is the DRAM they still share, it yields to placement rather than throttling, and placement is a parameter multikernel sets per instance as a matter of course. The kernel-side interference went to zero. The hardware-side interference went to a knob we hold.

You can spend the dial choosing which tenant loses. Or you can stop sharing the kernel.

Multikernel is open source. If your p99 has neighbors, we would love to hear from you at contact@multikernel.io.