The Fastest Lock Is One You Don't Share: Linux's Scaling Wall

Every year the core counts go up, and every year the same quiet assumption ships with them: that the operating system underneath will spread across those cores like water. It will not. A single Linux kernel is a single shared data structure at heart, and some of its locks are load-bearing walls. You cannot tune them away, because they are not misconfigurations. They are the architecture.

Here is the sharpest number we measured. On a 48-core Xeon, one process doing creat and unlink in a directory sustains 467,000 operations per second. Forty-eight processes, each on its own dedicated core, each working on its own file, sustain 188,000 between them. Not per process. Total. Every core you add past the first makes the machine slower at this job, until 48 cores deliver 40% of what one core managed alone.

This post does three things. It measures that wall with will-it-scale, the community’s standard scalability microbenchmark, the same one Intel’s 0-day robot uses to report kernel regressions. It names the locks responsible. And then it walks around the wall the only way that works: by holding the hardware exactly constant and splitting one kernel into two.

Benchmark Setup

The machine is the same dual-socket Intel Xeon Gold 5418Y from our KVM comparison: 24 cores per socket, SMT disabled, 128 GB of RAM split evenly across two NUMA nodes. One kernel build, 7.0.0-mk1+, boots every configuration in this post.

will-it-scale runs one benchmark loop in N processes, each pinned to its own dedicated core, and reports the summed throughput as N grows. Processes mode is deliberately the easy case for the kernel: no shared address space, no shared file descriptors, nothing shared in userspace at all. Whatever refuses to scale is shared inside the kernel. We ran seven tests:

Three configurations, and the fairness work is where the effort went. The single-kernel baseline runs on the host with all 48 cores; will-it-scale fills socket 0 first, so every comparison at 24 tasks and below is socket-local. The multikernel side uses kerf to boot spawn kernels on socket 1, with memory allocated on socket 1’s NUMA node, booted with the same mitigation flags as the host command line, running the identical binaries from the same rootfs, with tmpfs mounted over /tmp on both sides. At every comparison point the same number of cores on one socket are busy, so turbo bins match. Every number is the average over five one-second samples after warmup.

kerf create wis0 --cpus=<12 cores of socket 1> --memory=8192MB
kerf load wis0 --kernel=vmlinux --rootfs-dir=/root/wis-rootfs \
    --entrypoint="/bin/bash /run-wis.sh"
kerf exec wis0

One Kernel, 48 Cores: The Wall

Run the sweep on the single kernel and the seven tests split cleanly into three families.

The control behaves like a control: getppid1 reaches 35.4x at 48 cores, and the shortfall from 48x is the hardware’s all-core turbo bins, not the kernel; futex4 and poll2 land within a point of the same curve. These three lines are the proof that the machine itself scales, which converts every flat line below them from a suspicion into a verdict.

Then the wall. open1 climbs normally to 8 cores and stops: 9.6 million opens per second at 8 tasks, 9.6 million at 24, 8.4 million at 48. Sixteen additional dedicated cores add exactly nothing, then the cross-socket step subtracts. Every task opens its own file, but every open walks the same parent directory dentry and bumps the same shared reference counts on the way (on this host, apparmor’s file-open label accounting is on that path too). Nothing is held long enough to be called contention in a profiler. The cachelines just have one home and 48 visitors.

unlink1 is the same story with a write lock in it, and a write lock makes the story regress instead of plateau. Every creat and every unlink takes the parent directory’s i_rwsem exclusively. One directory means one rwsem; one rwsem means the whole machine advances one create or unlink at a time, while the lock’s cacheline ping-pongs between more and more cores:

That downward slope is worth staring at. It is not a benchmark artifact and it is not exotic: it is fs/namei.c doing what a shared mutable directory requires under one kernel. Mail spools, session stores, build systems, lock-file protocols, anything that churns files in a common directory lives on some part of this curve. And the threads-mode variants are crueler still: mmap1 with threads collapses to 0.09x of a single task at 48 cores, one mmap_lock strangling an entire address space.

The middle family, page_fault1 and mmap1 processes mode, scales but leaks: 15x and 26x at 48 cores, paying rent to the page allocator and to a commit-accounting counter we will meet properly in a moment.

Two Kernels, Same 24 Cores

Now the experiment this site exists for. Take 24 cores of one socket and 24 total tasks, and change only who manages them: one Linux kernel with 24 cores, versus two multikernel instances with 12 cores each, running 12 tasks each, concurrently, their throughputs summed. Same silicon, same clocks, same busy-core count, same binaries. If the wall is really made of shared kernel state, splitting the kernel should tear it down, and the control tests should not move at all.

Read the controls first, because they are what make the rest believable. getppid1 at 1.00x to three digits says the two configurations are running on indistinguishable hardware: same frequency, same cache, no spawn-kernel overhead on the syscall path. We also booted a single 24-core spawn as a second control, and it reproduces the single-kernel curves test for test, open1 plateau included. A spawn kernel is not faster Linux. It is just Linux, on fewer cores.

Which is the point. The 2.6x on unlink1 and 2.0x on open1 were not bought with a better kernel, a patched lock, or a tuned knob. Two kernels means two /tmp inodes, two rwsems, two dcaches, two of every cacheline that had one home before. The per-kernel curves tell you the headroom left: each 12-core kernel still flattens near its own 8-core mark on open1, so 4 kernels of 6 cores would stack another multiple on top. Partitioning turns a lock wall into a per-partition speed bump, and the aggregate compounds with every split. That compounding, incidentally, is the honest version of the pitch: at 48 cores against unlink1’s negative slope, two 12-core kernels using half the machine already beat the full 48-core kernel by 4x.

And unlike a sharded set of VMs, nothing here paid rent for the privilege: as we measured last week, each partition is a native kernel on its own cores, syscalls at native cost, sleeping in real C-states.

The Global Lock We Didn’t Order

Honesty section. Our first multikernel run did not look like the table above: mmap1 came in at 0.45x, twice as slow as one kernel, and page_fault1 sagged too. Wrong direction, and too interesting to ignore, so we profiled inside the spawn kernel. Sixty-eight percent of all cycles were inside percpu_counter_add_batch, spinning on one global spinlock: vm_committed_as, the counter behind the Committed_AS line of /proc/meminfo.

The mechanism is a heuristic in mm/mm_init.c that nobody thinks about because on big machines it never fires. The counter is per-CPU with a spill batch sized as totalram / ncpus / 4. Our first spawns had 4 GB and 12 cores: batch of 21,214 pages. mmap1 maps and unmaps 128 MB at a time, which is 32,768 pages, over the batch on every single call. Every mmap and every munmap on every core skipped the per-CPU fast path and took the same global lock. On the 128 GB host the batch is 169,000 pages and the lock is never touched: same kernel, same test, opposite behavior, decided by a division.

This is not a multikernel bug, and that is exactly why it earns this section: any Linux instance with that RAM-to-CPU ratio behaves this way, including a 4 GB, 12-vCPU cloud VM doing large mappings. Resize the spawn to 8 GB and the batch clears the mapping size; mmap1 jumps from 0.45x to the 1.23x in the table, and the lock vanishes from the profile. Two lessons came home with us. Practical: partition memory and cores together, keeping RAM / cores / 4 above your largest mapping. Upstream: under the default overcommit policy this counter is purely statistical, nothing ever reads it to make a decision, so the batch could be far more generous on small instances. That patch discussion is worth having on linux-mm.

Caveats

The multikernel result is a claim about share-nothing partitioning, stated plainly: in the two-kernel configuration the 24 tasks never share a directory, a dcache, or an allocator, because each kernel has its own. That is the product, not a trick, but it draws the boundary honestly: a workload that fundamentally needs one shared mutable directory, or one 48-core address space, cannot be split this way, and thread-mode benchmarks are the proof. Multikernel scales the machine for workloads built of independent processes, which is what most server fleets already are, sharded web workers, per-core network stacks, build farms, CI runners.

page_fault1 at 0.96x is page zeroing bound rather than lock bound at this core count, so parity is the expected result, and 0.96 is within the turbo noise we measured between runs. The single-kernel numbers above 24 tasks include a cross-socket step visible in the curves; every one-versus-two comparison in this post stays inside one socket to keep that out of the ratios. And kerf currently has no --node flag for the memory pool, so socket-local placement takes one line of its Python API; that flag is on our list.

Reproduce It

will-it-scale builds in a minute: clone, make, and each test is a standalone binary that takes a task count. The sweep is a shell loop. The multikernel side is kerf create, kerf load with a rootfs directory containing the same binaries, and kerf exec, with results read back from the DAXFS overlay on the host. Nothing in the method needs our hardware: pick the biggest box you have, run the single-kernel sweep, find your wall (open1 and unlink1 will find it for you), then split the same cores between two spawns and run it again. The absolute numbers will move with your core count and kernel version. The shape will not, because the shape is the architecture.

The Shape of the Wall

Add it up. A single Linux kernel turned 48 dedicated cores into 40% of one core on unlink, capped open at its 8-core number forever, and did both while the control tests scaled 35x on the very same silicon, which is the kernel confessing that the limit is its own shared state. Two kernels on the same cores gave back 2.6x and 2.0x immediately, with the controls at 1.00x certifying that nothing else changed, and the per-kernel curves showing another multiple still on the table at the next split.

The industry’s answer to this wall has been twenty years of heroic lock surgery, and the flat open1 line shows both its successes and its limit: the locks that remain are the ones that define what “one kernel” means. You do not shard those. You shard the kernel.

Multikernel is open source. If your machines have more cores than your kernel can honestly use, we would love to hear from you at contact@multikernel.io.