The Fastest Lock Is One You Don't Share: Linux's Scaling Wall

Every year the core counts go up, and every year the same quiet assumption ships with them: that the operating system underneath will spread across those cores like water. It will not. A single Linux kernel is a single shared data structure at heart, and some of its locks are load-bearing walls. You cannot tune them away, because they are not misconfigurations. They are the architecture.

Here is the sharpest number we measured. On a 48-core Xeon, one process doing creat and unlink in a directory sustains 467,000 operations per second. Forty-eight processes, each on its own dedicated core, each working on its own file, sustain 188,000 between them. Not per process. Total. Every core you add past the first makes the machine slower at this job, until 48 cores deliver 40% of what one core managed alone.

This post does four things. It measures that wall with will-it-scale, the community’s standard scalability microbenchmark, the same one Intel’s 0-day robot uses to report kernel regressions. It names the locks responsible. It walks around the wall the only way that works: holding the hardware exactly constant and splitting one kernel into two. And it stretches the wall across the socket boundary, where it grows taller and the same split pays twice as much. Running alongside all four are the benchmarks that should not move, and do not, because a mechanism claim that cannot fail is not a claim.

One Kernel, 48 Cores: The Wall

The machine is the same dual-socket Intel Xeon Gold 5418Y from our KVM comparison: 24 cores per socket, SMT disabled, 128 GB of RAM split evenly across two NUMA nodes. will-it-scale runs one benchmark loop in N processes, each pinned to its own dedicated core, and reports the summed throughput as N grows; it fills socket 0 first, so every curve below is socket-local through 24 tasks. Processes mode is deliberately the easy case for the kernel: no shared address space, no shared file descriptors, nothing shared in userspace at all. Whatever refuses to scale is shared inside the kernel. Every number in this post is processes mode unless a section says otherwise; the threads-mode results answer a different question and get their own section near the end. We ran ten tests:

Run the sweep on the single kernel and the ten tests split cleanly into three families.

The control behaves like a control: getppid1 reaches 35.4x at 48 cores, and the shortfall from 48x is the hardware’s all-core turbo bins, not the kernel; futex4 and poll2 land within a point of the same curve. These three lines are the proof that the machine itself scales, which converts every flat line below them from a suspicion into a verdict.

Then the wall. open1 climbs normally to 8 cores and stops: 9.6 million opens per second at 8 tasks, 9.6 million at 24, 8.4 million at 48. Sixteen additional dedicated cores add exactly nothing, then the cross-socket step subtracts. Every task opens its own file, and when we chased the ceiling to its exact cacheline it turned out not to be the dcache walk, which scales cleanly, but AppArmor’s file-open label accounting: one shared refcount bumped on every open, even for unconfined tasks. Boot the same kernel with that module off and the plateau simply disappears, 9.6M becoming 41M at 24 tasks. We keep the default-stack numbers throughout because they are what AppArmor distributions actually run, but the attribution matters for reading the tables below. Nothing here is held long enough to be called contention in a profiler. The cacheline just has one home and 48 visitors.

unlink1 is the same story with a write lock in it, and a write lock makes the story regress instead of plateau. Every creat and every unlink takes the parent directory’s i_rwsem exclusively. One directory means one rwsem; one rwsem means the whole machine advances one create or unlink at a time, while the lock’s cacheline ping-pongs between more and more cores:

That downward slope is worth staring at. It is not a benchmark artifact and it is not exotic: it is fs/namei.c doing what a shared mutable directory requires under one kernel. Mail spools, session stores, build systems, lock-file protocols, anything that churns files in a common directory lives on some part of this curve.

The middle family, page_fault1 and mmap1 processes mode, scales but leaks: 15x and 26x at 48 cores, paying rent to the page allocator and to a commit-accounting counter we will meet properly in a moment.

The network rows split along the same line. tcp_rr1 and udp1, one established loopback connection or socket pair per task, ride the control curve to 35x: data transfer on an established connection touches per-connection state only, and the kernel proves it by scaling it. tcp_conn1 is the one with a wall in it: connection setup climbs to 1.81 million connects per second at 24 tasks, 12.9x of one task against the control’s 17.7x at the same count, and then the cross-socket step subtracts, 32 tasks delivering less than 24 and 48 never recovering the peak. Where that contention actually lives has a surprising answer, measured in its own section below.

Two Kernels, Same 24 Cores

Now the experiment this site exists for. Take 24 cores of one socket and 24 total tasks, and change only who manages them: one Linux kernel with 24 cores, versus two multikernel instances with 12 cores each, running 12 tasks each, concurrently, their throughputs summed. Same silicon, same clocks, same busy-core count, same binaries, same kernel build; the full fairness checklist lives in the reproduction section at the end. If the wall is really made of shared kernel state, splitting the kernel should tear it down, and the control tests should not move at all.

Read the controls first, because they are what make the rest believable. getppid1 at 1.00x to three digits says the two configurations are running on indistinguishable hardware: same frequency, same cache, no spawn-kernel overhead on the syscall path. We also booted a single 24-core spawn as a second control, and it reproduces the single-kernel curves test for test, open1 plateau included. A spawn kernel is not faster Linux. It is just Linux, on fewer cores. And tcp_rr1 and udp1 join the parity block from the network side: an established connection’s state lives wherever the connection lives, so splitting kernels rightly buys it nothing.

Which is the point. The 2.6x on unlink1 was not bought with a better kernel, a patched lock, or a tuned knob. Two kernels means two /tmp inodes, two rwsems, two dcaches, two of every cacheline that had one home before. The open1 row carries an asterisk the others do not: its 2.02x comes from splitting AppArmor’s shared label, and disabling that module lifts both configurations to 40.8M and returns the ratio to 1.00x, so that win belongs to configuration, not architecture (unlink1 holds its win either way, 2.24x with the module off). Partitioning turns a lock wall into a per-partition speed bump, and the aggregate compounds with every split. That compounding, incidentally, is the honest version of the pitch: at 48 cores against unlink1’s negative slope, two 12-core kernels using half the machine already beat the full 48-core kernel by 4x.

And unlike a sharded set of VMs, nothing here paid rent for the privilege: as we measured last week, each partition is a native kernel on its own cores, syscalls at native cost, sleeping in real C-states.

The Global Lock We Didn’t Order

Honesty section. Our first multikernel run did not look like the table above: mmap1 came in at 0.45x, twice as slow as one kernel, and page_fault1 sagged too. Wrong direction, and too interesting to ignore, so we profiled inside the spawn kernel. Sixty-eight percent of all cycles were inside percpu_counter_add_batch, spinning on one global spinlock: vm_committed_as, the counter behind the Committed_AS line of /proc/meminfo.

The mechanism is a heuristic in mm/mm_init.c that nobody thinks about because on big machines it never fires. The counter is per-CPU with a spill batch sized as totalram / ncpus / 4. Our first spawns had 4 GB and 12 cores: batch of 21,214 pages. mmap1 maps and unmaps 128 MB at a time, which is 32,768 pages, over the batch on every single call. Every mmap and every munmap on every core skipped the per-CPU fast path and took the same global lock. On the 128 GB host the batch is 169,000 pages and the lock is never touched: same kernel, same test, opposite behavior, decided by a division.

This is not a multikernel bug, and that is why it earns this section: any Linux instance with that RAM-to-CPU ratio behaves this way, including a 4 GB, 12-vCPU cloud VM doing large mappings. Resize the spawn to 8 GB and the batch clears the mapping size; mmap1 jumps from 0.45x to the 1.23x in the table, and the lock vanishes from the profile. The lesson is to partition memory and cores together, keeping RAM / cores / 4 above your largest mapping; and since the counter is purely statistical under the default overcommit policy, a more generous batch on small instances is a patch discussion worth having on linux-mm.

The Walls No Flag Removes

unlink1’s wall invites an obvious mitigation, and will-it-scale ships the proof it works: unlink2 gives every task its own directory and the collapse disappears. So we went hunting for the walls no layout or boot flag can dodge, and found the first one missing from the suite entirely: there has never been a rename test, because the suite grew around locks people were actively fixing and this one has been accepted as unfixable for decades. We wrote it: rename1 is thirty lines in the standard harness, published with stat2 in our will-it-scale fork. Each task renames its own file between its own two private directories, nothing logically shared, yet every cross-directory rename takes s_vfs_rename_mutex, one mutex per superblock, there to keep concurrent renames from creating directory cycles. It sits under the atomic-replace idiom, the temp-file-rename of maildirs, package managers, and every editor that saves safely.

The single-kernel curve is the starkest in this post: throughput peaks at two tasks and decays until 24 dedicated cores deliver less than one task alone. Two kernels hold 2.14x, controls at 1.00x, and the security-module audit moves it 1%. The only single-kernel escape is one filesystem per worker, which surrenders the shared namespace, exactly the concession the multikernel split makes; partitioning is the only exit either way.

The second wall is the one your machine hits most often without noticing: every process start makes the loader stat and open the same shared libraries, every worker stats the same config file, all cores, one path, all day. stat2 (which we added alongside rename1) stats one shared file from every task; the suite’s open3 opens and closes it. The shared object is a single dentry, and every call takes a reference on it: one cacheline, updated by every core. One kernel caps stat2 at 2.8x of a single task and open3 at 2.3x, whatever the core count, and both pass the audit that open1 failed, moving about 1% with the security module off. We ran the suite’s open2 alongside them as the foil: a private file in a private directory, nothing logically shared at all. Two kernels give the shared-dentry pair back:

The open2 row is the audit cutting the other way: nothing shared, yet a 1.93x “win” that a single kernel reclaims entirely once the security module is off, the same story as open1. Structural or configurational: the whole post reduces to learning to tell them apart. And the structural rows carry honest semantics: a multikernel escapes the shared-dentry wall because each kernel genuinely has its own filesystem and its own copy of what its partition loads. The wall exists because one kernel insists all 48 cores agree on one dentry; two kernels simply have two.

The obvious objection is that our two kernels had two filesystems, so of course they had two dentries. Multikernel has an answer for workloads that genuinely need one filesystem: DAXFS, a shared-memory filesystem that multiple kernels mount simultaneously over one physical region. So we ran the strongest version of the experiment: open4, which is open3 with a fixed path so that every kernel opens the very same file, over one daxfs, one file existing exactly once in storage, both kernels mounting the same physical pages.

The split survives real sharing, controls at 1.00x, and the number says why: 7.41M is what the private-filesystem configuration scored too. The wall was never the storage. Every kernel builds its own dentries and inodes over whatever it mounts, so the refcount cacheline that serializes one kernel exists once per kernel, not once per file. DAXFS shares the bytes; it does not share the locks. One measured boundary keeps this honest: this is the read path. Two kernels mutating shared metadata through one daxfs is a different experiment, and until it is run, the shared-mutable-directory concession in the caveats stands.

The Wall Below the Namespace

The networking rows earn their own audit, because networking is where single-kernel tenancy has its most celebrated sharding tool: the network namespace. Every container platform leans on it, and it is the obvious objection to tcp_conn1’s row in the table above. Surely a netns per worker, with its own loopback, its own port space, its own everything, shards whatever connection setup contends on.

We wrote the test. tcp_conn2 is tcp_conn1 with one call added: each task unshares its own network namespace and brings up its own loopback before running the identical connect/accept/close loop, so nothing above the kernel is shared at all. The result is the cleanest null in this post: 1.80M connects per second at 24 tasks against tcp_conn1’s 1.80M, and identical at every task count on the way up. We audited the shared per-destination tcp_metrics entry the same way (net.ipv4.tcp_no_metrics_save=1) and it moved nothing either. The namespace splits the device, the port range, and the metrics cache, and none of it matters, because the contention lives below all of it: the established and bind hash tables with their bucket spinlocks, and the connect-time port search, are one structure per kernel, shared across every namespace that kernel hosts. Two kernels moved the number because two kernels means two of those structures.

That makes tcp_conn1 the strictest wall in this post. unlink1 falls to a per-task directory layout. rename1 falls to a filesystem per worker, at the price of the shared namespace. tcp_conn1 falls to nothing an administrator can reach: not a layout, not a mount, not a sysctl, not the namespace machinery built precisely to shard the network stack. The honest counterweight ships with it: at 24 socket-local cores the structural win is 1.16x, real but modest, the toll of bucket-lock cachelines rather than a serializing mutex. The single-kernel curve’s negative slope past one socket says the toll grows with distance, and the socket-aligned version of this experiment is the obvious next measurement.

Crossing the Socket

Everything above stayed inside one socket on purpose, to keep the lock wall isolated from NUMA effects. But almost every serious server is multi-socket, and a single kernel image spans all of it by default. So we ran the experiment again with the boundary moved: 24 tasks split 12 and 12 across the machine’s two sockets. One kernel managing both halves, versus two multikernel instances aligned to the sockets, each with 12 cores and memory on its own NUMA node. Same tests, same rules, twelve busy cores per socket in every configuration so the turbo bins match.

The most instructive number in this entire post came out of the smallest run. One task doing creat and unlink sustains 467,000 operations per second. Add one more task, on the other socket, in the same kernel, touching only its own file, and the two of them together sustain 213,000. The machine got slower than half of itself. Nothing in userspace is shared; the parent directory’s i_rwsem is, and its cacheline now crosses the UPI interconnect on every operation. A lock that costs nanoseconds inside a socket costs a coherence round-trip across one. Two kernels, one per socket, same two tasks: 1.01 million.

Load both configurations fully and the gap holds at every count, between 3.7x and 5.2x on unlink1 across the whole sweep:

Two readings of that table. First, the straddling kernel at 24 tasks is slower than the same kernel was with all 24 tasks packed on one socket: 231K versus 300K on unlink1, 9.1M versus 9.6M on open1, even though each socket now runs half loaded with correspondingly better clocks (the control shows 27% more per-core headroom than the packed configuration had). Spreading a lock-bound workload across sockets is negative-value parallelism: you pay for the second socket and receive less than you had. Second, the partitioning win grew from 2.6x to 4.1x for the same reason, because the partition boundary now coincides with the hardware boundary (open1’s row carries the same asterisk as before: its share of the win is the label cacheline, now bouncing over UPI). That is the rule this whole post has been circling. Shared state costs what the interconnect charges, and the further apart the sharers sit, the higher the toll. A multikernel does not negotiate the toll. It closes the bridge.

The Tests That Don’t Move

A fair reader of the tables above should ask whether we picked the benchmarks that flatter us. So we ran the rest of will-it-scale’s VM surface through the same two configurations as the first table: brk1 and malloc1 (the mmap_lock write path at two granularities), page_fault2 and page_fault3 (private and shared file-backed faults), and tlb_flush1 and tlb_flush3 (TLB shootdowns). Same 24 cores, same rules:

brk1 lands at 1.04x, tlb_flush1 and tlb_flush3 at 1.03x and 1.07x, page_fault3 at 0.98x, page_fault2 at 1.15x, malloc1 at 0.95x.

Parity almost everywhere, and the parity is the finding. brk1 and the TLB flush tests sit within 7% of 1x because the state they stress is per-process: a process’s mmap_lock lives in whichever kernel the process lives in, per-VMA locks shard it further within the process, and TLB shootdowns go only to the CPUs in the mm’s own cpumask. Splitting the machine into two kernels cannot speed up what one process does to itself, and the table says exactly that. This also answers a question we get about the per-VMA lock work: it is orthogonal to multikernel, not competing with it. Per-VMA locks shard contention inside a process; a multikernel shards it between processes. What neither can shard is contention inside one address space, which is the next section’s subject.

page_fault2 is the instructive row: a 1.15x win, but the mechanism is not locks. The single kernel plateaus at 2.1M faults per second from 12 cores onward, and each 12-core spawn plateaus at about half that while the two of them share the socket, summing to just above the single-kernel wall. That is a memory bandwidth ceiling wearing a benchmark costume. Partitioning cannot manufacture DRAM bandwidth, and a methodology that could not tell this row apart from open1’s would not deserve trust.

malloc1 is the row we owe the most honesty on: a 5% loss at 24 tasks, dipping to 15% at low counts. malloc of 128 MB is the mmap path in a glibc coat plus one page fault for the chunk header, so it should track mmap1’s win, and it does not. Profiling inside a spawn localizes the extra cycles to the page table install and teardown path (the page table lock in __pte_alloc, plus unmap_page_range), with clocks ruled out by the control tests and contention ruled out by the lock being per-process. It has the same signature as a stall we chased and cleared elsewhere in this work, an uncontended lock-prefixed instruction absorbing the cost of the stores ahead of it, but the residual here is real, bounded, and not yet explained, so it stays on the books as an open item rather than a footnote we buried.

Sixteen tests now. The wins appear exactly where kernel state is shared machine-wide, the parity appears exactly where it is not, and the one small loss ships with a profile attached. That is the shape an honest mechanism claim should have.

The Wall You Cannot Split: One Address Space

Everything above used processes because processes are what a multikernel can partition. Run the same tests as threads, all sharing one address space, and you meet a different wall, one we measured and can do nothing about, which is exactly why it deserves its own section instead of a footnote.

Three tests, three mechanisms, one root cause. mmap1 as threads collapses to 0.33x of a single task at 24 cores and 0.09x at 48: every map and unmap takes the one mmap_lock write-side, and an entire socket’s worth of threads advances one VMA operation at a time. tlb_flush1 shows the interrupt half: as 24 processes it scales 15.2x, as 24 threads only 2.6x, and the entire 6x gap is shootdown IPIs, because every unmap must broadcast to every CPU running the shared mm, while single-threaded processes flush only themselves and never send one. And page_fault3 as threads regresses from 15.0M faults per second at 12 cores to 10.6M at 24: shared-file faults serialize on the mapping’s rmap lock, the path the per-VMA lock work does not cover.

These numbers are the boundary of the multikernel claim, drawn in data: one address space lives inside one kernel, so none of this is fixable by splitting kernels. But notice what the boundary is made of. It is the same law that built every wall in this post, applied one level down: scaling is set by the granularity of sharing. One mm does not scale across threads; one kernel does not scale across processes; and the remedy at each level is the same move at a different radius. Per-VMA locks shard the address space, multi-process designs shard the mm entirely, and a multikernel shards the kernel. The workloads that could go share-nothing largely already have, nginx workers, postgres backends, sharded fleets of every kind, and our processes-mode tables show the wall they hit next: the kernel’s own shared state. The threads columns are not a counterargument to multikernel. They are rung one of the ladder multikernel finishes.

Caveats

The multikernel result is a claim about share-nothing partitioning, stated plainly: in the two-kernel configuration the 24 tasks never share a directory, a dcache, or an allocator, because each kernel has its own. That is the product, not a trick, but it draws the boundary honestly: a workload that fundamentally needs one shared mutable directory, or one 48-core address space, cannot be split this way, and the previous section measured that boundary. Multikernel scales the machine for workloads built of independent processes, which is what most server fleets already are, sharded web workers, per-core network stacks, build farms, CI runners.

page_fault1 at 0.96x is page zeroing bound rather than lock bound at this core count, so parity is the expected result, and 0.96 is within the turbo noise we measured between runs. The single-kernel numbers above 24 tasks include a cross-socket step visible in the curves; every one-versus-two comparison in this post stays inside one socket to keep that out of the ratios. And kerf currently has no --node flag for the memory pool, so socket-local placement takes one line of its Python API; that flag is on our list.

The cross-socket experiment ran its socket-aligned spawn pair in two phases with a reboot between them, because the multikernel pool is currently one contiguous physical range and therefore lives on one NUMA node per boot; that is fair here because the two sockets are independent turbo and memory domains, and each phase loaded its socket exactly as the single-kernel configuration did, but it is a tooling gap on our list next to the --node flag. And will-it-scale’s own core pinning ignores an external CPU mask, so the cross-socket runs used its no-affinity mode inside taskset masks, identically for both configurations; the cross-socket controls landed at 1.01x to 1.04x rather than 1.00x, and normalizing every ratio by the control still leaves 3.9x on unlink1 and 2.1x on open1.

Reproduce It

The fairness work is where the effort went, so here is the full setup. One kernel build, 7.0.0-mk1+, boots every configuration in this post. The single-kernel baseline runs on the host with all 48 cores. The multikernel side uses kerf to boot spawn kernels on socket 1, with memory allocated on socket 1’s NUMA node, booted with the same mitigation flags as the host command line, running the identical binaries from the same rootfs, with tmpfs mounted over /tmp on both sides. At every comparison point the same number of cores on one socket are busy, so turbo bins match. Every number is the average over five one-second samples after warmup.

will-it-scale builds in a minute: clone (our fork adds rename1, stat2, open4, pread4, and the network tests tcp_conn1, tcp_conn2, tcp_rr1, and udp1; every other test in this post, open2 and open3 included, is upstream), make, and each test is a standalone binary that takes a task count. The sweep is a shell loop. The multikernel side is kerf create, kerf load with a rootfs directory containing the same binaries, and kerf exec, with results read back from the DAXFS overlay on the host:

kerf create wis0 --cpus=<12 cores of socket 1> --memory=8192MB
kerf load wis0 --kernel=vmlinux --rootfs-dir=/root/wis-rootfs \
    --entrypoint="/bin/bash /run-wis.sh"
kerf exec wis0

Nothing in the method needs our hardware: pick the biggest box you have, run the single-kernel sweep, find your wall (open1 and unlink1 will find it for you), then split the same cores between two spawns and run it again. The absolute numbers will move with your core count and kernel version. The shape will not, because the shape is the architecture.

The Shape of the Wall

Every experiment in this post, one scoreboard. Ratios are two kernels against one on the same 24 cores unless noted; each row’s story lives in its section above.

The threads-mode walls have no ratio column because one address space cannot be split; their numbers live in their own section above.

Conclusion

Add it up. A single Linux kernel turned 48 dedicated cores into 40% of one core on unlink, capped open at its 8-core number forever, and did both while the control tests scaled 35x on the very same silicon, which is the kernel confessing that the limit is its own shared state. Two kernels on the same cores gave back 2.6x immediately, with the controls at 1.00x certifying that nothing else changed. A rename test the benchmark suite never had showed a mutex per filesystem doing the same at any directory layout, and the same split gave back 2.1x. Stat and open on one shared file, the shape of every exec storm, capped near 2.5x on one kernel however many cores arrived, and split to 2.1x as well. Even mounting one shared filesystem in both kernels, the same physical pages and the same file, kept the whole 2.18x, because the locks live in each kernel’s VFS, not in the storage. Stretch the one kernel across two sockets instead and the wall grows with the distance: two tasks ran slower than one, and the socket-aligned split paid back 4.1x. TCP connection setup hit its own wall at a smaller toll, 1.16x back from the split, and survived the strongest single-kernel counter there is: one network namespace per task, the sharding tool built for exactly this stack, changed the number by nothing at all. Six more VM benchmarks moved by at most 15% in either direction, and established-connection TCP and UDP sat at parity beside them, because the state they stress is per-process or per-connection: the mechanism claim passing its own control.

The industry’s answer to this wall has been twenty years of heroic lock surgery, and this post shows both its successes and its limit: some ceilings turn out to be configuration and fall to a boot flag, as open1’s did, but the locks that remain after that audit are the ones that define what “one kernel” means. You do not shard those. You shard the kernel.

Multikernel is open source. If your machines have more cores than your kernel can honestly use, we would love to hear from you at contact@multikernel.io.