We gave a Linux kernel 96 cores and asked it to launch containers as fast as it could. It launched 18 per second. Then we took half the cores away, and it launched 43.
Nothing else changed between those two runs: the same kernel, the same containerd, the same image, one worker per core in both. The 48 extra cores were not idle, either. They were busy the whole time, spinning on a lock that only one of them could hold.
That lock was reported a year ago. In November 2025, Netflix published Mount Mayhem at Netflix: Scaling Containers on Modern CPUs, describing nodes that stalled for tens of seconds when hundreds of containers started at once. The cause was the kernel’s mount locks, which are global to the machine. Netflix then worked with containerd upstream on a change that avoids most of the mount operations, and it shipped in containerd 2.2.
This post reports three measurements on our own hardware. First, the ceiling on one kernel, reproduced with containerd’s own stress tool. Second, what the userspace fix does to it. Third, what happens when the same cores are split between two kernels, both before and after the fix. The third is the one we care about, and its answer is smaller after the fix than before: two kernels launch 2.3 times as many containers as one on the older containerd, and 1.4 times as many on the current one.
What a Container Launch Asks of the Kernel
A container’s root filesystem is an overlayfs mount stacked from the image’s layers. When each container runs in its own user namespace, with its own range of host user IDs, the files in those layers have to appear to be owned by that range. The kernel feature for this is the idmapped mount.
Up to containerd 2.1, containerd created one idmapped bind mount per image layer, built the overlayfs on top of them, and then unmounted them again. Each layer costs three system calls (open_tree, mount_setattr, move_mount) plus an unmount, and each of those takes locks that cover every mount namespace on the machine. Netflix worked out the arithmetic for 100 containers with 50-layer images: 20,200 mount operations, all serialized.
The Setup
The machine is a single-socket Intel Xeon 6731E: 96 cores, no SMT, one NUMA node, 250 GB of memory, running Ubuntu 24.04 with the multikernel kernel, version 7.0.
The benchmark is containerd-stress, the stress tool that ships in the containerd source tree. Each of its workers loops: create a container from an image, start it, wait for it to exit, delete it. The container runs true, so the work is almost entirely setup and teardown. The score is containers completed per second.
The stock tool has no way to ask for user namespaces, so we added one; the change is described under Reproduce It. The image is an Alpine base layer plus 50 layers that each add one small file. containerd’s state lives on tmpfs, so the disk is not a factor. The runtime is runc 1.5.3. Unless noted, each number is from a 20-second run with one worker per core, and each configuration was run twice.
One Kernel: 20 Launches per Second at Any Core Count
Without user namespaces, one kernel scales acceptably: 5 containers per second with one worker and about 280 with 48. With user namespaces on containerd 2.1.4, it reaches 18 per second at 16 workers and stays there. Eighty more cores add nothing.
A profile at 96 workers shows where the time goes. 21% of all CPU cycles are spent spinning for a write lock on namespace_sem, reached from open_tree, move_mount and umount in nearly equal parts. Another 13% is spinlock contention, about half of it under mntput_no_expire. This is the same picture as the flame graph in the Netflix post.
“Add nothing” understates it. The extra cores make the machine slower. We took cores 48 to 95 away from the host by handing them to a multikernel instance that we left idle, and ran the same workload on the remaining 48. The host launched more than twice as many containers.
Each core past the point of saturation is one more contender for the same lock, and a lock with more contenders is handed over more slowly. The machine does less work with more cores.
The Userspace Fix
containerd 2.2 maps the common parent directory of all the layers once, so the number of mount operations per container no longer depends on the number of layers. We confirmed it by counting system calls: with user namespaces, containerd 2.1.4 makes about 155 move_mount calls per container with our image, and containerd 2.2.10 makes 3.
The effect on one kernel is large. With containerd 2.2.10, the host with all 96 cores launches about 250 user-namespaced containers per second, where it launched 18 before. That is a factor of 14 from a change in userspace, on the same kernel.
Anyone running containerd 2.2 or later does not have the 20-per-second ceiling. Everything in the previous section describes software that has since been fixed, and we want that stated before any two-kernel number appears.
Two Kernels, Before and After the Fix
A multikernel instance is a second Linux kernel booted on a subset of the machine’s cores and memory, beside the host kernel, with no hypervisor between it and the hardware. It has its own mount table and its own namespace_sem.
The comparison an operator would make is the machine as it is against the machine split in two. The single kernel is the Ubuntu host with all 96 cores, running two containerd instances pinned to 48 cores each. For two kernels, kerf moves cores 48 to 95 and 32 GB of memory into an instance, which runs one containerd; the host keeps cores 0 to 47 and runs the other. Both halves run at the same time, and each number is the total for the machine.
On containerd 2.1.4, two kernels launch 140 containers per second where one launches 18. On containerd 2.2.10, they launch 360 where one launches 252, a factor of 1.4.
The two halves do not disturb each other. On containerd 2.1.4 the host half launched 43 per second alone and 40 beside the busy instance; the instance launched 104 alone and 101 beside the busy host.
How Much of That Is the Kernel
The comparison above is not symmetric. The host runs systemd, AppArmor and a full distribution, and an instance runs a minimal Alpine userland. To separate the effect of splitting the kernel from the effect of a leaner environment, we ran a second comparison in which both sides are instances. The host keeps 8 cores and does nothing. One kernel is a single 88-core instance running two containerd instances pinned to 44 cores each. Two kernels are two 44-core instances running one containerd each. Same kernel build, same root filesystem, same cores.
On containerd 2.1.4, two kernels launch 2.3 times as many containers as one. On containerd 2.2.10, 1.4 times.
So of the factor of 7 to 8 in the as-deployed comparison on the older containerd, 2.3 comes from splitting the kernel. The rest is the host environment, which is four to five times slower than a minimal instance on the mount-heavy path. We have not isolated why. A host that watches its mount table, propagates mounts and runs a security module on every operation has more to do per mount, and all of it happens under the same locks. On containerd 2.2.10 the penalty is gone: the host and the 88-core instance perform about the same, and the as-deployed and like-for-like factors agree at 1.4.
The single 88-core instance also shows the interference directly. On containerd 2.1.4, one containerd running alone on its 44 cores launched 115 containers per second. Starting the second containerd on the other 44 cores brought the total for both down to 84. With two kernels, one half alone launched 106 and both together launched 196.
Two Remedies
The fix and the split work at different levels.
A userspace fix reduces the demand for one lock, in one application, after someone has found the lock in a flame graph. Netflix had to notice stalled nodes, trace them to the mount table, compare instance types, and work with upstream on a redesign. The result is excellent, and it is specific: namespace_sem is still one lock for the whole machine, and the next application that mounts heavily will find it again.
Partitioning does not reduce demand at all. It reduces how many cores share each lock, and it does that for every kernel-wide lock at once, including the ones nobody has profiled yet.
On this workload the first remedy was worth far more than the second: a factor of 14 against a factor of 2.3. They also add up. The best result we measured, 381 launches per second, is the fixed containerd on two kernels, and the remaining factor of 1.4 says that a single kernel is still losing something to sharing when 96 cores launch containers together. We have not yet profiled the 2.2.10 runs, so we cannot say which locks account for it.
What This Does Not Show
This machine has one socket, and that is the largest gap. Netflix’s most striking result was that a two-socket instance failed where a one-socket instance of the same CPU generation did not, because a contended lock costs more when its cache line crosses sockets. We could not test that here. We would expect a single kernel to do worse on a two-socket machine, and the gap to two kernels to be wider.
The benchmark is a launch loop, not a service. Containers that start once and run for days do not notice any of this. The result matters for nodes that fill up in bursts, for batch and CI systems, and for platforms that start a container per request.
The runs are short: 20 seconds each, two per configuration, with no variance estimate beyond the two values agreeing. The sweep chart used 10-second points.
We modified the benchmark. The user-namespace option is ours, not upstream’s, and every number for user-namespaced containers depends on it.
One cell in the summary is empty. The like-for-like single instance lost its runs without user namespaces to a benchmark client that hung, which we have not diagnosed.
Reproduce It
Everything here is public except our 30-line patch to containerd-stress, and the core of the patch is three lines. Each worker gets its own 65,536-ID host range and requests its snapshot with the same ID-mapping labels that the Kubernetes runtime interface uses:
idmap := []specs.LinuxIDMapping{{ContainerID: 0, HostID: uint32(1<<20 + w.id<<16), Size: 1 << 16}}
snapshot = containerd.WithNewSnapshot(id, w.image, containerd.WithUserNSRemapperLabels(idmap, idmap))
specOpts = append(specOpts, oci.WithUserNamespace(idmap, idmap))
The rest of the patch lets the tool use an image that is already imported, since the stock tool always pulls from a registry. Use the static containerd release tarballs for 2.1.4 and 2.2.10, build the stress tool from the matching source tree so that the client and the daemon mount the same way, and put containerd’s root on tmpfs.
The instance’s root filesystem is an Alpine minirootfs with the same containerd, runc and image copied in. kerf builds it into the instance’s root and boots it:
kerf init --cpus=<physical IDs of cores 48-95> --memory=40GB
kerf create half --cpus=<the same IDs> --memory=32GB
kerf load half --kernel=/boot/vmlinuz --rootfs-dir=./rootfs --entrypoint=/init.sh
kerf start half
kerf exec half containerd-stress <options>
To find the ceiling on your own machine, you do not need multikernel at all. Run the user-namespace workload on containerd 2.1 with one worker per core, then run it again with half the cores taken offline. Pinning to half the cores without removing them shows the effect too, though less of it: one containerd pinned to 44 of our 96 cores launched 33 containers per second.
Summary
Containers launched per second, whole machine, both halves busy.
| Configuration | 2.1.4, user namespaces | 2.2.10, user namespaces | 2.2.10, no user namespaces |
|---|---|---|---|
| One kernel: host, 96 cores | 16 to 20 | 242 to 262 | 292 to 324 |
| Two kernels: host 48 + instance 48 | 140 | 359 to 362 | 444 |
| One kernel: one 88-core instance | 84 | 266 to 275 | not measured |
| Two kernels: two 44-core instances | 196 | 381 to 382 | 459 to 460 |
| Two kernels over one | 2.1.4, user namespaces | 2.2.10, user namespaces |
|---|---|---|
| As deployed | 7 to 8x | 1.4x |
| Like for like | 2.3x | 1.4x |
Conclusion
A single Linux kernel launching user-namespaced containers with containerd 2.1 is held by its mount locks to about 20 launches per second on our 96-core server, and it launches more with 48 cores than with 96. containerd 2.2 removes most of the mount operations and lifts that to about 250.
Two kernels on the same cores do not share those locks. They launched 2.3 times as many containers as one kernel before the fix and 1.4 times as many after it.
We went looking for a workload where one kernel collapses and two do not, and we found one that a userspace fix had already rescued. We think that is the normal life cycle of a kernel-wide lock: it is found in production, worked around in the application that hit it, and left in place for the next application. Splitting the machine is the remedy that does not depend on finding the lock first.
Multikernel is open source, and so is kerf. If your nodes launch containers in bursts, or if a profile of yours ends in a kernel lock you cannot configure away, we would like to hear from you at contact@multikernel.io.