Hardware virtualization is supposed to be nearly free now. EPT removed shadow page tables. APICv moved interrupt delivery into silicon. Unrestricted guest mode erased the last software emulation from the boot path. The conventional wisdom is that a tuned KVM guest performs like bare metal, and for some workloads that is true.
We wanted the precise shape of “some.” So we ran the same experiment twice on one machine: the same kernel build, the same lmbench binaries, the same benchmark script, two cores and memory to spare on each side. The only variable was the layer underneath: a multikernel instance running natively on partitioned hardware, versus a KVM guest with every acceleration feature the host offers.
The result has three layers. Where the hardware does the work, the VM matches multikernel exactly. Where the kernel does the work, the VM pays between 1.04x and 2.5x at default settings, and it pays the most on the paths modern software hits the hardest. The third layer is what happens when you try to fix that, and it is the one worth reading for.
So take the caveat before the numbers, because it changes how you should read them. The 2x and 2.5x ratios below are measured against a guest running the default idle policy, HLT. Nearly all of that gap is one mechanism: a guest cannot execute mwait, so an idle vCPU halts, every wakeup becomes a VM exit and a re-entry, and a ping-pong benchmark pays that twice per iteration. It is buyable, and a stock distribution guest has already bought it. The haltpoll cpuidle driver most distro kernels ship spins before halting, and on wakeups this fast it closes most of the gap.
So the honest headline is not the 2.5x. It is that the guest reaches those numbers only by never sleeping, which costs a core and 19 W of package power, or by taking exclusive mwait control of physical cores away from the host. Multikernel reaches them while sleeping in real C-states. That is the whole argument, and the rest of this post is the measurement behind it. Underneath whichever option you pick, one smaller cost never comes off at all: roughly 30 ns on every kernel entry, in every configuration we could build.
Benchmark Setup
The host is a dual-socket Intel Xeon Gold 5418Y (Sapphire Rapids, 24 cores per socket, 48 KB L1d and 2 MB L2 per core, 45 MB shared L3, SMT disabled). Both configurations booted the same kernel build, 7.0.0-mk1+, out of one source tree and one config, packaged the two ways the two boot paths require: kerf takes the ELF vmlinux, QEMU’s -kernel takes the bzImage. Both ran the identical statically-built lmbench binaries from the same root filesystem, driven by the same shell script. Neither configuration touches an I/O device during measurement: every benchmark runs from memory.
Multikernel. A spawn instance created with kerf, given two dedicated physical cores and 1 GB of reserved memory carved out of the host at boot. The root filesystem is a DAXFS image in shared memory. The spawn kernel owns its cores outright: real page tables, real APIC, real IPIs, no hypervisor underneath.
kerf load lmbench --kernel=vmlinux --rootfs-dir=/root/lmbench-rootfs \
--entrypoint="/bin/bash /run-lmbench.sh"
kerf exec lmbench
KVM. The same kernel booted by QEMU with every hardware assist verified on: EPT, unrestricted guest, and APICv all enabled in kvm_intel. The guest gets -cpu host, two vCPUs pinned to two idle physical cores on the same socket, and its root filesystem as an initramfs (the kernel carries no virtio drivers, and for a pure CPU and memory benchmark it does not need any). The guest is given the same 1 GB.
taskset -c 10,12 qemu-system-x86_64 -enable-kvm -cpu host -smp 2 -m 1024 \
-kernel bzImage -initrd lmbench-initrd.gz \
-append "console=ttyS0 clocksource=tsc tsc=reliable rdinit=/vm-init.sh"
Both kernels use TSC as the clock source, so lmbench’s timing loops read the same hardware counter at the same cost in both worlds. Both sides run their default idle policy: the multikernel instance on its native intel_idle driver with mwait C-states (C1/C1E/C6), the guest on HLT, since KVM does not expose mwait by default. Our guest kernel is a minimal config with no haltpoll cpuidle driver, so plain HLT is genuinely what it falls back to. A stock distribution guest would spin first, and we measure that behavior as its own configuration further down rather than folding it into the default. This is otherwise a deliberately clean KVM setup: no overcommit, no ballooning, no noisy neighbors, no emulated devices in the measured path.
The Numbers at Default Idle
Every number in this post is the median of three full-suite runs per configuration.
| Benchmark | Multikernel | KVM guest | KVM / Multikernel |
|---|---|---|---|
| Null syscall | 0.070 µs | 0.099 µs | 1.42x |
| read() | 0.099 µs | 0.124 µs | 1.26x |
| write() | 0.082 µs | 0.114 µs | 1.39x |
| stat() | 0.354 µs | 0.379 µs | 1.07x |
| open()/close() | 0.554 µs | 0.586 µs | 1.06x |
| Signal handler install | 0.123 µs | 0.159 µs | 1.29x |
| Signal handler catch | 0.770 µs | 0.881 µs | 1.14x |
| Context switch (2 procs) | 1.37 µs | 3.42 µs | 2.50x |
| Pipe latency | 3.24 µs | 7.06 µs | 2.18x |
| AF_UNIX stream latency | 4.81 µs | 7.48 µs | 1.55x |
| fork + exit | 115 µs | 123 µs | 1.07x |
| fork + /bin/sh | 730 µs | 761 µs | 1.04x |
The pattern is not random, but it is two patterns stacked. The operations at the top are the ones dominated by wakeups. The ones at the bottom do enough real kernel work to dilute a fixed per-entry cost that every operation on the list pays.
Context switches pay 2.5x. A two-process context switch that costs 1.37 µs on multikernel costs 3.42 µs in the guest. This is the single most executed path in a busy kernel: every scheduler decision, every lock handoff, every producer-consumer pair crosses it. It is also the line item the guest can buy back, and the next section is about what the receipt says.
IPC pays 2x. A pipe round-trip more than doubles, from 3.24 µs to 7.06 µs, and an AF_UNIX round-trip pays 55%. These are the primitives that shells, supervisors, async runtimes, and every microservice sidecar are built from. Both are buyable for the same reason context switching is.
Even a null syscall pays 42%. A syscall does not exit to the hypervisor; the guest kernel handles it directly. Yet every kernel entry carries an extra 29 ns inside this guest, and about 28 ns even in the fastest guest we could assemble. This is the one number nothing bought back. It is the ambient cost of simply being a guest.
fork rides nearly free. fork+exit is a memory-management workload, thousands of page-table operations with few context switches, and it lands at 1.07x, down at the bottom of the chart with stat and open. EPT handles the page-table work in hardware, so what remains is mostly the per-entry tax. Which brings us to the other half of the story.
Where There Is No Tax
If virtualization overhead were a general slowness, it would show up in the memory system first. It does not show up at all.
This is lat_mem_rd walking working sets from 512 bytes to 128 MB. Both curves trace the identical staircase: 1.3 ns in L1, about 2 ns in L2, 11 ns in the shared L3, climbing to 32 ns in DRAM. The KVM guest, running behind a second layer of address translation, loses nothing measurable. EPT with huge pages has made nested translation genuinely free for well-behaved working sets, exactly as advertised.
Bandwidth tells the same story:
Sequential read, write, read-modify-write, and bzero agree within a few percent across all runs. The two copy variants consistently favor the guest (bcopy 9.6 vs 5.4 GB/s in the medians); we flag that as an open question rather than explain it away, but note its direction: a guest outrunning the native kernel is not virtualization overhead.
This parity is the interesting part. It rules out the lazy explanation for the latency numbers. The VM is not “slower hardware.” The silicon delivers identical service to both kernels. Everything the VM loses, it loses in the seams between contexts, and those seams are exactly where operating systems spend their time.
The Price of a Fast Wakeup
We said the gap is the idle loop. Here is the evidence, and the bill.
In a ping-pong benchmark like lat_pipe or lat_ctx, the partner process is blocked almost all the time, so its CPU is idle-waiting. On multikernel, that idle CPU executes mwait natively and a real IPI wakes it from C1 in a few hundred nanoseconds. Inside the guest, mwait is not available: idle means HLT, HLT means a VM exit, and every wakeup means the hypervisor re-entering the vCPU. That round trip, paid twice per ping-pong iteration, is most of the gap.
The evidence is that you can pay it off. KVM offers two ways, and we measured both:
Latency is only half of each bar’s story. We also measured what every configuration costs while the guest sits completely idle, host CPU utilization of the two cores and package power over the no-VM baseline:
| Idle policy | Ctx switch | Pipe | Host view of the idle cores | Idle power cost |
|---|---|---|---|---|
| Multikernel, C-states | 1.37 µs | 3.24 µs | ceded, sleeping in C1-C6 | baseline |
| KVM, HLT (default) | 3.42 µs | 7.06 µs | ~0% busy | baseline |
| KVM, idle=poll | 1.49 µs | 3.34 µs | 100% busy, spinning | +19 W |
| KVM, mwait passthrough | 1.58 µs | 3.49 µs | 100% busy, though the silicon is in C1 | +12 W |
Only one row gets low latency and cheap idle at the same time.
Boot the guest with idle=poll and the vCPU never sleeps: pipe latency lands within 3% of multikernel, the context switch within 9%, and the AF_UNIX round-trip actually beats it (3.40 µs vs 4.81 µs). The price is not hypothetical. With the guest completely idle, we measured both host cores at 100% utilization and package power up 19 W over baseline, spent on nothing. That defeats consolidation, power management, and every reason the hypervisor wanted those cores back.
This is also, roughly, where a stock distribution guest already lives. The haltpoll cpuidle driver spins for a few microseconds before halting, and for wakeups this fast it behaves like the poll column and spends the poll column’s CPU for the duration of the ping-pong. It is the better-engineered version of the same bargain: because it halts once the guest goes properly idle, it does not hold 19 W forever, and because it only wins by spinning, it does not get you a fast wakeup and a free core at the same time. Nothing does. That is the point of this table, and it is why the 2.5x in the section above is a fact about one configuration rather than a fact about virtualization.
Or launch QEMU with -overcommit cpu-pm=on, which passes mwait through so the guest can sleep for real. Latency comes within 4 to 15% of multikernel. But our measurements show the fine print. The guest only ever reached C1, the shallowest state, leaving package power 12 W over baseline. This is not fixable with another flag: the mwait instruction is passed through, but the knowledge of deeper states is not. The guest’s intel_idle never loads (no mwait sub-state CPUID leaf) and QEMU’s ACPI tables advertise no C-states, so the guest falls back to a C1-only idle loop. And the omission is deliberate, because C-states are a socket-shared resource: cores sleeping in C6 fund other cores’ turbo budget, so a guest with C-state control on shared silicon could modulate its neighbors’ frequency and park cores in states with wake latencies the host scheduler would have to eat. And because the vCPU thread now never exits on idle, the host sees those cores as 100% busy even while the silicon sleeps: it can no longer tell guest-idle from guest-work, cannot harvest idle cycles, and has ceded C-state policy of physical cores to the guest. Overcommitting such a vCPU stays technically possible and practically pointless; the first co-tenant that time-shares the core brings the scheduler back into every wakeup and the latency returns.
Those watts translate directly into money. A constant +12 W is 105 kWh a year; at a $0.10/kWh industrial rate behind a PUE of 1.4, call it $15 a year for one idle two-core guest, $23 for the polling variant. That sounds trivial until you remember it is a per-instance, always-on cost: a million latency-tuned guests are a standing 12 MW load, eight figures a year in electricity, spent executing nothing. And electricity is the smaller bill. Both fast configurations leave the host seeing two cores at 100% busy, so it can never place another tenant there: each such guest permanently strands the capacity a cloud would otherwise sell as a 2-vCPU instance.
Meanwhile the idle multikernel instance draws baseline power, because its cpuidle governor does what a native kernel’s does: C1 with its few-hundred-nanosecond wakeup during active ping-pong, C6 during long idle. Latency when it matters, deep sleep when it does not. The guest’s menu has no such entry: HLT (cheap, but 2.5x), polling (fast, +19 W, a core spent on nothing), or shallow mwait (+12 W and a core the host can no longer account for).
The industry has already seated itself around this menu, and the seating chart is in official documentation. Host-side, KVM ships its own polling layer: the host spins for up to a configurable window before scheduling out a halted vCPU (halt polling), trading host CPU for wakeup latency fleet-wide. Guest-side, the haltpoll driver and governor do the same from inside the VM, and Ubuntu ships that driver as a module in its stock kernels, which is why a stock distribution guest already lives near the poll column. And the mwait column is sold as a product feature under its own name: EC2 processor state control hands C-state control to guests on bare metal instances, on the largest sizes of the older instance families, and on all sizes of the newest generations, where a tenant’s vCPUs own their physical cores outright. The direction of that list is the point: sleep-state control is granted exactly as far as core ownership extends, and no further.
Notice what just happened. To make the VM fast we pinned its vCPUs to dedicated cores, then gave it exclusive control of their sleep states. Follow that ladder to the end and the guest is no longer sharing anything: it is a static hardware partition, hand-assembled from hypervisor exemptions. And even then the residue stays. A null syscall held near 1.4x and signal registration near 1.3x across the poll and mwait guests as well, with the entire virtualization stack still resident and still inside the trust boundary.
The Structural Difference
A KVM guest and a multikernel instance make the same promise: your own kernel, your own CPUs, your own memory, strong isolation from the neighbors. They keep it with different machinery.
The guest keeps it through interposition. The hypervisor owns the real APIC, the real timer, and the real idle loop, and every one of those interpositions costs latency that you can only buy back by carving out an exemption: pin these cores, pass mwait through, dedicate, dedicate, dedicate. Modern KVM is superbly engineered, EPT and APICv and IPI virtualization each erased a real cost, and our memory numbers show it. But the end state of a fully-exempted guest is a partition wearing a hypervisor, and the hypervisor still charges rent on every kernel entry.
A multikernel instance keeps the promise through partitioning from the start. The spawn kernel boots on cores the host kernel has permanently ceded, addresses memory the host will never touch, programs the real APIC, and sleeps in real C-states. An IPI is an IPI. A context switch is a context switch. Idle is actually idle, and fast is the default, not a configuration you assemble one exemption at a time.
The numbers above are that difference, measured.
Caveats
The medians above come from three full-suite runs per configuration, and the spread is worth stating. The multikernel side is strikingly stable across runs (its widest spread was the context switch, 1.01 to 1.38 µs); the guest’s wakeup numbers swing more (pipe ranged 5.8 to 7.2 µs across its default-idle runs), which is itself a symptom of the machinery underneath. The two configurations ran on different physical core pairs of the same socket (we verified all four cores share package 0, minding that kerf takes physical APIC IDs while taskset takes logical CPU numbers). Two metrics are excluded because the two sides boot different root filesystems: fork+execve loads the binary through the filesystem, where the guest is consistently faster (214 µs on tmpfs vs 315 µs on DAXFS), and lat_pagefault faults against it (0.15 µs vs 1.74 µs). Both measure DAXFS against tmpfs, not virtualization. fork+/bin/sh crosses those same filesystems but lands at 1.04x regardless, so we kept it. And a production VM sharing an oversubscribed host pays the measured floor plus scheduling on top.
Reproduce It Yourself
Everything runs from a stock lmbench built out of the autubrew/lmbench Docker image, exported once to a plain rootfs directory that both worlds share:
docker run --name lmbench-build --entrypoint /bin/bash \
autubrew/lmbench -c 'cd /root/lmbench/src && make -j8'
docker export lmbench-build | tar -x -C /root/lmbench-rootfs
The multikernel side loads that directory with kerf load --rootfs-dir and reads results from the DAXFS overlay on the host. The KVM side packs the same directory into an initramfs and replays the result file over the serial console. One driver script, two boots, one diff.
We encourage you to run this on your own hardware. The absolute numbers will move with CPU generation, kernel version, and idle policy. The shape will not: the memory system reaches parity because that was a hardware problem and hardware solved it; the wakeup paths reach parity only when you stop sharing the hardware; and the kernel-entry residue never reaches parity at all, because that one is the price of having a hypervisor in the loop.
Multikernel is open source. If you are rethinking what isolation has to cost, we would love to hear from you at contact@multikernel.io.