Hardware virtualization is supposed to be nearly free now. EPT removed shadow page tables. APICv moved interrupt delivery into silicon. Unrestricted guest mode erased the last software emulation from the boot path. The conventional wisdom is that a tuned KVM guest performs like bare metal, and for some workloads that is true.
We wanted the precise shape of “some.” So we ran the same experiment twice on one machine: the same kernel build, the same lmbench binaries, the same benchmark script, two cores and memory to spare on each side. The only variable was the layer underneath: a multikernel instance running natively on partitioned hardware, versus a KVM guest with every acceleration feature the host offers.
The result has three layers. Where the hardware does the work, the VM matches multikernel exactly. Where the kernel does the work, the VM pays between 1.04x and 2.5x at default settings, and it pays the most on the paths modern software hits the hardest. The third layer is what happens when you try to fix that, and it is the one worth reading for.
One caveat changes how the numbers should be read. The 2x and 2.5x ratios below are measured against a guest at the default idle policy, HLT, and nearly all of that gap is one mechanism: a guest cannot execute mwait, so every wakeup of an idle vCPU is a VM exit and a re-entry, paid twice per ping-pong iteration. That gap is buyable, and the price is the real story. The guest closes it only by never sleeping, a core and up to 19 W spent on nothing, or by taking mwait control of physical cores away from the host; multikernel posts the same numbers while sleeping in real C-states. And underneath whichever option you pick, about 30 ns on every kernel entry never comes off at all.
Benchmark Setup
The host is a dual-socket Intel Xeon Gold 5418Y (Sapphire Rapids, 24 cores per socket, 48 KB L1d and 2 MB L2 per core, 45 MB shared L3, SMT disabled). Both configurations booted the same kernel build, 7.0.0-mk1+, out of one source tree and one config, packaged the two ways the two boot paths require: kerf takes the ELF vmlinux, QEMU’s -kernel takes the bzImage. Both ran the identical statically-built lmbench binaries from the same root filesystem, driven by the same shell script. Neither configuration touches an I/O device during measurement: every benchmark runs from memory.
Multikernel. A spawn instance created with kerf, given two dedicated physical cores and 1 GB of reserved memory carved out of the host at boot. The root filesystem is a DAXFS image in shared memory. The spawn kernel owns its cores outright: real page tables, real APIC, real IPIs, no hypervisor underneath.
kerf load lmbench --kernel=vmlinux --rootfs-dir=/root/lmbench-rootfs \
--entrypoint="/bin/bash /run-lmbench.sh"
kerf exec lmbench
KVM. The same kernel booted by QEMU with every hardware assist verified on: EPT, unrestricted guest, and APICv all enabled in kvm_intel. The guest gets -cpu host, two vCPUs pinned to two idle physical cores on the same socket, and its root filesystem as an initramfs (the kernel carries no virtio drivers, and for a pure CPU and memory benchmark it does not need any). The guest is given the same 1 GB.
taskset -c 10,12 qemu-system-x86_64 -enable-kvm -cpu host -smp 2 -m 1024 \
-kernel bzImage -initrd lmbench-initrd.gz \
-append "console=ttyS0 clocksource=tsc tsc=reliable rdinit=/vm-init.sh"
Both kernels use TSC as the clock source, so lmbench’s timing loops read the same hardware counter at the same cost in both worlds. Both sides run their default idle policy: the multikernel instance on its native intel_idle driver with mwait C-states (C1/C1E/C6), the guest on HLT, since KVM does not expose mwait by default. Our guest kernel is a minimal config with no haltpoll cpuidle driver, so plain HLT is genuinely what it falls back to. A stock distribution guest whose hypervisor promises it dedicated cores would spin first, and we measure that behavior as its own configuration further down rather than folding it into the default. This is otherwise a deliberately clean KVM setup: no overcommit, no ballooning, no noisy neighbors, no emulated devices in the measured path.
The Numbers at Default Idle
Every number in this post is the median of three full-suite runs per configuration.
| Benchmark | Multikernel | KVM guest | KVM / Multikernel |
|---|---|---|---|
| Null syscall | 0.070 µs | 0.099 µs | 1.42x |
| read() | 0.099 µs | 0.124 µs | 1.26x |
| write() | 0.082 µs | 0.114 µs | 1.39x |
| stat() | 0.354 µs | 0.379 µs | 1.07x |
| open()/close() | 0.554 µs | 0.586 µs | 1.06x |
| Signal handler install | 0.123 µs | 0.159 µs | 1.29x |
| Signal handler catch | 0.770 µs | 0.881 µs | 1.14x |
| Context switch (2 procs) | 1.37 µs | 3.42 µs | 2.50x |
| Pipe latency | 3.24 µs | 7.06 µs | 2.18x |
| AF_UNIX stream latency | 4.81 µs | 7.48 µs | 1.55x |
| fork + exit | 115 µs | 123 µs | 1.07x |
| fork + /bin/sh | 730 µs | 761 µs | 1.04x |
The pattern is not random, but it is two patterns stacked. The operations at the top are the ones dominated by wakeups. The ones at the bottom do enough real kernel work to dilute a fixed per-entry cost that every operation on the list pays.
Context switches pay 2.5x. A two-process context switch that costs 1.37 µs on multikernel costs 3.42 µs in the guest. This is the single most executed path in a busy kernel: every scheduler decision, every lock handoff, every producer-consumer pair crosses it. It is also the line item the guest can buy back, and the next section is about what the receipt says.
IPC pays 2x. A pipe round-trip more than doubles, from 3.24 µs to 7.06 µs, and an AF_UNIX round-trip pays 55%. These are the primitives that shells, supervisors, async runtimes, and every microservice sidecar are built from. Both are buyable for the same reason context switching is.
Even a null syscall pays 42%. A syscall does not exit to the hypervisor; the guest kernel handles it directly. Yet every kernel entry carries about 30 ns of extra cost, in this guest and in the fastest guest we could assemble. This is the one number nothing bought back. It is the ambient cost of simply being a guest.
fork rides nearly free. fork+exit is a memory-management workload, thousands of page-table operations with few context switches, and it lands at 1.07x, down at the bottom of the chart with stat and open. EPT handles the page-table work in hardware, so what remains is mostly the per-entry tax. Which brings us to the other half of the story.
Where There Is No Tax
If virtualization overhead were a general slowness, it would show up in the memory system first. It does not show up at all.
This is lat_mem_rd walking working sets from 512 bytes to 128 MB. Both curves trace the identical staircase: 1.3 ns in L1, about 2 ns in L2, 11 ns in the shared L3, climbing to 32 ns in DRAM. The KVM guest, running behind a second layer of address translation, loses nothing measurable. EPT with huge pages has made nested translation genuinely free for well-behaved working sets, exactly as advertised.
Bandwidth tells the same story:
Sequential read, write, read-modify-write, and bzero agree within a few percent across all runs. The two copy variants consistently favor the guest (bcopy 9.6 vs 5.4 GB/s in the medians); we flag that as an open question rather than explain it away, but note its direction: a guest outrunning the native kernel is not virtualization overhead.
This parity is the interesting part. It rules out the lazy explanation for the latency numbers. The VM is not “slower hardware.” The silicon delivers identical service to both kernels. Everything the VM loses, it loses in the seams between contexts, and those seams are exactly where operating systems spend their time.
The Price of a Fast Wakeup
We said the gap is the idle loop. Here is the evidence, and the bill.
In a ping-pong benchmark like lat_pipe or lat_ctx, the partner process is blocked almost all the time, so its CPU is idle-waiting. On multikernel, that idle CPU executes mwait natively and a real IPI wakes it from C1 in a few hundred nanoseconds. Inside the guest, mwait is not available: idle means HLT, HLT means a VM exit, and every wakeup means the hypervisor re-entering the vCPU. That round trip, paid twice per ping-pong iteration, is most of the gap.
The evidence is that you can pay it off. KVM offers two ways, and we measured both:
Latency is only half of each bar’s story. We also measured what every configuration costs while the guest sits completely idle, host CPU utilization of the two cores and package power over the no-VM baseline:
| Idle policy | Ctx switch | Pipe | Host view of the idle cores | Idle power, alone on the socket |
|---|---|---|---|---|
| Multikernel, C-states | 1.37 µs | 3.24 µs | ceded, sleeping in C1-C6 | baseline |
| KVM, HLT (default) | 3.42 µs | 7.06 µs | ~0% busy | baseline |
| KVM, idle=poll | 1.49 µs | 3.34 µs | 100% busy, spinning | +19 W |
| KVM, mwait passthrough | 1.58 µs | 3.49 µs | 100% busy, though the silicon is in C1 | +12 W |
Only one row gets low latency and cheap idle at the same time.
Boot the guest with idle=poll and the vCPU never sleeps: pipe latency lands within 3% of multikernel, the context switch within 9%, and the AF_UNIX round-trip actually beats it (3.40 µs vs 4.81 µs). The price is not hypothetical. With the guest completely idle, we measured both host cores at 100% utilization and package power up 19 W over baseline, spent on nothing. That defeats consolidation, power management, and every reason the hypervisor wanted those cores back.
This is also, roughly, where a stock distribution guest lives once its hypervisor promises it dedicated cores. The haltpoll cpuidle driver spins for a few microseconds before halting, and for wakeups this fast it behaves like the poll column and spends the poll column’s CPU for the duration of the ping-pong. It is the better-engineered version of the same bargain: because it halts once the guest goes properly idle, it does not hold 19 W forever, and because it only wins by spinning, it does not get you a fast wakeup and a free core at the same time. Nothing does. That is the point of this table, and it is why the 2.5x in the section above is a fact about one configuration rather than a fact about virtualization.
Or launch QEMU with -overcommit cpu-pm=on, which passes mwait through so the guest can sleep for real. Latency comes within 4 to 15% of multikernel, but the fine print is expensive. The guest only ever reaches C1, leaving package power 12 W over baseline, and no flag fixes that: the instruction is passed through, the knowledge of deeper states is not (no mwait sub-state CPUID leaf for intel_idle, no C-states in QEMU’s ACPI tables), and every incentive says it stays withheld, because cores sleeping in C6 fund the whole socket’s turbo budget, a lever no hypervisor hands to one tenant among many. Meanwhile the vCPU thread never exits on idle, so the host sees the cores as 100% busy even while the silicon sleeps: it cannot tell guest-idle from guest-work, cannot harvest idle cycles, and overcommitting such a vCPU is pointless, since the first co-tenant brings the scheduler back into every wakeup.
Frequency Is Not Yours Either
C-states are only half of power policy; the other half is frequency, and there the guest holds even fewer cards. Inside our VM the cpufreq directory is simply empty: no driver, no governor, no P-state interface, and /proc/cpuinfo reports a constant nominal 2000 MHz whatever the silicon is doing. Frequency belongs to the host’s governor, which steers by host-visible load: a polling vCPU reads as flat-out busy and is held at turbo, which is nearly all of the poll column’s wattage, while a HLT-idle vCPU parks at its 800 MHz floor between wakeups, so the frequency ramp rides on top of the exit cost. We measured the frequency knob’s whole trade by capping the polling guest’s cores from the host:
| Polling guest clock | Package power over baseline | Pipe latency | Null syscall |
|---|---|---|---|
| 3.8 GHz (turbo, the default) | +13.7 W | 3.34 µs | 0.099 µs |
| 2.0 GHz (capped to base clock) | +9.5 W | 4.85 µs | 0.177 µs |
| 0.8 GHz (capped to the floor) | +0.7 W | not measured | not measured |
Capping to base saves 4 W and multiplies every clock-bound latency by exactly the clock ratio (1.9x, verified by L1 latency moving from 1.31 to 2.51 ns); capping to the floor erases the power cost along with the latency the polling existed to buy. A static frequency cap is the manual, always-on version of what a C-state does per microsecond; the native kernel gets the dynamic version, full clock at wakeup and floor power in idle, and the guest can only pick one half and keep it. The market rations this lever even more tightly than sleep: on EC2, P-state control remains confined to older whole-socket types and bare metal even where C-state control has spread, because turbo budget belongs to the whole package. The multikernel instance simply runs its own cpufreq governor on its own cores, though it partitions the control, not the physics. None of this skews our comparison: identical L1 latencies pin both sides at the same effective clock during the measured loops.
What the Watts Are Worth
Do the watts matter, then? As electricity, barely: a lone polling guest costs 14 to 19 W (almost entirely turbo, as the table above shows), and stacking shows each additional one adds only 1 to 5 W, because multi-core turbo lowers everyone’s clock as the socket fills, polling guests measurably stealing turbo budget from each other and from real work. At industrial rates behind a datacenter PUE that is about $20 a year in the worst case and a few dollars stacked. The watts earn their place here for a different reason: they are the only honest signal left. Host CPU accounting reads the same 100% for a spinning guest, a C1-sleeping guest, and a genuinely busy one; package power is how we could tell them apart at all. And what they point to is the actual bill: those cores read busy forever, so the host can never place another tenant on them, and each latency-tuned guest permanently strands capacity a cloud would sell as a 2-vCPU instance, $260 to $880 a year at retail, near $150 at amortized server cost. The wattage is the receipt; the stranded silicon is the price. The idle multikernel instance, meanwhile, draws baseline power because its cpuidle governor has the menu a native kernel has: C1’s few-hundred-nanosecond wakeup during active ping-pong, C6 during long idle. The guest’s menu has no such entry: HLT (cheap, but 2.5x), polling (fast, a core spent on nothing), or shallow mwait (a core the host can no longer account for).
Where the Industry Sits
The industry has already seated itself around this menu, and the seating chart is in official documentation. Host-side, KVM ships its own polling layer: the host spins for up to a configurable window before scheduling out a halted vCPU (halt polling), trading host CPU for wakeup latency fleet-wide. Guest-side, the haltpoll driver and governor do the same from inside the VM, and Ubuntu ships that driver as a module in its stock kernels. It loads itself only where the hypervisor advertises KVM’s realtime hint, the flag whose defined meaning is that vCPUs are never preempted, which is to say: on dedicated cores. So a stock distribution guest on an ordinary shared instance idles in the HLT column, and moves to the poll column exactly when its cores stop being shared. Even the polling workaround is rationed by core ownership. And the mwait column is sold as a product feature under its own name: EC2 processor state control hands C-state control to guests on bare metal instances, on the largest sizes of the older instance families, and on all sizes of the newest generations, where a tenant’s vCPUs own their physical cores outright. The direction of that list is the point: sleep-state control is granted exactly as far as core ownership extends, and no further.
Notice what just happened. To make the VM fast we pinned its vCPUs to dedicated cores, then gave it exclusive control of their sleep states. Follow that ladder to the end and the guest is no longer sharing anything: it is a static hardware partition, hand-assembled from hypervisor exemptions. And even then the residue stays. A null syscall held near 1.4x and signal registration near 1.3x across the poll and mwait guests as well, with the entire virtualization stack still resident and still inside the trust boundary.
Caveats
The medians above come from three full-suite runs per configuration, and the spread is worth stating. The multikernel side is strikingly stable across runs (its widest spread was the context switch, 1.01 to 1.38 µs); the guest’s wakeup numbers swing more (pipe ranged 5.8 to 7.2 µs across its default-idle runs), which is itself a symptom of the machinery underneath. The two configurations ran on different physical core pairs of the same socket (we verified all four cores share package 0, minding that kerf takes physical APIC IDs while taskset takes logical CPU numbers). Two metrics are excluded because the two sides boot different root filesystems: fork+execve loads the binary through the filesystem, where the guest is consistently faster (214 µs on tmpfs vs 315 µs on DAXFS), and lat_pagefault faults against it (0.15 µs vs 1.74 µs). Both measure DAXFS against tmpfs, not virtualization. fork+/bin/sh crosses those same filesystems but lands at 1.04x regardless, so we kept it. And a production VM sharing an oversubscribed host pays the measured floor plus scheduling on top.
Reproduce It Yourself
Everything runs from a stock lmbench built out of the autubrew/lmbench Docker image, exported once to a plain rootfs directory that both worlds share:
docker run --name lmbench-build --entrypoint /bin/bash \
autubrew/lmbench -c 'cd /root/lmbench/src && make -j8'
docker export lmbench-build | tar -x -C /root/lmbench-rootfs
The multikernel side loads that directory with kerf load --rootfs-dir and reads results from the DAXFS overlay on the host. The KVM side packs the same directory into an initramfs and replays the result file over the serial console. One driver script, two boots, one diff.
The complete scripts are published as a GitHub Gist, exactly as they ran: lmbench-full-daxfs.sh boots the multikernel instance, lmbench-vm.sh boots the KVM guest, both execute the shared run-lmbench.sh driver (the guest through vm-init.sh), and run-matrix.sh produced the three-run medians reported throughout.
We encourage you to run this on your own hardware. The absolute numbers will move with CPU generation, kernel version, and idle policy. The shape will not: the memory system reaches parity because that was a hardware problem and hardware solved it; the wakeup paths reach parity only when you stop sharing the hardware; and the kernel-entry residue never reaches parity at all, because that one is the price of having a hypervisor in the loop.
The Itemized Bill
Add up what this post measured. A guest with every hardware assist of 2026 matches native memory to the nanosecond, then pays 2.5x on the context switch, 2.2x on the pipe, and about 30 ns at every kernel entry. It can buy the wakeups back: spin and spend a core, or take mwait and blind its host. But after every purchase the receipt reads the same. The fastest virtual machine we could assemble was one that had stopped sharing anything, slept in a single shallow state it could not deepen, ran at a clock it could not see, looked permanently busy to the host that owned it, and still arrived 30 ns late to every syscall. That is not a slow VM. That is the best one.
The multikernel instance, meanwhile, held every number the tuned guest was chasing while doing none of the chasing: idle in C6, waking at full clock, drawing baseline power, its cores honestly accounted, because every decision was made by a kernel that owned the hardware it was deciding about. Fast was not a configuration. It was the default.
So the finding fits in one sentence. In 2026, the virtualization tax is no longer paid in performance; it is paid in the sharing, the sleep, and the honesty you must surrender to get the performance back. A hypervisor’s limit is a partition that pays rent. A multikernel starts as the partition, and keeps the rent.
Multikernel is open source. If you are rethinking what isolation has to cost, we would love to hear from you at contact@multikernel.io.