Same Wire, Ten Times the CPU

The previous post kept I/O off the table on purpose. It measured what a KVM guest pays on the kernel’s own hot paths, context switches and pipes and syscalls, and found that the memory system reaches parity while the wakeup paths do not. Every number ran from RAM.

Networking is where a hypervisor earns its keep, though, and where the VM stack has had fifteen years of engineering poured in: paravirtual devices, vhost, multiqueue, offload negotiation, interrupt posting. It is also the part of the multikernel story we had not built. An app-kernel that owns two cores and a gigabyte is not much use if it cannot talk to anything.

So we built it. mk_vnet is a virtio-net backend that lives in the host kernel and serves an unmodified virtio driver in an app-kernel over shared memory. Then we ran the experiment the previous post could not: the same kernel booted twice on one machine, once as a multikernel app-kernel with mk_vnet, once as a KVM guest with vhost-net and pinned vCPUs, both pushing iperf through the same 1 Gbit NIC to the same server two hops away.

Both saturated the wire on every test. That is the boring result, and it is the reason a gigabit link was the right instrument: with bandwidth pinned, the only thing left to measure is what each side spends to get there. Sending cost the VM about twice the CPU. Receiving cost it ten times. And the guest itself reported that it was 94% idle while doing it.

How an App-Kernel Gets a NIC

The design starts from a decision about what kind of kernels exist. There are exactly two. The device-kernel is the one firmware booted: it owns the PCI bus, the IOMMU, ACPI, and every physical device. An app-kernel is a spawn that runs one workload on cores and memory the device-kernel lent it, and it owns no hardware at all. Every device an app-kernel sees is a virtio device, driven by the unmodified virtio driver already in the Linux tree, served by the device-kernel.

That sounds like a VM’s device model, and the frontend half deliberately is. The backend half is where it stops being one.

The transport is a table in shared memory. Each app-kernel has a device table in a control block the host carves from the instance’s own memory before it boots. Each entry holds the feature bits, a status word, a fixed configuration area, and per-queue slots for the ring addresses the driver chooses. A device tree node points the spawn at each entry, and a 329-line platform driver, virtio_mk, plays the role virtio_mmio plays for a VM: it reads and writes the table and registers a virtio_device. The virtio-net driver above it does not know it is not in a VM.

The virtqueues live in the app-kernel’s memory. The driver allocates its descriptor, available, and used rings from its own RAM and writes their physical addresses into the table. There is no DMA API, no IOMMU mapping, and no address translation of any kind: both sides are CPUs on the same machine, and the host already has every byte of the spawn’s memory in its direct map. The host side is a 286-line split-ring implementation, mk_vring, that checks each address the driver hands over against the instance’s memory ranges once, when it walks the chain, and then works on plain pointers. A bad descriptor breaks that ring and nothing else.

Doorbells are a bit and an IPI. When the driver kicks a queue it sets a pending bit in the table entry and sends the bare multikernel IPI to the host CPU named in the table. The host’s IPI handler runs a 48-line scan over registered doorbells and calls the backend’s kick callback. Calls back to the driver go the same way in reverse. Nothing about the data plane touches the existing multikernel message ring, which stays what it was designed to be, a control plane with 64 four-kilobyte slots.

The backend is a netdev. mk_vnet is 562 lines in drivers/net, on the multikernel-virtio branch until it lands in the main tree. It registers with a small built-in device model that runs the status handshake and dispatches doorbells, and for every net entry it creates a host network device named after the instance, mk-iperf-0 in this test. That netdev is peered with the spawn’s eth0 the way one end of a veth pair is peered with the other. When the host stack transmits on it, mk_vnet writes the frame into the spawn’s receive ring in the caller’s context and rings the doorbell. When the spawn transmits, its kick schedules NAPI and the poll loop pulls frames out of the transmit ring into skbs in softirq. One copy per direction, done by whoever is already running, with no worker thread in between. The netdev outlives kill and re-exec, so addresses and routes configured on it survive an app-kernel reboot, and its carrier follows the spawn’s driver coming up and going down.

Offloads negotiate like real hardware. The backend offers checksum and TCP segmentation in both directions with 64 KB frames and mergeable receive buffers, and the host netdev’s own feature flags follow whatever the spawn accepted. A spawn that negotiates TSO hands the host one 64 KB frame per burst; the host stack segments it, or hands it to a NIC that will.

XDP takes the host stack out of the path. mk_vnet implements ndo_xdp_xmit and runs an attached XDP program on frames from the spawn. That lets kerf attach a pair of BPF programs when you ask for --virtio=net:ens7f1. On the NIC, the program looks up the destination IPv4 address of each arriving frame and, if it belongs to an app-kernel, redirects the frame straight into that spawn’s receive ring: no skb, one copy, done by the CPU that polled the NIC queue. On the app-kernel’s netdev, the program rewrites the source MAC to the NIC’s own and redirects the frame to the NIC’s transmit path. The switch sees one MAC per port, which is what campus ports, cloud instances, and wireless networks enforce, and the NIC does not need to be dedicated, put in promiscuous mode, or bridged. The cost of that speed is that an XDP program sees raw frames, so while one is attached the backend withdraws the spawn’s transmit offloads, exactly as virtio_net does in a VM.

Now list what is not in that description. There is no vhost worker thread, so no thread to schedule and no eventfd to signal it through. There is no tap device, because the backend is already a netdev. There is no bridge. There is no VM exit on a kick, because a kick is a store and an IPI, and no interrupt injection on a call, because a call is an IPI to a CPU that will take it natively. There is no vCPU scheduler between the interrupt and the driver, because the driver runs on a CPU it owns. And there is no daemon: kerf create makes the netdev, kerf exec boots the spawn, and the two find each other through the table.

One thing is also absent that a VM has, and it should be said plainly: a security boundary. The app-kernel’s memory is in the host’s direct map because both kernels are peers on one machine, and every shortcut above exists because that boundary does not. mk_vring makes the host robust against a broken ring; it does not make the host safe from a hostile one. This is a partition, not a sandbox.

Benchmark Setup

Same host as before: a dual-socket Intel Xeon Gold 5418Y with SMT off, and now an Intel 1 Gbit NIC driven by igb with native XDP. The iperf server, iperf 2 listening on TCP, is a campus host one router hop away; the host kernel itself reaches it with a 0.43 ms round trip and 942 Mbit/s of TCP, which is the wire. The kernel is one build, 7.0.0-mk2+, from the multikernel tree with mk_vnet, and both sides boot the identical bzImage and the identical root filesystem, an Ubuntu 20.04 image with iperf 2.0.13, driven by the same shell script. Both get two CPUs and 1 GB, on socket 0, and the same static address, so from the server’s point of view the two runs are indistinguishable.

Multikernel. One kerf instance on two dedicated cores, with a virtio-net device routed through the NIC by XDP:

kerf create iperf --cpus=2,4 --memory=1024MB --virtio=net:ens7f1
kerf load iperf --kernel=bzImage --rootfs-dir=/root/iperf-rootfs \
    --entrypoint='/bin/bash /run-iperf.sh' \
    --nic=eth0 --ip=198.51.100.99 --netmask=255.255.255.0 --gateway=198.51.100.1
kerf exec iperf

The spawn’s eth0 is the unmodified virtio_net driver over virtio_mk. Because XDP is attached, the spawn ran without transmit checksum or segmentation offload: every TCP segment it sent was checksummed and sized in software inside the app-kernel. Keep that handicap in mind when reading its CPU numbers.

KVM. The same bzImage under QEMU/KVM with -cpu host, two vCPUs pinned one-to-one to two idle cores, QEMU’s I/O threads and the vhost-net worker pinned to a third core so they never compete with the guest, and the guest’s memory bound to the same NUMA node. The device is virtio-net-pci with vhost=on, the fast path every cloud runs:

numactl -m 0 taskset -c 10,12,14 qemu-system-x86_64 -name iperf,debug-threads=on \
    -enable-kvm -cpu host -smp 2 -m 1024 -kernel bzImage -initrd iperf-initrd.gz \
    -append "console=ttyS0 ip=198.51.100.99::198.51.100.1:255.255.255.0::eth0:off rdinit=/vm-init.sh" \
    -netdev tap,id=n0,ifname=tap0,script=no,downscript=no,vhost=on \
    -device virtio-net-pci,netdev=n0,mac=52:54:00:53:00:63

The guest negotiated checksum and TSO with vhost, so unlike the spawn it did have its transmit offloads. Its tap could not be bridged onto the NIC, because the campus port drops frames from a second MAC, which is the same constraint mk_vnet’s XDP masquerade exists to satisfy. So the VM sits behind a routed tap: the host answers ARP for the guest’s address on the NIC with proxy ARP and forwards between tap and NIC through its own IP stack. This is libvirt’s routed mode, and it is the fair analog: on both sides the host mediates the address, XDP on one, forwarding on the other.

Measurement. iperf reports throughput. For CPU we did not trust either guest. Each guest logs its own /proc/stat around every test, and the host samples every CPU’s /proc/stat once a second and correlates by timestamp. For the VM the host view includes the two vCPU threads, the QEMU and vhost threads, and the forwarding softirqs. For multikernel the spawn’s cores are offline on the host, so its cost is the spawn’s own accounting plus what the host spent redirecting frames. CPU cost below is given as a percentage of one core: 100% is one core saturated for the whole test, 215% is a little more than two cores. Every number is the median of three 15-second runs.

Line Rate, Two Bills

Every TCP test on both sides landed between 941 and 942 Mbit/s. Here is what each side spent to get there.

Read the two KVM columns against each other first, because they are the story in miniature. During the download the guest believed it was using 9% of one core. The host was spending 215%, more than two full cores. That gap is not measurement noise; it is the definition of a hypervisor. The work of being a VM happens in the host, on the guest’s behalf, and the guest’s own accounting cannot see it. A team sizing this workload from inside the VM would have provisioned it at a twentieth of its real footprint.

Upload is a 2x story. A single TCP stream costs the app-kernel 13% of a core all in, and the VM 26%. The spawn’s own share, 9.3%, is higher than the guest’s visible 5.2% because the spawn had no TSO: it built and checksummed every 1500-byte segment itself while the guest handed vhost 64 KB frames. Even so, the host side of mk_vnet was 3.8%, the cost of XDP redirecting about 80,000 frames a second, and it is the only host cost there is. The VM’s host total is what remains after you take the guest’s 5.2% and add the vCPU threads’ exits and polling, vhost copying from the guest, and the forwarding softirqs.

Four streams is a 7.7x story. Multikernel does not move: 13% whether the traffic is one stream or four, because the work is the same bytes through the same ring. The VM climbs to a full core. Four streams means four sockets’ worth of ACKs arriving and four kicks interleaving, and each of those is an exit, a wakeup, or a poll.

Small writes are a 2.5x story, and the one place the guest looked expensive to itself. With 256-byte writes and Nagle off, iperf makes about 460,000 send calls a second. The guest’s own accounting jumped to 40%, twice the spawn’s 20%, because each of those sends ended in a kick to the device, and a kick from a guest is a trapping instruction. The spawn’s kick is a store and an IPI.

Download is a 10x story. Receiving 941 Mbit/s cost the app-kernel 21% of a core in total: 12% in the spawn and 9% on the host to redirect frames into its ring. It cost the KVM guest 215%, with both pinned vCPU threads at essentially 100% on the host while the guest reported itself 94% idle. That number was suspicious enough to deserve its own experiment.

Where Two Cores Went

We reran only the download test, three times in one guest, changing one host setting between runs and snapshotting KVM’s own counters from debugfs at every phase boundary.

The setting is KVM’s halt polling, and it is worth being precise about what it is, because it is neither of the two idle policies the previous post measured. The guest here runs the default HLT idle throughout; nobody passed idle=poll. When a vCPU executes HLT it exits to the host, as it must. The question is what the host does next. By default, before putting the vCPU thread to sleep, KVM spins for up to halt_poll_ns, 200 µs on every distribution we know of, checking whether an interrupt for that vCPU has become pending. If one arrives during the spin, the thread re-enters the guest without ever sleeping, and KVM counts a successful poll. The window is adaptive: it grows on success, up to the cap, and shrinks only on failure.

Now put 941 Mbit/s of incoming TCP through that. After the host’s GRO and vhost’s batching, the guest still gets a receive interrupt every few tens of microseconds, comfortably inside a 200 µs window. So every poll succeeds. In 30 seconds KVM attempted 464,000 halt polls and 458,000 of them succeeded, which means the vCPU threads reached the sleep step about 6,000 times. From the host’s view both threads are pegged. From the guest’s view it is idling in HLT, 94% of the time. Halt polling was designed to skip the scheduler round trip on an occasional wakeup, and under a sustained interrupt stream it degenerates into idle=poll implemented on the host side, invisible to the tenant, with the host paying the bill.

Disable it and the threads actually sleep: halt_wakeup climbs from 34,000 to 245,000, meaning the scheduler woke them a quarter of a million times, and the host total drops from 220% to 54% of a core at the same bandwidth. The guest’s visible CPU rises to 14%, because each real wakeup now carries the full exit, sleep, wake, and entry that polling was hiding.

That 54% is the honest floor for this VM, and it is worth itemizing, because “is the rest just VM exits?” is the natural question. The guest saw 14% of work; its two vCPU threads consumed 32% on the host. The 18% between those numbers is the host acting on the vCPUs’ behalf, and KVM’s counters say what it was: about 8,200 HLT exits and 4,600 other exits per second, the latter mostly the guest’s kicks on the virtio queues, which trap. That is roughly 14 µs per exit episode, far more than the 1 to 2 µs a bare exit and entry cost, because with polling off each HLT exit now drags the whole wake path behind it: the thread sleeps, the host core drops into idle, vhost’s irqfd fires, the scheduler wakes the thread, an IPI wakes the core, and the VM entry finally happens. The other 17% is the vhost worker and QEMU moving the bytes, and the last 5% is the host forwarding between tap and NIC.

Line the two columns up and the shape is clear. The two kernels do about the same work receiving TCP, 14% against 12%. Everything else on the VM side is machinery the app-kernel does not have: the exit and wakeup path, a longer copy path with a thread hop in it, and the forwarding. The multikernel number required no tuning at all. The spawn’s idle cores were in real C-states through intel_idle, and each doorbell IPI woke one of them directly into the driver. The split of the 18% between the exits themselves and the sleep and wake path around them is inferred from the counters and the accounting rather than measured with a profiler; the total is measured.

The Round Trip

Bandwidth was a tie by construction. Latency was not.

The host kernel reaches the server in 0.43 ms. The app-kernel adds about 0.10 ms to that round trip: an XDP redirect on the way in, a doorbell IPI, the spawn’s stack, and a redirect on the way out. The KVM guest adds 0.40 ms, four times as much, through vhost, the tap, the host’s forwarding path, and an interrupt injection and exit at each end. Under load the difference vanishes into the queue; iperf’s own TCP round-trip estimate during a full-rate upload was 1.54 ms for the spawn and 1.57 ms for the guest, both dominated by the buffering in front of a saturated gigabit link. It is the unloaded number that describes the path, and the path is four times longer for the VM.

Before You Quote These Numbers

A gigabit NIC is the instrument that made bandwidth a constant, and it also caps what this post can claim: we have not measured what either side does at 25 or 100 Gbit/s, where the host CPU per byte becomes the throughput limit rather than a footnote to it. The ratios above are per-byte costs at 1 Gbit/s; the tests that scale with packet rate rather than bytes, four streams and small writes, are the best hint of what faster links will show.

The spawn ran without transmit offloads because XDP was attached, and every one of its CPU numbers includes software checksumming and segmentation that the guest did not do. A spawn without XDP, or with a NIC program that preserves offload metadata, would post lower spawn numbers than these.

The VM’s host cost includes the host’s IP forwarding between tap and NIC, which a bridged VM would not pay. We could not bridge, for the same reason mk_vnet masquerades, so both sides carry a host-mediated address path. In the run with halt polling off, everything outside the vCPU, QEMU, and vhost threads, which is where forwarding lands, came to 5% of a core, so it cannot explain the difference between 54% and 220%.

The kernel under test routed every doorbell from every app-kernel to one host CPU; a per-instance doorbell CPU exists in the tree but was not in this build. And as before, medians of three runs: the multikernel side was steady to the third digit; the VM’s tradeoff runs, upload and download together, ranged from 118% to 127% of a core across the full matrix, and the download half from 220% to 221% in the focused experiment.

Reproduce It Yourself

Build the multikernel-virtio branch of the kernel with CONFIG_MK_VNET and CONFIG_VIRTIO_MULTIKERNEL, create an instance with --virtio=net:<your NIC>, and give it an address with --ip. Anything with a virtio-net driver boots against it.

For the VM side, the iperf rootfs is any image with iperf 2 packed as an initramfs, booted with the kernel’s ip= parameter and a routed or bridged tap with vhost=on. Pin the vCPU threads by name (-name debug-threads=on makes them CPU 0/KVM and CPU 1/KVM) and put QEMU’s other threads and the vhost-<pid> worker somewhere else. Sample /proc/stat per CPU on the host for the whole run; the guest’s own numbers are the least informative thing you will collect.

We would like to see this repeated against an iperf server that also listens on UDP, at 25 Gbit/s and above, and with the guest’s haltpoll cpuidle driver enabled, which moves the polling from host to guest and makes it visible to the tenant without making it cheaper.

The Second Bill

The previous post ended with an itemized bill for the kernel’s own paths: a tuned guest matches native memory, then pays 2.5x on a context switch and 30 ns at every kernel entry, and can buy the wakeups back only by spinning or by seizing mwait.

This post is the second bill, for the wire, and it reads the same way. Both kernels moved the same bytes at the same speed. One did it for 13% to 24% of a core and a tenth of a millisecond, without a hypervisor, a worker thread, a tap, a bridge, or an exit anywhere in the path, because the driver in the app-kernel writes a ring the host reads by pointer and rings a bell the host answers with an interrupt. The other did it for 26% to 215% of a core and four tenths of a millisecond, and hid nine tenths of that cost from the only party that could have seen it.

The mechanism this time was not mwait. It was 200 µs of host-side halt polling, a default that every cloud runs and no tenant can observe, turning two pinned vCPUs into two spinning cores under a gigabit of incoming traffic. Turn it off and the VM is honest and still 2.6x, and the remainder is itemized above: exits and the wakeups they force, a copy path with a thread in it, and forwarding. That is the shape we keep finding. Virtualization has moved its cost out of the throughput column and into the sharing, the sleep, and the accounting, and mk_vnet exists because an app-kernel should not have to pay it to reach the network.

Multikernel is open source, and so is kerf. If you are rethinking what isolation has to cost, we would love to hear from you at contact@multikernel.io.