Scaling Anonymous Page Faults on a 128-Core ARM NUMA Server

Modern ARM servers are shipping with core counts well beyond what was common a few years ago. At this scale, the core count alone says little about application throughput. NUMA topology, memory-management behavior, and kernel synchronization determine whether additional cores become additional capacity.

This post examines a 128-core ARM server with four NUMA nodes in a single physical socket, using the page_fault1_processes test from will-it-scale to measure how anonymous page fault throughput scales from one worker to all 128 cores. The central finding is that the workload scales poorly from core to core within a NUMA node, but the resulting per-node capacity scales almost linearly as additional nodes are added. Our previous will-it-scale post characterized the scaling wall on a 48-core x86 system and measured a multikernel split against it. This post is a single-kernel study on a different architecture with a considerably larger NUMA node, and it establishes where that wall stands on ARM.

Hardware Topology

The test system has 128 cores in one physical socket, organized as four NUMA nodes of 32 cores and approximately 64 GB of memory each:

NUMA node 0: CPUs   0-31
NUMA node 1: CPUs  32-63
NUMA node 2: CPUs  64-95
NUMA node 3: CPUs 96-127

Although the system contains only one socket, the processor exposes four NUMA domains. This topology is increasingly common in high-core-count parts, where a single package contains multiple compute dies, memory controllers, and interconnect segments. The NUMA distance matrix confirms a genuine internal hierarchy:

       0   1   2   3
0:    10  11  11  12
1:    11  10  12  11
2:    11  12  10  11
3:    12  11  11  10

A distance of 10 is local memory. Remote nodes report 11 or 12, so the processor distinguishes more than one tier of remote locality within the socket. These are relative topology costs rather than measured latencies, but they establish that the system is not a uniform 128-core machine.

Benchmark Method

page_fault1_processes has each worker map anonymous memory and touch it page by page. Every iteration passes through the fault handler, the page allocator, page zeroing, page table updates, memcg and per-node accounting, and LRU insertion. In processes mode, workers share nothing in userspace: each has its own address space and its own mappings. Any failure to scale is therefore inside the kernel.

Each run lasted 30 seconds. CPU placement was controlled with taskset, and the larger masks were chosen to align exactly with NUMA boundaries:

taskset -c 0-7    ./page_fault1_processes -s 30 -t 8
taskset -c 0-31   ./page_fault1_processes -s 30 -t 32    # node 0
taskset -c 0-63   ./page_fault1_processes -s 30 -t 64    # nodes 0-1
taskset -c 0-95   ./page_fault1_processes -s 30 -t 96    # nodes 0-2
taskset -c 0-127  ./page_fault1_processes -s 30 -t 128   # nodes 0-3

This design separates two questions: how the workload scales as cores are added within one NUMA node, and how aggregate throughput scales as additional nodes are brought online. The answers differ substantially.

Scaling Within One NUMA Node

The first experiment restricted execution to node 0, with 32 cores available:

Scaling from one to eight workers is reasonable: throughput rises from 0.91 million to 4.64 million operations per second, with efficiency declining but the curve still climbing. Beyond eight workers the behavior changes. Sixteen workers deliver 4.58 million, 24 deliver 4.53 million, and 32 deliver 4.46 million. The additional 24 cores contribute no throughput, and the curve declines slightly. With the node fully occupied, the workload reaches about 15% of ideal linear per-core scaling.

For this workload, a 32-core NUMA node does not behave like 32 independently scalable cores. Anonymous page fault throughput reaches a ceiling at approximately eight workers.

Scaling Across NUMA Nodes

The second experiment widened the CPU mask one node at a time:

Relative to the saturated single-node result, two nodes deliver 2.00x, three deliver 3.02x, and four deliver 4.03x. Aggregate throughput increases almost exactly with the number of NUMA nodes, each contributing approximately 4.5 million operations per second largely independently of the others.

This should not be read as linear 128-core scaling. The unit that scales is the NUMA node, and each node has already lost most of its per-core efficiency before it joins the aggregate. Against the 0.91 million single-worker baseline:

Throughput scales almost linearly from 32 to 128 workers while overall efficiency remains at roughly 15% throughout. There is no contradiction: the system adds approximately 4.5 million operations per second for every 32 cores. Measured per NUMA domain, that is strong scaling. Measured per core, it is not.

Below the Throughput Numbers

To characterize the saturation point, perf stat was used to compare eight workers against 32 on node 0:

Four times as many CPUs complete essentially the same number of page faults while consuming 5.7x the system CPU time. Per completed fault, system time rises from approximately 1.2 microseconds to 7.0 microseconds. The additional cores are fully utilized but contribute no useful work; they execute kernel code that exists only because of the added concurrency. IPC is nearly unchanged, which argues against a simple memory-bandwidth stall: the cores retire instructions at a normal rate, but the instructions are synchronization and accounting rather than fault handling.

A kernel-only perf profile of the 32-worker case identifies where that time goes:

Further down the profile are the remaining memory-management functions: lruvec_stat_mod_folio, folio_remove_rmap_ptes, count_memcg_events, handle_mm_fault, do_page_fault, __alloc_frozen_pages_noprof, and get_page_from_freelist.

Lock profiles require some care. A sample attributed to _raw_spin_unlock_irqrestore reflects where the CPU was when the sampling interrupt was delivered, which on ARM includes the point where interrupts are re-enabled after a critical section, so part of that 23% is time deferred from inside the lock rather than time spent waiting for it. Inlining and attribution affect the rest. Even so, the shape is clear: over a quarter of kernel samples land on spinlock primitives, roughly a tenth on memcg and per-node statistics, and the one unambiguously useful symbol, __pi_clear_page, accounts for 17%.

The evidence points to a kernel-side scalability bottleneck in the memory-management path rather than exhaustion of CPU capacity. Identifying the specific shared structure would require further profiling; the usual candidates are the allocator’s zone lock, the LRU lock, and the per-node and per-memcg counters updated on every fault.

Interpreting the Results

Physically, the system is one socket with 128 cores. For workloads that exercise kernel memory management, allocation, or shared kernel state, a more useful description is four domains of 32 cores, within each of which the kernel makes effective use of roughly eight. The second description predicts the benchmark results; the first does not.

The experiment also shows why measuring only the endpoints is insufficient. One worker at 0.91 million and 128 workers at 17.99 million is a 20x speedup, and the obvious conclusion is simply that the workload scales poorly. The intermediate points reveal the structure:

The NUMA boundaries explain why throughput resumes growing after the first node has saturated. Whatever is contended is largely per node, so each additional node brings a fresh instance of it. That is a useful clue in itself: the contention has a locality, and the locality is the NUMA domain.

Conclusion

The statement “the server scales” is incomplete without a unit. On this system, scaling at the core level stops at roughly eight workers per NUMA node. At the node level, each additional node contributes approximately 4.5 million operations per second, almost perfectly linearly. At the system level, 128 cores reach 18 million operations per second, about 15% of the single-worker rate multiplied out. All three statements describe the same run.

For performance engineering on high-core-count hardware, the practical lesson is to evaluate scalability against the NUMA topology rather than the total core count. A worker sweep should cross every node boundary, because those boundaries are where the curve changes shape. A single 128-core measurement reports how fast the machine is; measurements at 8, 32, 64, 96, and 128 workers report where that throughput comes from and where it is lost.

For multikernel, this is the relevant baseline. The x86 study showed that when a kernel’s shared state is the limit, partitioning the cores between kernels partitions the limit with them. On this system the hardware already defines four partitions, and the kernel’s contention respects them closely enough to scale by node. One kernel per NUMA node is the natural configuration to test on this machine, and it is the next experiment. We have not run it yet and make no claim about it here. What this post establishes is the number any such configuration must improve on: approximately 4.5 million operations per second per node, with eight of 32 cores doing useful work.

Reproducing the Measurement

The method does not depend on this hardware. Build will-it-scale, read the NUMA map from lscpu or numactl -H, and sweep page_fault1_processes with taskset masks that fill one node first and then add nodes whole. Run perf stat at the knee and at the full node width, and perf record on the saturated case. Absolute numbers will vary with core count and kernel version; the two-regime shape of the curve follows from where the kernel’s shared state lives and should not.

Multikernel is open source. If you are evaluating high-core-count servers and would like to discuss these results, contact us at contact@multikernel.io.