Call for Call

Post 14 got a uprobe to fire. Post 15 got it to fire on every core. Neither said whether the scopes that came out were right, and a capture of Box3D with four functions hooked answered that on its own: the main thread's scopes sat one level too deep from second seven to the end, and on four worker threads runs of joints sat at depth zero, outside the solver task that had called them. Ghosts. The obvious suspect was the kernel duplicating uprobe hits on thread migration, which C++ Orbit works around with a stack-pointer filter, and that filter was ported first. It dropped nothing. Lining every depth change up with the scheduler slices showed the opposite problem: at about one migration in ten the kernel lost a hit, an entry or a return, never doubled one. The pairing now uses the stack frame every hit carries to heal that, and this post is about the test that says whether it does.

A program that knows what it did

OrbitTestRust, the same binary the manual-instrumentation tour uses, has a second mode with no manual scopes in it at all. Three plain functions, kept out of line and out of tail position so that every call is a real entry and a real return at its own stack frame:

orbit_stress_outer(x)   calls orbit_stress_middle twice
orbit_stress_middle(x)  calls orbit_stress_inner three times
orbit_stress_inner(x)   eight multiply-adds, black-boxed

--stress-threads N --stress-hz F --stress-calls K spawns N threads that each call outer K times at F calls a second, paced by a spin-wait because a sleep cannot hold 5 kHz. So the capture must contain exactly N×K outer scopes at depth 0, 2×N×K middle at depth 1 and 6×N×K inner at depth 2, every middle inside an outer and every inner inside a middle, on N threads and no others. With --wait-go the program prints its pid and waits for a line on stdin before it starts, so the probes are armed before the first call and no call is in flight when the capture stops. Its last line of output says what it made.

The harness

The scenario loads the program's symbols, finds the three functions by name in the service's index, starts a capture with the three ids and the uprobe method, and polls the status line until the service says instrumenting 3 of 3 functions. Only then does it send the go line. When the program exits it waits out the pairing's 100 ms reorder window, stops the capture, reads the status line again — which since the frame pairing says N calls; a duplicate entries dropped, b entries with no return discarded, c returns with no entry dropped, plus any records the kernel reported lost — exports the bundle and hands it to check_stress.py.

The checker reads the events table with pyarrow, keeps the hooked scopes of the target pid, sorts each thread's by start, and walks them with a stack: for every scope it records the depth the timeline gave it against the depth the tree says, whether the scope on top of the stack is its parent and ends after it, and a count per function and per thread. It prints one JSON object; the scenario asserts on it.

What must hold

The kernel loses hits; the test cannot demand that nothing is missing. What it can demand is that the capture never contains something that did not happen, and that everything that is missing was reported missing. Six checks, in the order they would fail:

1. Every thread has scopes, and no thread that does not exist does.
2. No function has more scopes than calls. A duplicate hit that got through would show here.
3. No outer scope sits inside any other scope. An outer at depth one means an entry that never closed.
4. The status line's call count equals the number of scopes in the bundle: the service and the file agree.
5. Missing scopes = duplicates dropped + unclosed entries discarded + orphan returns dropped. Every hit the kernel loses costs exactly one scope and exactly one count in one of those three columns; if the pairing ever hides a loss (a ghost) or invents one, the two sides differ.
6. Scopes at the wrong depth, and scopes outside their parent, number at most eight per lost hit: an outer's two middles and six inners can sit one level off until the loss is healed, and no more.

A seventh check is a ceiling rather than an equality: under one percent of scopes lost, so a kernel that started dropping hits by the thousand would fail the run rather than merely be noted.

The fifth check is the one worth the post. Before the frame pairing, a lost return left an entry open until the capture ended; the bundle was short one scope and the status line said nothing, and everything after it on that thread was a level too deep. Now the discarded entry is a number, and the number has to match the hole. The same equation catches the opposite mistake, a pairing that discards live entries: the bundle would be short by more than the line admits.

Running it

python3 tools/e2e/orbit_e2e.py --only dyn-instr-stress --sudo --no-shots
ORBIT_E2E_STRESS=16,20000,20000 python3 tools/e2e/orbit_e2e.py --only dyn-instr-stress --sudo --no-shots

--sudo runs the service through sudo -n, which fails fast rather than prompting. Arming uprobes needs CAP_SYS_ADMIN: the kernel's uprobe PMU checks that capability in perf_uprobe_event_init before perf_event_paranoid is consulted, and it does not accept CAP_PERFMON, which every other perf path honours — a process holding only that one gets EACCES from the minimal uprobe attr, which this post's first draft, and the service's own refusal message, had wrong. Without the capability the scenario passes as a skip and says so, in 0.6 s. On the machine below the privileged runs go through a small root-owned wrapper (tools/sudo) that a sudoers rule allows without a password and that hands the service CAP_SYS_ADMIN, CAP_PERFMON and CAP_DAC_READ_SEARCH under the user's own uid; the manual says what that is worth in security terms. ORBIT_E2E_STRESS sets threads, rate and calls; the default is the 8 × 5,000 × 5,000 of the masthead, a second of work. The checker needs pyarrow, from ORBIT_E2E_PYARROW_PYTHON like the Python reader scenario.

Results

Five configurations, one second of work each, on a 32-core Linux 7.0 box; every number is in the metrics file with the per-function duration spread.

The first two rows are the ones the test was written for, and they are clean: 720,000 scopes, 1.44 million probe hits, every call seen once, at its depth, inside its parent, with nothing for the pairing to heal. The Box3D capture had predicted a few tens of migration losses per 360,000; the stress program's threads hardly migrate, and the one thing the pairing did in two runs was drop a single re-reported entry, at zero missing, which is the number that decides the duplicate filter's future: one real re-report in 720,000 hits, and no sibling taken for one.

That one duplicate also refined check 5. A dropped duplicate costs a scope only when the filter misjudged a real call, so the hole is bounded below by the unclosed and orphan counts and above by those plus the duplicates, rather than equal to their sum.

The last three rows are the second regression class from the list below, met for real. From 1.44 million scopes a second, 2.9 million hits, the kernel reports PERF_RECORD_LOST: the per-CPU rings (256 KB each at the time) overflow before the single drain thread, on a 5 ms poll, gets back to them. What the kernel drops never reaches the pairing, which is why its own counts stay in the hundreds while the hole runs to a million, and why the check only holds the floor once records are lost and leaves the verdict to the one-percent ceiling. The shape of the loss is instructive: the same 1.44 million on 8 threads at 20 kHz loses fifteen times more than on 16 threads at 10 kHz, because a thread at 20 kHz pours 360,000 hits a second into one CPU's ring, which fills in about 40 ms. The millisecond maxima in those rows are pairing artefacts across a hole, not slow calls. Bigger rings were the follow-up: raising the ring to 4 MB (post 20's sibling change) moved the clean ceiling to about 1.44 million scopes a second, and the pairing itself never fell behind a ring that had not overflowed.

The same session ran the other uprobe scenarios privileged for the first time: instrumentation (63,613 hooked spans on OrbitTestRust, longest 3.39 ms), hook-from-report (a Box3D function armed from the Flat report's context menu), thread-states and self-instrumentation, all passing.

What it would catch

Three regressions this test exists for. A pairing that pops by count again (the plain stack): the first lost return puts a thread a level too deep and checks 3, 5 and 6 fail at once. A ring too small for the rate, which the sweep above met at 1.44 million scopes a second: PERF_RECORD_LOST counts appear on the status line, thousands of scopes go missing in bursts, and the one-percent ceiling fails. And the duplicate rule taking a real sibling for a migration duplicate: two calls merge into one, which is one scope missing and one duplicate entries dropped — the bound still holds, and the run's note says how often it happened, which is the number that decides whether that rule should stay.