The request in the TODO reads: aggregate many machines' services to a single viewer, with precise clock synchronization, and an easy way to launch a service against the main viewer or connect to a remote one from the viewer itself. Before designing anything I went looking for what the Rust port already has, because the rail draws machine trees today and it would be easy to assume the model underneath is real. It is not. This post is what exists, what the design should be, what each phase delivers and how each is tested, and the five decisions I need from you before I start.
What exists, and what only looks like it exists
Three facts set the shape of the design.
One service, many viewers, already. orbit-service serves the wasm viewer and a /ws stream from the same port (44766 by default, bound to every interface). The stream is a tokio::sync::broadcast channel; each browser that connects gets hello_and_snapshot_frames(), a catch-up of names and interned strings, then the live batches. The viewer is same-origin: it talks to whatever window.location.host served it. So "connect a viewer to machine X" is already a solved problem, spelled open X's URL. What is missing is the converse.
Every timestamp is one host's CLOCK_MONOTONIC. Perf events (the perf attr sets clockid), manual scopes, the GPU helper, the collector, the API: all the same clock, deliberately, so a scope written by the target lines up with a kernel record without conversion. That is the right choice for one host and it is exactly why two hosts cannot be overlaid: monotonic is nanoseconds since that boot. The only bridge in the data is the SystemInfo record at the head of every capture, which samples CLOCK_REALTIME and CLOCK_MONOTONIC at the same instant. One pair, once, at capture start; nothing reads it for alignment. It says where the capture began on the wall clock to NTP precision. It says nothing about rate.
The machine tree is a demo fiction. MachineId lives in orbit-live-event/src/dev.rs, the viewer's self-profile and demo module. It is a two-valued enum, Local and Remote, and from_pid() assigns it by matching two constants, pids 20 and 21, that only the demo generator emits. The comment above them: "Spoofed remote-demo pids. LiveEvent has no machine field, the rail maps pid ranges." The reordering, collapsing and per-machine Scheduler nesting in tracks.rs are real and will carry over. The identity under them will not.
A service that is also a client
There are two places aggregation could happen. The browser could open one WebSocket per machine and merge in wasm. Or one orbit-service, the one whose URL you opened, could connect to the others as if it were a browser, stamp what arrives with a machine id, and push it into its own ring. I am proposing the second, and calling that service the hub. Any service can be a hub; nothing is configured in advance. The hub is whichever one you asked to add a machine.
Why not the browser? Four reasons, one of which is decisive on its own.
- Reports.
/api/sampling/report, the trees, function search, disassembly and source all run on the service that owns the process. A browser merging N machines would reimplement the report aggregator client-side or call N services and merge JSON. The hub just forwards the call to the machine the pid belongs to. - Connections. With N machines and M open tabs the browser design makes N×M links, each re-doing the catch-up snapshot. The hub makes N, once.
- Reach. A browser cannot run
ssh, cannot accept an inbound connection from a remote behind NAT, and cannot install anything. A service can do all three. - Clocks. The decisive one. Aligning two hosts needs round-trip timing at microsecond resolution. Browsers deliberately coarsen
performance.now()to 100 µs (5 µs when cross-origin isolated) as a Spectre mitigation. A probe from wasm would be an order of magnitude worse than the offset it is trying to measure. Server-to-server, withclock_gettimeon both ends andTCP_NODELAY, there is no such floor.
The cost is that a remote's events cross one more hop before a browser sees them. The hub does no decoding on the hot path: it takes a remote's event batch, rewrites the pid word (next section) and the timestamps (the section after), and re-broadcasts. That is an in-place pass over 32-byte records.
Three ways to reach a machine
You asked whether users would have to SSH-tunnel. They can, and that works with no code at all today (ssh -L 44766:localhost:44766 host, open localhost:44766). But the tunnel is the wrong unit. It gives you the remote's viewer, not the remote in your viewer, and it leaves the user managing a port per machine. The link kinds below all produce the same thing on the hub, a PeerLink carrying the existing frames, and differ only in who opens the socket and what carries it.
| Link | How you say it | Who dials | Needs on the remote | When it is the right one |
|---|---|---|---|---|
| Direct | render-01 or render-01:44766 | hub → remote, WebSocket to /ws + the HTTP API | a running orbit-service, port reachable, the hub holds its token | LAN, dev boxes, anything already serving its own viewer |
| SSH | user@render-01 | hub spawns ssh user@render-01 orbit-service --stdio | sshd. If orbit-service is missing, the hub offers to install it over the same connection | anything you can ssh to; the default for a machine you have never profiled before |
| Reverse join | on the remote: orbit-service --join hub:44766 --code 7Q4K | remote → hub, then the hub speaks the client protocol back over that socket | outbound TCP to the hub and the join code from the hub's banner | containers, CI runners, cloud instances, anything behind NAT or a firewall that allows nothing inbound |
SSH as a transport, the way git and rsync use it
The cleaner solution than a tunnel is to not have a port at all. ssh host command gives you an authenticated, encrypted byte pipe to a process on the remote. git, rsync, borg and VS Code Remote all work this way: the client runs the server as a subprocess over ssh and speaks its protocol on stdin/stdout. orbit-service --stdio is that mode: the same length-prefixed frames the WebSocket carries, on file descriptors 0 and 1, with the HTTP API multiplexed as request frames on the same pipe. Auth is ssh's. Nothing listens. The user types the same thing they would type to log in.
It also answers the deployment question almost for free. If ssh host orbit-service --stdio fails with "command not found", the hub knows the remote's OS and architecture from ssh host uname -sm, and it knows its own. When they match, the hub streams its own executable, std::env::current_exe(), the 12 MB static musl binary that has no runtime dependencies by construction, to ~/.orbit/bin/orbit-service on the remote, verifies the SHA-256 over the same connection, and retries. When they do not match (a Linux hub adding a Mac), it fetches the right artifact from the install base site/install.sh already defines, or, until that site exists, prints the one-line install and stops. Version skew is handled the same way: a remote whose Hello reports a different protocol version is offered the hub's binary. No package manager, no infra the project does not already have, and the install is exactly as trustworthy as the ssh session it rode in on.
This installs software on another machine, so it asks first, every time, per host, with the exact path and size, and it never elevates. Uprobes need CAP_SYS_ADMIN on the remote just as they do locally; the dialog says so when the hook engine is uprobes, and --sudo is passed through to the remote's tools/sudo only if that wrapper exists there and the user ticked the box. Frida-engine hooks and sampling of the user's own processes need nothing.
Reverse join
Some machines cannot be dialed. For those, the remote dials the hub: orbit-service --join hub:44766 --code 7Q4K opens a WebSocket to the hub's /api/peer/join, and once the code checks out the roles flip, the hub becomes the client over that socket, and from there on the link is indistinguishable from a direct one. The hub's startup banner, the one lan.rs already prints with the LAN URLs, gains one line: "Other machines can join with: orbit-service --join 192.168.1.10:44766 --code 7Q4K". Copy, paste, done. A remote that joins without a code shows up in the hub's Machines panel as waiting with an Accept button, so a colleague can join your session by asking.
Identity: a machine is a byte, a process is a word
The hub keeps a machine table: index 0 is itself, each link gets the next free index, and an entry records the hostname from SystemInfo, a service_id (a random 128-bit id the service generates once and persists in ~/.orbit/service-id, so two boxes both called build-01 stay distinct and a reconnect after a dropped link lands in the same row), the OS and architecture, and the remote's boot_id, which is what says whether a reconnected machine's monotonic clock is still the same clock.
The invasive question is pids. The viewer keys everything on a u32 pid: RowId::Process(u32), ThreadId { pid, tid }, machine_sort, the report queries, the settings window's process picker. Two machines will have a pid 1234 each. There are two ways to fix that and I want to recommend one.
LiveEvent keeps its layout and its spare _pad byte; single-machine captures are machine 0 and read exactly as before.Pack (recommended). At ingest the hub rewrites pid to machine << 24 | pid. Every place the viewer already asks MachineId::from_pid keeps working with a one-line body. Reports and code endpoints decode the machine from the pid the viewer sends, forward the low 24 bits to the owning remote, and stream the answer back. Display strips the top byte. Threads need nothing: a tid is only ever looked up with its pid. Saved bundles need nothing in the tables, only a machines entry in the manifest. The whole viewer change is the enum and the formatter.
Widen (alternative). Put the machine in _pad and make RowId::Process, ThreadId, the sort maps, the report query strings and the bundle schema carry it. Cleaner in principle, and it caps nothing at 255. But it touches every pid-keyed structure in the viewer and the Parquet schema, for a limit (255 machines in one live session) that is above anything the ring could ingest anyway. If a fleet mode ever wants thousands of hosts, it will want aggregate lanes rather than a tree per host, and that is a different viewer.
The Scheduler track is per machine: it is already emitted inside the machine loop in tracks.rs, and a remote's core ids stay its own. Interned string ids collide too, and are remapped at ingest through a per-link table, the same way names are already re-announced to late subscribers.
Clocks: what can be promised
This is the section to read critically. Everything above is plumbing; this is the part that decides whether a cross-machine timeline means anything.
Why the existing anchor is not enough
The SystemInfo pair says how the remote's monotonic clock related to its wall clock at one instant. With a second host's pair you can subtract and get an offset, to the precision of each host's NTP, typically a millisecond or more on a LAN, worse on a laptop. Orbit resolves scopes in microseconds; a millisecond of alignment error is three orders of magnitude too coarse to say which of two events across machines came first. And it is one point. Two crystals disagree by parts per million; even after NTP slews them, tens of ppm between a pair is ordinary. Ten ppm is 10 µs of drift per second, 600 µs per minute. Any capture longer than a few seconds needs a rate, not just an offset.
The exchange
The hub sends a small ClockPing frame on each link at intervals (fast at first, then every few hundred milliseconds), and the remote answers with a ClockPong. Four timestamps, two per side, all clock_gettime(CLOCK_MONOTONIC) taken as close to the socket as the code can get.
One sample's error is bounded by half its round trip and is dominated by whichever leg was queued behind something. So the hub keeps a window of samples and does what NTP, PTP and Perfetto's clock snapshots all do: trust the fast ones. Samples within a small margin of the window's minimum round trip are the ones that met no queue on either leg; those are fed to a least-squares line, θ(t) = a + b·t, offset and rate against the hub's clock. Each remote event is then shifted at ingest, start_ns += θ(start_ns), and the ring holds one time axis. The renderer, the reports and the bundle writer never learn that anything happened. Because the fit can move slightly as samples arrive, events already broadcast are not retroactively moved during a live session; the fit at the time of ingest is what a live viewer sees, and the raw timestamps plus every sample are kept so a saved bundle can be re-fitted from the complete record.
Two details matter more than they look. Pings go on the link as their own tiny frames, never appended to an event batch, because a ping queued behind a 64 KB batch measures the batch. And the fit must survive a dropped link: on reconnect the hub compares boot_id; same boot, same clock, the window continues; new boot, new clock, the fit restarts and the machine's earlier events keep the old fit.
The precision I expect, to be measured
| Path | Expected round trip | Expected alignment after the minimum filter | Good enough for |
|---|---|---|---|
| Two services on one box (the e2e case) | 20–60 µs | a few µs; the true offset is exactly 0, so this is the calibration | proving the estimator |
| Direct TCP on a wired LAN | 100–300 µs | ±20–50 µs | ordering RPCs between services, matching a frame's send to its receive |
| Over ssh on the same LAN | +50–150 µs of channel latency | ±50–100 µs | the same, with a wider band drawn |
| Across a WAN or VPN | milliseconds, asymmetric | ±0.5–2 ms and honest about it | seeing two machines' load side by side; not causality |
| Both hosts with a PTP-disciplined NIC (later) | — | sub-µs, from CLOCK_TAI//dev/ptp0 pairs sampled locally | a shared clock in the true sense |
These are expectations from how the mechanism behaves elsewhere, not Orbit measurements. Phase 0 exists to replace them, and the blog rule applies: no number in the follow-up post without a committed script. What I will not promise is the thing the phrase "precise clock synchronization" might be read to mean: that two hosts' scopes can be overlaid as if a single clock had stamped them. Over an ordinary network the floor is tens of microseconds, and the viewer should say so rather than hide it.
Uncertainty is part of the data
Each remote's machine header shows its current alignment band, ±37 µs, in the same place the thread rail shows a hooked count, amber if the last minute's fit is worse than a millisecond or the link has gone quiet. When a selection spans two machines, the readout states the band next to the delta. And the saved bundle gets a clocks table, one row per accepted sample (machine, t1, t2, t3, t4), plus the fit the hub used, in the manifest's new machines entry. A capture can be re-fitted offline with a better estimator than the one that recorded it, which is the same principle as keeping raw samples rather than a report.
The UX
The design goal is that the first machine costs one text field and the fifth costs the same. No configuration files, no mode switch, no separate "distributed" build.
ssh or a browser, and the panel decides which transport it means.Adding a machine. The field takes render-01, render-01:44766, pierric@render-01 or a pasted http://…. A bare host is probed directly first (200 ms budget), then over ssh; user@ goes straight to ssh. Progress reads in the row as it happens: connecting · found orbit-service 0.9.2 · syncing clocks · 41 k ev/s. If the remote has no service, the row turns into the install offer with the path, the size and one button. Nothing else in the UI is blocked while it happens.
The rail. Machines are what they already are on screen, a header row per host with its processes and threads under it and its own Scheduler track, draggable as a block. The hub is always first. The header carries the hostname, the alignment band, and a small link glyph that turns amber when the remote falls more than a second behind the hub's live edge, which is the honest way to show that a remote's most recent events are still in flight rather than draw a gap.
Recording. One Record button, one capture, N machines. Pressing it starts a capture on every armed machine; Stop stops them all; Save writes one bundle. The settings window behind the gear gains a machine strip across the top, so each machine has its own target process, hook list and sampling knobs, with the hub's settings as the defaults for a newly added machine. A machine can be disarmed to stay visible but not captured, which is how you keep a build box in view without profiling it.
Reports. Nothing changes on the surface. Select samples on a remote's thread and the flat, top-down and bottom-up reports come back; the hub forwarded the query to the machine that owns those samples and is streaming the answer. A selection that spans machines produces one report per machine, tabbed, because a merged callstack tree across two different binaries on two different hosts would be a lie in a table.
From the command line. Everything the panel does has a flag, so a saved shell line reproduces a setup: orbit-service --connect render-01,pierric@build-mac brings the hub up already linked; orbit-service --join hub:44766 --code 7Q4K on a remote; orbit-service --stdio is what ssh runs and is not meant to be typed. A URL with ?connect=pierric@build-mac lets a link in a chat message add the machine when opened, after the same confirmation.
Any viewer, any machine. Because a remote is an ordinary service, it still serves its own viewer, and its row's menu has open its own viewer for when you want the single-machine view. And because a hub is an ordinary service, a second person's hub can add the same remote; the remote streams to both, and if the first hub is mid-capture the second sees the machine as capturing for workstation, can watch, and cannot press Record on it until that capture stops. One capture owner per machine, many watchers.
Security, in one paragraph
Today the service binds 0.0.0.0 with no authentication; the comment in main.rs says so plainly. It has been acceptable because the exposure was one dev box on a trusted LAN. Peering makes the control plane worth thinking about, because a hub that can start captures and read disassembly on N machines is N times the target. The plan: every service generates a token beside its service-id, prints it in the banner, and requires it on /api/* and /ws from anything that is not loopback. The viewer it served gets the token in the page; a hub adding a direct link asks for it once and stores it; ssh links need none because ssh already authenticated; reverse join uses the hub's code. It is not a substitute for a network you trust, and the docs will say that, but it turns "anyone on the LAN can start a capture" into "anyone with the token can", which is the least a profiler that reaches across machines should do. This lands in phase 1, before any hub connects to anything.
What does not change
- The ring, the renderer, the track index, the 32-byte
LiveEventand its spare byte, the WebSocket frame format and its version. - The single-machine path. A service with no links is machine 0 and behaves byte for byte as it does today; the e2e suite's 36 scenarios are the regression test for that.
- The bundle's three Parquet tables. The manifest gains
machines; aclockstable is added only when there is more than one machine. Old bundles load as machine 0. - Any remote's code path. A remote is a stock
orbit-service; aggregation is something done to it by a client, and it cannot tell a hub from a browser except by the token it presented.
Phases, each with its test
Phase 0 · Measure before designing the estimator
A throwaway ClockPing/ClockPong on the existing WebSocket and a script that logs ten thousand round trips between the workstation and the Mac, direct and over ssh, and between two services on one box. Histograms of round trip, and the offset recovered on the one-box case where the truth is zero.
Done when: docs/blog/metrics/phase-17-clock-rtt.txt exists and the precision table above has real numbers in it, whatever they turn out to be.
Phase 1 · Token, peer link, identity
The token on the API. PeerLink in orbit-service: a WebSocket client to a remote's /ws (axum's WebSocket support already brings tokio-tungstenite into the tree; the client side is the same crate) plus HTTP forwarding for the API. The machine table, pid packing at ingest, string-id remapping, the MachineId rewrite, the Machines panel with the direct transport, --connect, capture fan-out, per-machine report forwarding, bundle manifest.
Done when: a new e2e scenario starts two services on one box, links the second from the first, captures orbit-test-rust on both from one Record, and checks by label through window.__orbit_ui that both machine trees are present, that a report on the remote's thread returns the remote's functions, and that Save then Open round-trips both machines. No timestamps are adjusted yet; on one box they do not need to be.
Phase 2 · Clock alignment
The ping frames, the minimum-filtered window, the offset-and-rate fit, application at ingest, the band in the header and the selection readout, the clocks table, boot_id handling on reconnect. Two debug knobs on the service, --debug-clock-offset-ns and --debug-clock-rate-ppm, that shift its reported timestamps and its pong stamps consistently, so a test can inject a known lie.
Done when: on one box with an injected offset of 12.345 ms and 30 ppm, under tc netem jitter of 200 µs, the hub's fit recovers both within the band it displays, and the band is under 50 µs. Second check, the causality invariant: orbit-test-rust gains a mode where one instance sends messages to another over TCP with a manual scope around each send and each receive; after alignment no receive scope starts before its send scope, across ten thousand messages, in both directions.
Phase 3 · SSH transport and install-on-demand
--stdio mode, the hub spawning ssh, the API multiplexed over the pipe, the install offer with the consent dialog, same-arch self-copy, cross-arch fetch from the install base, --sudo passthrough, the Mac as the first real remote.
Done when: from the workstation, with no Orbit on the Mac, typing pierric@build-mac ends with the Mac's processes in the rail after exactly one confirmation, and a capture of a Mac process shows manual scopes aligned to within the displayed band. An e2e version runs against a local sshd on the same box.
Phase 4 · Reverse join, settings strip, polish
--join and the code in the banner, the Accept tray, the per-machine settings strip, the disarm state, the amber lag glyph, the capturing for lock, ?connect=, the manual pages.
Done when: a container on the box with no inbound ports joins the hub, appears, is captured, and its process is profiled with its own hook list while the hub's own target keeps a different one.
Deliberately not in this plan: hardware PTP (worth doing when a user has the NICs; the clocks table is designed so it slots in as a better sample source), cross-machine flow arrows (needs an API for pairing a send with a receive by id, a feature in its own right), and a fleet view for dozens of hosts (a different rendering, not a bigger tree).
Five decisions I need from you
- Pack or widen? I recommend packing the machine into the pid's top byte, capping a live session at 255 machines. If you can see a use for more than that in one rail, widening is the time to do it, and it is cheaper now than after phase 1.
- Is a token acceptable on the LAN path? It is one paste per direct link, and it closes a hole that peering would otherwise widen. The alternative is to keep the API open and document the exposure. I would rather not build the hub on an unauthenticated control plane.
- Record fans out to every armed machine by default. The alternative is a Record per machine. One button matches how the timeline reads, one capture across hosts, but it means a stray click starts N captures.
- Install-on-demand ships the hub's own binary when the architecture matches. That needs no infrastructure, but it means a dev build on your workstation becomes what runs on the remote. Restrict it to release builds, or accept it for now and tighten when the site exists?
- The precision statement. The viewer will show a band and refuse to imply better. If the requirement behind "precise clock synchronization" is tighter than tens of microseconds on a LAN, the plan needs PTP in scope from the start and I should know that before phase 2.
Answer those and phase 0 starts the same day; it is a script and a frame, and its output decides how much of the clock section survives contact with the network.