Post 11 chose a fixed 32-byte record for every event and said, honestly, that it was 1.69× larger than protobuf. That was the right trade on the capture hot path, where decode CPU is the bottleneck and a local socket is not. It stopped being the whole story the day the viewer moved onto the network and a 100-second capture of a 32-core machine saved as a 127 MB file. This post is the measured answer to both, and the reasoning is short enough to write down in full.
What is in a 32-byte record
A live event is a start time and a duration as 64-bit nanoseconds, a pid and a tid as 32-bit, a kind, a depth and an extra byte, a padding byte, and a 32-bit name id. On a real capture almost all of those bytes are zero or repeated. Consecutive events in a batch start a few microseconds apart, so the top five bytes of every timestamp are the same as the previous event's. A batch of 2048 events comes from a handful of threads, so the eight bytes of pid and tid repeat thousands of times. A name id is under 65,536. Kind, depth and extra are genuinely one byte each.
The instinctive fix is protobuf's: varints, seven bits per byte with a continuation bit, so a small number takes one byte. It does not work on this data, and the reason is worth knowing before spending effort on it. A nanosecond timestamp since boot is around 4×1014. As a varint that is eight bytes, one fewer than the fixed field, and the duration is the same story. Measured on the real events, a protobuf-style layout — a tag byte and a varint per field — is 27.8 bytes per event. Varints save bytes on the small integers and nothing on the two big ones, and the two big ones are half the record.
Packed: send what changed
The version that works is one transform before the varints. Each event's start is written as the difference from the previous event's start, zigzag-encoded so an out-of-order event costs a sign bit rather than nine bytes. Inside a batch the difference is a few thousand nanoseconds: two or three bytes. The duration and the name id are varints as they are. The pid and tid pair goes into a small table at the head of the batch, and each event carries a one-byte index into it. Kind, depth and extra stay three plain bytes.
| 1,034,313 events, batches of 2048 | B/event | encode | decode | |
|---|---|---|---|---|
| raw (the 32-byte record) | 32.00 | 22 µs | 7 µs | per batch; decode is a memcpy |
| packed (delta, varints, thread table) | 11.06 | 38 µs | 19 µs | no dependency |
| packed + deflate (miniz, level 1) | 7.01 | 115 µs | 70 µs | pure Rust; runs in the browser's WASM |
| protobuf-style varints, no delta | 27.81 | estimate, same events |
Packed is 2.9× smaller than raw and 2.5× smaller than the protobuf layout, for sixteen microseconds more per batch. Deflate on top of the packed bytes finds what the transform left — the same name id and kind recurring, durations that cluster — and takes another 37 percent, at roughly a hundred microseconds each way. The decoder accepts all three; the producer chooses with orbit-service --wire raw|packed|deflate, /api/status reports which, and the viewer's stats line shows it after ws.
The default is packed, and the reason it is not deflate is latency. The service pushes a batch per capture pass, about every five milliseconds, so packing adds one percent of a pass and deflate four. On the link the arithmetic goes the other way: a 2048-event batch is 65.5 KB raw and 22.8 KB packed, which at 100 Mbit/s is 5.2 ms against 1.8. Packed pays fifty microseconds to save three and a half milliseconds on a slow link. Deflate pays 135 more to save 0.65. On a ten-gigabit link deflate is CPU for nothing, and on loopback so is packing, which is why the honest next step is to send raw to a loopback peer and packed to everyone else. That is a few lines in the socket handler and it is not done yet.
Would ssh -C have done this?
A fair question, because ssh -C is one flag and no code. It is zlib at level 6 over the whole connection, and a WebSocket's built-in permessage-deflate is the same thing per message. Both were measured on the raw record stream, since that is what they would see.
| zlib level 6 on the raw records | B/event | CPU per batch | |
|---|---|---|---|
| per message | 8.38 | 1,324 µs | permessage-deflate |
| whole-stream context | 8.32 | 1,958 µs | ssh -C |
| packed + deflate level 1, for comparison | 7.01 | 115 µs |
In bytes ssh lands between packed and packed-plus-deflate, and that is the "almost optimal by default" impression. In CPU it costs twelve to seventeen times more, on both ends, because it is grinding 65 KB of mostly-zero bytes at level 6 instead of 23 KB of already-packed bytes at level 1. A generic compressor sees consecutive timestamps as eight-byte words that mostly differ; it does not know they are small deltas, and that one-line transform is the half of the gain it cannot recover. Two practical points finish it: ssh's default is compression off, so the flag and a port forward are both needed, and the viewer is a browser tab that talks WebSocket to the service directly, so on the LAN there is no ssh in the path at all.
On disk: the wrong file, then the right encodings
The saved capture is a zip with the events, samples and frames tables and a manifest of thread and process names. Its first version stored the tables as Arrow IPC with no compression, which is how a 100-second capture came to 127 MB. The first fix that morning was the obvious one: deflate the zip entries with miniz_oxide, already in the tree through parquet, pure Rust, so the static service binary gains no C. The million-event table went from 38.1 to 7.6 MB in 0.87 seconds.
Then the second question: would Parquet compress well? It compresses better than that with no compressor at all, and the way it does so is the wire trick again, per column.
| the same table, 1,034,313 events | size | write | read |
|---|---|---|---|
| Arrow IPC (in the first zip) | 38.0 MB | ||
| Arrow IPC in a deflate zip | 7.6 MB | 0.87 s | |
| Parquet, plain, no dictionary, no codec | 46.2 MB | 0.03 s | 0.15 s |
| Parquet, dictionary, no codec | 18.0 MB | 0.05 s | 0.01 s |
| Parquet, delta on the ns columns + dictionary, no codec | 7.9 MB | 0.04 s | 0.01 s |
| … + snappy | 7.3 MB | 0.04 s | 0.01 s |
| … + gzip | 7.0 MB | 0.16 s | 0.01 s |
The two rows that matter are the plain one and the delta one. Plain Parquet is larger than the IPC file, and it was what the ?format=parquet export had been writing since the day it was added: a thing worth saying because "we export Parquet" sounds like a size claim and was not one. With DELTA_BINARY_PACKED on the two nanosecond columns and RLE_DICTIONARY on the other seven, the file is the same size as the deflated IPC, written twenty times faster and read in ten milliseconds because nothing has to be inflated first. Delta on a column of timestamps is exactly the delta the wire uses; a dictionary of pids is exactly the thread table. A codec on top buys five to ten percent, and the only codec in the Rust parquet crate that is not pure Rust is zstd, so there is none.
So the bundle is now three Parquet tables in a stored zip, and the plain export uses the same properties. In Rust, on the same slice, the stored Parquet bundle is 7.91 MB in 0.14 seconds; deflating the zip on top of it gets 6.97 MB for double the time and is not the default. Bundles from the first pass, Arrow tables in a deflated zip, still open, because the reader takes both. The verification was end to end: a fresh capture exported, pyarrow reading back the column encodings from the file's own metadata, the same file posted to /api/capture/import and coming back with the same 15,878 events.
One idea, three places
Every number in this post comes from the same observation about the data: a capture is a long run of nearly-identical records whose only fast-changing field is a timestamp that moves by a few thousand at a time. The wire sends the delta and a table index. The file stores the delta and a dictionary. A generic compressor, ssh or deflate, gets most of the way by pattern-matching but pays an order of magnitude in CPU for not knowing what it is looking at, and on a local link the right amount of compression is none. The fixed record from post 11 is still what the ring holds and what the capture path writes, because there the cost that matters is the decode, and a memcpy is still the fastest decoder there is.
What is left
Raw on loopback, packed elsewhere, chosen per connection, so that local profiling pays nothing. A stateful deflate across batches would recover the 0.06 bytes per event between ssh's two rows above, which is not worth a stateful decoder in the browser. And the dataset directory that --out-arrow writes is still Arrow IPC while the bundle is Parquet; the Python example reads either, but one format for both would be tidier.