hraness
Theme
Appearance

hegel: stateful tests that find the three-step bug

crash between commit and fsync, then come back

drafted with ai assistance by ben guo

the rest of this lesson is free: add your email to keep reading.

A unit test calls a function once. Most interesting state bugs are not in a call; they are in a sequence: write, checkpoint, crash, restart, replay, and only then does the corruption show. Writing such sequences by hand is exactly the work humans are bad at: we write the sequences we can already imagine, which are the ones the code already handles.

sequences, not calls

Consider the shape of a real durability bug. A journal commits a pin in two writes: rename, then directory sync. Kill the process between them, reopen, and the journal must decide whether the pin committed. A unit test can call commit and recover separately; the bug lives in calling them in the middle of each other. The space of "one call at a time" is linear; the space of interleaved sequences is where systems actually fail.

Stateful testing generates those sequences. Instead of writing a case per sequence, you write a model (the operations the system supports, plus crash/restart as an operation), a generator that draws sequences of operations, and assertions that run after every step. The test stops being a script and becomes a space of scripts.

what hegel does

Hegel is the Rust-native stateful testing library the workspace uses: property tests where each case is a sequence of operations against the real system, driven by a seeded generator inside the test loop. Because the seed is data, a failing sequence is reproducible exactly, and the shrinker can reduce it to the smallest sequence that still fails.

That last property is what makes the bug actionable. A failure that arrives as "somewhere in a 400-step random sequence the journal lost a pin" is a puzzle. The same failure shrunk to "rename, kill, recover, retry: pin applied twice" is a bug report with an address.

where it lives

vhalla's ledger tests are the clearest case. recovery_hegel.rs drives the ledger through appends, checkpoints, snapshots, restores, and replay attempts, with every choice drawn inside the loop from the seeded generator. Sixty-four generated sequences run in milliseconds because the ledger is a pure core: no sockets, no wall clock, no filesystem the test cannot control. The deterministic-core pattern (its own lesson) is what makes this cheap; Hegel is what makes it cover ground.

Gobstopper's vault surgery tests are the second case. Compaction, snapshot, and recovery are sequences over an on-disk vault, and the claims that matter ("the source is preserved", "a copy can be recovered", "protected output survives") are all sequence properties. The tests inject the crash at different steps of the same surgery and check the postcondition each time.

the honest limits

Three caveats keep this honest.

First, a generated sequence only covers what the model can express. If the test's operation set lacks "the OS reorders the fsync" or "the allocator fails," those failures stay outside the space. The model is a claim about which operations matter; like the TLA+ spec, it earns that claim by being updated when a new operation class appears.

Second, Hegel is a Rust-side tool. The portfolio's TypeScript projects use other stateful machinery (seeded property tests, Direct compositions), and the technique here is the general one: generated sequences plus shrinking plus replayable seeds. Naming one library would undersell the pattern and overclaim its coverage.

Third, the sequences that matter are the ones the system can actually produce. A generator that permits operations the real caller can never perform spends its budget on fiction and can report bugs the system cannot have. The fix is the same fix as everywhere in this series: bound the model to what the boundary admits, and write the bound down.

The deeper reason the technique earns its place: the three-step bug is precisely the bug code review cannot see. Reviewers read functions; sequences are behavior. A tool that writes the sequences for you, and shrinks the failure into a trace you can hold, converts "the system sometimes loses state" into "rename-then-kill-then-retry double-applies." That conversion is most of what debugging distributed state actually is.

keep reading: free for subscribers

the rest of this lesson is free. enter your email to subscribe, and every subscriber lesson unlocks in this browser.

already subscribed? enter the same email to unlock.