hraness
Theme
Appearance

two implementations, one spec: parity as a test oracle

when typescript and rust disagree, one of them is lying

drafted with ai assistance by ben guo

the rest of this lesson is free: add your email to keep reading.

How do you test that a system is correct when no oracle exists? A unit test needs expected outputs, and for a whole runtime (a parser, an evaluator, a replay engine) the expected output is the system itself. The cheap answer is to write it twice: two implementations of the same contract, each serving as the other's oracle.

the oracle problem

An oracle is the thing that tells a test what "correct" means. For most code the oracle is the programmer's intent encoded in examples. For a deterministic system (a manifest interpreter, a state machine, a projection), there is a stronger option: any second faithful implementation. Run the same input through both and require bit-for-bit agreement; a divergence is, by construction, a bug in exactly one of them.

The asymmetry is the power. A bug in implementation A is a bug in A; the same bug in both is nearly impossible, because independent implementations of a shared contract share mistakes only where the contract itself is ambiguous, and the contract is the thing you then fix.

what parity is

Parity in the portfolio means the same artifacts driven through both runtimes with identical receipts. ALGAL is the fullest case: the TypeScript reference runtime and the Rust kernel each execute the bundled example organisms, and the parity harness compares run receipts byte for byte. The verify direction runs too: a receipt produced by the TypeScript CLI must replay and verify under the Rust CLI, and vice versa, so "the run was deterministic" is itself tested across the language boundary.

Oh's parity suite applies the same law to the memory kernel: operation sequences run against the model and the implementation and must agree, which is the same contract expressed in one language against a model instead of two against each other.

how the suites work

The mechanics are boring by design. A parity suite enumerates the example set (every bundled organism, every CLI surface that emits receipts, every replayable lifecycle), runs each through both implementations, and diffs canonical output. The suites are split by surface so a divergence arrives with an address: store parity, CLI parity, inference parity, application lifecycle parity.

The yield is the class of bugs no single-runtime test can find: a subtle difference in float formatting, a JSON serialization order that differs, a hash computed over slightly different bytes, a boundary condition read differently by two implementations of the same spec. Each divergence forces the contract to be more precise, which is the real deliverable: the parity suite is a spec-precision machine.

the honest limits

Parity proves agreement, not correctness. If the spec itself is wrong (the manifest semantics misdesign a capability rule), both runtimes can implement the same wrong thing with perfect parity. The suite detects divergence; it cannot detect shared mistaken intent. That is the claims-ledger lesson's territory: parity is evidence for the claim "the implementations agree on the spec," full stop.

The second limit is coverage of the example set. Parity holds on the inputs the suite drives; an input class outside the examples is unverified in both runtimes simultaneously. The bundled examples are the coverage claim, which is why the suites enumerate them explicitly rather than sampling.

The third is cost asymmetry: two implementations means every contract change lands twice, and the parity suite fails on every incomplete port until both sides land. That friction is the price of the oracle, and it is exactly why the portfolio keeps parity for cores that justify it (a replayable runtime, a memory kernel) rather than everywhere.

Within those bounds, though, parity is the strongest cheap evidence available for deterministic systems: it converts "we believe the implementation matches the spec" into "here are two implementations that cannot share a bug unless the spec let them." For software an agent largely wrote, that is about as good as evidence gets without proofs.

keep reading: free for subscribers

the rest of this lesson is free. enter your email to subscribe, and every subscriber lesson unlocks in this browser.

already subscribed? enter the same email to unlock.