hraness
Theme
Appearance

property tests everywhere: parsers, projections, and round trips

a thousand adversarial inputs beat a hundred examples

drafted with ai assistance by ben guo

the rest of this lesson is free: add your email to keep reading.

An example test says: for this input, expect this output. It is a story about one case. A property test says: for every input of this shape, this relationship must hold: parse(serialize(x)) == x, sorted output is a permutation of input, decode never panics. It is a law, and the test is the prosecution's attempt to break it a few thousand times.

the law, not the example

The difference matters most for code that faces foreign input. A parser's bug almost never lives at an input a developer thought to write down; it lives at the input that is almost-valid: the IDN that normalizes oddly, the path with a drive-letter segment, the JSON with a duplicate key. A property test's generator produces those systematically where an example list produces them never.

Across the portfolio, the convention pairs concrete regressions with property tests at four boundaries: parsing (round trips and adversarial acceptance), resolution (paths and identities stay confined), ordering (output order is a function of input, not of map iteration), and confinement (nothing escapes its directory or scope).

the conventions

The shape is recognizable in every repository. Ghostget's URL admission is the clearest: candidate URLs run through a comparedParse that checks the TypeScript admission predicate against the Rust url crate as an oracle, because two independent implementations of the same standard diverging is exactly the failure a single implementation cannot see in itself. The property is "the two agree"; the exclusions are written down as known deviations with reasons, not silently ignored.

Oh's parity suite runs the same law differently: the memory kernel's operations have a documented contract, and the property tests generate operation sequences that must behave identically against the model. Accounts and design-kit run bounded-input properties for the same reason: the catalog's types reject what they cannot name, and the tests prove the rejection is total rather than sampled.

xcb's task history and AI Charts' benchmark projections use ordering and determinism properties: the same input history must project to the same output, byte for byte, so a replay is itself a proof of determinism.

the corpus

The second convention is what happens after a counterexample is found. A random generator that finds a bug once will probably never find it again, which makes the fix unverifiable. The portfolio's answer is a recorded corpus: Ghostget keeps seeds/corpus.json where each entry names a seed and a path so the exact counterexample replays deterministically inside the suite.

That converts a flake into an asset. CI once found a divergence between the runtime's URL admission and the Rust oracle on an input involving a backslash-normalized drive-letter segment: the fix was not to patch the case but to correct the exclusion rule in comparedParse, and the seed is now pinned so the case replays every run. The corpus is the property test's memory.

the honest limits

Property tests sample. With enough runs they approximate coverage, but they are still a search, and the seed budget is finite: a counterexample that appears once in a million cases can pass a thousand-run suite and fail production. That is why the technique is layered with the others in this series: proofs where the space permits, oracles where two implementations exist, claims ledgers where none do.

A property is also only as good as the law it states. "Parse never throws" is true of a parser that accepts garbage; "round trip preserves bytes" is false by design for a normalizing parser unless the law is stated on normalized input. Writing the law is the actual work, and the test only enforces whatever law was actually written down.

The last limit is generator quality. A generator that produces only well-formed inputs tests the happy path thoroughly and the hostile path never. The convention that keeps generators honest is the same one that keeps specs honest: draw inputs from the hostile boundary (arbitrary strings, truncated bytes, structurally wrong shapes), not from examples that already pass.

Done well, the property suite becomes the cheapest review artifact in the repository: the laws read as a specification, the corpus reads as a list of past adversarial inputs, and the CI run is evidence the laws held on this exact build. That is most of what "tested" should mean at this scale of agent-written code.

keep reading: free for subscribers

the rest of this lesson is free. enter your email to subscribe, and every subscriber lesson unlocks in this browser.

already subscribed? enter the same email to unlock.