A benchmark case that cannot be solved is not a hard test; it is no test at all. It measures luck, noise, and the evaluator’s imagination rather than the submitted program. The failure is usually invisible: the case is generated, nobody can solve it, everyone scores zero, and the zeros get averaged into the results as if they meant something.
Platonik’s generated-challenge system addresses this with a mechanism that is older than benchmark design and still widely skipped: the feasibility witness. Before a generated case may be published, a known public program must actually run it to completion under the same rules, budgets, and event schedule the entrant will face. The mechanism lives in crates/platonik-core/src/challenge.rs, and it generalizes cleanly to any evaluation harness that generates its own test data.
connectivity is not feasibility
The engine design doc, docs/engine.md, puts the requirement in its sharpest form: a challenge “must retain an alternate route and have a successful checked witness run from its starting state under the same event schedule, body bounds, duration, and resource limits. Connectivity alone is insufficient: travel time and traffic can make an apparently connected world impossible.”
Read that twice, because it names the trap. Static analysis of a generated world can verify that a path exists (the grid is connected, the beacon is reachable) and still be wrong about whether the mission is possible. A world can be connected and infeasible: the route exists but is too long for the tick budget, the corridor exists but the drain schedule starves the beacon before a courier can cross it, the plan exists but costs more fuel than the cap allows. Feasibility is a property of the whole contract (map, schedule, time limit, fuel, quotas), and the only honest way to establish it is to run a program that wins.
This is the difference between the space of valid inputs and the space of solvable inputs. Validation (validate_experiment) answers the first: the JSON is well-formed, the rules are legal, the budgets are in range. A witness answers the second: at least one legal trajectory reaches the goal. Conflating them is how benchmarks ship unsolvable items labeled “hard.”
the witness mechanism
The challenge generator is a pure derivation: stream(index, stream_id) mixes SplitMix64 bits into candidate worlds deterministically, so platonik challenge challenge-0007 produces the same bundle on any machine. Generation alone would give you plausible worlds. Admission is provided by witnessed_case:
let witnessed = fixtures::replace_program(&experiment, editable, witness.clone());
if crate::sim::run(&witnessed).is_ok_and(|result| result.outcome.passed) {
// admit this case
}The function grafts the family’s public witness program into the editable cell, executes the real simulator (not a checker, not a sketch) and admits the case only if result.outcome.passed is true. The run happens under the identical event schedule, fuel, activation limits, and tick horizon an entrant will face. Candidates that can’t be solved by the witness are never published; the generator retries within bounded limits (MAX_CASE_ATTEMPTS = 512, MAX_PLACEMENT_ATTEMPTS = 64) and moves to the next derivation.
Each family ships its own witness because feasibility is skill-relative. The crossing family (indices 1–32) uses resilient_courier, the public right-wall follower from fixtures.rs that also serves as platonik challenge reference <id> resilient. The switchboard family (indices 33–64) uses switchboard_keeper, a stationary routing program matched to that family’s depot-valve mechanics. The choice is load-bearing: the witness must be a plausible participant, not a magic solver, so its success demonstrates that the case is reachable by the kind of program the challenge invites.
what a witness proves
A passed witness establishes exactly one thing: at least one legal program passes this case under the published limits. It is a lower bound on feasibility, an existence proof with the program attached as the constructive evidence.
Everything it does not prove matters equally. It does not prove the case is easy, fair, or typical; the witness may succeed where every submitted program fails. It does not prove your program can find the solution; the gap between “solvable” and “solved by you” is the entire point of the benchmark. It does not even prove the case is solvable by another route; the docs are explicit that “an individual colony can still strand itself.” And it does not prove the witness is optimal; the leaderboard’s whole premise is that better programs exist.
The negative control makes the boundary vivid. docs/challenges.md records that the crossing witness grafted into the keeper cell (reference:resilient) fails every switchboard eval case. That failure is the families measuring different skills, kept on the public board rather than deleted. A witness is a floor for feasibility, not a ceiling on performance, and not a transferable one across families.
There is also a deliberate calibration choice hiding in the witness’s competence. The witness need not be the strongest program; it needs to be representative. A superpowered oracle solver would admit cases so hard that every realistic submission fails, and the benchmark would silently measure nothing. resilient_courier is a hand-readable right-wall follower, the kind of program a participant could plausibly write; its success says “a modest, legal strategy suffices,” which is exactly the claim a fair benchmark needs. The Challenge.witness field documents this in the type itself: “the public policy used for admission; it is not part of a case.”
witnesses everywhere in the pipeline
The same pattern repeats at every layer of the project, which is the sign that it is a principle rather than a patch. docs/design-validation.md makes it an admission rule for capability: “every new capability needs a finite, independently checkable scenario contract and a known feasible example. Publish the legal observations, world events, initial resources, limits, success conditions, and failure conditions. Test the example, a simpler baseline, and deliberately damaged variants. Do not infer feasibility from map connectivity alone.”
The diagnostics run that way. Before any agent is measured, each study qualifies its fixtures with a reference run: all eight ark worlds passed on the first captured round before a single candidate was submitted; all eight construction habitats completed construction and service; all eight port worlds fulfilled both commitments. Qualification is the witness pattern at study scale: it proves the instrument can register a pass before it is used to score anything.
The gate is applied uniformly to both halves of a challenge bundle. In challenge.rs, the four public training cases and the four reserved eval cases are each produced by witnessed_case. A case that the witness cannot pass does not become a practice example either, so “trainable” and “testable” are the same property here. And the hosted season evaluator keeps the gate at competition scale: withheld cases derive from a committed salt rather than the public (generator, index) stream, but season.rs is explicit that “every window member must still derive a witnessed public bundle under this generator”: generation is the admission check. An entrant’s unseen exam still contains only cases the public witness passed.
the general rule
The generalization is a one-liner: admission is a property of the artifact, not a hope about the input. Any generator of test cases (a fuzzer, a benchmark suite, a level generator, an interview-question bank) needs a constructive feasibility check that executes the real acceptance test with a known-good solver under the real constraints. Without it, your benchmark contains an unknown mixture of “hard” and “impossible,” and no amount of downstream statistics can separate them.
Three implementation details are worth stealing. The witness runs the production simulator (crate::sim::run, the same code that scores entrants), so the admission test cannot drift from the scored test. The witness is public (platonik challenge reference <id> resilient prints it), so feasibility is itself auditable rather than asserted. And the search is bounded (512 case attempts, 64 placement attempts), so generation cannot loop forever hunting a feasible draw; the budget forces the generator’s difficulty distribution to stay where the witness can reach.
The final framing belongs to the repository: a case is admitted when a witness passes it, and that is all admission ever means. The question “can this be done?” is answered by something doing it (under the same clock, the same budget, and the same rules), and the artifact of that run is what makes the challenge a test instead of a trap.
sources
- platonik:
crates/platonik-core/src/challenge.rsforwitnessed_case,stream, and the attempt bounds;crates/platonik-core/src/fixtures.rsforresilient_courierandswitchboard_keeper;crates/platonik-core/src/sim.rsforvalidate_experiment - docs/engine.md: “connectivity alone is insufficient” and the witness admission rule
- docs/challenges.md: the two families, their witnesses, and the
reference:resilientnegative control - docs/design-validation.md: the capability-admission contract
- docs/ark-evaluation.md and docs/ports-evaluation.md: reference qualification before measurement