“Emergent behavior” is the easiest claim in computing and the least informative: the system did something its designer didn’t type in. Every cellular automaton, every market simulation, every neural network produces it. The question that actually separates a benchmark from a demo is narrower: whether you can show that the observed behavior is caused by the thing you claim causes it, and say what the result does not show. Platonik’s evaluation documents are worth reading as a worked answer, because the project treats every measurement as an experiment with controls, and publishes the accounting.
the benchmark is an experiment
The method statement is in docs/complexity-and-scale.md, and it reads like an experimental protocol because it is one: before confirming anything, “publish the comparator, smallest improvement worth pursuing, case count, uncertainty analysis, and fixed evaluation budget. Preserve failures and unsuccessful searches.” A result is not a number; it is a number produced by a declared procedure that someone else can re-run.
The engine’s design makes this cheap. Because every run is a deterministic receipt (the same input always produces the same trace), a control is not a statistical estimate but a counterfactual rerun: take the passing world, remove one component, and see what breaks. Because the cost model charges for everything including failed attempts, “the intervention was worse” is a measured statement, not a vibe. And because selection happens on a declared training split before transfer cases run, the protocol makes it visible whether the winners were tuned to the test.
the four comparisons
The complexity doc prescribes four kinds of comparison before a claim travels:
- Ordinary engineering: hand-written simple policies, random search, a direct implementation of the task, and a flat controller restricted to the same information and delay model. This is the “would a dumb thing work” control; a rich mechanism that can’t beat a simple baseline hasn’t earned its complexity.
- Composition: the candidate assembly, uniform constituent colonies, and copies with suspected capabilities removed, under matched total resources with interfaces charged. This is the “does the assembly exceed its parts” control.
- Transfer: freeze the candidate, then test new map families and damage schedules, separating further adaptation from unchanged transfer. This is the “did it learn a habit or memorize a map” control.
- Practical export: an independent implementation and an external benchmark, with input conversion and real runtime counted. This is the “does it survive outside our cost table” control; the doc is blunt that “an improvement visible only through a favorable simulator cost table is a game result.”
The component-removal control deserves the emphasis because it is the one that catches honest self-deception. You do not ask whether the system works; you ask which part is load-bearing, by removing each in turn under identical resources and watching which removal collapses the outcome. A result that survives every removal is a result about the world, not about your mechanism.
a taxonomy of claims
docs/research.md gives the archive three escalating statuses: Observed (it happened in an identified run, trace preserved), Replicated (a frozen protocol reproduced it on new declared cases with controls and uncertainty reported), and Reviewed contribution (independent review found the evidence sound relative to existing work). It insists that “untested,” “passed these cases,” “refuted within this scope,” and “proved under these assumptions” remain distinct. The same page lists refused claims: no P=NP progress meter, finite benchmark success is not a polynomial-time result, a checked trace is not cognition, and oracle or advice inputs must be declared.
The discipline is visible in how the docs write their own results. The navigation courier that passes all 44 worlds is “a bounded habit repaired,” not a navigator. The ark selector’s arithmetic is checked on all 256 input pairs (a complete claim about the declared input set), while the docs still note that success “does not establish that every submitted selector uses the designated bit internally.” The claim is always stated at the scope the controls actually cover.