saved

MirrorCode: What's the largest software project AI can complete on its own?

by Tom Adamczewski, David Owen and David ReinEpoch AI

read the original source

gist

MirrorCode tests whether an AI agent can reproduce an entire program from observable behavior without seeing its source code or using the internet. Its 25 targets span utilities, interpreters, data tools, bioinformatics, cryptography, and compression, with hidden end-to-end tests and budgets measured in days and billions of tokens. The results show meaningful whole-program capability, including a near-complete 16,000-line bioinformatics toolkit, while strict completion, cost, and possible training-data contamination remain important limits.

ideas

  • Whole-program behavior is the unit. MirrorCode asks agents to recreate complete programs rather than repair a bug or implement one isolated feature.
  • Task-scale budgets change the evidence. Attempts can run for days and consume billions of tokens, matching the scope of work that would take a human engineer weeks or months.
  • Isolation makes plausibility insufficient. Agents have no internet or original source access, and held-out end-to-end tests require exact observable behavior.
  • Near-complete is not complete. The gotree reimplementation passed 2,000 of 2,001 tests, demonstrating substantial capability while still failing the benchmark's strict solve condition.
  • Contamination remains unresolved. Memorization screens reduce one explanation for success, but open-source targets may still have appeared in model training data.

quotes

“In a MirrorCode task, AI models are tasked with reimplementing an entire program end-to-end, without access to the original source code.”

Tom Adamczewski, David Owen, and David Rein, defining the benchmark's unit of work.

“one of the largest MirrorCode tasks cost $2,600 for a single run and involved AI working for 19 days without human intervention.”

Tom Adamczewski, David Owen, and David Rein, describing the inference scale used for a serious attempt.

“The best-scoring AI gotree implementations passed 2000/2001 tests, but failed a single edge-case test for a niche command to manipulate date annotations.”

Tom Adamczewski, David Owen, and David Rein, distinguishing near-completion from a strict solve.

“we cannot rule out the possibility that memorization contributes to AI performance.”

Tom Adamczewski, David Owen, and David Rein, preserving the benchmark's principal uncertainty.