Self-Play Pretraining with Zero Data

gist

Cowsik, Dolev, Li, and coauthors show self-play over a universal Turing machine can pretrain with zero natural data and predictable zero-shot scaling. A generator proposes Brainfuck-like programs whose byte outputs train a learner by next-token prediction, while reinforcement learning rewards programs at the learner's frontier. Across text, images, audio, music, DNA, and code, compute-optimal frontiers follow power laws with exponents comparable to ordinary pretraining; the learner also develops in-context learning. The authors frame this as generating universal predictive structure, not as a substitute for contingent world facts.

ideas

• Two models co-evolve from random weights. A generator proposes Brainfuck-like programs for a universal Turing machine; a learner trains on the emitted byte sequences by next-token prediction.
• Learning-progress reward beats raw difficulty. The generator is reinforced when learner gradients align with recent parameter movement, avoiding programs that are hard only because they inject noise.
• Zero-shot loss scales as a power law in compute. Across text, images, audio, music, DNA, and code, compute-optimal frontiers improve without any natural-data gradient steps.
• Adaptive search beats a fixed universal prior. Sampling the same program space without adapting the curriculum scales much more slowly; self-play also discovers Fibonacci, geometric, and polynomial sequences far earlier.
• Universal structure is not contingent knowledge. The authors treat the method as generating transferable predictive regularities and as a warm start that speeds later natural-data pretraining, not as a replacement for world-specific facts.

quotes