Pretraining at scale without a pretraining team

gist

Mirendil says one researcher working with AI agents pretrained Mirendil-Base, a 100B–500B coding model, in 3.5 months for under $10M in all-in compute. On held-out Mirendil code it scores 0.2803 bits per byte, behind only Kimi-K3 (0.2387) among five MoE peers, and its larger siblings, still in training, already reach 0.2177 and 0.2181.

ideas

• Agents wrote the code; the researcher chose what to trust. AI agents wrote over 99% of the code, ran experiments, and diagnosed failures, starting from an empty repository on top of PyTorch and JAX, with no distillation, synthetic data, or purchased data.
• The budget is all-in. The under-$10M figure covers research, small-scale experiments, the final run, agent token spend, and data processing.
• The recipe scales. Mirendil-Base-Large (500B–2T) is about 80% trained and Mirendil-Base-XL (2T–10T) about 19%, and both already beat Kimi-K3 on the held-out code set.
• On fresh external repositories it is second to Kimi-K3. Across seven repositories published after training, Mirendil-Base averages 0.222 bits per byte against 0.193 for Kimi-K3 and 0.242 to 0.247 for GLM-5.3, DeepSeek-V4-Pro-Base, and Nemotron-550B-Base, and it ties Kimi-K3 at 0.127 on json-render.
• The bottleneck moves from people to compute, data, and judgment. Mirendil argues domain teams could train their own base models, organizations could scale with compute rather than hiring, and its next model will be built partly by Mirendil models, while stressing this is not yet autonomous recursive self-improvement.

quotes