44% on ARC-AGI-1 in 67 cents

gist

Mithil Vakde trains a small transformer from scratch at test time on ARC-AGI and scores 44% on ARC-1, plus 7% on ARC-2, for 67 cents of 5090 compute, matching TRM and HRM without recursion. The lift comes from 3D RoPE and per-task embeddings, a modern SwiGLU and RMSNorm stack, NorMuon, fewer augmentations, and supervised output-only loss. Ablations fall to about 24% without those representations. He argues ARC should measure sample efficiency, ban synthetic pretraining, and stop mixing LLM and from-scratch models on one cost-per-task chart.

ideas

• Sample efficiency is the research target. ARC is useful because it has few puzzles, is unsaturated without synthetic data, and is cheap enough for GPU-poor iteration.
• Representations, not recursion, carry the score. 3D RoPE plus per-task embeddings do the work; drop either and the model saturates near 24%. Recursion's confirmed benefit is extra compute without extra memory movement.
• Val-loss chasing can fail. Switching to supervised output-only loss made test loss worse yet raised the score from 40% to 44% with less variance.
• Do not mix model classes on one cost chart. Lifetime compute (init plus inference on all tasks) is 67 cents; amortizing across tasks and omitting LLM pretraining makes comparisons unfair.
• 65% looks reachable inside transformers. Open-source mdlARC already unions to 55% across runs. He asks others to improve score or cost, not add training data.

quotes