What if automating AI R&D triggers an intelligence explosion?

gist

Alan Chan, Sören Mindermann, and coauthors say automating AI R&D could trigger an intelligence explosion within a few years. Their Cambridge Programme on AI Science and Policy (CASP) working paper reviews preliminary evidence that software feedback loops could overcome diminishing returns, compute and data limits, and hard-to-automate tasks. They say the high stakes warrant urgent visibility into lab automation, tools to pace an explosion, and advance preparation for loss of control and eroded checks on power.

ideas

• AI systems already write most of the code at frontier labs. Compared with a year earlier, staff at leading companies delegate core R&D tasks and some delegate all coding; Anthropic reports approved-code share above 80 percent by May 2026.
• A software-driven explosion has two steps. Better AI expands the effective R&D workforce, and that workforce then produces still-better systems that expand it again.
• Frictions may not stop the loop. Historical returns-to-research estimates above one suggest tenfold faster progress within about 1.5 years after full automation, though compute, data, and long training runs remain uncertain.
• The paper names three extreme risks. Capabilities could outrun society's ability to steer; automated R&D could weaken human oversight; a decisive lead could erode checks on power within and between states.
• Policy has to be ready before the window closes. The authors want visibility into internal automation, tools to pace and isolate automated R&D, and advance adaptation for labor, geopolitics, and misuse.

quotes