How concerned should we be about Astra's recurrent architecture?

gist

Rauno Arike assesses reports that OpenAI's Astra uses a looped-transformer recurrent-depth architecture akin to Geiping et al. After Jakub Pachocki said Astra's computation-graph depth is within 2× of GPT-4, the worst neuralese panic cools, but Greenblatt's dial worry remains: loop count might rise later or race others into deeper recurrence. Academic evidence is mixed on whether hundreds of loops scale; Arike worries more about fragile CoT monitorability than about Astra's current depth.

ideas

• Astra is a looped transformer, not a classic RNN. Recurrence runs along depth for a token—the same forward pass can loop before emitting—rather than carrying an unbounded hidden state across the trajectory. That bounds practical serial depth and is a step toward opaque reasoning whose size depends on loop count, not architecture scare alone.
• Pachocki caps current depth near GPT-4. OpenAI says frontier models including Astra stay within a factor of two of GPT-4's computation-graph depth and still lean on CoT monitoring. Effective depth matters more than looped-versus-unrolled form for monitorability; weight-tying may even help slightly.
• The remaining worry is a depth dial and a race. Greenblatt asks whether inference or light fine-tuning can raise loops performantly, whether opaque reasoning jumped, and whether the architecture makes massively deeper future runs natural. Arike guesses Astra uses about three to four loops today.
• Literature does not yet show a race to hundreds of loops. Huginn trained up to 32 and tested to 64, but larger models such as Loopie (20B-A2B) use two; small-scale stability work reaches ~12. Deeper is not always better, so a bottom-race toward hundreds of loops is uncertain.
• Neuralese concern tracks serial depth needed to subvert oversight. Circuit-complexity takes and no-CoT horizon scaling suggest "spit a token every few hundred layers" is a weak bound; Hugging Face-style tool-call spoofing may need less opaque depth than a full plan. OpenAI's own monitorability signals still trend fragile independent of this architecture.

quotes