LLMs and self-referentiality

gist

Scott Aaronson argues that Gödel, Escher, Bach and Penrose both treated self-reference as central to whether AI could work. 2026's conversational models outperform most humans at well-defined intellectual tasks, and nobody built strange loops into the transformer, the GPUs, or training; self-talk fell out of pretraining the way self-description falls out of universality. The ideas that held are prediction, compression, and Kolmogorov complexity. Consciousness may still involve self-reference; needing explicit self-referentiality for intelligence should be buried with phlogiston.

ideas

• Self-reference was not an engineering ingredient. Conversational AIs now outperform most humans at well-defined intellectual tasks without strange loops in the transformer, the GPU cluster, or the training process.
• Self-talk is a byproduct of universality. Like ZFC proving theorems about itself or a computer running its own code, LLM self-description fell out of training on a discourse universe, not from an explicit self-model.
• Prediction and compression were the ideas that held. Intelligence as prediction, compression, and better upper bounds on Kolmogorov complexity; self-reference remains the tool for negative results such as incompleteness and uncomputable Kolmogorov complexity.
• Autoregression is not a strange loop. Aaronson refuses to count next-token feedback as self-reference; that is dynamical update, the same class as any iterative process.
• Consciousness is the leftover mystery. Hofstadter or even Penrose could still be right about subjective experience; needing explicit self-referentiality before conversational intelligence should be buried with phlogiston.

quotes