Can LLMs Grow Their Own Capabilities?

gist

Percepta Research argues that shipped language models freeze the world they represent, so new facts never enter the weights. Spotlight keeps a small fixed intelligence module and an unbounded writable memory that each token indexes sparsely, so access cost does not grow with what is stored. A hand-built Python interpreter in that memory runs real programs with no external runtime, holds throughput flat out to tens of millions of tokens, and can install or revise packages by editing cells. Capabilities, including an MNIST classifier larger than the module, live in memory rather than in retrained weights.

ideas

• Unbounded memory, constant access. Spotlight is arbitrarily sparse: every token touches a fixed handful of cells no matter how large the store grows, unlike full attention's rising cost or a fixed recurrent state that forgets.
• Skills live beside facts. The intelligence module stays under 100K matrix parameters and does not change; new procedures are written into memory, so capability is not capped by weight size.
• Python runs inside the model. A hand-constructed interpreter executes real programs autoregressively with no external Python runtime, holding the language, packages, code, and runtime state in Spotlight memory.
• Constant work across long traces. Project Euler solutions stay at 134K–144K tokens per second from a 6.5M-token trace to a 47.6M-token one, and tests cover recursion 2,000 calls deep.
• Updates rewrite cells, not weights. Installing or revising a package, including the owed_tax function, leaves the query and weights unchanged; an MNIST classifier stored in memory then exceeds the module's own parameter count.

quotes