saved
mini-AGI: Continual learning on an 8 GB laptop GPU
Hraness cites a source capture. The source author remains the source.
gist
Alexey Borsky's mini-AGI is a toy continual-learning byte-level language model that trains from scratch on one 8 GB VRAM GPU, pages experts from disk, and never freezes a base: reading a stream and taking a gradient are the same path. Adaptive-depth MoE routing, growth/pruning, and a trunk learning rate at 0.1× the experts' rate cut massed-subject forgetting to +0.0067 nats. Weights are not published yet; the README frames it as an experiment that almost anyone can keep training on their own files and hardware.
ideas
- Own the model, keep training it. Frozen bases plus thin fine-tunes stop learning at ship; mini-AGI trains end-to-end on a consumer card and treats every read as another training step.
- VRAM holds a working set, not the pool. Experts live on disk with their Adam moments; only about 32 experts (~109M of ~540M params) sit resident so the pool can grow on 8 GB.
- Trunk LR, not the pool, stops forgetting. On a 524k-character chess probe, trunk at 0.1× expert LR leaves unread subjects at +0.0067 nats (99.84% retained); freezing the working set alone still collapses.
- Architecture assembles per character. Two dense preludes plus a recurrent block applied up to 24 times with top-8 expert routing and a PonderNet-style halting head spend depth only when needed.
- Dead means unaddressed, not small gate. Growth and pruning key off how long since anything routed to an expert; gate size is anti-predictive because busy sinks look dead and idle high-gate experts look alive.
quotes
“There is no separate fine-tuning regime and no frozen base: reading and being trained are the same event.”
“The trunk learning rate is the mechanism.”
“+0.0067 nats after half a million characters of a single subject.”
“Dead means unaddressed.”