The sleep framework: A new way for LLMs to retain knowledge in continual learning

gist

Google Research's sleep framework gives LLMs offline phases that move fragile in-context memory into new long-term parameters. The aim is learning without forgetting. Consolidation distills short-term memory into newly unlocked low-rank experts. A dreaming stage then rehearses on curated synthetic data from randomly routed experts. The authors report gains in continual learning, 10M-token BABILong recall, knowledge incorporation and math, with modest margins on math.

ideas

• Learning gets a wake and sleep cycle. During the active phase the model stores new information in fast short-term memory; during sleep it stops taking input and spends its compute on consolidation and self-improvement.
• Consolidation grows capacity instead of overwriting. Low-rank MoE experts are unlocked in slower memory levels and seeded from short-term memory with on-policy distillation plus RL-based imitation learning. The temporary high-frequency parameters are then reset.
• Dreaming is curated self-training. Randomly routed experts generate synthetic samples, gradient-based selection keeps the most useful ones, and the model fine-tunes on those that improve downstream performance.
• The reported gains are broad but uneven. Sleep tops class-incremental baselines (97 vs 93 for Hope on CLINC with Llama3-8B) and keeps both new languages in sequential translation. On Qwen3-8B math it scores 79.2 / 69.0 / 46.1 on AIME-24/25 and HMMT-25, against 76.6 / 67.4 / 45.1 for the best baseline.
• The substance is the arXiv paper. Language Models Need Sleep (Behrouz, Hashemi, Javanmard, Mirrokni) has been on arXiv since June 2026, and on OpenReview since September 2025. All results are the authors' own.

quotes