Google Research has published a continual-learning framework that schedules "sleep" for large language models. The paper is by Ali Behrouz, Farnoosh Hashemi, Adel Javanmard and Vahab Mirrokni, with Behrouz listed under both Google and Cornell University. The idea is borrowed from the human brain: take in information during the day, then reorganize short-term memory into long-term memory at night.
Once a model is deployed, its parameters usually stop changing, so its knowledge is frozen at the training cutoff. Fine-tuning it directly tends to cause catastrophic forgetting, where new knowledge overwrites old capabilities. In-context learning only lasts until the session ends; once the window is cleared, nothing is retained. The sleep framework tries to sidestep both problems at once.
Take notes awake, rewrite parameters asleep
The framework switches the model between two phases. In the awake phase, the model processes input as usual, and new information first goes into high-frequency short-term memory, such as the context. In the sleep phase, the model stops accepting new input and does two things.
The first is memory consolidation. The model reserves a set of low-rank experts in advance inside a mixture-of-experts (MoE) structure. They stay locked during normal operation and are unlocked during sleep, attached to a long-term memory layer that updates less frequently, while the old parameters are left untouched. Content in short-term memory is moved "upward" into these new experts through distillation. One route is on-policy distillation, in which the student model aligns with the teacher's token distribution on sequences it samples itself. The other, called "learning to imitate," has the student continue the prefix of text generated by the teacher, scores the result by semantic similarity and edit distance, and trains with reinforcement learning. After the transfer, the high-frequency temporary parameters are reset to zero to make room for the next round; the authors compare this step to synaptic pruning.
The second step is what the authors call dreaming. When generating synthetic data, the router deliberately activates experts that are not usually paired, mixing different domains to produce a batch of "dream samples." Gradient-based data selection then picks the portion most likely to help, which is used for supervised fine-tuning, and a final check confirms whether downstream performance has improved. Throughout the process, the model's core architecture and training rules stay unchanged.
The scorecard and the gaps
In a continual translation test, the researchers had the model learn, stage by stage, languages such as Manchu and Kalamang that barely appear in training corpora. The blog says the sleep method showed almost no performance drop, while in-context learning, Cartridges and SFT all suffered severe forgetting. On the BABILong long-context benchmark, which runs up to 10 million tokens, the results were close to perfect and ahead of GPT-4 and Titans. In math reasoning, it scored 79.2% on AIME 24, 69.0% on AIME 25 and 46.1% on HMMT 25. On factual knowledge, it reached 48.9% on single-task SQuAD and 46.2% in the continual pre-training setting.
These numbers lack comparisons. The blog does not list the specific scores of each baseline on math and factual QA, and the continual translation and class-incremental learning results are given only as charts, without values. The text does not say how large the model is or how much compute a single sleep costs. There is no dedicated limitations section, and no external reproduction has appeared so far.
The third step from the same team
Placed on the research line of Behrouz and Mirrokni, the thread is fairly clear. In late 2024 they proposed Titans, which added to the Transformer a neural long-term memory module that can also update at inference time. The Hope architecture that followed split the model into multiple memory layers that update at different frequencies. This sleep framework is built directly on top of Hope and MoE. Across the three steps, the question has shifted from "how to stretch the context longer" to "how to settle what is in the context into the parameters."
There is related work with a similar direction, such as MIT's SEAL, in which a model generates its own fine-tuning data and updates its weights; the blog lists it among the comparison baselines. What the sleep framework adds is the mechanism of reserved experts plus periodic resets, which uses structure to keep new and old knowledge apart.
It is still some way from a product. If a production model "slept" and changed its parameters every night, safety evaluation and version management would have to be redesigned, and a user could face two slightly different models today and tomorrow. Google has not disclosed any deployment plans; the paper is already on arXiv.
Sources: Google Research blog, CocoLoop, arXiv paper; the scoring basis for AIME, HMMT and SQuAD and the context lengths of the BABILong test were cross-checked.