Recirculation: Teaching Transformers to Remember What They Learned Too Late
This presentation explains recirculation, a novel inference-time intervention that addresses a fundamental limitation in feedforward transformers: deep contextual information becomes unavailable to shallow layers when processing later tokens. By adding a lightweight recurrent feedback path that transfers normalized deep-layer activations back to earlier layers across processing steps, recirculation enables persistent state tracking without retraining. The method achieves dramatic perplexity reductions up to 35% on language modeling tasks and improves reasoning performance, offering a computationally efficient alternative to architectural redesign.Script
A transformer processes the word bank in a sentence about geography, but by the time it figures out this means riverbank and not a financial institution, that clarification lives six layers deep. When the next word arrives, the shallow layers still think we're talking about money. The architecture has learned something crucial too late to use it.
Recirculation solves this by creating a feedback channel from layer 11 back down to layer 4, carrying a normalized snapshot of what the deep layers just learned. Unlike looping, which just runs the model twice to get more depth, recirculation lets the same layer hold an evolving state that gets updated with each new token.
The authors swept 25 source layers against 15 destinations on Gemma 1B and found a smooth favorable region: destination layer 4 works best when fed from layers 9 through 13, roughly 5 to 7 layers higher. This pattern held across arXiv, PG-19, and C4, cutting perplexity by an average of 4.72 percent with just a tenth-strength mix of the deep signal.
Scaling up, the 4B model hit 15.95 percent maximum reduction, while the 12B model reached 35.4 percent on PG-19. That last number is striking but contextualized: the 12B baseline was comparatively weak as a language model, so the large relative gain reflects both the method's power and the baseline's starting point.
Token-level analysis revealed that recirculating early tokens, especially the first ten, can actually hurt because the model hasn't yet built useful state. But from position 20 onward, the effect is persistently helpful out to a lag of 256 tokens, strongest for verbs, adjectives, and adverbs, weakest for determiners and pronouns. The pattern suggests content-dependent state accumulation, not generic smoothing.
Adaptive recirculation takes this further by training a tiny MLP on just 250 documents to predict when and along which dimensions to transfer the deep state. The foundation model stays frozen, but the learned controller cuts mean perplexity by 23 percent, slightly beating full fine-tuning while updating a fraction of the parameters. You can explore the full mechanism, layer landscapes, and benchmark results at EmergentMind.com, where you can also turn any paper into a video like this one.
