Create a Video View Paper

T²MLR: Temporal Middle-Layer Recurrence for Reasoning

This presentation explores T²MLR, a novel Transformer architecture that introduces temporal recurrence specifically at the middle layers where abstract reasoning occurs. By recycling deep-layer representations from the previous token into shallow layers of the current token, T²MLR enables persistent latent computation without significant inference cost, consistently outperforming standard Transformers on multi-hop reasoning tasks while requiring minimal architectural modification.

Script

Every time a Transformer generates a token, it throws away its rich internal reasoning and squeezes everything through a narrow vocabulary bottleneck. T²MLR introduces a temporal shortcut that recycles middle-layer representations from one token to the next, keeping reasoning alive across steps.

The architecture is elegant. A recurrent cache captures the output of layer 22 at the previous token and fuses it into layer 14 at the current token. A gated module controls the blend, and crucially, the recurrent pathway starts inactive at initialization, ensuring stable training from a standard Transformer starting point.

Here's the striking result. Localizing recurrence to just 20% of the middle layers outperforms looping over all 30 layers. This confirms that abstract reasoning lives in the middle strata of the network, not uniformly distributed across all layers.

Training at scale requires parallelism. T²MLR uses a Jacobi-style approximation that computes the recurrent cache across all tokens simultaneously, performing multiple fixed-point refinement passes through the middle layers. This balances approximation fidelity with the efficiency needed for long sequences.

You can even retrofit T²MLR onto existing models. Inserting the recurrent fusion module into a pretrained SmolLM model and briefly fine-tuning boosts GSM8K accuracy from 35.8 to 39.9, with only 8% inference overhead per generated token.

T²MLR reveals that the position and specificity of recurrence matter more than its mere presence. By targeting the middle layers where reasoning actually happens, it keeps latent computation alive across autoregressive steps without expensive full-stack looping. Visit EmergentMind.com to explore more research like this and create your own video summaries.