Create a Video View Paper

One Layer Is Enough: Rethinking RL Training in Transformers

This presentation reveals a surprising discovery about reinforcement learning in large language models: nearly all adaptation gains come from a small subset of transformer layers. Through systematic experiments across multiple architectures and tasks, the research shows that training just one carefully chosen layer can match or exceed full-parameter RL training, fundamentally challenging our assumptions about how these models learn from rewards and opening new paths for efficient AI training.

Script

When you train a language model with reinforcement learning, only a handful of layers in the middle of the network actually absorb the improvement. Everything else? Nearly inert.

The researchers measured layer contribution, the fraction of RL gain you can recover by training only one layer at a time. The best single layer routinely captures 80 to 100 percent of the improvement from training all parameters, and sometimes exceeds it.

This concentration is not an accident of one model or task. It holds across architectures from Qwen2.5 to Qwen3, across RL algorithms including GRPO and Dr. GRPO, and across domains from math reasoning to code generation.

Even more striking, the layer rankings correlate strongly across completely different tasks. The layers that matter for math also matter for code, suggesting this is an intrinsic property of how transformers are structured, not what they are being trained to do.

When the authors applied layer-adaptive learning rates, boosting the rate only for high-contribution layers, performance consistently exceeded the full-parameter baseline. Selective training of just the best few layers outperformed both uniform updates and random layer selection.

This work overturns the assumption that RL adaptation requires touching every parameter. Most of the learning happens in a compact subspace, and targeting it delivers better results at a fraction of the compute cost. Explore more breakthroughs like this and create your own video summaries at EmergentMind.com.