Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Abstract: Reinforcement learning (RL) has become a central component of post-training LLMs, yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.

• The paper finds that single-layer RL updates, particularly in middle layers, can match or exceed full-parameter adaptation gains.
• It introduces and quantifies the layer contribution metric, revealing non-uniform adaptation across transformer layers.
• Targeted strategies like layer-adaptive learning rates and selective training provide more efficient and scalable RL post-training.

Training a Single Transformer Layer for RL Post-Training: An Expert Analysis

Introduction and Motivation

Understanding the locus and dynamics of gain during reinforcement learning (RL) post-training of LLMs is critical for efficient adaptation. The paper "Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training" (2607.01232) addresses a foundational question: does RL adaptation require modifying the entire parameter space, or is the salience of improvement concentrated within a limited subspace? The authors conduct an exhaustive empirical analysis across multiple transformer-based LLMs, RL algorithms, and tasks, introducing the metric layer contribution, defined as the fraction of RL-induced improvement on benchmarks recoverable by training only one layer at a time.

The dominant finding is that RL adaptation is highly non-uniform: most of the improvement is attributable to a small subset of transformer layers, with single-layer updates not only matching but occasionally exceeding the gain of full-parameter RL post-training. This structural regularity persists across architectures, RL schemes, datasets, and tasks, robustly localizing the core adaptation to middle layers.

Layer Contribution: Quantification and Structural Regularity

The layer contribution $\mathcal{C}(k)$ of a layer $k$ is computed as the fraction of post-training improvement attained when only $\theta_k$ is updated and all other parameters are frozen. Systematically, the authors perform per-layer RL training sweeps, benchmarking models after adaptation of each individual layer.

Figure 1: Layer contribution profiles across seven models, showing consistent concentration of RL-induced gains in the middle layers.

The core empirical result is that $\mathcal{C}(k)$ varies dramatically across depth, with the highest contributions systematically emerging from central layers (typically 40–60% of stack depth), whereas initial and final layers exhibit much weaker or even deleterious adaptation when trained in isolation. The best single-layer training frequently achieves $\mathcal{C}(k) \geq 1.0$, i.e., matching or surpassing full-parameter RL training.

This middle-layer concentration persists regardless of model family (Qwen3, Qwen2.5), RL optimization objective (GRPO, GiGPO, Dr. GRPO), or task domain (mathematical reasoning, code generation, agentic tasks), indicating an emergent property of transformer geometry and pretraining recipe rather than a contingent artifact.

Figure 2: Layer contribution for Qwen2.5-Math-1.5B trained with Dr. GRPO; middle-layer dominance is robust to changes in family and RL algorithm.

Cross-Dataset and Cross-Task Consistency

Layer contribution rankings display high Spearman correlation not only across datasets with the same underlying task (e.g., different math benchmarks) but also across fundamentally distinct domains (math vs. code).

Figure 3: Cross-dataset consistency; per-layer contribution rankings are strongly correlated for different datasets and tasks.

This indicates that the "salient" layers for RL adaptation are an intrinsic consequence of the model's pretraining history and architecture, not the specifics of an RL task or reward. Such cross-task coherence supports transferability of layer selection strategies, which can be exploited for efficient adaptation to novel domains.

Layer-Guided RL Training Strategies

In light of the observed inhomogeneity, the paper systematically evaluates layer-aware RL training protocols:

1. Layer-adaptive learning rates: Increasing the learning rate for high-contribution layers consistently enhances downstream performance relative to the full-parameter baseline.
2. Selective layer training: Freezing all but a small cohort of top-contributing layers ("best $k$") consistently achieves higher task accuracy than both uniform training and random (or "worst $k$") selection.
3. Profiling-free heuristic: Simply updating central layers by architectural position (without costly profiling) yields most of the benefits of contribution-guided selection.

   Figure 4: Layer contribution-guided strategies outperform full-parameter RL, both for adaptive learning rate and selective layer training.

These interventions demonstrate that targeted parameter adaptation is not only sufficient but can exceed the performance of uniform RL updates—an insight with immediate implications for training efficiency, compute allocation, and memory overhead.

Diversity and Complementarity among Layer-Adapted Models

Although high-contribution layers independently reach similar aggregate scores, the sets of examples they solve are largely non-overlapping. Ensembles obtained by majority voting over multiple independently layer-trained models outperform all single experts—including the full-parameter baseline—on challenging evaluation sets.

Figure 5: Majority voting across layer-specialized models yields gains beyond self-consistency or any single model.

This result indicates structural and functional diversity in the solution manifolds reached by different layers' parameter subspaces, suggesting untapped potential for ensemble-based RL post-training and further analysis of solution diversity in the weight space.

Per-Layer Weight Change Analysis

A salient control is the relationship between per-layer weight change and contribution. The paper shows that, during full-parameter RL, the L2 magnitude of parameter updates is nearly uniform across depth, in stark contrast with the highly non-uniform distribution of contribution. During single-layer RL, all layers undergo similar-scale updates, but their capacity to absorb signal is profoundly different, again emphasizing the criticality of subspace orientation over movement magnitude.

Figure 6: Under full RL training, parameter change per layer is uniform; layer contribution, however, is highly peaked in the middle.

This dissociation contributes to a deeper understanding of the functional geometry of RL adaptation.

Theoretical and Practical Implications

The findings advocate for a revision of current RL post-training practices. Instead of the canonical uniform update, targeted intervention in high-contribution layers yields more efficient and frequently higher-quality adaptation, reducing the parameter update budget by orders of magnitude.

The study raises pointed questions for theoretical analysis: why do central layers dominate in RL adaptation, and what specific representational or computational roles do they serve that makes them more "plastic" or reward-absorbent? These remain open but could be informed by in-depth representational analyses such as probing or attention flow tracing across layers.

Moreover, given the transferability of layer rankings and the diversity of induced experts, future work could explore generalization to parameter-efficient RL finetuning, structured expert mixtures, or dynamic per-layer RL schedules. Additionally, these findings have profound relevance in scenarios constrained by compute, where layer-aware RL could deliver most of the performance gain at a fraction of the cost.

Conclusion

This paper provides rigorous, multi-dimensional evidence that RL post-training gains are highly concentrated within a compact middle-layer subspace in transformer networks. Single-layer training frequently suffices to reach or surpass full-parameter adaptation. This overturns conventional assumptions about distributed plasticity in transformers during RL adaptation and motivates a new research axis in RL-efficient transformer training. The study lays groundwork for principled, resource-efficient parameter allocation and new theoretical inquiry into the modularity and plasticity of transformer depth (2607.01232).