Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies (2512.19673v1)

Abstract: Existing reinforcement learning (RL) approaches treat LLMs as a single unified policy, overlooking their internal mechanisms. Understanding how policy evolves across layers and modules is therefore crucial for enabling more targeted optimization and raveling out complex reasoning mechanisms. In this paper, we decompose the LLM policy by leveraging the intrinsic split of the Transformer residual stream and the equivalence between the composition of hidden states with the unembedding matrix and the resulting samplable policy. This decomposition reveals Internal Layer Policies, corresponding to contributions from individual layers, and Internal Modular Policies, which align with the self-attention and feed-forward network (FFN) components within each layer. By analyzing the entropy of internal policy, we find that: (a) Early layers keep high entropy for exploration, top layers converge to near-zero entropy for refinement, with convergence patterns varying across model series. (b) LLama's prediction space rapidly converges in the final layer, whereas Qwen-series models, especially Qwen3, exhibit a more human-like, progressively structured reasoning pattern. Motivated by these findings, we propose Bottom-up Policy Optimization (BuPO), a novel RL paradigm that directly optimizes the internal layer policy during early training. By aligning training objective at lower layer, BuPO reconstructs foundational reasoning capabilities and achieves superior performance. Extensive experiments on complex reasoning benchmarks demonstrates the effectiveness of our method. Our code is available at https://github.com/Trae1ounG/BuPO.

• The paper introduces BuPO, a novel approach exploiting internal layer and modular policies to enhance reinforcement learning training in language models.
• The analysis reveals distinct entropy dynamics, with lower layers driving exploration and upper layers ensuring convergence.
• Empirical results show BuPO significantly improves performance on reasoning benchmarks by carefully calibrating early internal policy optimization.

Bottom-up Policy Optimization for LLMs: Internal Policies and Their Role in Reinforcement Learning

Introduction

The paper "Bottom-up Policy Optimization: Your LLM Policy Secretly Contains Internal Policies" (2512.19673) presents a novel policy decomposition for Transformer-based LLMs, revealing the existence of distinct, samplable policies at intermediate layers and modules. The authors introduce the concepts of Internal Layer Policies and Internal Modular Policies and provide a detailed entropy-based analysis of their dynamics across model families. Central to the work is the proposition of Bottom-up Policy Optimization (BuPO), a reinforcement learning paradigm that targets these internal policies early in training, rather than optimizing only the top-layer policy.

Internal Policy Decomposition

The intrinsic structure of the Transformer allows the LLM policy to be formally decomposed based on the residual stream. This decomposition yields:

• Internal Layer Policies ($\pi^l_\text{Layer}$): The cumulative policy up to a given layer $l$, computed by unembedding the hidden states after layer $l$.
• Internal Modular Policies ($\pi^l_\text{ATTN}$, $\pi^l_\text{FFN}$): The policies attributable to the outputs of self-attention (ATTN) and feed-forward network (FFN) modules.

  Figure 1: A schematic of the residual stream in Transformers, illustrating how hidden states interface with attention, FFN, and the unembedding matrix to form policies at each stage.

By treating the outputs of every intermediate layer (or module) as samplable policies, this decomposition enables a fine-grained, policy-centric interpretability of the model’s computations, distinct from standard logit-lens analysis. This view is not only more aligned with RL but also empirically avoids the instability in entropy metrics observed with layer normalization present in classical logit-lens approaches.

Entropy Dynamics and Reasoning Patterns

The paper systematically analyzes Internal Policy Entropy across popular LLM families (Qwen, Llama, DeepSeek-Math). Major observations include:

• Layerwise Entropy Flow: Early layers display high policy entropy (exploration), which contracts and converges at upper layers (refinement).
• Architectural Divergence: There are marked differences in entropy flow across model families. Llama models exhibit abrupt entropy collapse in their final layers, while Qwen-series models (notably Qwen3) exhibit a more gradual, progressive reduction in entropy, reflecting human-like reasoning phases.

Figure 2: Token-level entropy traced through layers for multiple model families, exemplifying universal exploration in lower layers and high-variance convergence trends in upper layers.

To probe module-specific contributions, the authors define entropy change $\Delta H^l$ for attention and FFN modules. Qwen3 consistently shows positive $\Delta H^l_\text{ATTN}$ and a three-stage Exploration–Integration–Convergence (EIC) structure in FFN: exploration expands in the lower FFNs, knowledge integration stabilizes entropy in mid layers, and upper FFNs drive convergence.

Figure 3: Entropy change $\Delta H^l$ across layers, indicating the shifting balance between exploration and contraction; the EIC pattern in Qwen3 FFN modules is especially pronounced.

Influence of Internal Modules on the Residual Stream

The study further examines residual cosine similarity to quantify how attention and FFN modules at each layer contribute to (or interfere with) existing hidden representations. Qwen3's attention modules consistently amplify features (strong positive cosine similarity), whereas Qwen2.5's attention is weaker. FFN modules in Qwen3 are structurally organized—injecting orthogonal features in early layers, suppressing noise for integration in the middle, and amplifying for convergence at the top.

Figure 4: Cosine similarity between module outputs and preceding residual streams, highlighting differential module function and inter-series variance in information routing.

Internal Policy Optimization: Bottom-up Approach

The authors propose InterGRPO, extending standard group-wise RL policy optimization to operate directly on internal policies of arbitrary layers. Notably, optimizing these internal policies independently exhibits distinct learning dynamics. Lower-layer policies, when optimized, consistently lead to internal feature refinement: their output representations become increasingly similar to those of later, higher layers, forcing early capture of high-level reasoning signals.

However, optimizing only internal policies can destabilize final performance if not properly tuned—over-alignment results in rapid policy collapse, underscoring the need for balance in bottom-up alignment.

Figure 5: Training curves of reward, entropy, and output length for various policies during direct internal policy optimization. Lower-layer policies show more robust, stable adjustment than top layers.

Figure 6: Left: hidden-state similarity between an optimized internal layer and subsequent layers. Middle: corresponding entropy change stabilizes and converges. Right: model perplexity as over-alignment leads to collapse beyond optimal intervention.

Bottom-up Policy Optimization (BuPO) Framework

Drawing on the observed progressive emergence of reasoning, the BuPO training schedule optimizes internal layer policies for a controlled number of steps before switching to standard top-level policy RL (GRPO) optimization. The optimal selection of which layer to align, and for how long, is empirically determined via the entropy transitions charted earlier.

BuPO consistently results in performance gains across multiple mathematical reasoning benchmarks (MATH, AMC, AIME) relative to PPO, GRPO, and other leading RL baselines. On Qwen3-4B, BuPO provides a $+4.58$ Pass@$K$ gain on AIME24 and $+2.49$ on MATH500 over direct RL. Llama models, despite their less progressive reasoning structure, also benefit from controlled bottom-up intervention.

Figure 7: Average Pass@$K$ evaluation on four benchmarks demonstrates stable and consistent improvement using BuPO across all models and $K$ ranges.

BuPO’s efficacy is explained by the improved early-stage entropy profile: modest bottom-up optimization drives increased, more stable exploration in the LLM policy, forming a “wider base” for subsequent RL refinement.

Figure 8: Training entropy curves of BuPO and GRPO, illustrating that intermediate policy optimization enhances initial exploration without destabilizing late-stage convergence.

Theoretical and Practical Implications

This work has several direct implications for LLM optimization and interpretability:

• Internal Policy Structure: The existence of well-formed, samplable internal policies, latent in the Transformer residual stream, motivates design and analysis at a finer granularity than top-layer outputs.
• Progressive Reasoning Model Selection: The progressive EIC pattern is strongly correlated with models’ capacity to absorb knowledge post-training, suggesting that architectural design should explicitly promote staged reasoning phases.
• RL Algorithms for LLMs: Bottom-up optimization principles, especially in models with clear progressive internal structure, can substantially outperform monolithic top-policy optimization. However, over-optimization of internal policies can hurt final model performance, indicating that layer and step selection must be empirically calibrated.
• Interpretability Methods: Policy-centric entropy analysis provides actionable signals for selecting which layers to intervene on, as well as for diagnosing learning failures and structural limitations in the Transformer stack.

These results open new avenues for mechanistic alignment, automated layer selection in policy optimization, and fundamentally, the design of RL curricula that echo cognitive staged learning.

Conclusion

This paper establishes that Transformer-based LLMs encode distinct, samplable internal policies at intermediate layers and modules, which underlie the stepwise construction of reasoning. Through policy entropy analysis, it is shown that Qwen-series models realize a progressive Exploration–Integration–Convergence architecture, while Llama models tend to abrupt late-layer convergence. The Bottom-up Policy Optimization method directly exploits this layer structure, efficiently improving performance in complex reasoning tasks without adverse trade-offs, provided intervention is properly controlled. This work yields actionable interpretability tools and policy optimization paradigms, advancing RL for LLMs both practically and theoretically.