RAGEN-2: Reasoning Collapse in Agentic RL

Abstract: RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.

• The paper identifies template collapse, where RL agents generate superficially diverse yet input-agnostic reasoning traces.
• It introduces an MI-proxy suite and SNR analysis to diagnose failures in entropy-based diagnostics and quantify task performance correlations.
• SNR-Aware Filtering is proposed as a lightweight method that reallocates policy updates to high-signal prompts, enhancing agent stability.

Diagnosing and Mitigating Template Collapse in Multi-Turn Agentic RL: An Information-Theoretic and Signal-to-Noise Perspective

Introduction

"RAGEN-2: Reasoning Collapse in Agentic RL" (2604.06268) rigorously analyzes the limitations of entropy-based reasoning diagnostics in reinforcement learning (RL) training of multi-turn LLM agents. The paper identifies a prevalent failure mode termed template collapse, wherein agents produce superficially diverse but input-agnostic reasoning traces. Through an information-theoretic dissection and a mechanistic signal-to-noise ratio (SNR) analysis, the authors establish mutual information (MI) as a superior proxy for input-conditioned reasoning and introduce SNR-Aware Filtering—a lightweight intervention to preserve input sensitivity and consequently agent performance across a broad set of domains, optimizers, and modalities.

Template Collapse: Information-Theoretic Characterization

The current RL agent training paradigm monitors reward and entropy as proxies for behavioral and process stability. However, entropy, quantifying within-input diversity, is fundamentally blind to the distinction between genuinely input-driven reasoning and high-diversity template-based outputs that ignore cross-input variation. The authors establish the decomposition

$H(Z) = I(X;Z) + H(Z|X),$

where $H(Z)$ is the marginal entropy of reasoning, $H(Z|X)$ the conditional entropy (diversity within a single prompt), and $I(X;Z)$ the MI (input dependence of reasoning). Crucially, RL training can leave $H(Z|X)$ high while $I(X;Z)$ drops, yielding template collapse.

Figure 1: Input-driven reasoning (left) exhibits output adaptation to input content, while templated reasoning converges to input-agnostic, reusable explanations; four reasoning regimes can be visualized along conditional entropy and MI axes.

This collapse regime is concerning because entropy-based stabilization mechanisms—including entropy bonuses and regularization schedules—are unable to detect or prevent it. Empirically, the paper shows MI correlates strongly with final task performance (Spearman $+0.39$), while entropy metrics are weakly or negatively correlated (near zero or $-0.11$ to $-0.14$). Template collapse thus represents a class of silent and degenerate failure modes omitted by standard RL diagnostics.

MI Proxy Suite: Empirical Measurement of Input Dependence

Obtaining tractable and actionable MI estimates on high-dimensional reasoning traces is non-trivial. The paper designs an efficient batch-level MI-proxy suite derived from in-batch cross-scoring: per-token or per-sequence log-likelihoods of each rollout $Z$ under both its true input $H(Z)$0 and the uniform batch mixture. This enables retrieval accuracy (discrete input-reasoning matching), MI difference, and z-score stabilization, all with negligible framework overhead and no external models.

Retrieval accuracy under template collapse approaches chance (e.g., $H(Z)$1 for $H(Z)$2), defining a clear failure threshold. The paper validates MI-proxy tracking as an early-warning diagnostic for reasoning drift, showing that MI decay precedes both task degradation and entropy change.

SNR View: Gradient Dynamics and the Mechanism of Collapse

Critically, the authors move beyond surface-level measurements to a mechanistic account of collapse grounded in RL optimization dynamics. Empirical gradient analyses reveal that for prompt groups with low within-input reward variance,

• The task gradient norm $H(Z)$3 sharply decreases, yielding weak and noisy updates
• Regularization gradient norm (from entropy and KL) is invariant across prompt reward variance, dominating the total update in low-SNR regions

  Figure 2: Left: Gradient updates decompose into informative task-aligned and regularization-aligned components. Right: High within-input reward variance (high SNR) is essential for coherent learning; low SNR regimes manifest as noisy, input-agnostic update drift.

  

  Figure 3: Prompt reward-variance buckets expose the monotonic scaling of the task gradient with SNR; regularization is uniform, leading to noise-dominated, input-agnostic updates in low-variance regions.

A formal decomposition confirms that SNR, measured as the ratio of the norm of the gradient driven by reward-discriminative signal to that driven by sampling noise and reward-agnostic regularization, tightly controls the degree of input-conditioned reasoning maintained by the agent over training. When SNR collapses, parameter updates random-walk toward generic, high-entropy templates, mathematically driving $H(Z)$4 while $H(Z)$5 remains artificially inflated.

SNR-Aware Filtering: Algorithm and Effectiveness

To mitigate template collapse, the paper introduces SNR-Aware Filtering. At each RL update, the method ranks prompt groups by within-input reward variance and retains only the highest-SNR (i.e., high-variance) subset for policy gradient updates. This mechanism adaptively reallocates update capacity to prompts with informative reward structure while excluding those whose gradients are regularization-dominated and task-agnostic.

Figure 4: SNR-Aware Filtering workflow: rollouts are grouped and scored by reward variance, top-p prompts are selected, and only high-signal samples update policy parameters—preventing reinforcement of noisy or degenerate reasoning.

The filtering strategy is analogous to nucleus sampling along the reward variance axis, and supports both fixed-count (Top-k) and mass-based (Top-p) selection rules. Across controlled experiments, Top-p filtering outperforms both Top-k and no-filter baselines, adaptively suppressing input-agnostic updates and preserving input sensitivity under SNR contraction.

Figure 5: Top-p SNR-Aware Filtering best preserves both MI and process entropy during training; MI degradation is evident without filtering even when entropy remains high.

Figure 6: Top-p filtering consistently outperforms alternative strategies across representative environments by maintaining input-dependent reasoning.

Comprehensive ablations confirm that:

• Filtering by reward variance (not simply entropy or length) best preserves both MI and task performance
• The performance gain is especially marked in the presence of prompt reward variance heterogeneity (as validated using RV statistics)
• At high environmental stochasticity, the filtering signal degrades commensurately, as predicted by the SNR mechanism

Empirical Validation Across Domains, Scales, Algorithms

A large testbed spanning Sokoban (irreversible planning), FrozenLake (stochastic navigation), MetaMathQA/Countdown (mathematical reasoning), text-based QA, web navigation, and code generation validates the pervasiveness of template collapse and the efficacy of SNR-Aware Filtering. The approach is effective across LLM sizes (from 0.5B to 7B+), reinforcement learning optimizers (PPO, DAPO, GRPO, Dr. GRPO), and both text and vision modalities.

Figure 7: Systematic behavioral collapse: reasoning length monotonically declines as agents converge on template outputs across environments.

Figure 8: MI-proxy metrics strongly predict task performance across interventions; entropy-based measures correlate weakly or negatively, confirming the diagnostic value of MI and the irrelevance of entropy for input-dependent reasoning.

Analysis: Filtering vs. Regularization and Metric Separability

The paper distinguishes SNR-driven interventions from standard regularization schedules. Tuning entropy and KL coefficients affects process diversity but does not rescue MI, as their implicit action is to control within-input entropy, not cross-input dependence. In contrast, SNR-Aware Filtering directly targets the signal content of gradients. This orthogonality is evident in diagnostic space trajectories during intervention sweeps (see detailed training dynamics visualizations).

Format-based validity or length regularity metrics are also shown empirically orthogonal to MI and unfit as collapse diagnostics, reinforcing the necessity of content-sensitive, information-theoretic analysis.

Theoretical and Practical Implications

The theoretical implication is a principled, information-theoretic reformulation of agent-level collapse modes, with precise quantification via MI-proxy diagnostics. The SNR framework provides a tractable explanation for why agentic RL training drifts toward input-agnostic templates under low task SNR—an explanation that can power new lines of inquiry into stabilizing self-improving or self-aligning language agents. Practically, SNR-Aware Filtering is low-overhead, broadly compatible, and yields tangible agent performance and robustness gains.

Future Directions

Open problems include the generalization of SNR mechanisms to multi-agent interaction, development of adversarial countermeasures for agents that might game the variance-driven filtering signal, dynamic adaptation of the filtering mass to optimize exploration-exploitation tradeoff, and extension to extremely sparse- or high-noise reward environments. Further, augmenting MI estimation with model-based or counterfactual rollouts could yield more powerful input-dependence diagnostics.

Conclusion

This work demonstrates that entropy is an unreliable proxy for input-dependent reasoning in RL-trained language agents. Template collapse—the drift toward highly diverse but input-agnostic templates—occurs pervasively, silently degrading the interpretability and controllability of RL-finetuned agents. By formalizing the phenomenon with mutual information, analyzing its mechanistic SNR drivers, and providing a practical SNR-Aware Filtering method, the paper sets a new standard for closed-loop agentic RL process diagnostics and training stabilization (2604.06268).