Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR

Abstract: Reinforcement learning with verifiable rewards (RLVR) has been extended from single-domain training to multi-domain reasoning suites spanning mathematics, programming, and science. However, the training curriculum (how often each domain is sampled) is typically fixed or hand-tuned, even though reasoning skills transfer unevenly across domains. Existing learnability-based curricula adapt to where the policy is currently improving, but are blind to whether a gradient step on the selected domain benefits the remaining domains. In this paper, we propose Transfer-Aware Curriculum (TAC), a bandit-style online curriculum that prioritizes domains whose updates broadly benefit the rest of the training suite. TAC repurposes signals already produced by RL training: per-domain advantages capture local learnability, and projected gradients, taken from the GRPO step being computed, estimate cross-domain transferability via gradient-geometry alignment, at negligible cost (<1% wall-clock overhead). Across a six-domain reasoning suite, TAC achieves the best macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B, outperforming proportional random sampling, a hand-designed schedule, and a learnability-only bandit, and improving over the last of these by up to 2.8 points (10% relative). Ablations show performance degrades sharply when the transferability term is removed, and TAC remains robust on imbalanced training mixtures where learnability-only curricula over-commit to dominant domains. Our findings establish cross-domain transferability as a key signal for curriculum design in multi-domain RLVR.

• The paper introduces a transfer-aware curriculum that balances local learnability and cross-domain transferability to improve RLVR performance.
• It employs random projections on gradients and cosine similarity to compute a transferability signal efficiently.
• Empirical evaluations show up to +2.8 points improvement on Qwen3-1.7B, ensuring robust generalization with negligible computational overhead.

Transferability-Aware Curriculum for Multi-Domain RLVR: Formal Overview and Analysis

Motivation and Problem Statement

Multi-domain reinforcement learning with verifiable rewards (RLVR) has emerged as a critical method for post-training LLMs on diverse reasoning tasks. Prior approaches predominantly rely on fixed or hand-tuned curricula for domain sampling, neglecting the heterogeneity in cross-domain transfer effects. Empirical analyses confirm highly uneven transferability: RL-trained policies on one domain can yield vastly different generalization profiles to other domains (Figure 1).

Figure 1: Cross-domain transfer matrix, showing disparate accuracy gains on target domains after single-domain RL training.

Existing adaptive curricula primarily leverage local learnability signals, such as on-policy advantage or reward variance, to prioritize domains where the policy is actively improving. However, these approaches are intrinsically myopic—they remain agnostic to whether an update in one domain benefits others. The risk is overfitting to domains with high immediate learning signal but low global transfer utility.

Methodology: Transfer-Aware Bandit Curriculum

The paper proposes a transfer-aware curriculum (TAC) formalized as a multi-armed bandit policy. Each domain is treated as an arm, and domain selection at each RL step is governed by a composite feedback signal $S_m^{(t)}$, representing both local learnability and cross-domain transferability.

Learnability is quantified via mean absolute GRPO advantage (Equation (6)), normalized by an EMA z-score (Equation (7)). For transferability, the method leverages projected gradient representations, computed using Johnson-Lindenstrauss random projections on gradients from the last $N$ transformer layers (Figure 2), and cosine similarities as directional alignment proxies.

Figure 2: Projection method schematic for gradient vector compression via random projections.

Each domain maintains an EMA of unit-normalized projected gradients, updated only when sampled. At periodic intervals, pairwise cosine similarities between these EMAs are computed, and a per-domain transferability score $T_m^{(t)}$ is produced using cross-domain min-max normalization.

The composite curriculum signal is:

$$S_{m_t}^{(t)} = \beta\, \hat{L}_{m_t}^{(t)} + (1-\beta)\,T_{m_t}^{(t)}$$

where $\beta$ controls the balance between learnability and transferability. Bandit Q-values are updated accordingly and sampling probabilities are derived from Boltzmann distributions augmented with UCB exploration bonuses.

Figure 3: Ablation of mixing coefficient $\beta$, highlighting performance sensitivity to the balance between learnability and transferability.

Empirical Evaluation

The method is extensively benchmarked on the GURU suite, covering mathematics, codegen, logic, simulation, tables, and stem reasoning. TAC demonstrates:

• Superior macro-averaged accuracy across both Qwen3-1.7B and Llama3.2-3B backbones compared to random, hand-designed (Math-to-Others), and pure-learnability (SEC) curricula.
• Gains of up to $+2.8$ points ($+10\%$ relative) over SEC on Qwen3-1.7B, especially pronounced in benchmarks with imbalanced dataset composition.
• Ablations indicate sharp performance degradation when transferability is removed ($\beta=1$), confirming its necessity.

Figure 4: Curriculum dynamics, showing domain sampling probabilities under TAC evolving adaptively in response to learnability and transferability.

Figure 5: Per-domain learnability $\hat{L}_m^{(t)}$ across training steps, illustrating dynamic signal evolution.

Figure 6: Per-step time breakdown demonstrating minimal computational overhead of the transferability signal (<1\% wall-clock).

Figure 7: Pairwise projected-gradient cosine similarity trajectories, evidencing domain-to-domain gradient alignment patterns.

Key empirical findings include the following strong claims:

• The most learnable domains are often not the most transferable. TAC corrects myopic bandit anchoring onto locally rich but globally narrow domains.
• Math and codegen, while commonly prioritized in prior RLVR, exhibit the lowest transferability—TAC down-weights them accordingly but still improves their generalization due to transfer signal exploitation.
• The transferability signal is computed online solely from training gradients, incurring negligible computational cost.

Theoretical Interpretation and Algorithmic Justification

The transferability component is justified by first-order Taylor expansion of the GRPO surrogate loss, where the direction-only cosine similarity between gradients predicts aggregate loss improvement on other domains. Random projections serve as efficient sketches that preserve inner products in expectation.

Two-phase value updates allow unsampled domains to reallocate sampling mass via updated transferability signals, enabling efficient exploitation of transient cross-domain alignment and mitigating starvation of domains with temporarily low learnability.

Practical and Theoretical Implications

Practically, TAC enables robust and efficient RLVR post-training across heterogeneous domains, improving both sample efficiency and overall reasoning performance. In imbalanced settings, TAC dynamically corrects for over-representation, making it suitable for real-world deployments with non-uniform data budgets.

Theoretically, TAC operationalizes transferability as a first-class curriculum signal, enabling principled exploitation of gradient geometry and domain interactions. This represents an orthogonal axis to optimizer- or loss-level interventions (e.g., curvature-guided policy optimization, multi-task GRPO loss reweighting).

Speculation on Future Directions

• Extension to domains lacking verifiable rewards, where reward proxies or model-based verification are required.
• Combined curriculum-optimizer strategies, integrating transfer-aware domain selection with gradient-alignment objectives in loss functions.
• Application to even larger heterogeneous mixtures as scaling laws for reasoning transfer become better understood.
• Investigation of transferability signals for multi-modal or agentic settings, where domain boundaries are less well-defined.

Conclusion

This work establishes cross-domain transferability as a foundational signal for automated curriculum design in multi-domain RLVR. Empirically, transfer-aware bandit curricula outperform hand-designed and learnability-only baselines with minimal overhead, demonstrating substantial improvements in generalization and reasoning over a broad set of tasks. This principle is anticipated to become central in future developments of robust and versatile reasoning systems for LLMs.