Cascaded Domain-wise RL

• Cascaded domain-wise RL is a modular framework that applies sequential reinforcement modules to isolate domain-specific reward structures and improve training efficiency.
• It leverages reusable skill vectors and attribute modules to enable efficient cross-domain transfer while reducing catastrophic forgetting.
• Empirical studies report performance gains up to 11 points and significant reductions in compute cost compared to monolithic RL approaches.

Cascaded domain-wise reinforcement learning (RL) refers to frameworks in which RL is applied sequentially or modularly across domains, attributes, or task subspaces, rather than as a monolithic or blended process. This paradigm addresses heterogeneity in task structure, reward specification, and training constraints by designing and composing either policy modules or parameter updates in a stage-wise or attribute-wise manner. Cascaded domain-wise RL frameworks have been realized in LLMs, multi-domain reasoning agents, and hierarchical control policies, offering improved modularity, sample efficiency, and zero-shot generalization (Tang et al., 16 Jan 2026, Wang et al., 15 Dec 2025, Chang et al., 2020).

1. Foundational Principles and Motivation

Cascaded domain-wise RL is motivated by the limitations of “blended” RL approaches, which mix heterogeneous prompts, rewards, and data domains in joint optimization loops. Such approaches require unified reward models and hyperparameters, complicate engineering, slow convergence, and often degrade per-domain performance. Instead, cascaded domain-wise RL orchestrates RL in a sequence, isolating policy or parameter optimization for each domain, attribute, or task constraint. The organizing hypothesis is that composite skills or behaviors can be decomposed into domain- or attribute-local updates that are either orthogonal in parameter space or modular in execution space.

Key motivations include:

• Simplification of domain-specific reward computation and evaluation pipelines, allowing each RL stage to exploit tailored verifiers and data.
• Improved sample efficiency and transfer, as modularly trained skills or attribute-heads can be reused or composed in new domains without global retraining.
• Resistance to catastrophic forgetting, as domain separation and sequential ordering reduce cross-domain interference and preserve or even improve earlier-acquired capabilities (Wang et al., 15 Dec 2025, Chang et al., 2020).

2. Framework Instantiations and Methodologies

2.1. Parametric Skill Transfer (PaST)

PaST treats factual knowledge acquisition (via supervised fine-tuning, SFT) and reasoning-skill acquisition (via RL, e.g. PPO or GRPO) as orthogonal parameter updates. SFT injects new knowledge by minimizing cross-entropy loss, producing parameters $\theta_\mathrm{SFT}$. RL further refines $\theta_\mathrm{SFT}$ to obtain $\theta_\mathrm{RL}$, targeting procedural or reasoning skills through problem-specific reward signals. Empirically, SFT and RL updates ($\Delta\theta_\mathrm{SFT}$, $\Delta\theta_\mathrm{RL}$) are nearly orthogonal, enabling extraction of a domain-agnostic skill vector $S = \Delta\theta_\mathrm{RL}$. In any target domain, lightweight SFT on domain data is performed, then the pre-extracted $S$ is linearly injected:

$$\theta_\text{final} = \theta_T^\mathrm{SFT} + \alpha S \,,$$

with $\alpha$ controlling the injection magnitude. This procedure allows robust cross-domain reasoning behavior without re-running costly RL per domain (Tang et al., 16 Jan 2026).

2.2. Sequential Domain-wise RL for Large Models

Cascaded domain-wise RL, as realized in Nemotron-Cascade, proceeds as a strict sequence of RL stages: (1) SFT on mixed corpus, (2) RLHF for alignment, (3) instruction-following RL (IF-RL), (4) math RL, (5) code RL, and (6) software engineering (SWE) RL. Each stage utilizes a domain-specific verifier, reward model, and curriculum. Core algorithmic steps include batch rollout sampling, advantage normalization, and REINFORCE-style gradient updates without KL penalization. Domain separation allows for asynchronous verification (e.g., for code), tailored reward signals, and gradual extension of model capabilities. RLHF not only aligns model preferences but also improves coverage of reasoning-oriented metrics by reducing verbosity and improving token efficiency (Wang et al., 15 Dec 2025).

2.3. Cascade Attribute Networks (CAN)

The CAN framework formalizes compound RL control tasks as MDPs decomposed along $n+1$ attributes, each corresponding to a constraint or skill. Each attribute module $\pi_i$ is a neural network accepting state and previous module’s output, producing an additive compensatory action. Base module $\pi_0$ solves the fundamental task (e.g., target-reaching), while each add-on module $\pi_i$ learns to minimally adjust upstream actions for its specific constraint (e.g., obstacle avoidance). Modules are stacked in a cascade,

$$u_0 = \pi_0(s_0),\quad u_1 = u_0 + w_1\pi_1([s_1,u_0]),\quad ...,$$

allowing zero-shot assembly for any attribute subset by simply “wiring up” the desired modules and gates (Chang et al., 2020).

3. Mathematical Structure and Training Procedures

PaST Orthogonality and Skill Vector

Parameter updates are defined relative to a shared base $\theta_\mathrm{base}$. The skill vector $S$ is given by

$$\Delta\theta_\mathrm{SFT} = \theta_\mathrm{SFT} - \theta_\mathrm{base}\,,\quad \Delta\theta_\mathrm{RL} = \theta_\mathrm{RL} - \theta_\mathrm{SFT}\,,\quad S = \Delta\theta_\mathrm{RL}$$

Orthogonality (cosine similarity) of SFT and RL updates is measured as:

$$\text{orth}(\Delta\theta_\mathrm{SFT}, \Delta\theta_\mathrm{RL}) = \frac{(\Delta\theta_\mathrm{SFT})^\top \Delta\theta_\mathrm{RL}}{\|\Delta\theta_\mathrm{SFT}\| \, \|\Delta\theta_\mathrm{RL}\|} \,.$$

Linear skill-injection enables scalable and modular reasoning adaptation (Tang et al., 16 Jan 2026).

Cascaded RL Stage Procedure (Nemotron-Cascade)

Each RL stage executes:

1. Sample prompts $\{q_j\}$ from domain data $D$.
2. For each $q_j$, generate $G$ rollouts $\{o_{j,i}\}$ using current $\pi_\theta$.
3. Compute rewards $r_{j,i}$ via domain verifier.
4. Compute normalized advantage $A_{j,i} = (r_{j,i} - \mu)/\sigma$.
5. Update $\theta$ by gradient descent on

$$L(\theta) = -\frac{1}{B G}\sum_{j,i} A_{j,i}\log \pi_\theta(o_{j,i}|q_j).$$

This yields robust stage-wise domain adaptation with minimal interference (Wang et al., 15 Dec 2025).

CAN Module Training

Base module $\pi_0$ is trained by PPO with GAE under standard base-task reward. Add-on module $\pi_i$ is trained with upstream modules frozen, optimizing for $R_0 + R_i$, with a compensation regularizer $L^c_i = \lambda \mathbb{E}_t[\|a^c_i(t)\|^2]$ to prevent unnecessary drift. Curriculum learning strategies gradually increase initial state complexity, promoting robustness (Chang et al., 2020).

4. Empirical Outcomes and Comparative Performance

Cascaded domain-wise RL frameworks consistently demonstrate:

• Substantial gains in accuracy and success metrics over monolithic or blended baselines.
• Strong zero-shot generalization for new domains or constraint compositions.
• Reduced RL compute in new target domains, since skill vectors or attribute modules are reused without retraining.

Key empirical observations include:

• PaST achieves up to 9.9-point gain over state-of-the-art self-editing SFT baselines on SQuAD, and +10.3 points in average zero-shot success on ToolBench across 20 unseen categories (Tang et al., 16 Jan 2026).
• Nemotron-Cascade models attain +11 point improvement on LiveCodeBench v5/v6 over their SFT teachers, outperform DeepSWE-32B on SWE-bench (43.1% vs. 42.2%), and win silver in the 2025 IOI with superior performance on individual algorithmic benchmarks (Wang et al., 15 Dec 2025).
• CAN enables success rates of 10/10 in zero-shot multi-attribute robot control and >10× faster curriculum progression relative to flat PPO (Chang et al., 2020).

5. Scalability, Transfer, and Robustness

Cascaded domain-wise RL provides compelling scalability properties:

• PaST scales to hundreds of domains and extremely long context lengths (24k+ tokens) with only a single source-domain RL cost; further improvements are possible through adaptive skill vector injection and multi-vector composition (Tang et al., 16 Jan 2026).
• In Nemotron-Cascade, domain-wise RLVR stages are empirically resistant to catastrophic forgetting, as (i) on-policy RL continues to reinforce reward-aligned behaviors from previous domains, (ii) reward structure overlaps generalize, not overwrite, previous skills, (iii) strict domain separation mitigates destructive interference, (iv) domain order prevents specialist objectives from degrading global alignment (Wang et al., 15 Dec 2025).
• In CAN, modular policy design grants interpretable, reusable behaviors and linear scaling in training time for new attribute conjunctions. However, additive decoupling is assumed; highly entangled constraints can challenge the cascade assumption, and deep cascades may accumulate compounding errors (Chang et al., 2020).

6. Extensions, Limitations, and Future Directions

Avenues for extension and further research include:

• Multi-skill composition: learning a bank of orthogonal skill vectors for different reasoning or behavioral paradigms, and developing ensemble selection procedures (Tang et al., 16 Jan 2026).
• Adaptive and per-layer skill injection: parameterizing the injection coefficient $\alpha$ by layer or domain to optimize cross-domain transfer (Tang et al., 16 Jan 2026).
• Continuous or attention-based module routing: in CAN, developing modules that accept continuous parameters or are dynamically weighted by state, increasing flexibility in policy composition (Chang et al., 2020).
• Applications to hierarchical and multi-agent systems, where skills and modules can be cascaded at several abstraction levels (Chang et al., 2020).
• Exploration of the cascade paradigm beyond Qwen2.5-7B architectures and robust hyperparameter tuning strategies for diverse domain shifts (Tang et al., 16 Jan 2026).
• Automated domain selection for skill vector or module distillation and rigorous ablations on catastrophic forgetting resistance and compositional robustness (Wang et al., 15 Dec 2025).

7. Significance and Impact

Cascaded domain-wise RL frameworks mark a transition from homogenized, monolithic RL training toward stage-wise, modular skill acquisition applicable to both LLMs and policy networks. Empirical studies attribute gains to modularization, domain-aware reward engineering, and skill reuse, with demonstrated benefits in zero-shot generalization, compute efficiency, and cross-domain robustness. Current evidence suggests that these frameworks can match or surpass the performance of substantially larger monolithic models and offer new pathways for continual adaptation in dynamic multi-domain environments (Tang et al., 16 Jan 2026, Wang et al., 15 Dec 2025, Chang et al., 2020).