---
title: 'Nemotron-Cascade: Scalable Domain-wise RL'
url: https://www.emergentmind.com/topics/nemotron-cascade
type: topic
---

# Nemotron-Cascade: Scalable Domain-wise RL

Nemotron-Cascade refers to a scalable methodology for domain-wise reinforcement learning (RL) in the context of large general-purpose language models, with an explicit focus on robust reasoning and task specialization across mathematical proof, instruction following, code generation, and software engineering. The Nemotron-Cascade workflow emphasizes sequential, domain-specific RL stages—rather than joint multi-domain optimization—addressing infrastructure bottlenecks, training curriculum complexity, and cross-domain interference frequently encountered in large-scale RLHF and RLVR pipelines. The architecture supports both "instruct" and "deep thinking" modes under a unified schema, and achieves state-of-the-art results on multiple academic benchmarks [2512.13607].

## 1. Architectural Principles and Motivation

Nemotron-Cascade is designed to address severe cross-domain heterogeneity intrinsic to general-purpose reasoning models. The primary motivation arises from the observation that reasoning tasks—including Q&A, instruction following, proof generation, competitive coding, and software patching—present domain-specific verification costs and divergent response-length requirements. For example, mathematical proof verification can leverage fast, rule-based verifiers, while code generation necessitates slow execution-based testing, and alignment via RLHF relies on high-latency reward models.

Mixing domains within a single RL stage results in curriculum inefficiencies, infrastructure bottlenecks (waiting on slow verifiers), and brittle hyperparameter tuning. Nemotron-Cascade circumvents these limitations with sequential, domain-wise RL, enabling each stage to utilize custom verifiers, curricula, and reward shaping without cross-domain deadlock.

## 2. Cascade RL Workflow and Algorithmic Details

The core methodology is Cascade RL—a fixed-order, sequential RL pipeline where each domain is optimized with its own reward function and verifier:

1. **RLHF (Reinforcement Learning from Human Feedback):** General human-preference alignment using a 72B reward model trained on ~82K preference pairs.
2. **Instruction-Following RL:** Enforcement of explicit instruction satisfaction using deterministic verifiers. Rewards combine rule-based IF checks and a scaled RLHF signal.
3. **Math RL:** Symbolic math reasoning with dynamic token-budget curricula and rule-based answer checkers.
4. **Code RL:** Execution-based code verification (unit tests), employing higher sampling temperature for exploration.
5. **SWE (Software Engineering) RL:** Software patching (GitHub repair) tasks with execution-free, hybrid lexical-semantic reward computed by a 72B LLM.

Each stage employs an on-policy, token-level REINFORCE-style objective without a KL penalty. Rollout-level rewards $r_i$ are normalized per-group:
\[
\tilde r_i = \frac{r_i - \mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\})}
\]
The gradient-regularized policy optimization (GRPO) loss is:
\[
J_{\mathrm{GRPO}}(\theta) = -\sum_{i=1}^G \sum_{t=1}^{T_i} \tilde r_i \log \pi_\theta(o_{i,t} \mid q, o_{i,<t})
\]
where $q$ is the prompt and $o_{i,t}$ is the token at timestep $t$ of rollout $i$.

Pseudocode for one RL stage:
```python
# Domain-specific RL stage (condensed)
for step in range(N_steps):
    q_batch = sample_prompts(D)
    rollouts = [generate_rollouts(q, G, model=θ) for q in q_batch]
    rewards = [R(q, o) for (q, o) in zip(q_batch, rollouts)]
    norm_rewards = normalize(rewards, groupby=q_batch)
    θ = update_weights(θ, policy_gradient(norm_rewards, rollouts))
```

Prompt sets are disjoint across domains (e.g., removing math prompts from RLHF), reducing destructive interference and catastrophic forgetting. The RL process is resistant to forgetting because the on-policy nature ensures relevant capabilities remain sampled unless they are non-rewarding [2512.13607, §4.1.1].

## 3. Unified Generation Modes and ChatML Templates

Nemotron-Cascade models operate in both instruct and deep thinking modes, selectable at generation time via ChatML template flags:
- **“/no_think”**: Instruct mode (instant answer, no chain-of-thought).
- **“/think”**: Deep thinking mode (model emits a `<think>...</think>` block with explicit reasoning steps prior to final answer).

Batches are split (e.g., RLHF "Half-Half" training) to assign roughly equal probability mass to both prompt types, driving robust cross-mode transfer.

## 4. Training Regimes, Data, and Curricula

### 4.1 Supervised Fine-Tuning (SFT)

A two-stage SFT process establishes a broad base:
- Stage 1 (16K tokens): General domain data with math, code, science in thinking mode.
- Stage 2 (32K tokens): Addition of tool-calling and software-engineering data, primarily in thinking mode, but with parallel non-thinking general responses.

Extensive cleaning (n-gram decontamination, cross-validation) and balancing yield approximately 2.8M general, 2.8M math, 1.4M code, and 634K science samples.

### 4.2 RL Stages

Domain-specific data and hyperparameters:

| RL Stage        | Data                              | Key Hyperparams             |
|-----------------|-----------------------------------|-----------------------------|
| RLHF            | 82K preference pairs              | 8 rollouts/prompt, max 12K, LR 2e-6    |
| IF-RL           | ~100K constraints                 | 8 rollouts, 2 stages (8K/16K), 2K+1K steps |
| Math RL         | 14K Olympiad problems             | 3 stages (24K/32K/40K tokens), T=0.8–1.2 |
| Code RL         | 9.8K code prompts, unit test suite| Single stage (44K/48K), T=1.0, LR 4e-6   |
| SWE RL          | 127K repair tasks                 | 2 context stages (16K/24K), 16 rollouts  |

Dynamic curriculum (e.g., longer token budgets, filtering solved/unsolved items) is employed in Math RL and SWE RL.

## 5. Empirical Performance and Benchmark Outcomes

Nemotron-Cascade models achieve the following representative results (all on unified instruct/thinking models):

- **MMLU/Pro EM:** Up to 85.1/77.0 (14B think model), $+10\sim11$ points over SFT teacher.
- **ArenaHard alignment:** Up to 95.7% (8B).
- **IFEval:** 90.2% (14B).
- **Math (AIME24/25):** 90.4%/83.3% (14B).
- **Code (LCB v6):** 71.1% (8B), 74.6% (14B-think), exceeding DeepSeek-R1-0528.
- **SWE-bench Verified:** 43.1% (14B), outperforming specialty models >32B.
- **Competitive Coding (IOI 2025):** "Silver-medal" performance (343.37/438.30 gold threshold) using a feedback-driven test-time scaling mechanism—incorporating up to 128K total tokens, 20 candidates per round, and iterative verdict-embedding in the prompt.

Ablation reveals each RL stage rarely degrades prior-stage benchmarks and frequently confers small cross-domain improvements.

## 6. Stability, Scalability, and Limitations

Cascade RL’s modular pipeline permits domain-level verifier and curriculum selection, enabling scalable infrastructure: each stage customizes its own batch sizes, token budgets, curriculum schedules, and reward shaping. On-policy GRPO with token-level loss and no KL penalty (vs reference model) yields consistently stable training across all stages.

- **Resistance to forgetting:** Sequential order (RLHF → IF → Math → Code → SWE) with disjoint prompt pools and on-policy RL results in minimal catastrophic forgetting.
- **Unified model advantages:** Cross-mode transfer is effective; unified 8B models perform comparably to dedicated "thinking" versions while supporting both generation modes.
- **Known limitations:** IFEval sometimes degrades after RLHF (prompt overlap), long-context capacity beyond 32K is underutilized (Math/SWE rely on up to 40K), and further improvements require stronger reward models and advanced attention mechanisms.

## 7. Data Availability and Future Directions

All model weights, SFT, RL-stage recipes, and benchmark scripts are publicly released at https://huggingface.co/collections/nvidia/nemotron-cascade to facilitate community benchmarking and extendibility [2512.13607]. Plans include scaling up reward model capacity, enhancing long-context attention, and developing joint multi-stage RLHF+RLVR optimization. A plausible implication is that modular Cascade RL architectures will enable further advances as task heterogeneity and evaluation complexity increase in next-generation reasoning models.

Source: https://www.emergentmind.com/topics/nemotron-cascade