---
title: 'CAMPO: Context-Aware Multi-Stage Optimization'
url: https://www.emergentmind.com/topics/context-aware-multi-stage-policy-optimization-campo
type: topic
---

# CAMPO: Context-Aware Multi-Stage Optimization

Context-Aware Multi-Stage Policy Optimization (CAMPO) is a reinforcement learning (RL) methodology that integrates sequential curriculum, context-sensitive reward shaping, and multi-stage policy update schedules to optimize complex, context-dependent tasks. The approach is characterized by explicit multi-phase training, context-dependent regularization or penalties, and advanced group-normalized policy optimization techniques. CAMPO was introduced and formalized in the context of mathematical reasoning for open-source reasoning language models by the MiroMind-M1 project [2507.14683], and closely related multi-stage or context-aware RL approaches are documented in question answering [2512.14465], in-context visual localization [2605.31145], and multi-agent architectures [2601.05465]. 

## 1. Core Objective and Theoretical Foundations

CAMPO generalizes standard actor-critic and policy-gradient RL objectives such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) to address high-variance settings and complex reward structures. The canonical CAMPO objective incorporates:

- **Multi-stage length-aware clipping schedules**: At each stage $s$, distinct clipping bounds $(\varphi_{\text{low}}(s), \varphi_{\text{high}}(s))$ control the surrogate policy ratio per-token, decoupling gradient updates per context length regime.
- **Per-token normalization**: The conventional sum over all sampled output tokens $\sum_{i}|o_i|$ is used in the denominator to eliminate bias towards longer/shorter generations; this ensures balanced credit assignment across outputs of varying lengths.
- **Adaptive, context-dependent penalties**: Distinct from generic entropy or KL penalties, CAMPO integrates penalties derived from sequence properties (e.g., repetition) or task-specific failures.

The stage-specific surrogate objective is:
\[
J_{\text{CAMPO}}(\pi_\theta) = 
\mathbb{E}_{(q, a), \{o_i\} \sim \pi_{\theta_{\text{old}}}} \left[ 
\frac{1}{\sum_i |o_i|} \sum_i \sum_{t=1}^{|o_i|} 
\min\left( r_{i,t}(\theta) \cdot \widehat{A}_{i,t}, 
\operatorname{clip}\big(r_{i,t}(\theta), 1 - \varphi_{\text{low}}, 1 + \varphi_{\text{high}}\big) \cdot \widehat{A}_{i,t} \right) 
\right]
\]
with group-normalized advantage with adaptive penalty:
\[
\widehat{A}_{i,t} = \frac{[r(o_i,a) - f(o_i)] - \mu_r}{\sigma_r}
\]

Here, $f(o_i)$ is a context-induced penalty (e.g., first detected repetition loop in output), $\mu_r,\ \sigma_r$ denote the mean and std across the batch, and $r_{i,t}(\theta)$ is the likelihood ratio for the $t$-th token of $o_i$ under new vs. old policy. Only "partially solved" examples, for which not all sampled outputs are correct or incorrect, are retained to focus optimization on ambiguous regimes [2507.14683].

## 2. Length-Progressive Multi-Stage Training

CAMPO implements a structured curriculum by progressively increasing the maximum allowed generation length $L_{\max}(s)$ across $M$ stages. For each curriculum stage:

- **Rollout length cap** $L_{\max}(s)$ is enforced: any sampled sequence exceeding this length is automatically assigned zero reward, treating verbosity as a failure.
- **Advancement criteria**: Early-stage progression is either step-count-based (e.g., advance after $1\ 200$ steps) or conditioned on average model output length saturation on a held-out set, typically at $0.9\cdot L_{\max}(s)$.
- **Benefits**: Early, short-length training stages yield faster rollouts, reduce GPU idle time, and prioritize efficient, concise reasoning patterns. Later stages enable the model to generalize to arbitrarily long, multistep solutions, crucial for higher-order reasoning [2507.14683].

Curriculum progression schedules in MiroMind-M1, for instance, use two stages ($16\,384 \rightarrow 32\,768$ tokens) for 7B models and three ($16\,384 \rightarrow 32\,768 \rightarrow 49\,152$) for 32B, empirically accelerating convergence and improving short-budget accuracy.

## 3. Adaptive Repetition and Degeneracy Penalties

To prevent pathological behaviors such as infinite loops or excessive output repetition—a known failure mode in long-form sequence modeling—CAMPO introduces an explicit, context-sensitive penalty $f(o)$:

\[
f(o) = \frac{\text{\# tokens in the first detected repetition loop}}{T}
\]
where $T$ is the total length of $o$. Upon detecting the earliest repeating sub-sequence inside the generation, $f(o)$ is computed and subtracted from the binary correctness reward prior to normalization.

Ablations on MATH and AIME tasks demonstrate that omitting this term leads to abrupt training divergence, whereas its inclusion yields stable and efficient learning curves, especially in the absence of KL regularization [2507.14683].

## 4. Implementation and Algorithmic Structure

The end-to-end CAMPO pipeline comprises:

1. **Supervised Fine-Tuning (SFT)**: The base model is first trained on large, human-verified datasets with chain-of-thought (CoT) trajectories.
   - Hyperparameters: Peak learning rate $5\times10^{-5}$ (cosine decay), batch size 128, 3 epochs on verified math CoT pairs, no-packing preferred.
2. **Multi-stage Reinforcement Learning**: 
   - For each stage $s$:
     - Cap rollout length at $L_{\max}(s)$
     - Set decoupled clip bounds $(\varphi_{\text{low}}(s), \varphi_{\text{high}}(s))$
     - Sample batches, generate $G$ outputs per instance, filter for partially solved examples.
     - Compute per-token clipped surrogates and group-normalized advantages with penalty.
     - Inner PPO-style optimization with $1$–$4$ epochs per mini-batch.
   - Hyperparameters: Const. RL LR $1\times 10^{-6}$, $G=16$ rollouts/sample, batch size 32, temperature 1.0, no KL penalty.
3. **Pseudocode**: The complete update algorithm and batch filtering protocol is formalized (see [2507.14683], Algorithm 1).

## 5. Empirical Benchmarking and Token Efficiency

CAMPO-trained models (MiroMind-M1-RL-7B/32B) achieve state-of-the-art or highly competitive accuracy and improved token efficiency among Qwen-2.5-base open-source reasoning language models:

| Model                    | AIME24 | AIME25 | MATH500 |
|--------------------------|--------|--------|---------|
| MiroMind-SFT-7B          | 60.4   | 45.0   | 94.6    |
| MiroMind-M1-RL-7B        | **73.4** | **57.8** | **96.7**  |
| DeepSeek-R1-Qwen-32B     | 70.8   | 52.1   | 95.8    |
| Skywork-OR1-32B-Prev     | 77.1   | 68.2   | 97.5    |
| MiroMind-M1-RL-32B       | **77.5** | 65.6   | 96.4    |

Token efficiency metrics establish that, at identical rollout caps, CAMPO-optimized models produce $20$–$30\%$ shorter outputs with equal or better accuracy compared to leading alternatives [2507.14683].

## 6. Generalization Across Modalities and Tasks

The essential CAMPO paradigm, namely staged context progression, context-sensitive penalties, and adaptive group-advantage policy optimization, generalizes across domains:

- **Long-context QA**: In "Context-Picker," a two-stage curriculum (recall-oriented followed by precision-oriented) applies staged redundancy penalties and uses GRPO for stable RL; this delivers superior evidence selection and answer accuracy across five benchmarks [2512.14465].
- **Vision-Language Localization**: The "FOCUS" approach introduces a first-stage attention-constrained objective to ensure visual grounding, followed by RL refinement (GRPO) on IoU-based rewards, enforcing context-sensitive localization that outperforms pure scaling [2605.31145].
- **Multi-agent Multi-hop QA**: In "PRISMA," two-stage GRPO first specializes planning and solving, then a context-aware inspector agent applies observation-augmented residual policy optimization (OARPO) to perform targeted recovery, achieving modularity and state-of-the-art generalization [2601.05465].

A plausible implication is that the distinctive combination of staged objectives and context-aware regularization in CAMPO drives both robustness and efficiency across highly structured generative or reasoning tasks.

## 7. Ablations, Limitations, and Open Directions

Ablations in MiroMind-M1 and related systems show:

- **No-packing vs. packing**: No-packing yields a $2$–$6$ point accuracy increase in SFT.
- **Stage curriculum**: Multi-stage scheduling trains faster and marginally outperforms single-stage models for short budget settings.
- **Repetition penalty**: Omitting yields convergence failure; inclusion is necessary for stability.
- **Verifier quality**: Cascade verifiers reduce noisy reward flips and yield more concise, correct completions.

Current limitations include reliance on consistent, high-quality verifiers, susceptibility to credit assignment issues for highly ambiguous cases despite group normalization, and the need to hand-craft context-sensitive penalties for each modality. Open research problems include formal analysis of convergence under non-stationary stage transitions and extension to unbounded curriculum or dynamic stage adaptation [2507.14683]. 

CAMPO thus represents a modular, extensible framework for robust, efficient, and context-sensitive policy optimization at scale.

Source: https://www.emergentmind.com/topics/context-aware-multi-stage-policy-optimization-campo