---
title: 'MARGO: Mixed-Mode Advantage Regularization'
url: https://www.emergentmind.com/topics/margo
type: topic
---

# MARGO: Mixed-Mode Advantage Regularization

Searching arXiv for recent papers on “MARGO” and closely related terms to ground the article.
arXiv search query: "MARGO Mixed-Mode Advantage Regularization for Grounded Optimization"
MARGO most commonly denotes **Mixed-Mode Advantage Regularization for Grounded Optimization**, a reinforcement-learning framework for improving the factual reliability of large reasoning models (LRMs) on knowledge-intensive question answering. It is introduced in the context of factuality-oriented QA, where explicit reasoning traces can improve average performance but can also induce factual drift by overturning correct direct answers. MARGO addresses that failure mode by comparing “thinking” trajectories against “non-thinking” rollouts from the same model on the same question, so that the training signal reflects whether explicit reasoning adds factual value beyond direct answering [2607.05861].

## 1. Conceptual setting and problem definition

MARGO is motivated by what is termed **thinking-induced hallucination**: a model answers correctly in direct-answer mode, but becomes incorrect after being prompted to generate an explicit reasoning trace. The paper studies this phenomenon in factuality-oriented QA with LRMs such as Qwen3 models, each run in two prompted modes. In **thinking mode**, the model generates an explicit reasoning trace and then a final answer. In **non-thinking mode**, the prompt already contains an empty `<think></think>` block, and the model emits the final answer directly [2607.05861].

The instance-level phenomenon is formalized by comparing correctness in the two modes. Let $c^{\mathrm{N}}(x), c^{\mathrm{T}}(x)\in\{0,1\}$ denote correctness for question $x$ under non-thinking and thinking, respectively. The paper defines transition groups
$$
\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},
$$
with transition ratios
$$
r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.
$$
The critical group is $\tau_{1,0}$, where the same model is correct without thinking but wrong after thinking. On TriviaQA, this effect is substantial across Qwen3 scales. For Qwen3-8B, thinking fixes $11.74\%$ of originally wrong non-thinking answers ($r_{0,1}$) but also breaks $7.50\%$ of originally correct ones ($r_{1,0}$). Similar patterns appear on NQ\_Open and PopQA, while on GSM8K the harmful transition remains below $2\%$ [2607.05861].

The paper interprets this as a property of factual QA rather than derivation-heavy reasoning. In factual QA, reasoning does not create new evidence; it can either recover latent knowledge or introduce unsupported associations. This suggests that explicit chain-of-thought is not uniformly beneficial at the instance level, even when it improves average performance.

## 2. Residual formulation of explicit thinking

The central conceptual move in MARGO is to model explicit thinking as a **residual over the model’s direct-answer behavior**. Given question $x$, an output sequence $o=(o_1,\dots,o_T)$ is generated autoregressively as
$$
P_\theta(o \mid x) = \prod_{t=1}^{T} P_\theta(o_t \mid x, o_{<t}).
$$
In thinking mode, the output has the structure
$$
o = [<think>,\, r,\, </think>,\, y],
$$
where $r$ is a non-empty reasoning trace and $y$ is the final answer [2607.05861].

For each mode $m\in\{\mathrm{T},\mathrm{N}\}$, the expected factual reward is
$$
\mu_m(x) = \mathbb{E}_{y^{m}\sim \pi_\theta(\cdot|x,m)} \left[ R(x,y^{m}) \right].
$$
The **residual value of explicit thinking** is then defined as
$$
\Delta_{\mathrm{res}}(x) = \mu_{\mathrm{T}}(x)-\mu_{\mathrm{N}}(x).
$$
If $\Delta_{\mathrm{res}}(x) > 0$, thinking adds factual value; if $\Delta_{\mathrm{res}}(x) < 0$, thinking introduces harmful residual contamination [2607.05861].

This formulation is technically important because standard reasoning RL typically evaluates thinking trajectories against other thinking trajectories. Such within-mode comparison does not determine whether reasoning improves factuality relative to the model’s own direct answer. MARGO instead asks whether the thinking mode is beneficial *relative to the same model’s non-thinking mode on the same input*. That shift in reference point is the method’s defining feature.

## 3. Mixed-mode advantage regularization

MARGO is a **mixed-mode GRPO-style RL framework**. For each question $x$, it constructs a rollout group containing both thinking and non-thinking trajectories:
$$
\mathcal{G}(x) = \{y^{\mathrm{T}}_1,\ldots,y^{\mathrm{T}}_{K_T}\} \cup \{y^{\mathrm{N}}_1,\ldots,y^{\mathrm{N}}_{K_N}\}.
$$
In experiments, the rollout group contains **8 rollouts total per prompt: 6 thinking and 2 non-thinking**, so
$$
\alpha = \frac{K_T}{K_T+K_N} = 0.75.
$$
The expected mixed reward baseline is
$$
\mu_{\mathrm{mix}}(x) = \alpha\mu_{\mathrm{T}}(x) + (1-\alpha)\mu_{\mathrm{N}}(x).
$$
This mixed baseline induces the residual comparison [2607.05861].

For intuition, the expected relative advantage of a thinking trajectory under the mixed baseline is
$$
A_{\mathrm{mix}}^{\mathrm{T}}(x) \propto R(x,y^{\mathrm{T}})-\mu_{\mathrm{mix}}(x)
= \left(R(x,y^{\mathrm{T}})-\mu_{\mathrm{T}}(x)\right) + (1-\alpha)\Delta_{\mathrm{res}}(x).
$$
The appendix gives the full decomposition:
$$
A_{\mathrm{mix}}^{\mathrm{T}}(x,y^{\mathrm{T}}) \propto \left(R(x,y^{\mathrm{T}})-\mu_{\mathrm{T}}(x)\right) + (1-\alpha)\Delta_{\mathrm{res}}(x),
$$
$$
A_{\mathrm{mix}}^{\mathrm{N}}(x,y^{\mathrm{N}}) \propto \left(R(x,y^{\mathrm{N}})-\mu_{\mathrm{N}}(x)\right) - \alpha\Delta_{\mathrm{res}}(x).
$$
Thus, if thinking has negative residual value relative to direct answering, thinking trajectories are relatively penalized; if thinking helps, they are relatively rewarded [2607.05861].

The empirical group-normalized advantage used in training is
$$
A_i = \frac{r_i - \mathrm{mean}(\{r_j\}_{j=1}^{|\mathcal{G}(x)|})}
{\mathrm{std}(\{r_j\}_{j=1}^{|\mathcal{G}(x)|})},
\qquad
\rho_i(\theta)= \frac{\pi_\theta(y_i\mid x,m_i)}
{\pi_{\theta_{\mathrm{old}}}(y_i\mid x,m_i)}.
$$
The clipped policy objective is
$$
\mathcal{J}_{\mathrm{GRPO}}(\theta) =
\mathbb{E}_{x,\mathcal{G}(x)} \left[
\frac{1}{|\mathcal{G}(x)|} \sum_{y_i\in \mathcal{G}(x)}
\min\Big(
\rho_i(\theta)A_i,\,
\mathrm{clip}\big(\rho_i(\theta),1-\epsilon,1+\epsilon\big)A_i
\Big)
\right].
$$
KL regularization with coefficient $0.001$ is also applied to keep the updated policy close to a reference model [2607.05861].

The reward is
$$
r = r_{\mathrm{corr}} + r_{\mathrm{fmt}},
$$
where
$$
r_{\mathrm{corr}} =
\begin{cases}
+1, & \text{if the response is judged correct},\\
-1, & \text{otherwise},
\end{cases}
$$
and
$$
r_{\mathrm{fmt}} =
\begin{cases}
+0.05, & \text{if the response format is valid},\\
0, & \text{otherwise}.
\end{cases}
$$
A format-valid response contains properly closed `<think>` and `</think>` blocks. There is no explicit reward for reasoning quality or length [2607.05861].

## 4. Training pipeline, data construction, and implementation

The training pipeline begins from a base LRM, specifically Qwen3-4B or Qwen3-8B, and constructs model-specific training data from TriviaQA training questions. The paper deduplicates TriviaQA from 138,384 pairs to 76,523 unique questions. For each question and each mode, it samples $N=6$ responses and computes empirical correctness ratios
$$
S_{\mathrm{T}}(x) = \frac{1}{N} \sum_{k=1}^{N} c^{\mathrm{T}}_k(x), \qquad
S_{\mathrm{N}}(x) = \frac{1}{N} \sum_{k=1}^{N} c^{\mathrm{N}}_k(x),
$$
with
$$
b(x) = \frac{S_{\mathrm{T}}(x)+S_{\mathrm{N}}(x)}{2},
$$
$$
A_{\mathrm{T}}(x) = S_{\mathrm{T}}(x)-b(x), \qquad
A_{\mathrm{N}}(x) = S_{\mathrm{N}}(x)-b(x),
$$
and factuality gap
$$
\Delta(x) = S_{\mathrm{T}}(x)-S_{\mathrm{N}}(x).
$$
The method keeps examples with clear mode-dependent factuality gaps [2607.05861].

For **Qwen3-4B**, non-thinking-favored examples satisfy
$$
\Delta(x) \leq -0.7 \quad \text{and} \quad S_{\mathrm{N}}(x) \geq 0.7,
$$
while thinking-favored examples satisfy
$$
\Delta(x) \geq 1.0 \quad \text{and} \quad S_{\mathrm{T}}(x) \geq 1.0.
$$
For **Qwen3-8B**, the corresponding criteria are
$$
\Delta(x) \leq -0.5 \quad \text{and} \quad S_{\mathrm{N}}(x) \geq 0.5,
$$
and
$$
\Delta(x) \geq 0.7 \quad \text{and} \quad S_{\mathrm{T}}(x) \geq 0.7.
$$
This yields **5,660** training examples for Qwen3-4B and **5,721** for Qwen3-8B [2607.05861].

Thinking and non-thinking differ only in prompt template. During evaluation and training, thinking uses temperature $0.6$, top-$p=0.95$, top-$k=20$, while non-thinking uses temperature $0.7$, top-$p=0.8$, top-$k=20$. Maximum generation length is **8192** tokens for factual QA and **32,768** for math evaluation. The judge is **Qwen3-32B**, which receives the question, the gold answer, and the model prediction, and determines whether the prediction is semantically consistent with the gold target [2607.05861].

Optimization uses the **verl** framework on **8 NVIDIA H200 GPUs**, with 4 GPUs for policy training and 4 for reward model serving. Training uses **1 epoch**, **batch size 64**, learning rate **$1\times 10^{-6}$**, **8 total rollouts per prompt**, thinking-ratio coefficient **$\alpha=0.75$**, and KL coefficient **0.001** [2607.05861].

## 5. Empirical behavior, baselines, and limitations

The evaluation covers six factuality-oriented QA datasets—**SimpleQA**, **SimpleQA-Verified**, **TriviaQA**, **NQ\_Open**, **PopQA**, and **HotpotQA**—as well as three mathematical benchmarks—**AMC23**, **AIME24**, and **AIME25**—to test reasoning preservation. Baselines include fixed-mode systems, adaptive mode-selection methods, and fixed-mode RL variants: **Fixed\(_{NoThink}\)**, **Fixed\(_{Think}\)**, **Adaptive\(_{CLS}\)**, **Adaptive\(_{SFT}\)**, **Adaptive\(_{RL}\)**, **Fixed\(_{NoThink+RL}\)**, and **Fixed\(_{Think+RL}\)** [2607.05861].

MARGO achieves the best average factual QA accuracy on all six factuality benchmarks at both model scales. For **Qwen3-4B**, **Fixed\(_{Think}\)** obtains **22.81%** average, **Fixed\(_{Think+RL}\)** obtains **23.31%**, and **MARGO** reaches **25.49%**. For **Qwen3-8B**, the corresponding values are **27.59%**, **27.92%**, and **29.57%**. The gains are especially large on **SimpleQA** and **SimpleQA-V**, where unsupported reasoning can easily induce drift [2607.05861].

The transition analysis is central. Relative to **Fixed\(_{Think}\)**, **Fixed\(_{Think+RL}\)** only marginally changes the transition ratios, whereas **MARGO** consistently reduces $r_{1,0}$ and increases $r_{1,1}$ on **TriviaQA**, **NQ\_Open**, and **PopQA**. The paper illustrates the mechanism with the question “Christmas Island is administered by which country?” In that example, **Fixed\(_{NoThink}\)** answers **Australia**, **Fixed\(_{Think}\)** drifts to **United Kingdom**, and **MARGO** preserves the correct answer [2607.05861].

On mathematical benchmarks, MARGO preserves or slightly improves reasoning ability. Average **Mean@16** rises from **78.0** to **78.5** for Qwen3-4B and from **79.2** to **80.7** for Qwen3-8B. The paper summarizes this as **+0.5%** average gain at 4B and **+1.5%** at 8B over the base thinking mode [2607.05861].

The paper also reports a data-selection ablation. Training on a random subset, rather than carefully selected gap-rich examples, yields much weaker gains. For Qwen3-4B, **MARGO + Random** is only slightly above or near **Fixed\(_{Think}\)**, whereas full **MARGO** is substantially better, including **SimpleQA: 6.73% vs 3.70%**, **SimpleQA-V: 10.00% vs 4.50%**, and **TriviaQA: 49.12% vs 46.14%**. The generation-length analysis further indicates that the method is not merely shortening reasoning: on Qwen3-4B MARGO is slightly shorter than **Fixed\(_{Think}\)**, but on Qwen3-8B it is slightly longer [2607.05861].

The stated limitations are equally specific. The method is evaluated only for factuality-oriented QA and math preservation, not for open-ended generation, multi-turn dialogue, retrieval-augmented systems, or multimodal reasoning. It relies on a large automatic judge, uses model-specific training data construction, draws training data from TriviaQA only, provides no formal convergence theory beyond the advantage decomposition, and does not add new evidence through retrieval or tools. An open question is how the mixed-mode reference should be defined when grounding comes from external evidence rather than the model’s own non-thinking response [2607.05861].

## 6. Terminological ambiguity and unrelated uses of “Margo” and nearby names

The string **MARGO** is not unique across technical literature. In the LRM factuality paper, it is the exact acronym for **Mixed-Mode Advantage Regularization for Grounded Optimization** [2607.05861]. By contrast, in multi-object tracking, the related 2022 paper introduces a **marginal-inference-based data association module** and a tracker called **MTracker**; the paper explicitly does **not** introduce the acronym MARGO, even though the query may informally refer to it that way. That work replaces raw Re-ID distance–based association with a marginal probability over feasible one-to-one assignments and reports about one point improvement in **IDF1** when plugged into existing trackers [2208.03727].

Outside machine learning, **Margo** also denotes **asteroid (1175) Margo**, an outer main-belt asteroid. Photometric observations over **8 nights over 4 months in 2015** support a **precise period near 6.0143 h**, a measured color index $V-R = 0.46 \pm 0.02\ \mathrm{mag}$, and magnitude–phase behavior that the authors interpret as confirming a **moderate-albedo S-type** classification [2105.09763].

Two additional near-matches are sometimes conflated with MARGO but are distinct. **“Margin Play”** is a six-agent MARL system for public-policy analysis in the Brazilian Equatorial Margin; the exact term **MARGO** does not appear in that paper [2606.02614]. **“MARGIN”** is a separate method, **Multi-Agent Runtime Grading via Incremental Normalization**, for online calibration of confidence in multi-agent foundation-model coordination; it is likewise not MARGO, although the spelling is close [2605.22949].

In current arXiv usage, therefore, **MARGO** most precisely refers to the mixed-mode RL framework for suppressing thinking-induced hallucination in LRMs. Other appearances of “Margo” or similar names belong to unrelated domains and should be disambiguated by context.

Source: https://www.emergentmind.com/topics/margo