MARGO: Mixed-Mode Advantage Regularization

• MARGO is a reinforcement learning framework that reduces thinking-induced hallucination by comparing explicit reasoning and direct-answer modes in factual QA.
• It models explicit thinking as a residual over direct-answer behavior, using a mixed-mode rollout group to dynamically adjust rewards and suppress factual drift.
• Empirical results show that MARGO improves accuracy on benchmarks like TriviaQA by reducing harmful transitions while preserving reasoning quality.

Searching arXiv for papers on “MARGO” and closely related terms to ground the article. arXiv search query: "MARGO Mixed-Mode Advantage Regularization for Grounded Optimization" MARGO most commonly denotes Mixed-Mode Advantage Regularization for Grounded Optimization, a reinforcement-learning framework for improving the factual reliability of large reasoning models (LRMs) on knowledge-intensive question answering. It is introduced in the context of factuality-oriented QA, where explicit reasoning traces can improve average performance but can also induce factual drift by overturning correct direct answers. MARGO addresses that failure mode by comparing “thinking” trajectories against “non-thinking” rollouts from the same model on the same question, so that the training signal reflects whether explicit reasoning adds factual value beyond direct answering (Wang et al., 7 Jul 2026).

1. Conceptual setting and problem definition

MARGO is motivated by what is termed thinking-induced hallucination: a model answers correctly in direct-answer mode, but becomes incorrect after being prompted to generate an explicit reasoning trace. The paper studies this phenomenon in factuality-oriented QA with LRMs such as Qwen3 models, each run in two prompted modes. In thinking mode, the model generates an explicit reasoning trace and then a final answer. In non-thinking mode, the prompt already contains an empty > </think> block, and the model emits the final answer directly (Wang et al., 7 Jul 2026).

The instance-level phenomenon is formalized by comparing correctness in the two modes. Let $c^{\mathrm{N}}(x), c^{\mathrm{T}}(x)\in\{0,1\}$ denote correctness for question $x$ under non-thinking and thinking, respectively. The paper defines transition groups

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$

with transition ratios

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$

The critical group is $\tau_{1,0}$, where the same model is correct without thinking but wrong after thinking. On TriviaQA, this effect is substantial across Qwen3 scales. For Qwen3-8B, thinking fixes $11.74\%$ of originally wrong non-thinking answers ($r_{0,1}$) but also breaks $7.50\%$ of originally correct ones ($r_{1,0}$). Similar patterns appear on NQ_Open and PopQA, while on GSM8K the harmful transition remains below $2\%$ (Wang et al., 7 Jul 2026).

The paper interprets this as a property of factual QA rather than derivation-heavy reasoning. In factual QA, reasoning does not create new evidence; it can either recover latent knowledge or introduce unsupported associations. This suggests that explicit chain-of-thought is not uniformly beneficial at the instance level, even when it improves average performance.

2. Residual formulation of explicit thinking

The central conceptual move in MARGO is to model explicit thinking as a residual over the model’s direct-answer behavior. Given question $x$0, an output sequence $x$1 is generated autoregressively as

$x$2

In thinking mode, the output has the structure

$x$3

where $x$4 is a non-empty reasoning trace and $x$5 is the final answer (Wang et al., 7 Jul 2026).

For each mode $x$6, the expected factual reward is

$x$7

The residual value of explicit thinking is then defined as

$x$8

If $x$9, thinking adds factual value; if $$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$0, thinking introduces harmful residual contamination (Wang et al., 7 Jul 2026).

This formulation is technically important because standard reasoning RL typically evaluates thinking trajectories against other thinking trajectories. Such within-mode comparison does not determine whether reasoning improves factuality relative to the model’s own direct answer. MARGO instead asks whether the thinking mode is beneficial relative to the same model’s non-thinking mode on the same input. That shift in reference point is the method’s defining feature.

3. Mixed-mode advantage regularization

MARGO is a mixed-mode GRPO-style RL framework. For each question $$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$1, it constructs a rollout group containing both thinking and non-thinking trajectories:

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$2

In experiments, the rollout group contains 8 rollouts total per prompt: 6 thinking and 2 non-thinking, so

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$3

The expected mixed reward baseline is

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$4

This mixed baseline induces the residual comparison (Wang et al., 7 Jul 2026).

For intuition, the expected relative advantage of a thinking trajectory under the mixed baseline is

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$5

The appendix gives the full decomposition:

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$6

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$7

Thus, if thinking has negative residual value relative to direct answering, thinking trajectories are relatively penalized; if thinking helps, they are relatively rewarded (Wang et al., 7 Jul 2026).

The empirical group-normalized advantage used in training is

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$8

The clipped policy objective is

$$\tau_{i,j}=\{x \mid c^{\mathrm{N}}(x)=i,\ c^{\mathrm{T}}(x)=j\}, \quad i,j\in\{0,1\},$$9

KL regularization with coefficient $r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$0 is also applied to keep the updated policy close to a reference model (Wang et al., 7 Jul 2026).

The reward is

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$1

where

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$2

and

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$3

A format-valid response contains properly closed <think> and `` blocks. There is no explicit reward for reasoning quality or length (Wang et al., 7 Jul 2026).

4. Training pipeline, data construction, and implementation

The training pipeline begins from a base LRM, specifically Qwen3-4B or Qwen3-8B, and constructs model-specific training data from TriviaQA training questions. The paper deduplicates TriviaQA from 138,384 pairs to 76,523 unique questions. For each question and each mode, it samples $r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$4 responses and computes empirical correctness ratios

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$5

with

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$6

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$7

and factuality gap

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$8

The method keeps examples with clear mode-dependent factuality gaps (Wang et al., 7 Jul 2026).

For Qwen3-4B, non-thinking-favored examples satisfy

$r_{i,j} = \frac{|\tau_{i,j}|}{|\mathcal{D}|}.$9

while thinking-favored examples satisfy

$\tau_{1,0}$0

For Qwen3-8B, the corresponding criteria are

$\tau_{1,0}$1

and

$\tau_{1,0}$2

This yields 5,660 training examples for Qwen3-4B and 5,721 for Qwen3-8B (Wang et al., 7 Jul 2026).

Thinking and non-thinking differ only in prompt template. During evaluation and training, thinking uses temperature $\tau_{1,0}$3, top-$\tau_{1,0}$4, top-$\tau_{1,0}$5, while non-thinking uses temperature $\tau_{1,0}$6, top-$\tau_{1,0}$7, top-$\tau_{1,0}$8. Maximum generation length is 8192 tokens for factual QA and 32,768 for math evaluation. The judge is Qwen3-32B, which receives the question, the gold answer, and the model prediction, and determines whether the prediction is semantically consistent with the gold target (Wang et al., 7 Jul 2026).

Optimization uses the verl framework on 8 NVIDIA H200 GPUs, with 4 GPUs for policy training and 4 for reward model serving. Training uses 1 epoch, batch size 64, learning rate $\tau_{1,0}$9, 8 total rollouts per prompt, thinking-ratio coefficient $11.74\%$0, and KL coefficient 0.001 (Wang et al., 7 Jul 2026).

5. Empirical behavior, baselines, and limitations

The evaluation covers six factuality-oriented QA datasets—SimpleQA, SimpleQA-Verified, TriviaQA, NQ_Open, PopQA, and HotpotQA—as well as three mathematical benchmarks—AMC23, AIME24, and AIME25—to test reasoning preservation. Baselines include fixed-mode systems, adaptive mode-selection methods, and fixed-mode RL variants: Fixed$11.74\%$1, Fixed$11.74\%$2, Adaptive$11.74\%$3, Adaptive$11.74\%$4, Adaptive$11.74\%$5, Fixed$11.74\%$6, and Fixed$11.74\%$7 (Wang et al., 7 Jul 2026).

MARGO achieves the best average factual QA accuracy on all six factuality benchmarks at both model scales. For Qwen3-4B, Fixed$11.74\%$8 obtains 22.81% average, Fixed$11.74\%$9 obtains 23.31%, and MARGO reaches 25.49%. For Qwen3-8B, the corresponding values are 27.59%, 27.92%, and 29.57%. The gains are especially large on SimpleQA and SimpleQA-V, where unsupported reasoning can easily induce drift (Wang et al., 7 Jul 2026).

The transition analysis is central. Relative to Fixed$r_{0,1}$0, Fixed$r_{0,1}$1 only marginally changes the transition ratios, whereas MARGO consistently reduces $r_{0,1}$2 and increases $r_{0,1}$3 on TriviaQA, NQ_Open, and PopQA. The paper illustrates the mechanism with the question “Christmas Island is administered by which country?” In that example, Fixed$r_{0,1}$4 answers Australia, Fixed$r_{0,1}$5 drifts to United Kingdom, and MARGO preserves the correct answer (Wang et al., 7 Jul 2026).

On mathematical benchmarks, MARGO preserves or slightly improves reasoning ability. Average Mean@16 rises from 78.0 to 78.5 for Qwen3-4B and from 79.2 to 80.7 for Qwen3-8B. The paper summarizes this as +0.5% average gain at 4B and +1.5% at 8B over the base thinking mode (Wang et al., 7 Jul 2026).

The paper also reports a data-selection ablation. Training on a random subset, rather than carefully selected gap-rich examples, yields much weaker gains. For Qwen3-4B, MARGO + Random is only slightly above or near Fixed$r_{0,1}$6, whereas full MARGO is substantially better, including SimpleQA: 6.73% vs 3.70%, SimpleQA-V: 10.00% vs 4.50%, and TriviaQA: 49.12% vs 46.14%. The generation-length analysis further indicates that the method is not merely shortening reasoning: on Qwen3-4B MARGO is slightly shorter than Fixed$r_{0,1}$7, but on Qwen3-8B it is slightly longer (Wang et al., 7 Jul 2026).

The stated limitations are equally specific. The method is evaluated only for factuality-oriented QA and math preservation, not for open-ended generation, multi-turn dialogue, retrieval-augmented systems, or multimodal reasoning. It relies on a large automatic judge, uses model-specific training data construction, draws training data from TriviaQA only, provides no formal convergence theory beyond the advantage decomposition, and does not add new evidence through retrieval or tools. An open question is how the mixed-mode reference should be defined when grounding comes from external evidence rather than the model’s own non-thinking response (Wang et al., 7 Jul 2026).

6. Terminological ambiguity and unrelated uses of “Margo” and nearby names

The string MARGO is not unique across technical literature. In the LRM factuality paper, it is the exact acronym for Mixed-Mode Advantage Regularization for Grounded Optimization (Wang et al., 7 Jul 2026). By contrast, in multi-object tracking, the related 2022 paper introduces a marginal-inference-based data association module and a tracker called MTracker; the paper explicitly does not introduce the acronym MARGO, even though the query may informally refer to it that way. That work replaces raw Re-ID distance–based association with a marginal probability over feasible one-to-one assignments and reports about one point improvement in IDF1 when plugged into existing trackers (Zhang et al., 2022).

Outside machine learning, Margo also denotes asteroid (1175) Margo, an outer main-belt asteroid. Photometric observations over 8 nights over 4 months in 2015 support a precise period near 6.0143 h, a measured color index $r_{0,1}$8, and magnitude–phase behavior that the authors interpret as confirming a moderate-albedo S-type classification (Shevchenko et al., 2021).

Two additional near-matches are sometimes conflated with MARGO but are distinct. “Margin Play” is a six-agent MARL system for public-policy analysis in the Brazilian Equatorial Margin; the exact term MARGO does not appear in that paper (Filho et al., 26 May 2026). “MARGIN” is a separate method, Multi-Agent Runtime Grading via Incremental Normalization, for online calibration of confidence in multi-agent foundation-model coordination; it is likewise not MARGO, although the spelling is close (Armstrong, 21 May 2026).

In current arXiv usage, therefore, MARGO most precisely refers to the mixed-mode RL framework for suppressing thinking-induced hallucination in LRMs. Other appearances of “Margo” or similar names belong to unrelated domains and should be disambiguated by context.