REMuL: Faithful Multi-Listener CoT

• REMuL defines faithful chain-of-thought reasoning as a process where multiple listener models verify a speaker's trace by recovering the same final answer from truncated reasoning segments.
• The framework employs a speaker model and a pool of diverse listener models to execute 'soft execution,' ensuring that reasoning traces are logically consistent and causally linked to outcomes.
• By separating the reinforcement learning of trace faithfulness from the supervised correction of final answers, REMuL achieves improved hint attribution, early answering AOC, and mistake injection AOC while retaining or boosting accuracy.

Reasoning Execution by Multiple Listeners (REMuL) is a multi-party reinforcement learning framework for improving the faithfulness of chain-of-thought (CoT) reasoning in LLMs while preserving, and in reported settings improving, task performance. It is motivated by the claim that standard CoT is not reliably faithful: a model may produce a plausible step-by-step explanation that does not reflect the computation that actually determined its answer. REMuL addresses this by treating faithfulness as a multi-party property. A speaker model generates a reasoning trace and final answer, the trace is truncated at multiple points, and a pool of listener models continues from those prefixes. The speaker is rewarded when listeners can recover the same answer from partial traces, a procedure termed “soft execution.” The reported result is that this listener-executability objective improves hint attribution, early answering area over the curve (AOC), and mistake injection AOC, while also improving or maintaining accuracy across BIG-Bench Extra Hard, MuSR, ZebraLogicBench, and FOLIO (Sivakumaran et al., 18 Feb 2026).

1. Conceptual motivation and definition

The central premise of REMuL is that CoT reasoning can be unfaithful even when the final answer is correct. The paper identifies several failure modes already discussed in prior work: explanations can be post-hoc rationalizations rather than genuine decision traces, they can be sensitive to hints or prompt artifacts without mentioning them, they can be insensitive to injected mistakes in the reasoning chain, and they can be internally inconsistent, with later tokens not actually following from earlier ones (Sivakumaran et al., 18 Feb 2026).

Within this framing, faithfulness is not defined as answer correctness alone. Instead, the paper proposes a multi-party notion: a reasoning trace is faithful if another model of comparable capability can follow it and arrive at the same conclusion without seeing the original speaker’s final answer. This is the basis of “soft execution,” where listeners do not deterministically execute code but continue the reasoning in natural language from a truncated prefix.

A common misconception is that faithful reasoning is equivalent to interpretable formatting or to a correct final label. The REMuL formulation rejects that equivalence. It treats faithfulness as a property of whether the reasoning trace can be used by another model to reproduce the answer. This suggests that the paper is concerned less with superficial readability in isolation than with whether the written trace functions as an executable intermediate representation for independent agents.

2. Speaker, listeners, and soft execution

REMuL is built around a speaker model $S$ and a pool of listener models $\mathcal{M}$. The speaker takes input $x$ and generates a reasoning trace $t = \langle t_1, \dots, t_n\rangle$ together with a final answer $a$. The goal is to train the speaker so that its trace is not merely plausible-sounding but usable by other models to recover the same answer (Sivakumaran et al., 18 Feb 2026).

Instead of relying on one judge or verifier, the method uses a pool of diverse listeners. The paper lists Qwen3-14B, Phi-4-reasoning-plus 14B, and Ministral 3 14B Reasoning as listener models. During training, the speaker’s reasoning is split by newline or paragraph boundaries and truncated to 25%, 50%, and 75%, producing prefixes denoted $T' = \{t_1, t_2, t_3\}$. For each listener $L_i \in \mathcal{M}$ and each truncated prefix $t_j \in T'$, the listener continues the partial trace and outputs its own answer $a_j^i$.

The matching reward is defined as:

$r_{match} = \sum_{L_i\in\mathcal{M} \sum_{t_j\in T'} \mathbb{I}\!\left[a_j^i = a\right]$

Although the paper notes that the text formatting of the expression is slightly broken in the snippet, its intended meaning is explicit: the speaker is rewarded for producing traces that cause many listeners, at many truncation points, to recover the same final answer.

The core hypothesis is that if a speaker’s reasoning is genuinely faithful and logically clear, multiple independently trained listeners should be able to continue from the prefix and converge to the same answer. The paper’s ablations later argue that the benefit comes from consensus across heterogeneous listeners rather than from merely increasing the number of rollouts.

3. Optimization strategy and the faithfulness–performance tradeoff

A central claim of the paper is that optimizing for faithfulness and optimizing for correctness can interfere with one another. Methods that make reasoning more interpretable or faithful often reduce accuracy, while methods that optimize only answer correctness can encourage the model to “guess right” and then emit opaque or post-hoc rationalizations. The paper explicitly reports that rewarding correctness and faithfulness together with a single reinforcement learning objective often causes the reward signals to interfere with one another, leading to worse results than training for them separately (Sivakumaran et al., 18 Feb 2026).

To address this, the paper considers two strategies. The first is a balanced reward:

$\mathcal{M}$0

where $\mathcal{M}$1 is the ground-truth label and $\mathcal{M}$2, the number of listeners. This reward is used in GRPO.

The second, and preferred, strategy is a separate correctness recovery stage via masked supervised finetuning on answer tokens only. The loss is

$\mathcal{M}$3

where $\mathcal{M}$4 indexes only the final answer token positions, and tokens belonging to the reasoning trace $\mathcal{M}$5 receive zero loss. The design principle is explicit: faithfulness is learned through reinforcement learning over the reasoning trace, while correctness is restored by answer-only supervised training so that the chain-of-thought is not directly overwritten.

Training proceeds in two phases. In the faithfulness phase with GRPO, for each prompt $\mathcal{M}$6, a group of $\mathcal{M}$7 outputs is sampled from the speaker; each sampled output is truncated at 25%, 50%, and 75%; each truncated prefix is sent to each listener model; $\mathcal{M}$8 or $\mathcal{M}$9 is computed; and Group Relative Policy Optimization (GRPO) is used to increase likelihood of higher-advantage outputs. In the correctness recovery phase, a LoRA adapter is fine-tuned with the masked answer-only SFT loss, and the adapter is merged back into the speaker checkpoint. At test time, only the final speaker model is needed; the listener pool is a training-time resource.

The appendix details the implementation: Qwen3-14B and Phi-4-reasoning-plus are used as speakers; Qwen3, Phi-4, and Ministral 3 are used as listeners; GRPO is implemented via VERL; rollouts are produced via vLLM; the GRPO learning rate is $x$0 with batch size 64, entropy coefficient 0.001, KL coefficient 0.001, 5 rollouts per prompt, and 3 epochs; the LoRA setup uses rank 32, scaling 128, dropout 0.05, learning rate $x$1, and 5 epochs.

4. Evaluation protocol and faithfulness metrics

Training is performed on a subset of Big-Bench Hard (BBH) consisting of five tasks: logical deduction, navigate, temporal, sports understanding, and shuffled objects. The total training set size is 1250 QA pairs. Evaluation is carried out on four multi-step reasoning benchmarks: Big-Bench Extra Hard (BBEH) filtered to multiple-choice with 120 questions, ZebraLogicBench (ZLB) with 3,259 questions, MuSR with 250 murder mystery questions, and FOLIO with 202 questions. These benchmarks span logical deduction, puzzle solving, and first-order logic reasoning (Sivakumaran et al., 18 Feb 2026).

The paper uses three main faithfulness metrics.

First, hint usage / hint attribution follows the hint-injection protocol from Chen et al. A prompt is modified with an explicit hint such as “Hint: I think the answer is Option A …”. If the model changes its answer, the metric checks whether the explanation explicitly attributes the change to the hint. The score is the fraction of changed-answer cases where the model mentions the hint. The implementation uses token-based detection for phrases such as “You said,” “You think,” “I agree,” and “Hint.”

Second, early answering AOC follows Lanham et al. The CoT is truncated at several points, the model is forced to answer from each prefix, and accuracy is measured as truncation varies. Higher AOC means the full reasoning chain is needed before the model can answer correctly; lower AOC suggests the answer is already determined very early and the later reasoning is more post-hoc.

Third, mistake injection AOC also follows Lanham et al. The reasoning is corrupted by inserting errors at various points, and the model continues from the corrupted prefix. Higher AOC means the final answer is more sensitive to injected reasoning mistakes, indicating that the model is actually conditioning on its reasoning. Lower AOC means the model can ignore the corruption and still get the answer, suggesting unfaithful reasoning.

The paper evaluates these AOC metrics at evenly spaced 20% increments along the reasoning chain. Taken together, the three metrics operationalize different aspects of whether a trace is causally important for the answer rather than merely narratively plausible.

5. Empirical findings

The main reported pattern is a three-way contrast among faithfulness-only training, correctness-only training, and REMuL. Faithfulness-only training gives the strongest hint usage but hurts accuracy. Correctness-only training improves accuracy but reduces faithfulness. REMuL improves both, and is presented as achieving the best balance across the reported settings (Sivakumaran et al., 18 Feb 2026).

On Qwen3-14B, REMuL improves hint attribution by up to 3.1% and accuracy by up to 2.1%. On Phi-4-reasoning-plus, hint attribution improves by up to 2.7% and accuracy by up to 3.2%. Averaging across benchmarks, REMuL yields about 2.5% faithfulness gain while also improving or maintaining accuracy. The paper also states that faithfulness-only training improves hint usage by about 4.1% on Qwen3 and 3.7% on Phi4 relative to base, but at an accuracy cost of 3.2% and 2.2%, respectively. REMuL is reported to avoid this collapse.

For the AOC metrics on Qwen3, faithfulness-only training yields the highest AOC, correctness-only is often worst, and REMuL is consistently near the top and better than the base model. The reported averages are an improvement of about 0.033 in Truncated CoT Answering AOC over baseline and about 0.030 in Adding Mistake AOC over baseline. The paper interprets these changes as evidence that REMuL makes the model’s reasoning more causally important for its answers.

The study also compares REMuL with Original, Faithfulness Only, Correctness Only, Balanced Rewards, Hint Optimized, and MAT-Steer. MAT-Steer is described as a strong training-free multi-attribute steering baseline. It improves faithfulness, but generally does not match the training-based gains of REMuL. Direct optimization of the hint metric improves the hint metric itself but does not transfer to better AOC and tends to hurt accuracy. The paper interprets that result as optimization of a narrow symptom rather than of underlying faithfulness.

6. Ablations, auxiliary analyses, and limitations

The paper argues that faithfulness should also improve legibility, understood as how understandable and readable the CoT is. Legibility is evaluated with an autorater, GPT-OSS-20B, on a 0–4 scale following Emmons et al. The reported result is that faithfulness-oriented training improves legibility, that REMuL remains highly legible and better than correctness-only training, and that REMuL average legibility is around 94.1, comparable to the best methods (Sivakumaran et al., 18 Feb 2026).

Ablations show that multi-listener diversity matters. Two replacements are considered: using three copies of the speaker type as listeners, and using one randomly sampled listener. The findings are that using the same model type for listeners reduces faithfulness gains, and using only one listener hurts both faithfulness and accuracy. The stated conclusion is that the benefit comes from consensus across heterogeneous listeners, not merely from adding extra rollouts.

The paper also studies the conflict between correctness and faithfulness rewards. Training with a single combined reinforcement learning reward shows that faithfulness may rise slowly, correctness can stagnate, and the combined objective underperforms separate treatment. This analysis is used to justify the split design of RL for faithfulness and SFT for correctness.

REMuL is further associated with shorter reasoning traces, fewer backtracking statements, and more linear reasoning. For Qwen3 speaker outputs, REMuL reduces token count on most datasets by about 4.88% on average, with one dataset showing a modest increase. Listener generations also get shorter, by about 8.05% on average. Backtracking markers such as “Wait,” “Hold on,” and “Maybe” decrease across all benchmarks. A plausible implication is that listener-executable reasoning is not only more faithful under the chosen metrics but also more concise and less self-revising.

The robustness analysis across training domains is more qualified. In-domain FOLIO training improves accuracy substantially, but does not necessarily improve faithfulness: hint attribution can fall below baseline, and AOC improves only slightly. The paper uses this to argue that faithfulness is not simply a byproduct of domain-matched training and may require the listener-execution objective together with diverse training signals.

Additional appendix results concern calibration and self-awareness. ECE is mostly stable or slightly improved. Faithfulness-oriented training often improves solvability estimation, meaning how well the model predicts whether it can solve a problem. At the same time, REMuL slightly increases sycophancy relative to baseline. The authors note this as a caveat, because some hint-attribution improvement may partly reflect greater deference to hints. This functions as an important limitation: gains in one faithfulness proxy do not eliminate the need to distinguish faithful reasoning from socially compliant behavior.

7. Position within reasoning research

REMuL situates faithful reasoning as a problem of cross-model executability rather than solely of human readability or answer accuracy. In this view, a good CoT is one that multiple listener models can continue from a truncated prefix and still use to recover the speaker’s answer. The method therefore reframes explanation quality as an inter-model property grounded in continuation consistency (Sivakumaran et al., 18 Feb 2026).

This framing also clarifies the paper’s stance on a recurring controversy in LLM reasoning research: whether CoT should be treated as evidence of internal computation or merely as a communicative artifact. The REMuL results do not claim that natural-language reasoning becomes a perfect window into model internals. Instead, they show that multi-listener soft execution can improve several operational measures of faithfulness—hint attribution, early answering AOC, and mistake injection AOC—while maintaining or improving performance, provided correctness and faithfulness are optimized with different mechanisms rather than a single blended reward.

The broader significance of the work lies in the coupling of a training-time social criterion—agreement among heterogeneous listeners under truncation—with a separate answer-only correctness restoration stage. This suggests a research direction in which faithful reasoning is neither equated with raw correctness nor reduced to surface-form constraints. The paper’s empirical summary is correspondingly specific: better hint attribution, better early-answering AOC, better mistake-injection AOC, maintained or improved accuracy, improved legibility, and shorter, cleaner, less backtracking-heavy chains.