- The paper introduces an inference-only generator–validator pipeline where heterogeneous language models critique intermediate reasoning and trigger up to three rounds of targeted regeneration.
- The framework achieves 93.56% accuracy on GSM8K with a 20B heterogeneous validator, improving over single-shot systems by up to approximately 13 percentage points and outperforming several reported baselines.
- The results indicate that structured critique contributes more than validator scale, although higher reliability, inference cost, and generalization beyond grade-school arithmetic remain open challenges.
Overview and motivation
Single-model LLM reasoning pipelines remain vulnerable to hallucinations, incoherent chains of thought, and error propagation, where an early arithmetic or logical mistake persists through the remainder of a solution. This paper addresses these weaknesses for mathematical reasoning by combining two design choices: heterogeneous multi-agent collaboration and an explicit critic (validator) that evaluates intermediate reasoning and feeds structured critiques back to the generator. The work is positioned against prior multi-agent systems—collaborative frameworks such as Chain-of-Agents and reflective schemes like COPPER, heterogeneous debate systems (A-HMAD), routing and orchestration approaches (MasRouter, ORCH), and single-model structured reasoning frameworks such as RDoLT. The authors argue that existing systems are either homogeneous, rely on majority voting or debate without verifying intermediate steps, or use heuristic orchestration that cannot detect and repair reasoning errors in real time. The central research question is whether an adaptive, critique-driven mechanism over heterogeneous agents can suppress cascading errors in mathematical problem solving.
Methodology
The framework is a generator–validator pipeline evaluated entirely at inference time on the full GSM8K test set (1,319 examples), with no fine-tuning. The generator is fixed across all experiments as llama-3.1-8b-instant, producing solutions of three to four reasoning steps ending in a parseable final answer. The validator varies across experiments and returns a structured JSON verdict with boolean flags for reasoning-step validity and answer correctness, optionally accompanied by a critique. Correctness is cast as a binary classification requiring both mathematically valid reasoning and a correct final answer, with a programmatic override that strictly compares predicted and gold answers numerically.
In the critic-guided mode, a failed validation triggers regeneration conditioned on the validator's critique, repeated for up to three rounds; a problem is credited as solved if any round satisfies both criteria. Generation uses temperature 0.2 with a 512-token cap; validation uses temperature 0.0 for deterministic judgment. All runs execute through a Groq-hosted, OpenAI-compatible API with retry and fallback strategies, and structured logging (results.jsonl, experiment.json) supports reproducibility.
Experimental design
Eight configurations isolate two factors: validator scale/heterogeneity and the presence of the critic loop. Validators span 8B, 20B, 70B, and 120B parameters, in homogeneous (validator = generator) and heterogeneous settings. Single-shot experiments (1A–2B) serve as the ablated baseline; critic-guided experiments (3A–4B) constitute the full method. The generator, dataset, prompts, and decoding parameters are held constant, so the single-shot versus critique comparison isolates the effect of the feedback loop.
Results
Accuracy ranges from 72.55% to 80.14% without critique and rises to 85.44%–93.56% with critique, an improvement of up to approximately 13 percentage points:
| Setting |
Critic |
Large validator |
Acc (%) |
Small validator |
Acc (%) |
| Homogeneous |
No |
70B (1A) |
80.14 |
8B (2A) |
72.55 |
| Heterogeneous |
No |
120B (1B) |
80.06 |
20B (2B) |
78.85 |
| Homogeneous |
Yes |
70B (3A) |
85.44 |
8B (4A) |
88.63 |
| Heterogeneous |
Yes |
120B (3B) |
92.04 |
20B (4B) |
93.56 |
Three findings stand out. First, validator scale alone yields marginal gains: the 70B and 120B validators achieve nearly identical single-shot accuracy (80.14% vs. 80.06%), and the 8B-to-20B validator upgrade contributes only +6.30% (no critic) and +4.93% (with critic)—far below the +12–16% attributable to critique. Second, the critic effect is the dominant factor: the best critic-guided configuration (20B heterogeneous validator) reaches 93.56%, exceeding the best reported RDoLT result with ChatGPT-4o (90.98%) by +2.58% absolute, and outperforming vanilla CoT (84.7%), CoT-SC (89.4%), and ReAct (90.5%) on GSM8K—all without a proprietary frontier model. Third, retry logs show most problems are solved in round one (73–78%), but critique-driven recovery adds 161–210 problems per configuration, indicating that a large share of failures are recoverable intermediate errors rather than fundamental capability gaps; residual failures drop to 85 in the best configuration.
The scale–critique interaction is notable: without critique, larger validators help substantially in homogeneous settings (+7.59%) but barely in heterogeneous ones (+1.21%); with critique, smaller validators match or exceed larger ones, particularly in heterogeneous configurations. The practical implication is that structured critique plus model heterogeneity can substitute for validator scale, reducing inference cost.
Limitations and open questions
Several caveats temper these conclusions. The evaluation is restricted to GSM8K, a grade-school arithmetic benchmark; generalization to harder mathematical domains (e.g., MATH, olympiad-level problems) is not demonstrated. The generator is fixed to a single 8B model, so the claimed benefits of heterogeneity are established only on the validator side. The comparison against RDoLT and other baselines uses reported figures obtained with different backbone models (ChatGPT-4o), so the cross-framework comparison confounds method with model capability and decoding budget; the critic loop also triples worst-case inference cost per problem, and the paper does not report a cost-normalized comparison. Validator reliability is an acknowledged dependency of critique-based regeneration—an inaccurate critic could reinforce errors—and the paper does not measure validator judgment accuracy separately. Finally, the mechanism by which critique reduces the importance of validator scale is hypothesized rather than mechanistically explained.
Conclusion
This paper presents an inference-only, critic-guided heterogeneous multi-agent framework in which a validator critiques and drives regeneration of failed solutions. On GSM8K, the critique loop delivers a 12–16 percentage-point improvement over single-shot generation, reaching 93.56% accuracy with a 20B validator, and ablations attribute the gains to the feedback mechanism rather than model scale. The results support the claim that structured intermediate-step verification is a more reliable path to accurate mathematical reasoning than validator scaling, though the evidence is confined to grade-school arithmetic and cross-model baseline comparisons.
