ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models

Abstract: We present ThinknCheck, a 1B-parameter verifier for grounded claim verification that first produces a short, structured rationale and then a binary verdict. We construct LLMAggreFact-Think, a 24.1k reasoning-augmented training set derived from LLMAggreFact, and fine-tune a 4-bit Gemma3 model to follow this format. On LLMAggreFact, ThinknCheck attains 78.1 balanced accuracy (BAcc), surpassing MiniCheck-7B (77.4) with 7x fewer parameters; removing the reasoning step reduces BAcc to 57.5. On SciFact, ThinknCheck reaches 64.7 BAcc, a +14.7 absolute gain over MiniCheck-7B. By contrast, zero-shot chain-of-thought on the base Gemma3-1B harms accuracy relative to direct answers, and preference optimization with a simple format+accuracy reward underperforms supervised reasoning. To probe the latter, we introduce GSMClaims and a domain-specialized variant, ThinknCheck-Science, which improves across benchmarks, including 61.0\% accuracy on GSMClaims. Overall, explicit, supervised reasoning enables compact verifiers that are competitive while remaining resource-efficient and interpretable.

• The paper demonstrates that explicit, supervised reasoning significantly improves document-grounded claim verification accuracy.
• It employs a compact 1B-parameter Gemma3-1B model fine-tuned with LoRA adaptation on the curated LLMAggreFact-Think dataset.
• Empirical results reveal that optimal reasoning length and targeted supervision are critical for both in-domain and out-of-domain performance.

ThinknCheck: Compact, Reasoning-Driven Claim Verification at 1B Scale

Overview

The paper, "ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models" (2604.01652), introduces ThinknCheck, a 1B-parameter fine-tuned verifier designed for efficient, interpretable document-grounded claim verification. It leverages explicit, structured reasoning to enhance both predictive accuracy and transparency, while maintaining a resource profile suitable for deployment in compute- or privacy-constrained environments.

A central contribution is LLMAggreFact-Think, a 24.1k instance training set whose construction incorporates high-quality, explicit reasoning traces. This work presents strong empirical results demonstrating that explicit, supervised reasoning with a compact model dramatically improves both in-domain and out-of-domain performance over larger and earlier methods. The paper substantiates this with extensive ablations, cross-benchmark evaluations, and a focused study of rationale length versus model accuracy.

Figure 1: A sample from the LLMAggreFact-Think dataset, illustrating claim verification with cogent, explicit reasoning followed by a verdict.

Methods and Datasets

Explicit Reasoning for Claim Verification

ThinknCheck extends the standard document-grounded claim verification task by implementing a reasoning-annotated generation process: given a $(\text{claim}, \text{document})$ pair, the model first generates a concise rationale, then emits a binary label. This design aims to improve interpretability and force the model to engage in deeper inference, rather than shortcutting through shallow lexical overlap.

Construction of LLMAggreFact-Think

The LLMAggreFact-Think dataset augments the LLMAggreFact benchmark by leveraging zero-shot prompting with GPT-4o-mini to generate step-by-step reasoning for each (document, claim) pair. Filtering for label alignment reduced the pool by ~21%, yielding high-quality rationale+label pairs.

Model Architecture and Variants

ThinknCheck uses a 4-bit quantized Gemma3-1B backbone, fine-tuned on LLMAggreFact-Think with LoRA adaptation. The paper also implements several baselines and ablation models:

• ThinknCheck-nothink: Solution-only variant (no reasoning emitted).
• Gemma3-1B base/CoT: Untuned and CoT-prompted base models.
• Preference-optimized GRPO variants: With reward functions based on output format and accuracy.

Experimental Results

Core Verification and Generalization

On the 29.3k-instance LLMAggreFact test set, ThinknCheck achieves 78.1 BAcc, outperforming MiniCheck-7B (77.4) with 7x fewer parameters, and also exceeding several zero-shot closed models (GPT-4, GPT-4o, Claude-Sonnet-3.5). On SciFact, ThinknCheck yields 64.7 BAcc—a +14.7 point improvement over MiniCheck-7B.

The reasoning step is critical: removing it drops BAcc to 57.5 (LLMAggreFact) and 21.7 (SciFact). Zero-shot CoT on the base model further reduces accuracy, corroborating recent findings that small models do not inherently benefit from uninstructed CoT (Li et al., 17 Feb 2025, Liu et al., 2024).

Performance of preference-optimized (GRPO) variants lags SFT: GRPO from the base reaches 52.6 BAcc, warm-started GRPO peaks at 74.2, both below reasoning-supervised SFT. Manual audits reveal GRPO-learned policies exploit lexical-overlap shortcuts instead of robust evidence synthesis.

Dataset and Domain Generalization

To probe arithmetic and scientific reasoning, the paper introduces GSMClaims (reframed GSM8K problems as claim verification instances) and trains ThinknCheck-Science, a domain-adapted ThinknCheck augmented with SciFact and GSMClaims supervision. ThinknCheck-Science achieves the strongest cross-benchmark performance—61.0% accuracy on GSMClaims (a 17% relative increase over base ThinknCheck-1B), 79.2 BAcc on LLMAggreFact, and 66.4 BAcc on SciFact, highlighting the value of targeted supervision.

Reasoning Length and Performance

A detailed analysis relates output rationale length to balanced accuracy:

Figure 2: ThinknCheck-1B's accuracy as a function of reasoning token length; mid-length rationales maximize BAcc, while both short and excessive chains degrade results in characteristic regimes.

Accuracy forms an inverted U-curve: mid-length rationales most reliably yield correct answers, while short rationales show high recall but poor precision (liberal YES-bias), and long rationales exhibit the opposite (conservative, lower recall). The authors find length itself is not causal; longer rationales correlate with increased instance difficulty and aggregation requirements.

Error Analysis

A unified error taxonomy reveals distinctive domain profiles:

• Lexical Overlap Bias: Most prevalent in general (LLMAggreFact); GRPO amplification reflects shortcut learning.
• Arithmetic Errors: Dominate GSMClaims, underscoring the challenge of numerical inference.
• Overcautiousness and Aggregation Failures: Prominent in SciFact and LLMAggreFact, linked to long rationales and missed synthesis.
• Negation/Temporal Confusions: Significant in SciFact, negligible for math claims.

Figure 3: Error distributions by dataset reveal that lexical overlap is dominant for general claims, arithmetic mistakes for math claims, while aggregation and temporal reasoning are critical in scientific domains.

These results suggest domain-specific augmentation and targeted prompting as priority directions for further mitigation.

Practical and Theoretical Implications

The results decisively demonstrate that explicit, supervised reasoning can close or outperform parameter-mismatched, specialized models and even approach closed LLMs on document-grounded verification, with interpretability and computational advantages. The study advances the thesis that small models, when configured to reason before deciding, are practical for real-world deployment.

The architecture, training procedures, and data—openly licensed—lower the barrier for principled, transparent claim verification in legal, scientific, or safety-critical contexts. Error analyses expose weaknesses in current benchmarks (inadequate testing of synthesis/arithmetic), motivating future dataset design.

Future Directions

The paper proposes:

• Constructing harder, better-balanced datasets requiring multi-sentence evidence aggregation and arithmetic.
• Adding lightweight tool-use (e.g., integrated calculators) to alleviate arithmetic bottlenecks.
• Adapting rationale length dynamically at inference for optimal performance.
• Improving output confidence calibration—a known challenge in compact models (Liu et al., 20 Mar 2025).

Conclusion

ThinknCheck provides compelling empirical evidence for the effectiveness of explicit, supervised reasoning in small verifiers. At 1B parameters, it not only achieves state-of-the-art accuracy on LLMAggreFact but also demonstrates robust out-of-domain generalization, interpretability, and resource efficiency. Its framework, in conjunction with targeted data construction, sets a standard for future research in verifiable, deployable claim verification.

All datasets and models associated with this work are released under an open license, providing valuable resources for further advancement in interpretable, grounded AI systems.