CompassVerifier: LLM Answer Verification Model

• CompassVerifier is a specialized lightweight verifier for LLMs that assesses answer correctness across domains such as mathematics, factual knowledge, and multi-step reasoning.
• It employs advanced augmentation strategies including Complex Formula and Error-Driven Adversarial Augmentation to enhance robustness and accurately match varied answer formats.
• Serving as both an evaluation metric and an RL reward oracle, CompassVerifier drives automated feedback and iterative improvement in diverse LLM tasks.

CompassVerifier is a specialized lightweight verifier model for LLMs that provides both answer verification and outcome reward functionality across mathematics, knowledge, and general reasoning tasks. Developed to address the lack of systematic benchmarks and the limited robustness of existing verification solutions, CompassVerifier utilizes an extensive, manually curated dataset ("VerifierBench") and advanced data augmentation techniques to deliver multi-domain evaluation and serve as a reward model for LLM optimization. Its architecture and methodology are centered around enabling reliable, generalizable verification of varied answer types—including multi-part responses, formulas, and complex sequences—and robust detection of abnormal or invalid outputs (Liu et al., 5 Aug 2025).

1. Objectives and Scope

CompassVerifier's primary function is unified answer verification: systematically assessing the correctness and validity of unstructured LLM outputs by comparing them to ground-truth references. Unlike traditional frameworks that depend heavily on domain-specific regex matching or prompt manipulations, CompassVerifier operates as a stand-alone verifier model trained for generalizability across domains (mathematical, factual, multi-step reasoning) and answer formats (enumerations, formulas, free text, sequences).

A secondary but critical objective is to serve as an explicit reward model in reinforcement learning (RL) settings for LLMs. By generating reliable reward signals based on precise answer assessment, CompassVerifier enables outcome-driven model iteration without dependence on fragile, hand-tuned reward heuristics or prompts.

2. Model Architecture and Methodologies

Model Formulation and Algorithms

CompassVerifier comprises a suite of lightweight verifier models parameterized to different scales (e.g., 3B, 7B, and 32B parameters). Each verifier processes triples of (question, reference answer, model response) and outputs a judgment over three classes: correct, incorrect, or invalid/abnormal response. The modeling leverages both natural language understanding and domain-specific canonicalization (for mathematics and symbolic tasks).

The training regime of CompassVerifier integrates three key augmentation strategies:

1. Complex Formula Augmentation
   • Canonicalizes reference answers using symbolic manipulation.
   • Employs tools such as DeepSeek-V3 to generate multiple equivalent representations of formulas, enhancing the model’s capacity to identify semantically correct responses with notational variance.
2. Error-Driven Adversarial Augmentation
   • Human annotators analyze model failure cases, clustering over 20 observed metaerror types (such as misformatting, premature truncation, or logical inconsistencies).
   • Structured templates and synthetic samples are generated to stress-test the verifier, priming it for edge-case discrimination.
3. Generalizability Augmentation
   • Trains on data augmented by prompt rewriting, structural perturbations, and dynamic answer types, expanding beyond fixed patterns and bolstering resilience against domain and formatting variability.

This leads to a model that not only matches reference answers with high precision but also robustly rejects anomalous or adversarial model outputs.

Mathematical and Reinforcement-Learning Integration

For RL reward modeling, CompassVerifier incorporates an explicit loss function involving normalized advantage estimation and reward normalization. For example, the paper references an expected cost formula:

$C = \mathbb{E}_{(q,a) \sim D, f}[\dots]$

where the loss is designed to balance raw rewards and normalized advantages, integrated seamlessly with reinforcement updating rules.

Canonicalization procedures standardize both numerical precision and the symbolic structure of mathematical responses, accommodating algebraic transformations and notation variants as part of the matching process.

3. VerifierBench Benchmark and Data Resources

VerifierBench is an extensive, multi-domain dataset curated to provide a robust evaluation foundation for CompassVerifier. It includes over 1.3 million triplets (question, reference, response) across mathematics, reasoning, knowledge, and scientific tasks, gathered via the OpenCompass framework. The dataset is iteratively cleaned and annotated using:

• Automated multi-expert voting.
• Verification under multiple prompt templates.
• Final human adjudication.
• Cataloguing >30 metaerror patterns directly informing model augmentation and evaluation.

This comprehensive resource underpins both systematic training and fair, reproducible comparative evaluation of verification models.

4. Empirical Results and Performance Analysis

CompassVerifier demonstrates strong cross-domain performance, substantially outperforming both general-purpose LLMs (e.g., GPT-4.1, Qwen2.5, DeepSeek-V3) and other specialized verifiers (e.g., xVerify, Tencent-Qwen2.5-RLVR).

Key numerical results:

Model Variant	Accuracy (%)	F1 Score (%)
CompassVerifier-32B	84.1 – 95.1	80.8 – 94.8
CompassVerifier-3B	+10.6 over GPT-4.1 (F1)	–
CompassVerifier-7B	+40 (abs F1 over 7B baselines)	–

Detailed ablation indicates that both Complex Formula Augmentation and Error-Driven Adversarial Augmentation independently yield 2–3 percentage point gains in both accuracy and F1, with the combination delivering the highest robustness.

5. Applications and Implications

CompassVerifier is applicable both as an LLM evaluation metric and as an RL reward oracle:

• Automated LLM Evaluation:

Fine-grained differentiation among categories of correctness; strong generalization across changing prompts, question domains, and answer types.

• Reinforcement Learning:

Direct reward feedback for outcome-driven optimization, supporting advanced RL paradigms by eliminating the need for handcrafted evaluation rules.

The system’s architecture and data-driven augmentation suggest a significant reduction in the need for per-domain customization—a key bottleneck of previous methods. A plausible implication is broader applicability to future domains and new answer types as LLM capabilities evolve.

6. Limitations and Prospective Directions

Despite significant performance improvements, CompassVerifier still exhibits limitations in domains such as fine-grained mathematical verification and sequential or highly intricate formula-based answers. The manual enumeration of metaerror classes, though effective, may miss long-tail edge cases.

Future directions outlined in the work include:

• Further refinement of domain-specific checking logic.
• Expansion of error-driven augmentation to capture unobserved or minority error classes.
• Systematic extension to more diverse datasets and answer types.

The codebase and VerifierBench are publicly available (https://github.com/open-compass/CompassVerifier), inviting further research and benchmarking by the community.

7. Significance in the LLM Evaluation Ecosystem

CompassVerifier represents a transition from ad-hoc, handcrafted answer verification toward systematically benchmarked, augmentation-driven, and reward-model-integrated solutions for LLM-centric NLP pipelines. By explicitly targeting multi-format responses, abnormal/invalid output detection, and robust mathematical reasoning, it addresses key gaps in the present verification methodology landscape, providing a foundation for both reliable evaluation and RL-driven LLM improvement (Liu et al., 5 Aug 2025).