BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

Abstract: Accurate evaluation is central to the LLM ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model's true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judgments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthetically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling tradeoff between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge's performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption.

• The paper demonstrates that BERT-as-a-Judge significantly improves evaluation accuracy, achieving near-perfect scores on multiple-choice benchmarks.
• It employs an encoder-based framework that is robust against variations in answer formatting and effectively handles domain shifts.
• The approach reduces computational overhead and better aligns model rankings with human judgments compared to traditional regex-based methods.

Efficient and Robust Evaluation of LLMs: BERT-as-a-Judge

Motivation and Lexical Evaluation Limitations

The assessment of LLMs in zero-shot generative settings relies fundamentally on reference-based evaluation, which must robustly extract and judge answers regardless of output formatting. The prevalent reliance on regex-based (lexical) extraction and comparison conflates problem-solving ability with formatting compliance—a confounding that systematically distorts both accuracy and rankings across model families and task types. Empirical analysis reveals weak correlation between regex-based evaluation and human judgments, with parsing failure rates varying significantly with model scale, task category, and family. For example, context extraction and open-form mathematics tasks are particularly susceptible to answer formatting inconsistencies, resulting in parsing failures exceeding 20–60% for certain models (Figure 1).

Figure 1: Regex parsing failures quantified across Llama-3, Gemma-3, and Qwen-3 model families, highlighting the dependence on model scale and task.

Performance degradation impacts both absolute accuracy and leaderboard positioning; models with superior reasoning often underperform in rankings due to trivial violations of formatting heuristics. Even in high-compliance cases (e.g., Gemma-3 on context extraction), verbose outputs that technically adhere to instructions but diverge lexically from references yield substantial negative accuracy deltas.

BERT-as-a-Judge: Encoder-Based Evaluation Framework

To address these limitations, the authors introduce BERT-as-a-Judge, an encoder-based approach that leverages bidirectional attention for structured text classification. The method trains a EuroBERT 210M encoder on 1M synthetically labeled question-candidate-reference triplets spanning multiple-choice (e.g., MMLU, ARC), context extraction (e.g., SQuAD-v2, HotpotQA), and open-form mathematics tasks (e.g., GSM8K, MATH). The encoder consistently achieves higher assessment accuracy—near-perfect on multiple-choice (99.7% ARC-Easy, 98.5% MMLU), and robust (>93%) on mathematics and context extraction benchmarks—aligning closely with synthetic and human judgments, and outperforming both regex and small-scale LLM-as-a-Judge baselines.

Figure 2: Comparison of regex-based and BERT-as-a-Judge evaluation; the bottom pane shows altered model rankings due to rigid lexical heuristics.

The framework is specifically robust to variations in answer phrasing and formatting guidelines. For out-of-domain tasks and novel model generations (i.e., models excluded from the training mixture), BERT-as-a-Judge maintains high accuracy, evidencing strong generalization capacity.

Comparative Analysis: Encoders vs. Generative Judges

Direct comparison with LLM-as-a-Judge approaches reveals a pronounced computational and accuracy gap. Small generative models (e.g., Qwen-3 0.6B) underperform even regex baselines (<60% on most tasks), highlighting the necessity of scale for generative evaluation. Scaling judge models and inference budgets only closes the gap at the cost of substantial inference FLOPs (Figure 3).

Figure 3: BERT-as-a-Judge vs. LLM judges (Qwen-3 and Gemma-3 families) across model sizes and inference modes, demonstrating the efficiency and reliability of encoder-based assessment.

The encoder's inference latency ($\approx$200 ms on commodity hardware) and training efficiency (high accuracy with only 100K samples) emphasizes its practical suitability. The method reliably separates correctness classes, reducing the need for task-specific threshold tuning (Figure 4), and offers robustness to free-form answer formats and omitted contextual prompts, maintaining substantially better results than lexical tools in all tested configurations.

Figure 4: BERT-as-a-Judge accuracy as a function of decision threshold values, showing stability and strong class separation.

Theoretical and Practical Implications

The findings challenge the reliance on surface-level format adherence for model evaluation, providing strong evidence that rule-based lexical assessment can systematically undervalue true problem-solving abilities. The encoder-based approach offers a scalable, low-cost, and semantically robust protocol that closely approximates expert human judgment. Its robustness to domain shift—both in tasks and model families—makes it immediately applicable for rapid assessment in open research and deployment pipelines.

Practically, the method addresses two central needs: (1) reducing computational load in large-scale benchmark evaluations as the model zoo expands, and (2) enabling reproducible, fair, and accessible evaluation that does not penalize models for stylistic or prompt-derived artifacts. The generalization to free-form and multimodal scenarios is feasible via encoder architectures, suggesting extensibility toward vision and speech tasks or cross-modal reference-based evaluation.

Directions for Future Work

The scope of BERT-as-a-Judge is currently confined to fact-based, English-language benchmarks with objectively verifiable answers. Extending encoder-based evaluation to open-ended generation tasks (summarization, code, translation) and multilingual and multimodal contexts presents a promising avenue for developing unified evaluation frameworks. Direct application or adaptation of learned metrics (e.g., InfoLM, BLEURT, COMET) may enhance semantic fidelity in less strictly referential tasks. Further investigation into scalability for extreme context lengths and cross-modal conditioning is warranted as multimodal foundation models proliferate.

Conclusion

BERT-as-a-Judge provides a robust, efficient, encoder-driven protocol for reference-based LLM evaluation that resolves the conflation of reasoning and formatting compliance inherent to regex-based methods. It achieves superior accuracy and generalization across tasks and models, enabling reliable, reproducible, and scalable leaderboard assessments without incurring the inference overheads associated with LLM-as-a-Judge frameworks. The approach offers an immediate path toward fairer and more semantically valid evaluation protocols—critical for future advances in foundation model benchmarking and deployment (2604.09497).