AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models (2511.13029v1)

Abstract: Existing LLM evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.

• The paper introduces the AA-Omniscience benchmark and Omniscience Index to measure LLMs' factual reliability by penalizing hallucinations and rewarding safe abstentions.
• It employs a 6,000-question dataset across 42 topics to assess model accuracy, domain-specific performance, and cost-efficiency trade-offs.
• Results reveal that no frontier model consistently achieves high reliability, highlighting the need for domain-specific evaluation and improved calibration.

AA-Omniscience: A Comprehensive Benchmark for Factuality and Calibration in LLMs

Introduction

The AA-Omniscience benchmark addresses persistent weaknesses in factual recall and knowledge calibration across LLMs. Unlike traditional accuracy-centered evaluations, AA-Omniscience is designed to explicitly penalize hallucinations and reward abstention, measuring reliability through the Omniscience Index (OI), a bounded metric ($[-100, 100]$) that jointly evaluates correctness, calibration, and safe non-response. This work introduces a 6,000-question dataset covering 42 topics within six economically relevant domains, enabling granular model comparisons not just by aggregate performance but by domain-specific reliability.

Figure 1: Omniscience Index results for evaluated LLMs, illustrating the overall distribution and the scarcity of models with positive factual reliability.

Dataset Construction and Domain Coverage

AA-Omniscience’s dataset aggregates questions from authoritative academic and industry sources. Domains include Business, Humanities & Social Sciences, Health, Law, Software Engineering, and Science/Engineering/Mathematics—together representing high economic value and real-world LLM deployment density. The automated question generation agent synthesizes challenging, unambiguous, and context-independent questions, which are then filtered for relevance, ambiguity, and answer uniqueness.

Figure 2: Distribution of the 6,000 question AA-Omniscience dataset across the six domains and 42 categories.

First answerable date bias toward recent facts ensures relevance and reduces legacy knowledge effects that often confound factuality benchmarks.

Figure 3: Distribution of first answerable date by domain, confirming a recency skew and domain-specific temporal granularity.

Evaluation Metrics

Omniscience Index

OI is defined as:

$\text{OI} = 100 \cdot \frac{c - i}{c + p + i + a}$

where $c$ is correct, $i$ is incorrect, $p$ is partially correct, and $a$ is abstention. Abstention when unsure is incentivized over guessing, counteracting the reward structure of existing evaluations that reinforce overconfident hallucination.

Accuracy and Hallucination Rate

Accuracy measures $\frac{c}{c + p + i + a}$, while hallucination rate computes $\frac{i}{p + i + a}$, yielding orthogonal insights into model prediction and calibration behaviors.

Cost-to-run

Token consumption and associated API cost for complete benchmark execution are recorded, enabling practical trade-off analyses between performance and deployment economics.

Experimental Results

Cross-model Performance and Calibration

AA-Omniscience reveals that most models fail to achieve positive reliability; only three attain $OI > 0$. High-accuracy models such as Grok 4 and GPT-5 exhibit excessive hallucination rates (64–81%), which severely degrade their OI.

Figure 4: Relationship between Omniscience Accuracy and Omniscience Index among tested models.

Calibration is critical: Claude 4.1 Opus achieves highest OI (4.8) not due to maximal accuracy but lower hallucination—outperforming models with higher factual recall but poor self-regulation.

Figure 5: Real-world chat application outcome comparing models, showing failure modes of limited embedded knowledge and deficiency in abstention.

Domain Granularity

No evaluated model dominates all domains; reliability is heterogeneous across topic verticals. For example, Claude 4.1 Opus excels in Law, Software Engineering, and Humanities/Social Sciences, whereas GPT-5.1 is optimal for Business, and Grok 4 for Health and STEM fields.

Figure 6: Normalized Omniscience Index across domains, highlighting intra- and inter-model domain performance variation.

This underlines the necessity of domain-specific evaluation and selection when factual reliability is critical; aggregate scores provide insufficient guidance for specialized deployment.

Model Scale, Intelligence, and Knowledge Reliability

Factual accuracy follows a direct correlation with parameter count among open-weight models, but OI does not; scaling alone does not guarantee reduction in hallucination or improvement in reliability.

Figure 7: Accuracy vs. Model Parameters for open-weight models, demonstrating a scaling effect.

Figure 8: Omniscience Index vs. Model Parameters, revealing outlier models with strong calibration at lower scale.

Similarly, overall intelligence (measured by Artificial Analysis Intelligence Index) poorly predicts OI; several smaller or mid-capacity models outperform more generalist, high-intelligence baselines on factual reliability.

Figure 9: Omniscience Index vs. Artificial Analysis Intelligence Index, substantiating poor aggregate intelligence-to-reliability mapping.

Hallucination Analysis

Frontier models generally exhibit high hallucination rates under challenging factuality conditions, which deflates reliability independent of embedded knowledge capacity.

Figure 10: Hallucination Rate distribution across tested models, illustrating calibration failure prevalence.

Cost Efficiency

OI shows a positive correlation with operational cost, though efficiency outliers exist (e.g., Claude 4.5 Haiku), demonstrating that optimal calibration and reliability are not strictly dependent on resource expenditure.

Figure 11: Omniscience Index vs. USD cost to execute benchmark, quantifying cost-performance trade-offs.

Methodological Innovations and Benchmark Comparisons

AA-Omniscience advances existing benchmarks by automating question generation and filtering via LLM agents, facilitating both rapid scaling and continual dataset update. Uniquely, the benchmark offers fine-grained domain-level reporting rather than aggregated metrics.

In contrast to prior work (e.g., LawBench (Fei et al., 2023), FActScore (Min et al., 2023), HalluLens (Bang et al., 24 Apr 2025)), AA-Omniscience uses task-driven, real-world-relevant queries verifiable through authoritative sources, and sharply penalizes incorrect “confident” guesses (2511.13029).

Limitations and Future Directions

AA-Omniscience is currently monolingual (English-centric) and geographically skewed, limiting cross-cultural applicability. The question generation process depends on a single LLM family, introducing potential bias toward evaluation-familiar model architectures. The metric’s abstention reward can temporarily inflate models that safely defer, though empirical attempt rates mitigate this.

Automatic question generation could be diversified via multi-model pipelines, and global multilingual expansion would increase relevance. Modifying the penalty structure for incorrect answers could better reflect risk tolerance in deployment contexts.

Conclusion

AA-Omniscience offers a rigorous multi-domain benchmark for evaluating factuality and calibration in LLMs, balancing correctness, abstention, and hallucination penalties through the Omniscience Index. Key findings show no current frontier model achieves sustained high reliability across all domains. Model choice for knowledge-driven applications must therefore be use-case and domain-specific, rather than guided by aggregate general capability. These results indicate a need for further research into model calibration, domain adaptation, and cost-efficient improvements in factual reliability.

Figure 12: Calibration of results across public and full AA-Omniscience question sets, substantiating transferability for model assessment and comparison.