Omniscience Index (OI): A Unified Knowledge Metric

• Omniscience Index (OI) is a rigorously defined metric for quantifying knowledge reliability and optimal information exchange across diverse domains.
• It integrates principles from information theory, submodular optimization, and algorithmic techniques to assess factual recall and calibration.
• Applications include benchmarking large language models and optimizing distributed source coding, with improvements in computational efficiency.

The Omniscience Index (OI) is a rigorously defined metric used across disparate research domains to quantify knowledge reliability and optimal information exchange. In modern contexts, OI appears as a normalized factual-recall score for LLMs, while in classical information theory, it denotes the minimum communication rate required for distributed users to collectively recover an entire random source. Its recent adoption as a benchmark for LLMs marks a convergence of statistical, combinatorial, and algorithmic methodologies under a unified metric for knowledge completeness and calibration.

1. Formal Definition: LLM Evaluation

The Omniscience Index for cross-domain factual recall is formally defined using four counts:

• $c$: number of fully correct answers
• $p$: number of partially correct answers
• $i$: number of incorrect answers
• $a$: number of abstentions (questions for which the model refused an answer)

Given $N = c + p + i + a$ as the total number of questions, the OI is computed as

$$\mathrm{OI} = 100 \times \frac{c - i}{c + p + i + a}$$

Correct answers contribute $+1$, incorrect $-1$, partial and abstention $0$ in the numerator, while all categories contribute to the denominator.

The OI is bounded in $[-100, 100]$, with $0$ implying neutral factual reliability—correct and incorrect answers are balanced, or the model abstains entirely. Positive OI denotes more correct than incorrect answers, negative the reverse. Models that abstain when uncertain are implicitly rewarded: abstention enlarges the denominator without direct penalization, while guessing incorrectly is penalized equivalently to a correct answer's reward (Jackson et al., 17 Nov 2025).

2. Methodological Foundations: Submodular Minimization and Information Theory

In distributed source coding, the Omniscience Index (often notated as $\alpha^*$) is the minimum sum-rate required for a group of users, each observing part of a multiple random source, to jointly recover the full source through communication. For a finite user set $V$ and source $Z_V = (Z_i : i \in V)$, achievable rates $r_V$ must satisfy Slepian–Wolf constraints:

$$r(A) \geq H(V) - H(V \setminus A), \quad \forall A \subsetneq V$$

OI is defined via convex optimization:

• Asymptotic:

$$\alpha^* = \min \{ r(V) : r_V \in \mathbb{R}^{|V|}, r(A) \geq H(V) - H(V \setminus A)\}$$

• Non-asymptotic:

$$\alpha^* = \min \{ r(V) : r_V \in \mathbb{Z}^{|V|}, \ldots \}$$

Here, $H(\cdot)$ is the entropy function, which is submodular, permitting tractable solution via submodular function minimization (SFM) (Ding et al., 2019).

3. Algorithmic Approaches to OI Calculation

Algorithmic advances rely on submodular function minimization and partitioning techniques:

• The CoordSatCap algorithm iteratively saturates rate coordinates and merges source blocks via SFM.
• The Parametric (PAR) Algorithm uses strict strong-map properties to compute the principal sequence of partitions (PSP) and critical rates in $O(|V| \cdot \text{SFM}(|V|))$ time, a significant improvement over classical approaches (Ding et al., 2019).

Dilworth truncation is used for hierarchical info-clustering and optimal rate assignment:

$$F_\alpha(V) = \min_{P \in \Pi(V)} \sum_{C \in P} (\alpha - H(V) + H(C)),$$

where $\Pi(V)$ is the set of all partitions of $V$.

4. Scenario Analysis and Benchmarking

The characterization of OI across hypothetical scenarios demonstrates its calibration and penalization properties (LLM context):

Scenario	$c$	$i$	$p$/$a$	OI Value
Always answer, half right	$N/2$	$N/2$	$0$	$0$
Always abstain	$0$	$0$	$N$	$0$
80% perfect coverage	$0.8N$	$0$	$0.2N$	$80$
40% correct, 60% wrong (always guess)	$0.4N$	$0.6N$	$0$	$-20$
Balanced correct and wrong w/ abstain	$0.3N$	$0.3N$	$0.4N$	$0$
Highly conservative, accurate	$0.2N$	$0.05N$	$0.75N$	$15$

These scenarios illustrate that random or overconfident guessing leads to strong penalties, while appropriate abstention fosters higher OI (Jackson et al., 17 Nov 2025).

5. Relation to Classical and Modern Metrics

Traditional metrics such as accuracy ($c / N$) and hallucination rate ($i / (p + i + a)$) fail to capture calibration and abstention. OI synthesizes factual recall, penalization of incorrect responses, and explicit reward for uncertainty recognition, providing a more holistic metric for both LLMs and multi-user information systems.

In communication for omniscience, OI links directly to related problems:

• Secret capacity: $C(V) = H(V) - \alpha^*$, where $\alpha^*$ is the OI.
• Network strength: In PIN-graph models, $\alpha^* = H(V) - 2\sigma(V)$, so efficient calculation of OI yields maximum spanning-tree packing.
• Info-clustering: The PSP, computed during OI minimization, provides a hierarchical clustering according to mutual information (Ding et al., 2019).

6. Practical Impact and Model Benchmarking

Recent frontier models evaluated using AA-Omniscience show that only a minority exhibit positive OI, with the highest reported being Claude 4.1 Opus at 4.8. These models manifest domain-dependent variability and persistent weaknesses in factual accuracy and calibration. The implication is that model selection should be use-case-dependent and not solely based on general task performance when reliable knowledge is critical (Jackson et al., 17 Nov 2025). Differences in OI scores elucidate the value of building models capable of calibrated abstention rather than indiscriminate guessing.

7. Computational Complexity and Advances

Classical algorithms for OI minimization rely on $O(|V|^2 \cdot \text{SFM}(|V|))$ operations, whereas the PAR algorithm leverages parametric SFM for an improved $O(|V| \cdot \text{SFM}(|V|))$ complexity. This advancement underpins scalable OI computation for large-scale info-clustering, network analysis, and source coding problems. References for relevant algorithms include Fleischer–Iwata's push–relabel PSFM [17], Nagano [32], and Iwata–Fleischer–Fujishige [33].

---

In summary, the Omniscience Index constitutes a unified metric for assessing knowledge reliability, optimal information exchange, and the trade-off between coverage and calibration. Its rigorous definition, algorithmic foundations, and practical benchmarking roles position OI as a foundational criterion for both distributed information systems and frontier AI models (Jackson et al., 17 Nov 2025, Ding et al., 2019).