AuthorityBench: Dual LLM Authority Evaluations
- AuthorityBench is a dual benchmark project that defines authority through citation signals and source reliability, addressing distinct evaluation challenges in LLM behavior.
- The citation-based benchmark uses a balanced 2x2 design over claim and citation veracity to quantify hallucination rates across multiple domains.
- The RAG-oriented benchmark evaluates models' ability to discern domain and entity authority, demonstrating that authority-guided filtering improves retrieval accuracy.
AuthorityBench is the shared title of two distinct 2026 benchmark projects that study “authority” in LLMs under different operational definitions. One benchmark, introduced in “Authority, Truth, and Citation Bias: A Large-Scale Multi-Domain Benchmark for Studying Epistemic Susceptibility in LLMs,” isolates how citation-based authority signals alter model epistemic behavior independently of factual content, using a 220,564-prompt fully balanced design over claim veracity and citation veracity across general knowledge, science, law, and medicine (Khurana et al., 11 Jun 2026). The other, introduced in “AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation,” evaluates whether models can perceive information authority in retrieval settings, using domain-level and entity-level authority proxies together with a downstream RAG task (Yao et al., 26 Mar 2026). This suggests that the same label is being used for two non-equivalent research programs: one concerns susceptibility to authoritative-looking citations, while the other concerns discrimination of authoritative sources.
1. Dual meaning of “AuthorityBench”
The two benchmarks differ in what counts as authority, what is being measured, and what failure mode is central.
| Benchmark | Authority notion | Core evaluation object |
|---|---|---|
| “Authority, Truth, and Citation Bias” (Khurana et al., 11 Jun 2026) | Citation-based authority signals | Hallucination and false denial under citation presence |
| “Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation” (Yao et al., 26 Mar 2026) | Source authority and entity authority | Correlation with authority labels and downstream RAG accuracy |
In the citation-focused benchmark, the question is whether a model treats a citation as evidence to evaluate or as authority to defer to. In the RAG-focused benchmark, the question is whether a model can distinguish high-authority from lower-authority sources or entities, beyond semantic relevance alone. The shared name is therefore taxonomically ambiguous, but the two projects are complementary in one respect: both treat authority as a distinct variable rather than collapsing it into ordinary correctness or relevance.
2. Citation-based AuthorityBench: benchmark design and measurement
The citation-focused AuthorityBench is a 220,564-prompt multi-domain benchmark built to isolate citation-based authority effects from ordinary factual difficulty (Khurana et al., 11 Jun 2026). Its core design is a fully balanced factorial crossing claim veracity with citation veracity, yielding four main conditions: true claim + real citation, true claim + fabricated citation, false claim + real citation, and false claim + fabricated citation. Each condition is balanced to about 55,000 prompts, and each base claim also appears in a no-citation baseline.
The benchmark spans four domains: general knowledge, science, law, and medicine. It uses 40 prompt templates across seven structural categories—prefix, mid-sentence, suffix, minimum salience, venue-first, author-first declarative, and footnote-style. It also varies venue prestige across four tiers using 480 venues in total, and for fabricated citations it uses a country-coded author name dataset. The design further includes controlled same-domain versus cross-domain citation placement and citation years spanning roughly 1980 to the present.
Seven models are evaluated: Gemma 3 4B, Llama 3.1 8B Instruct, Phi-4 Mini Instruct, Gemma 4 31B, Claude Haiku 4.5, GPT 5.4 mini, and DeepSeek V3.2. Three models are run on the full dataset, while four are evaluated on a balanced 15K subset for cost or access reasons. The study is organized around 12 structured research questions covering citation authority effects, override of internal knowledge, false-citation-induced denial of true claims, prestige effects, domain sensitivity, structural prompt effects, author demographics, temporal framing, and cross-model comparisons.
Its primary metric is hallucination rate per condition, defined as the proportion of outputs labeled hallucinated over all non-refused responses; refusals were below 2% and excluded from the denominator. The paper also reports “lift,” meaning absolute percentage-point difference from the relevant baseline, and Cohen’s , with confidence intervals from a normal approximation to the binomial. The no-citation baseline is central because the benchmark asks not only whether responses are wrong, but whether citation presence itself increases hallucination.
3. Citation-based AuthorityBench: principal findings
Across all seven models, citation presence, whether real or fabricated, increases hallucination relative to the no-citation baseline (Khurana et al., 11 Jun 2026). The strongest effect occurs in the true-claim + fabricated-citation condition, denoted TCFC in the paper, which is the worst-performing condition in every model tested. Fabricated citations paired with true claims raise hallucination by 3 to 22 percentage points over the true-claim baseline. Reported TCFC lifts include pp for Llama 3.1 8B, pp for GPT 5.4 mini, pp for Claude Haiku 4.5, and pp for Gemma 4 31B.
The general knowledge domain is the most vulnerable. Under TCFC, hallucination rates in that domain reach 35% to 77%, including 0 for Llama, 1 for Claude, and 2 for GPT 5.4 mini. A central interpretation offered in the paper is that citation pressure can push models into near-ceiling hallucination even when the underlying claim is true, indicating a failure of epistemic reasoning under authority pressure rather than a simple lack of factual knowledge.
The legal domain is comparatively robust. Across models, legal claims tend to show the smallest citation-induced lifts, often non-significant, and same-domain legal citations are the least disruptive source of citation pressure. By contrast, medicine and science vary more by model and baseline headroom. Venue prestige has negligible impact: hallucination rates are essentially flat across prestige levels, with no monotonic gradient. Author demographics also have negligible impact: country-coded surname regions produce only tiny spreads, typically around 0.7 to 1.9 percentage points in the full-dataset models. Model size does not guarantee robustness; some larger or more capable models are among the most susceptible.
The benchmark’s broader conclusion is that LLMs are epistemically susceptible to citation authority. They often defer to the mere appearance of source backing, even when the citation is fabricated and even when the claim is actually true. The no-citation comparison is therefore not a minor control but the mechanism by which citation presence itself becomes measurable as a source of epistemic distortion.
4. RAG-oriented AuthorityBench: datasets, authority proxies, and judging protocols
The RAG-oriented AuthorityBench evaluates whether LLMs can perceive information authority rather than only semantic relevance (Yao et al., 26 Mar 2026). It contains three datasets. DomainAuth has 10K web domains and uses PageRank-based authority as the domain-level authority proxy. EntityAuth has 22K entities across basketball players, movies, and songs and uses Wikipedia sitelink counts as the popularity-based authority proxy. RAGAuth has 120 yes/no queries, each paired with 10 retrieved documents of varying authority, for downstream evaluation.
The benchmark explicitly distinguishes source authority from entity authority. Source authority is publisher- or domain-level reliability and influence, operationalized by PageRank. Entity authority is recognition or prominence of a person, institution, or similar object, operationalized by Wikipedia sitelink counts. For DomainAuth, Google Toolbar PageRank scores are used as 0–9 labels, with score 10 excluded because it is extremely rare; the paper also provides a coarse 5-level mapping. For EntityAuth, the scoring binning is defined as
3
Five LLMs are evaluated: Qwen3-8B, Qwen3-14B, Qwen3-32B, Llama-3-8B-Instruct, and Llama-3.1-8B-Instruct. Temperature is set to 0, and Qwen3’s thinking function is disabled for fairness. The benchmark compares three judging paradigms. PointJudge scores one item at a time. PairJudge takes two items and either directly ranks them or assigns scores to both. ListJudge evaluates an entire list and either outputs a ranking or assigns scores and sorts by score. Each judge is tested both without text, using only the domain or entity name, and with text, including webpage text snippets.
The evaluation uses Spearman’s 4 and Kendall’s 5 for listwise correlation and paired-preference accuracy for pairwise evaluation. The paper distinguishes direct ranking outputs from PointScore outputs and further distinguishes AverageScore from BubbleSort when deriving rankings from PairJudge scores. In the appendix, the main-paper use of PointScore refers to the point-based averaging method.
5. RAG-oriented AuthorityBench: empirical findings and downstream implications
Across DomainAuth and EntityAuth, the strongest methods are ListJudge + PointScore and PairJudge + PointScore (Yao et al., 26 Mar 2026). PairJudge + PointScore often achieves the best raw correlation with ground-truth authority, but PairJudge has the highest inference cost. ListJudge is nearly as strong and is identified as the best trade-off between performance and cost. PointJudge is usually weaker than the other two.
A central result is that adding webpage text usually degrades authority judgment performance, especially for ListJudge and PairJudge. The paper treats this as evidence that authority is distinct from textual style, fluency, or narrative richness. When only the domain is shown, the model relies on structural or source cues; when text is added, the model can become distracted by verbosity, polish, or persuasive style. There is one stated exception: in hard pairwise cases where authority differences are small, text can help because it provides extra evidence when structural cues are ambiguous.
The downstream RAG experiments test a simple pipeline: score or filter retrieved documents, keep top-6, and generate the final yes/no answer. The benchmark compares no filtering, relevance filtering, utility filtering, and authority filtering, all using ListJudge + PointScore for fair comparison. Authority-guided filtering gives the best answer accuracy across models and 7. Reported examples include Qwen3-8B improving from 53.33 to 66.67 / 69.16, Qwen3-14B from 58.33 to 76.67 / 75.00 / 73.33, Qwen3-32B from 55.00 to 70.00, Llama-3-8B from 50.83 to 64.17, and Llama-3.1-8B from 57.50 to 71.76.
The practical recommendation is therefore not merely to retrieve semantically relevant documents, but to filter by authority as well. In this benchmark, authority is treated as a content-agnostic property of the source rather than a property of the writing style. The practical significance lies in the claim that authority-aware retrieval improves final answer accuracy and reduces misinformation risk.
6. Adjacent benchmarks and the wider authority-evaluation landscape
The two AuthorityBench projects sit within a broader cluster of benchmarks that treat authority, authorization, citation reliability, or governance as first-class evaluation targets. LegalCiteBench studies closed-book citation recovery, citation verification, and case matching in legal LLMs; across 21 LLMs, no model exceeds 7/100 on citation retrieval or completion, and Misleading Answer Rate exceeds 94% for 20 of 21 evaluated models on retrieval-heavy tasks (Chen et al., 11 May 2026). OrgAccess evaluates whether LLMs can reason about role-based access control and organizational hierarchy constraints; on its hardest split, even GPT-4.1 only achieves an F1-Score of 0.27 (Sanyal et al., 25 May 2025).
Partial Evidence Bench measures authorization-limited evidence in enterprise agents rather than direct permission breaches. It reports that silent filtering is catastrophically unsafe across all shipped families, with unsafe completeness rate 8 for the silent_filter baseline and 9 for fail-and-report (Tallam, 6 May 2026). AgentSecBench formalizes instruction-integrity, retrieval-confidentiality, and capability-integrity under a noninterference-style framework and argues that prompt annotations are descriptive, whereas provenance projection, capability restriction, and output validation can enforce a boundary (Alpay et al., 25 May 2026). AgentBound extends this logic to runtime governance by composing delegated authorization, owner-signed behavioral constitutions, and site action contracts, and by producing cryptographically verifiable governance receipts through AgentBound-Bench (Kaul et al., 29 Jun 2026).
Other neighboring efforts operate at different layers of the stack. AuditBench evaluates alignment auditing methods on models with hidden behaviors, including “Hallucinates Citations,” and reports a tool-to-agent gap together with strong performance from scaffolded black-box tools (Sheshadri et al., 26 Feb 2026). BenchGuard shifts attention from model behavior to benchmark integrity and shows that automated auditing can identify author-confirmed defects in benchmark infrastructure itself (Tu et al., 27 Apr 2026). AuAu audits authoritarian alignment in LLM outputs through psychometric questions, contextual behavior vignettes, and realistic user prompts, explicitly decomposing Authoritarian Aggression, Authoritarian Submission, and Conventionalism (Einwiller et al., 15 Jun 2026). At a philosophical level, “Automatic Authorities: Power and AI” defines “Automatic Authorities” as automated computational systems used to exercise power over people by substantially determining what they may know, what they may have, and what their options will be, and distinguishes substantive justification, procedural legitimacy, and proper authority (Lazar, 2024).
Taken together, these works indicate that “authority” in contemporary LLM evaluation is not a single construct. It can denote citation appearance, source credibility, institutional role, access permission, governance legitimacy, or ideological deference. The two benchmarks named AuthorityBench are therefore best read not as redundant projects, but as anchors for two separate research questions: whether models are misled by authority signals, and whether they can correctly identify authoritative sources when retrieval quality depends on that distinction.
