LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review

Abstract: LLMs are increasingly used to produce test oracles, the part of a test that decides whether observed behavior is correct. Yet a clear account of where these oracles draw their authority is missing. Prior secondary studies organize the area by oracle form or by LLM technique. None organizes it by the source of the verdict's authority, the property that governs how far a verdict can be trusted. This article presents a systematic literature review, conducted and reported under the PRISMA 2020 guidelines. From 2,436 records, an LLM pre-filter followed by independent dual human screening (reviewer agreement, a Cohen's kappa of 0.79) and full-text assessment yielded 54 included studies. We analyze these along three axes: the source of an oracle's authority, the form it takes, and the mechanism that adjudicates it. We characterize the landscape of domains, languages, models, and adaptation strategies. Specification-derived authority, though the most common single source, covers about half of the studies (28 of 54). The remaining 26 reach a verdict with no specification at all. The source of authority and the adjudication mechanism cross-cut: the same source is checked by several mechanisms and one mechanism serves several sources, so a label such as LLM-as-a-judge names a mechanism rather than a basis for trust. We further report how these oracles are evaluated and how they fail, and read the sparse and empty regions of the taxonomy as a research agenda. The protocol, search query, and per-study coding sheet are released as supplementary material.

• The paper introduces a seven-category taxonomy of authority sources and analyzes 54 studies, showing that 26 use no external specification and that authority, rather than mechanism, determines trust limits.
• The review finds that assertions and runtime execution dominate oracle design, while LLM-as-a-judge appears in only 13 studies; hallucination is reported in 46 studies and weak assertions in 24.
• The paper shows that evaluation remains weakly fault-grounded, with only 6 studies using mutation analysis, and recommends shared mutation-scored benchmarks and explicit authority-based risk assessment.

Motivation and framing

Test oracles—the component of a test that decides whether observed behavior is correct—have historically been the bottleneck of test automation, a difficulty formalized as the oracle problem in Barr et al.'s survey [barr2015]. LLMs have changed the practical landscape in two ways: they can generate an oracle artifact (assertions, metamorphic relations, properties), or they can be the oracle, judging outputs directly. Existing secondary studies characterize this literature by oracle form (assertion vs. metamorphic relation) or by technique ("LLM-as-a-judge"), but not by the property that actually bounds trustworthiness: where the verdict's authority comes from. This systematic literature review (SLR), conducted and reported under PRISMA 2020, supplies that missing lens. Its central analytical move is to separate three questions that prior work conflates:

1. Source of authority: where correctness ultimately comes from;
2. Form: what shape the oracle takes;
3. Mechanism: how pass/fail is computed.

The paper argues, and demonstrates empirically, that only the first sets the ceiling on how far a verdict can be trusted—and that "LLM-as-a-judge" names a mechanism, not a basis for trust.

The seven-source taxonomy

The framework distinguishes seven sources of authority, extending Barr et al. into the LLM era:

• Specification-derived (formal postconditions, semi-formal rules/Gherkin, natural-language docstrings and requirements);
• Implementation-derived (the code or its existing tests; catches regressions but not pre-existing bugs);
• Reference-differential (a separate implementation, including the recent variant where several LLM-written implementations vote by consensus);
• Model-parametric (the model's own knowledge, no external anchor);
• Human-elicited, regression-from-prior-version, and implicit-intrinsic (crashes/well-formedness).

The taxonomy splits cleanly on one question: does the verdict require an external specification? Only specification-derived oracles do, by definition. Crucially, source and mechanism cross-cut rather than coincide—a specification-derived oracle may be checked by assertion execution, exact match, or model judgment, and one mechanism serves many sources.

Review method

The search (frozen 31 May 2026, over Scopus, IEEE Xplore, ACM DL) identified 2,436 records, de-duplicated to 2,245. Screening used a two-stage design disclosed under PRISMA item 8: an LLM pre-filter (Claude Opus 4.8) reduced the corpus to 178 records, followed by independent blind dual human screening over a 385-record verification set (all retained records plus a stratified 10% sample of exclusions). Inter-rater agreement was substantial: Cohen's $\kappa = 0.79$ (95% CI $[0.72, 0.85]$) on the binary retain/exclude decision, $\kappa = 0.64$ on the three-level scale. The pre-filter achieved 99.1% recall against human consensus, with a sampled false-omission rate of 0.48%, projecting to roughly nine missed relevant records across unaudited exclusions—a residual risk the authors carry explicitly as a threat to validity. Full-text assessment yielded 54 included studies from 114 unique full-text records; the dominant exclusion reason (55 of 60) was criterion E2, i.e., LLM involvement without an LLM-supplied oracle (input generation, static proofs, crash-only signals). A stated boundary rule: an LLM-generated invariant executed against a running system is an oracle; the same artifact inside a static proof is verification and is excluded.

Principal findings

Specification independence is nearly half the corpus. Specification-derived authority is the largest single source but covers only 28 of 54 studies; the remaining 26 reach a verdict with no specification at all (implementation-derived: 11; model-parametric: 9; reference-differential: 4; human-elicited: 2). Departures from the idealized split amount to two boundary cases. This is the review's most consequential result: most LLM-based oracles are useful precisely because everyday software lacks specifications, but model-parametric oracles can be wrong with no external signal that they are wrong.

Source and mechanism cross-cut. Assertions dominate form (31 co-occurrences) and runtime assertion execution dominates mechanism (33). In 41 of 54 studies the LLM generates an artifact checked by something else; in only 13 is the LLM itself the judge. Notably, in all 13 LLM-as-judge cases the underlying authority is specification-derived, model-parametric, or human-elicited—never implementation-derived or reference-differential. The authors correctly flag this as a corpus pattern (n=13), not an established constraint.

Failure modes concentrate on oracle weakness and hallucination. Hallucination is engaged with in 46 of 54 studies (44 name it as a risk, 31 observe it empirically, 35 apply a mitigation)—but it is named more often than measured. Weak or missing assertions appear in 24 studies versus only 4 for over-assertion, making under-constrained oracles the most frequently reported correctness risk. Mitigations center on execution-based filtering, self-consistency voting, and human review, rarely evaluated for residual risk.

Evaluation is resemblance-based, not fault-grounded. 59% of studies use custom benchmarks; Defects4J (9) and HumanEval (4) are the largest shared ones. Ground truth is mostly human-provided (28) or existing tests (14). Only 6 of 54 use mutation analysis to verify that the oracle catches injected faults; 3 assess oracle quality not at all. A lightweight eight-item quality appraisal (median 6/8) confirms a literature mature in reporting but thin on fault-grounded validation.

Landscape characteristics. 85% of included studies appeared in 2025 or later; general-purpose unit-level software dominates (30/54); closed-weight models outnumber open-weight (32 vs. 12); prompting dominates fine-tuning (only 9 studies fine-tune); Java (21) and Python (13) lead; 42 of 54 release code or data. Model-parametric oracles concentrate in interactive/visual domains (mobile, GUI, autonomous driving), plausibly because these lack executable specifications.

Gaps read off the taxonomy

Two of the seven sources are unoccupied as primary authorities—regression-from-prior-version and implicit-intrinsic—though both appear as secondary signals in compositional systems. The authors note candidly that these cells are empty partly by construction of their scope (criterion E2 excludes designs where non-LLM tooling supplies the comparison), leaving open whether an LLM adds value where it is currently bypassed. Other gaps: LLM-as-judge is never applied to implementation- or reference-differential authority (whether a judge briefed with the reference could beat exact matching for semantically equivalent outputs is untested); human-elicited authority is the sparsest occupied source (2 studies); and the field lacks a shared, mutation-scored benchmark for oracle strength.

Relation to prior reviews

Against 14 overlapping secondary studies, none organizes the literature by source of verdict authority; thirteen sort by form or technique, one by mechanism. None reports inter-reviewer agreement. The only oracle-dedicated secondary study is a non-systematic roadmap. This review is thus positioned as the first simultaneously oracle-focused, systematically searched, source-of-authority-organized, and reliability-reported survey of the area.

Limitations

The authors disclose these plainly: the automation-assisted screening leaves a projected ~nine-record recall risk with wide confidence intervals around a single observed omission; Web of Science, Springer Link, and preprint servers were not searched, so preprint-only work is under-represented (deliberately, via criterion E5); hardware-verification assertion generation is excluded as an adjacent literature; and the source-of-authority coding is itself an interpretive judgment at category boundaries, mitigated by releasing per-record decisions, codes, and anchoring quotations for audit. Effectiveness numbers are deliberately not pooled, since 59% custom benchmarks make cross-study aggregation unsound.

Conclusion

This SLR reframes the LLM-based oracle literature around a question prior surveys skip: where does the verdict's authority come from? Across 54 studies, just under half of LLM-based oracles need no specification, the source almost entirely determines specification dependence, and source cross-cuts mechanism so thoroughly that mechanism-only labels discard the information that governs trust. The dominant reported risks—an oracle too weak to catch the bug, and hallucinated behavior—are compounded by evaluation practices that measure resemblance to a reference oracle rather than fault detection. For practitioners, the actionable takeaway is that the first question about any LLM oracle should be its source of authority, which predicts failure modes more reliably than its mechanism does.