How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

Abstract: How much do LLMs actually hallucinate when answering questions grounded in provided documents? Despite the critical importance of this question for enterprise AI deployments, reliable measurement has been hampered by benchmarks that rely on static datasets vulnerable to contamination, LLM-based judges with documented biases, or evaluation scales too small for statistical confidence. We address this gap using RIKER, a ground-truth-first evaluation methodology that enables deterministic scoring without human annotation. Across 35 open-weight models, three context lengths (32K, 128K, and 200K tokens), four temperature settings, and three hardware platforms (NVIDIA H200, AMD MI300X, and Intel Gaudi 3), we conducted over 172 billion tokens of evaluation - an order of magnitude beyond prior work. Our findings reveal that: (1) even the best-performing models fabricate answers at a non-trivial rate - 1.19% at best at 32K, with top-tier models at 5 - 7% - and fabrication rises steeply with context length, nearly tripling at 128K and exceeding 10% for all models at 200K; (2) model selection dominates all other factors, with overall accuracy spanning a 72-percentage-point range and model family predicting fabrication resistance better than model size; (3) temperature effects are nuanced - T=0.0 yields the best overall accuracy in roughly 60% of cases, but higher temperatures reduce fabrication for the majority of models and dramatically reduce coherence loss (infinite generation loops), which can reach 48x higher rates at T=0.0 versus T=1.0; (4) grounding ability and fabrication resistance are distinct capabilities - models that excel at finding facts may still fabricate facts that do not exist; and (5) results are consistent across hardware platforms, confirming that deployment decisions need not be hardware-dependent.

• The paper precisely quantifies hallucination rates by evaluating 35 open-weight models over 172 billion tokens using a novel, ground-truth-first RIKER framework.
• It reveals significant impacts of context length and model family on fabrication rates, with longer contexts leading to drastic drops in accuracy.
• The results underscore that deliberate model selection and specific anti-fabrication training are essential to boost factual grounding in large-context QA.

How Much Do LLMs Hallucinate in Document Q&A Scenarios? A Large-Scale and Multivariate Analysis

Introduction

This paper conducts an expansive and methodologically rigorous evaluation of LLM hallucination in document-grounded question answering (Q&A). Prior assessment paradigms were limited by benchmark contamination, reliance on LLM-as-judge with biased scoring, and insufficient statistical power due to small sample regimes. This study introduces a definitive answer to the rate and structure of hallucination (defined as fabrication of non-existent facts) using the RIKER framework—a ground-truth-first methodology for scalable, contamination-resistant, and deterministic evaluation. Over 172 billion tokens were evaluated across 35 open-weight models, three context lengths (32K, 128K, 200K), four temperatures, and three distinct hardware backends. The resulting analysis provides nuanced insights into fabrication floors, context length scaling, temperature effects, architectural and training influences, and hardware invariance for the document QA hallucination problem.

RIKER: Methodological Overview

The RIKER framework represents a paradigm inversion compared to conventional evaluation designs. Instead of extracting ground truth from real documents (suffering from static contamination or annotation error), RIKER first specifies relational ground truth and then generates document corpora de novo alongside query sets. Deterministic and template-based corpus construction guarantees known answers to all queries, including fabrication probes where correct behavior is refusal or negation (for entities intentionally absent from the corpus). This structuring enables precise, contamination-immune, and arbitrarily large-scale generation of evaluation data.

Figure 1: The RIKER methodology contrasts traditional human-annotated static benchmarks (left) with ground-truth-driven deterministic QA and document generation (right), delivering regenerable, contamination-resistant, and scalable evaluation.

The RIKER benchmark scaffolds twelve levels of QA task spanning single-document extraction, multi-document aggregation, and explicit hallucination detection. Four primary metrics are used: overall accuracy, single-document grounding accuracy, cross-document aggregation accuracy, and fabrication rate (false positives on hallucination probes). Truncation (uninformative/infinite output) rates are additionally tracked to account for generation coherence loss.

Experimental Scope

RIKER was executed over 35 open-weight models (variants from Qwen, GLM, Llama, DeepSeek, MiniMax, Granite), at three context lengths (up to 200K tokens), on NVIDIA H200, AMD MI300X, and Intel Gaudi3 platforms. For every model–context combination, four temperatures (0.0, 0.4, 0.7, 1.0) were independently benchmarked, and each run was replicated eight times for robust estimation, producing 172B total processed tokens. Notably, hardware backend, serving stack, and prompt implementations were stringently controlled to isolate hardware-dependent effects.

Major Results and Empirical Trends

Hardware Effects Are Negligible

Model behavior was highly consistent across tested GPU/accelerator platforms (H200, MI300X, Gaudi3). Between-hardware accuracy differences were universally below 1 pp—smaller than even minor temperature-induced or run-to-run stochasticity effects. This result generalizes LLM hallucination rates for inference deployment, allowing practitioners to optimize hardware for cost/performance without fidelity concerns.

Strong Model Selection Effects, Minimal Size Correlation

Overall accuracy and fabrication rates display a 70+ percentage point spread across models. Model family is a far stronger predictor of hallucination resistance than parameter count—e.g., GLM family models universally achieve low fabrication, whereas Llama 3.x models routinely combine high grounding with dramatically high fabrication rates. Mixture-of-experts (MoE) Qwen3 models (e.g., Next 80B-A3B) maintain competitive accuracy at radically lower per-query active parameter footprints.

Hallucination Rates

At 32K context (shortest/easiest), the best-case observed fabrication rate is 1.19% (GLM 4.5), with only 2/35 models below 5%. At 200K context, no model stays below 10% fabrication; the best (Qwen3 Next 80B-A3B) fabricates to 10.25% of hallucination probes. The median model's fabrication rates approach 25% at 32K and degrade further with context length. Increasing context length is universally detrimental: models lose, on average, 10–30 pp in overall accuracy between 32K and 128K, and in extreme cases over 50 pp by 200K context.

Grounding and Fabrication Decouple

Grounding and fabrication are found to be empirically uncorrelated across models. Many models that excel at extracting gold facts from documents (e.g., Llama 3.1 70B with 90% grounding) simultaneously fabricate non-existent answers at rates exceeding 40–50%. Fabrication resistance thus does not emerge from improvements in retrieval or comprehension architecture alone but requires deliberate alignment or regularization in training objectives.

Figure 2: Grounding accuracy versus fabrication rate for 32K context; models above the diagonal (dashed) show excess fabrication relative to their retrieval skills. Green: well-calibrated, orange: mid-tier, red: fabrication-prone.

Aggregation and Long-Context Degradation

The hardest QA tasks are cross-document aggregation queries, especially as context grows. Most models show a sharp drop in aggregation accuracy by 128K/200K. Anecdotally, GLM 4.6 exemplifies catastrophic collapse, with its fabrication rate rising from ~7% at 32K to over 70% at 200K and overall accuracy plummeting below 40%.

Temperature Effects: Accuracy, Fabrication, and Coherence Trade-offs

Temperature $T=0.0$ (greedy decoding) yields best accuracy for only ~60% of model–context pairs. Higher temperatures sometimes lower fabrication rates and more generally suppress coherence loss (infinite output loops), which afflict some models with up to $48\times$ increased failure rates at $T=0.0$ (notably at long context, e.g., Llama 3.1 8B at 128K). Optimal temperature settings thus require task-specific and model-dependent tuning. Universal "use $T=0.0$ for factual QA" guidance is refuted.

Practical and Theoretical Implications

Practically, no open-weight model is free of hallucination in document QA—even top performers fabricate at unacceptable rates by deployment standards, especially at contexts meaningful for real-world tasks. Model selection (by training procedure/family) is overwhelmingly more impactful than hardware, parameter size, or decoding temperature. Long-document question answering requires evaluation and countermeasures at deployment-specific context windows, not merely at short-benchmark regimes. Benchmarking solely on grounding is dangerously misleading; explicit fabrication assessment is essential.

Theoretically, the results suggest that hallucination resistance is not a monotonic function of scaling nor a natural consequence of improved cross-document retrieval; it is an independent axis requiring explicit training or alignment strategies. Architectural advances (e.g., MoE routing) may assist context scaling, but do not obviate hallucination.

Future Directions

The study motivates several avenues: (1) fine-grained analysis of repetition penalty versus temperature for coherence management, (2) detailed investigation of family-specific anti-fabrication training signals, (3) extension of ground-truth-first frameworks like RIKER to multilingual and even multimodal document regimes, and (4) large-scale, automatic hallucination detection benchmarks for frontier API models and new open-weight architectures at million-token context scales.

Conclusion

The study establishes robust fabrication floors for SOTA open-weight LLMs in document QA. Even under ideal conditions, hallucination is persistent and non-trivial; context expansion exacerbates the effect. Practitioners must integrate holistic and model-specific countermeasures, and the research community must recognize the critical distinction between retrieval/grounding and genuine fabrication resistance. Progress toward trustworthy, large-context, document-grounded LLMs will require concerted methodological and training-level innovations.