---
title: Package Hallucination Rate (PHR)
url: https://www.emergentmind.com/topics/package-hallucination-rate-phr
type: topic
---

# Package Hallucination Rate (PHR)

Package Hallucination Rate (PHR) is a quantitative metric for characterizing the tendency of generative AI systems, including large language models (LLMs), to recommend, cite, or reference non-existent or unsupported “packages.” Across code generation, shell command synthesis, scientific summarization, and in-context learning, PHR has become the de facto standard for measuring the frequency of such fact-conflicting errors, especially as they pertain to supply chain security, code reliability, and trustworthy automated knowledge synthesis. Although originally developed in the context of LLM-generated package dependencies, the metric has been adapted for rigorous evaluation in domains including code recommendation, summarization, and Bayesian in-context reasoning.

## 1. Formal Definition and Mathematical Formulation

The canonical definition of Package Hallucination Rate is the ratio of hallucinated (non-existent or unsupported) package references to the total number of package recommendations or claims. The mathematical formality and operationalization of PHR depends on domain and granularity:

- **Code Generation Context**: For code samples indexed by $i=1,\ldots,N$, let $r_i$ denote the number of packages recommended, and $h_i$ those that are hallucinations. Then

  $$
  R = \sum_{i=1}^N r_i,\qquad H = \sum_{i=1}^N h_i,\qquad
  \mathrm{PHR} = \frac{H}{R}\times 100\%
  $$

- **Shell Command/Go Ecosystem**: Given $G$ as the multiset of generated package references and $H\subset G$ the subset failing existence checks,

  $$
  \mathrm{PHR} = \frac{|H|}{|G|}
  $$

- **Language-Agnostic, Multi-Model Context**: For languages $l\in P$ and coding prompts $Q$, repeated $K$ times, with $G_l$ as the “known-good” package set,

  $$
  PHR(m,l) = \frac{1}{|Q|K} \sum_{q\in Q} \sum_{i=1}^K H_{m,q,i}
  $$

  where $H_{m,q,i}=1$ iff any generated import $s\notin G_l$.

- **Summarization/Knowledge Synthesis**: For $N$ abstract “packages” each with $m_j$ claims, $H_{j,i}\in\{0,1\}$ marks a hallucinated claim,

  $$
  \mathrm{PHR} = \frac{1}{N} \sum_{j=1}^N \sum_{i=1}^{m_j} H_{j,i}
  $$

- **Bayesian In-Context Learning**: Letting $y$ denote a generated prediction, $f$ the latent mechanism, and $Q_\epsilon(f,D)$ the $\epsilon$-quantile threshold,

  $$
  h_\epsilon(D) = \mathbb{E}_{f} \mathbb{E}_y \left[ \mathbf{1}\{ \log p(y|D,f) < Q_\epsilon(f,D) \} \right]
  $$

In all cases, the quantity is typically reported as a percentage or mean for comparability.

## 2. Domain-Specific PHR Measurement Protocols

PHR measurement requires systematic extraction and verification of candidate “package” references. Specific protocols vary:

- **Code-Generating LLMs**: Packages are extracted from code snippets via installed package statements (e.g., `pip install`, `npm install`) and module import patterns. Package names are cross-referenced against authoritative registries (PyPI, npm) as of the model’s training date. Hallucinations are defined as names not present in these registries [2406.10279].

- **Shell Command/Go Ecosystem**: Shell commands such as `go get …` are parsed for URL-based Go module paths. Existence is checked by resolving HTTP queries or invoking package managers; unresolved paths are hallucinatory [2512.08213].

- **Security-Focused PHR across Languages**: Package names are extracted from code via language-specific regular expressions. The generated names are compared against historical indexes per language and cutoff (PyPI/NPM/crates.io), marking as hallucinations any package not found. Both natural and adversarial (“induced”) hallucinations can be tested [2501.19012].

- **Scientific Summarization**: Summaries are subdivided at the claim level (typically sentences with citations). Each claim is checked via an automated or model-based “Factored Verification” procedure to determine whether it is supported by the cited source material. Unsupported claims increment the hallucination count per package [2310.10627].

- **ICL/Generative Modeling**: In Bayesian settings, sampled responses with log-likelihood below a quantile threshold, conditioned on a latent mechanism, are classed as hallucinations. Monte Carlo estimators use repeated sampling to approximate the posterior-averaged PHR [2406.07457].

Each protocol imposes domain-specific caveats: infallible registry support is assumed, extraction heuristics may lead to false negatives or positives, and prompt-context mismatch can skew rates.

## 3. Empirical Findings and Patterns

Large-scale empirical evaluation reveals PHR is non-negligible and exhibits strong systematic patterns:

- **Typical Rates**:
  - Commercial code LLMs (OpenAI GPTs) show $5.2\%$ average PHR; open-source LLMs reach $21.7\%$ [2406.10279].
  - In Go LLMs, full-precision models show $30\%$–$46\%$ PHR, with aggressive quantization ($4$-bit) yielding up to $96\%$ [2512.08213].
  - Security survey finds per-language means: JavaScript $14.7\%$, Rust $24.7\%$, Python $23.1\%$, with best models below $2\%$ [2501.19012].
  - In summarization, ChatGPT and GPT-4 produce $0.62$–$0.84$ hallucinations per summary, decreasing with critique-enhanced workflows [2310.10627].

- **Dependency on Model Family, Size, and Precision**: Larger parameter counts reduce PHR, as does access to up-to-date training data. Quantization increases PHR, with $8$-bit models showing moderate ($+2\%$–$4\%$) increase, while $4$-bit models induce catastrophic hallucination frequency except for top-scale models [2512.08213].

- **Sampling Temperature and Prompt Recency**: Increased temperature exacerbates PHR sharply, more than doubling rates at extreme values. Prompts referencing recent or esoteric packages see $\sim 10\%$ higher PHR [2406.10279].

- **Persistence and Specificity**: Hallucinations are often persistent across generations from the same prompt and model—$43\%$ of hallucinated packages reappear in repeated samples [2406.10279]. Most hallucinated names arise only in a single model.

- **String Structure**: In code, hallucinated packages are rarely typo-variants of real names; $48.6\%$ differ by at least $6$ edit distances from all real packages [2406.10279]. For Go, over $80\%$ hallucinated packages take plausible URL form with correct domain but non-existent user or subpath [2512.08213].

- **Correlation to Code Quality**: There is a strong negative correlation ($\rho=-0.79$) between HumanEval (code correctness) and PHR: higher code quality begets lower hallucination [2501.19012].

## 4. Mitigation Strategies and Trade-offs

Multiple approaches have been evaluated for PHR reduction, each with trade-offs:

- **Retrieval-Augmented Generation (RAG)**: Augment prompts with package-to-task facts, e.g., vector-indexed corpora. Observed $24$–$49\%$ relative reduction in PHR across open LLM baselines [2406.10279].

- **Self-Refinement**: Post-hoc LLM self-validation of package recommendations, with up to $19\%$ reduction. Effectiveness varies by model [2406.10279].

- **Supervised Fine-Tuning**: Model retraining on prompt$\to$valid-package data can offer $83\%$ (DeepSeek) and $61\%$ (CodeLlama) lower PHR, but at the expense of code generation quality (HumanEval pass@1 drops by half in some cases) [2406.10279].

- **Quantization-Aware Practices**: $8$-bit quantization is generally safe with minor PHR cost; $4$-bit demands aggressive post-filtering and validation layers to block “slopsquatting” attacks—malicious registration of hallucinated package names [2512.08213].

- **Deployment Safeguards**: Integrate registry exist checks, prompt hardening, explicit dependency provisioning, and internal “sinkholing” of high-risk names. Pre-deployment integration of PHR detection into code completion platforms can provide early warning [2501.19012].

Combining these techniques in ensemble reduces PHR by up to $85\%$ (DeepSeek) [2406.10279].

## 5. Security, Reliability, and Broader Impact

Elevated PHR presents a significant software supply chain security risk. Hallucinated names, especially those not yet registered in public package indices, create “zero-day” attack surfaces: adversaries can publish malicious code under them (“slopsquatting”), which is then consumed by downstream developers acting on LLM suggestions.

PHR is directly actionable as a security metric:

- **Model Selection**: High PHR models, even if otherwise performant, should be disfavored for security-critical code synthesis [2501.19012].
- **Continuous Monitoring**: Registry operators and security teams may proactively reserve or monitor common hallucinated names [2501.19012].
- **Automated Vetting**: Tooling can flag, require explicit approval for, or auto-remediate suspicious package suggestions.

There is no evidence that current model development best practices consistently optimize for low PHR in tandem with code quality: Pareto-optimality in error-hallucination space is sparsely populated [2501.19012]. *A plausible implication is joint consideration of coding benchmarks and hallucination-centric benchmarks should shape future model architecture and training set curation.*

## 6. Extensions, Limitations, and Adaptability

PHR has been adapted from code to summarization and generative in-context learning:

- **Summarization**: Factored Verification measures per-package hallucination count at claim level (mean per summary/package, or proportion with $\ge1$ hallucination), with further correction for verifier accuracy [2310.10627].
- **Bayesian Modeling**: Posterior Hallucination Rate tracks the probability of response-generation with log-likelihood below the $(1-\epsilon)$ region for the latent data-generating mechanism, estimated via black-box Monte Carlo sampling [2406.07457].

Practical limitations include:

- **Lower Bound Bias**: Registry-index-based measurement only establishes a lower bound; actors may have registered hallucinated names post-model-training [2406.10279].
- **Extraction and Classification Error**: Regex-based or heuristics-based extraction can misclassify modules as packages, or miss certain classes entirely [2406.10279].
- **Domain Specificity**: Results are contingent on model, ecosystem (Python, JavaScript, Go, Rust), prompt set, and date of registry snapshots. Rates and optimal mitigations may differ significantly elsewhere [2406.10279][2501.19012].
- **Adaptive Attack Risk**: Adversarial prompts can sharply increase hallucination frequency, especially for code-specialized and small models [2501.19012].

Despite these limitations, PHR offers a transferable, interpretable, and robust metric for quantifying hallucination-related threats and guiding the design of secure, supply-chain–conscious AI systems.

## 7. Cross-Domain and Methodological Evolution

The principles underlying PHR have influenced evaluation standards across LLM-centric research, including but not limited to supply chain security, code recommendation reliability, factual correctness in summarization, and ICL trustworthiness. Algorithmic refinements—including claim weighting, token- or entity-level granularity, and confidence-threshold–modulated PHR—extend its utility in matching the statistical structure of various generative tasks [2310.10627].

*This suggests that PHR, once narrowly tailored for code package hallucination, has become integral to a broader epistemic and security paradigm for generative AI evaluation.*

---

**Principal sources:**  
- "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs" [2406.10279]  
- "Secure or Suspect? Investigating Package Hallucinations of Shell Command in Original and Quantized LLMs" [2512.08213]  
- "Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities" [2501.19012]  
- "Estimating the Hallucination Rate of Generative AI" [2406.07457]  
- "Factored Verification: Detecting and Reducing Hallucination in Summaries of Academic Papers" [2310.10627]

Source: https://www.emergentmind.com/topics/package-hallucination-rate-phr