When LLMs Invent Rust Crates: An Empirical Study of Hallucination Patterns and Mitigation

Abstract: LLMs have become powerful tools for code generation, yet they remain prone to hallucinations-producing plausible but incorrect or fabricated outputs. Among these, package hallucination, where an LLM suggests non-existent dependencies, poses an emerging security risk to the software supply chain. While previous studies focus on popular languages like Python or JavaScript, in this work we present the first large-scale empirical study on crate hallucination in LLM-generated Rust code. We construct a multi-source dataset combining coding tasks from Stack Overflow, GitHub, and LLM-generated tasks, and evaluate both commercial and open-source models under various decoding settings. Our analysis reveals that, unlike prior findings in Python and JavaScript, hallucination behavior in Rust follows a distinct pattern: different models exhibit surprisingly consistent hallucination rates, and these rates show minimal sensitivity to model parameters. Furthermore, we investigate prompt engineering strategies to mitigate hallucinations without sacrificing code quality. This study provides new insights into the reliability and security implications of LLM-assisted Rust development, offering guidance for future research and safer model deployment in software engineering workflows.

• The paper presents an empirical evaluation of Rust crate hallucinations, reporting an average hallucination rate of 20.23% across leading LLMs.
• It characterizes hallucination patterns, showing that over half arise from module-crate confusions and minor lexical variations.
• It evaluates mitigation techniques like self-refinement and RAG, demonstrating modest CHR reductions and advocating for toolchain-integrated solutions.

Empirical Analysis of Rust Crate Hallucination in LLM Code Generation

Introduction

This paper presents a comprehensive empirical study on crate hallucination in LLM-generated Rust code, substantially advancing understanding of package hallucinations beyond popular languages such as Python and JavaScript. By constructing a large, multi-source dataset and evaluating a range of leading commercial and open-source LLMs, the authors investigate the prevalence, recurring patterns, and mitigation of crate hallucinations in Rust. The findings reveal that crate hallucination in Rust is highly structured, typically manifests as module-crate confusion or near-miss names, and exhibits distinctive behaviors compared to established results in other language ecosystems.

Dataset and Model Selection

Three sources were integrated to construct the coding task dataset: Stack Overflow (1,001 tasks), function-level GitHub extraction with RustDoc comments (793 tasks after curation), and 1,000 LLM-generated tasks based on crate descriptions. This amalgamation ensures broad domain coverage, from user-driven requests to real-world project artifacts and synthetic prompts focused on dependency-relevant code.

The evaluation includes six high-performing LLMs: three commercial (Claude 4 Opus, GPT-5, Gemini 2.5 Pro) and three open-source (Qwen2.5-Coder, DeepSeek-Coder-V2, OpenCoder), with architectural and parameter variants to facilitate cross-family and intra-family comparisons.

Figure 1: The transformation pipeline from a real-world Rust function into an actionable coding task.

Detection and Measurement of Crate Hallucination

Hallucinated crates were detected via a comprehensive reference list incorporating all crates on crates.io, Rust standard library crates, rustc internal crates, and extracted aliases. Source code was statically analyzed through patterns including use statements, path-prefixed references, and extern crate statements, enabling precise quantification of hallucinations at the source reference level.

Crate Hallucination Rate (CHR) was formalized as the ratio of hallucinated crates to all extracted crate references:

$$\mathrm{CHR} = \frac{\text{Number of hallucinated crates}}{\text{Total number of extracted crates}}$$

Prevalence and Model-Level Effects

The CHR analysis across 16,764 generations (48,494 recommendations) demonstrates:

• Prevalence: All models exhibited non-trivial crate hallucination rates, with a macro CHR around 20.23%. The best (Gemini 2.5 Pro) achieved 16.18% and the worst (Claude 4 Opus) 26.90%.
• Model Family and Size: Variance across commercial and open-source models is modest; model family is weakly predictive of CHR. Notably, within-family experiments over Qwen2.5-Coder, DeepSeek, and OpenCoder showed no statistically significant reduction in CHR with larger parameter counts.
• Decoding Parameters: Adjusting temperature and other decoding hyperparameters produced only marginal and inconsistent changes in CHR Figure 2. Unlike in Python/JavaScript, there is no monotonic relationship in Rust.

Figure 3: Crate hallucination rates for evaluated commercial and open-source models.

Figure 2: CHR stability across temperatures for multiple models; low sensitivity to decoding parameterization.

These findings jointly show that, for Rust, crate hallucination in LLM output is relatively insensitive to both model size and decoding stochasticity under current data, arch, and prompting constraints.

Characterization of Hallucination Patterns

Structural Error Taxonomy

Analysis reveals that over half of hallucinated crate names are legitimate Rust modules, erroneously referenced as external crates. This is fundamentally a hierarchical naming misalignment, often originating from overgeneralized import rules (e.g., Python-style module/package conflation) being transferred to Rust, which enforces stricter distinctions.

• Cross-model Consistency: Module hallucinations are strikingly consistent across models (Jaccard ≈ 0.44), suggesting common upstream data or language-specific gaps. By contrast, non-module hallucinations are highly model/instance-specific (Jaccard ≈ 0.16).
• Name Similarity: Most hallucinated crate names are minor lexical variations of valid crates, quantified by edit distance (mean ≈ 3.42). 86.48% are within five edits of a real crate Figure 4, with frequent omissions or substitutions of hyphens robustly observed.

Figure 4: Distribution of Levenshtein distance between hallucinated and authentic crate names demonstrates systematic near-miss generation rather than random string sampling.

• Cross-ecosystem Borrowing: 38.9% of frequently hallucinated non-modules originate in other programming or API ecosystems (e.g., “queryparser” from Python), further portraying crate hallucination in Rust as a structured, data-driven transfer phenomenon.

Mitigation Evaluation

The study systematically investigates two no-training, pre-processing prompt engineering methods: self-refinement and Retrieval-Augmented Generation (RAG):

• Self-Refinement: Applies an automated review and regeneration loop, with explicit instructions to the model to check and avoid hallucinated dependencies.
• RAG: Augments prompts with retrieved context from a curated Top-10,000 crates database and a grounded STD-list of standard library capabilities.

Experimental results show that both methods reduce CHR, but improvements are modest and far from eliminating source-level crate hallucination. Self-refinement achieves an average reduction of 2-3 percentage points in CHR (10–15% relative), while RAG exhibits weaker effectiveness and is confounded by retrieval precision.

Theoretical and Practical Implications

The findings challenge several assumptions derived from Python and JavaScript studies. In Rust:

• Hallucination patterns are highly structured, being primarily driven by module/crate confusions and closely related to the idiosyncrasies of Rust's namespace and package systems.
• Model scaling and decoding hyperparameters have minimal effect, pointing toward the critical influence of training data scarcity and representativeness for low-resource ecosystems.
• Lightweight prompting interventions offer marginal gains; effective mitigation likely requires toolchain or verification-level integration, especially with manifest cross-checking, namespace validation, and edit-distance–based correction mechanisms.

Future Directions

The results strongly motivate Rust-specific, toolchain-integrated mitigation. Future research should address:

1. Namespace-aware and registry-validated output filtering post-generation.
2. Incorporating semantic and manifest-level verification (auto-correlation of Cargo.toml and .rs source references).
3. Augmenting LLM pretraining corpora to enrich Rust coverage and reduce cross-ecosystem transfer artifacts.

Conclusion

This work delivers the first systematic empirical analysis of Rust crate hallucination in LLM-generated code, demonstrating that hallucination is both prevalent and structurally patterned, with characteristics distinct from high-resource ecosystems. The comprehensive evaluation across modern LLMs and multiple mitigation strategies establishes important empirical baselines and offers actionable insights for both LLM developers and the supply chain security community. The challenge of reducing crate hallucination in Rust ultimately demands tailored, toolchain-aware solutions beyond traditional prompt engineering.