Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summaries

Abstract: LLMs are increasingly used to generate summaries of software bug reports, including sections such as Steps-to-Reproduce (S2R), Actual Behavior (AB), and Expected Behavior (EB). However, these models frequently produce hallucinations that can be convincing but unsupported by the source report. This can mislead developers and reduce trust in automated maintenance tools. Existing hallucination detection approaches typically evaluate outputs at the full-response level and do not consider the structure of technical documents. An initial exploratory study on 80 structured bug report summaries found that approximately 47.9% contained missing information, while 12.3% included fabricated content, highlighting the need for systematic hallucination analysis in bug report summarization. In this work, we empirically investigate hallucinations in LLM-generated bug report summaries from a section-aware perspective. Using the BugsRepo dataset, derived from Mozilla OSS projects, we introduce controlled synthetic hallucination injection to construct a benchmark for training and evaluation. We propose a section-aware hallucination detection approach that jointly predicts whether a summary contains hallucinated content, identifies affected sections, and classifies hallucination types. Experimental results across multiple pretrained LLMs show that the proposed approach achieves strong performance across all tasks, with the best model obtaining 0.89 report-level Macro-F1, 0.83 section-level Macro-F1, and 0.84 hallucination-type Macro-F1. We further analyze common hallucination patterns and model failure modes to better understand limitations of current LLM-generated bug report summaries. The findings highlight the importance of section-aware hallucination analysis for improving the reliability of LLM-assisted bug report summarization in software maintenance workflows.

• The paper introduces a framework to detect hallucinations in AI-generated bug report summaries by analyzing structured document sections.
• Utilizes a multi-task learning paradigm on the BugsRepo dataset for precise report and section-level hallucination detection and classification.
• Achieves 0.89 macro-F1 score in detection tasks, revealing varied vulnerability to hallucinations across document sections.

Section-Aware Hallucination Analysis and Detection in LLM-Generated Bug Report Summaries

Introduction

This paper presents a comprehensive empirical framework for analyzing and detecting hallucinations in LLM-generated bug report summaries, focusing specifically on the structural segmentation of reports into Steps-to-Reproduce (S2R), Actual Behavior (AB), and Expected Behavior (EB) components. Hallucinations, defined as plausible yet unsupported or fabricated content, pose significant reliability concerns for the adoption of LLM-generated artifacts in software maintenance workflows. While hallucination detection in open-domain NLP has received considerable attention, its investigation in the context of structured technical documents for Software Engineering (SE)—and particularly, within distinct report sections—remains underexplored.

Exploratory Analysis of Hallucination Phenomena

A preliminary analysis was performed on summaries generated via Llama 3.2 on a curated selection from the BugsRepo dataset. Human annotation demonstrated that hallucinations, including both missing (incomplete) and fabricated information, are prevalent and exhibit section-dependent behavior. Notably, approximately 47.9% of summary segments exhibited missing information while 12.3% contained fabricated content, with overall hallucination rates exceeding 60% across all summary segments.

Figure 1: Empirical distribution of hallucination types (missing and fabricated) across summary segments in LLM-generated bug report summaries.

Section-dependent hallucination rates indicate that patterns of error generation are closely related to the contextual and functional role of each segment in the summary. For example, S2R segments are particularly prone to omission, while declarative EB fields are more resistant to hallucinations.

Methodological Framework and Section-Aware Detection

The core contribution is a novel section-aware hallucination detection architecture employing a multi-task learning paradigm. The framework leverages a benchmark constructed from the BugsRepo dataset, where hallucinations are synthetically injected into individual sections to provide ground truth at both the report and section levels. The multi-task model jointly performs:

1. Report-level binary hallucination classification,
2. Section-level hallucination localization,
3. Hallucination type (addition, removal, reorder) classification.

   Figure 2: Architecture of the section-aware hallucination detection system, highlighting input structuring, multi-task prediction heads for detection, localization, and hallucination-type classification.

This design is motivated by evidence from intra- and inter-section self-attention distributions in transformer architectures, revealing systematic under-attention to certain segments, which correlates with empirical hallucination sensitivity.

Empirical Investigation: Attention Allocation and Dataset Curation

Self-attention analysis over transformer models—including BERT, RoBERTa, DistilBERT, SciBERT, and DeBERTa—quantifies the distribution of focus across S2R, AB, and EB fields. EB receives the highest average attention, followed by AB, with S2R consistently suffering from the lowest attention, providing a quantitative basis for section-specific vulnerability to hallucination.

Figure 3: Mean section-wise attention allocations across different transformer models, indicating structural disparities in representation focus that drive hallucination susceptibility.

These findings inform the design of the synthetic benchmark, with higher proportions of S2R-type hallucinations injected to reflect higher risk in these segments. The dataset curation process employs a modified PARENT metric for filtering and evaluating structured-to-unstructured conversions, ensuring rigorous preservation of field-level information.

Figure 4: (a) Distribution of $\mathrm{PARENT}$ scores for generated bug report descriptions; (b) Filtering strategy visualization for retaining high-quality unstructured report conversions.

Model Performance and Error Profiling

The proposed fine-tuned section-aware detectors—especially the Ministral-3B transformer—achieved strong numerical results: 0.89 macro-F1 at the report level, 0.83 macro-F1 on section localization, and 0.84 macro-F1 for hallucination-type classification. Notably, mastery of the hallucination detection task did not increase monotonically with parameter count, as Ministral-3B consistently outperformed or matched larger architectures such as LLaMA-8B across all evaluation axes.

• Contradictory claim: Increasing model size did not result in superior hallucination-type classification, as Ministral-3B outperformed models with more parameters.

Qualitative error analysis highlights significant residual challenges, particularly for removal-type hallucinations. Manual coding of mispredictions shows that the main failure mode—accounting for 44.8% of errors—is the model's inability to detect omitted or implicit information in structured fields, especially in AB and EB segments.

Figure 5: Frequency distribution of manual explanation labels assigned during qualitative error analysis, demonstrating error concentration in omission and implicit information recognition.

Implications and Future Directions

These results underscore the necessity of section-sensitive approaches for hallucination analysis in LLM-generated SE artifacts. The empirical evidence demonstrates that a multi-task detection strategy supports fine-grained and interpretable prediction, better capturing the complex provenance and propagation of errors across structured fields in bug report summaries.

Practically, integrating section-aware hallucination detection enhances the reliability of LLM-powered support for software maintenance, facilitating trust in automated summarization pipelines. Theoretically, the demonstrated correspondence between attention allocations and hallucination vulnerability suggests promising avenues for attention-regularized training methods, model interpretability research, and adaptive prompting strategies. Future extensions could benchmark larger-scale models, evaluate on organically hallucinated real-world samples, and explore cross-domain generalization to issue tracking systems with variable schema conventions.

Conclusion

This study delivers an end-to-end empirical pipeline for section-aware hallucination analysis in LLM-generated bug report summaries. The introduction of synthetic, benchmarked errors and rigorous multi-task detection advances the methodological foundations for both practical deployment and theoretical understanding of factual errors in LLM-mediated SE workflows. Structural document context is shown to be central for reliable hallucination detection, with the framework and released dataset enabling further development in AI-assisted software maintenance (2605.24137).