Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation

Abstract: In this paper, we focus on automating two of the widely used Verification and Validation (V&V) activities in the Software Development Lifecycle (SDLC): Software testing and software inspection (also known as review). Concerning the former, we concentrate on automated test case generation using LLMs. For the latter, we enable inspection of the source code by LLMs. To address the known LLM hallucination problem, in which LLMs confidently produce incorrect outputs, we implement a Retrieval Augmented Generation (RAG) pipeline to integrate supplementary knowledge sources and provide additional context to the LLM. Our experimental results indicate that incorporating external context via the RAG pipeline has a generally positive impact on both test case generation and code inspection. This novel approach reduces the total project cost by saving human testers'/inspectors' time. It also improves the effectiveness and efficiency of these V&V activities, as evidenced by our experimental study.

• The paper presents a novel framework that integrates retrieval-augmented generation with large language models for automating test case generation and code inspection.
• The methodology employs a two-stage process combining semantic retrieval and tailored prompt engineering to boost test coverage and error detection.
• Empirical results show that RAG-augmented LLMs improve line and branch coverage by over 3% and achieve up to 90.57% accuracy in code inspection, surpassing human performance.

Retrieval-Augmented LLMs for Automated Software Testing and Inspection

Introduction

The paper "Enhancing LLMs with Retrieval Augmented Generation for Software Testing and Inspection Automation" (2604.15270) presents a rigorous framework for automating key verification and validation (V&V) activities in the Software Development Lifecycle (SDLC) using Retrieval-Augmented Generation (RAG)-based LLMs. The work targets two critical tasks: automated test case generation and source code inspection. Motivated by the persistent challenge of LLM hallucinations and the associated risks for software quality, the approach integrates external knowledge bases into the LLM generative process. The authors deliver both a formal methodology and extensive experimental evaluations, quantifying accuracy, coverage, and runtime. This essay reviews the theoretical and practical contributions, experimental results, and the future implications of this research.

RAG-Enhanced Automation Pipeline

The proposed framework adopts a two-stage methodology combining semantic retrieval and LLM generation. Source code or testing instructions serve as queries, which are first converted to embeddings (using MiniLM-L6-V2) for top-K relevant context retrieval from a vectorized external knowledge base (e.g., MBPP). This retrieved information is embedded in the prompt to the LLM, which then performs the downstream code analysis or test case generation.

Figure 1: Overview of the RAG-based code inspection and test generation pipeline, illustrating knowledge retrieval and prompt enrichment for both tasks.

Code inspection and test generation are unified within this architecture via abstraction over the query and context management mechanisms. The inspection task evaluates error type classification on Bug In the Code Stack, while test generation is assessed on TestEval, measuring coverage and correctness.

Prompt Engineering

The performance of both tasks is highly dependent on prompt design and retrieval quality. Custom templates are defined for each activity to inject retrieved context into the LLM prompt.

Figure 2: Prompt template for code inspection, demonstrating how retrieved external knowledge is incorporated.

Figure 3: Prompt template for test case generation, enabling knowledge-enhanced test synthesis.

Empirical Results

Test Case Generation

Experimental results reveal that RAG augmentation significantly improves downstream metrics on the TestEval benchmark, for both GPT-3.5 Turbo and GPT-4o. Line and branch coverage, including the cov@k metrics, consistently increase with RAG. Notably, GPT-3.5 Turbo with RAG attains 96.64%/93.20% for line/branch coverage, an increase of over 3% relative to the baseline. Compilation and execution success rates rise correspondingly, validating the reduction of hallucinations and the improved factual alignment enabled by RAG.

Figure 4: Comparative efficiency of automated test generation by LLM and RAG configuration.

Efficiency results show mixed trends: for GPT-3.5 Turbo, RAG reduces runtime (by ~3.5 minutes), whereas retrieval overhead can offset or negate gains for GPT-4o, underscoring the importance of context filtering and model-retriever synergy.

Code Inspection

For automated code inspection, LLMs with RAG display a substantial accuracy improvement over non-RAG baselines and human inspectors. On the Bug In the Code Stack dataset, GPT-3.5 Turbo with RAG attains 90.57% accuracy (vs. 66.78% without RAG and the 60% human benchmark), and similar trends are observed for GPT-4 and GPT-4o.

Figure 5: LLM code analysis predictions with RAG context, including accurate bug categorization and correct identification of error-free snippets.

Figure 6: Inspection accuracy outcomes across models, benchmarking standard LLMs against their RAG-enhanced counterparts.

Error analysis details a sharp reduction in hallucination-induced misclassification—e.g., for "keyword as identifier" and "mismatched quotation" bugs, mismatch rates drop by an order of magnitude. However, variability persists: some simple syntax errors (e.g., missing colon, bug-free code) occasionally show less pronounced improvement, exposing residual LLM weaknesses even with RAG.

Figure 7: Runtimes for automated code inspection by model and RAG status, indicating varying trade-offs between speed and accuracy.

Comparative Error Type Distribution

Mismatch count analyses across model variants and bug types confirm that RAG consistently lowers erroneous predictions in otherwise challenging categories.

Implications and Future Directions

The integration of RAG pipelines into LLM-centric V&V directly addresses core practical issues: hallucination reduction, knowledge grounding, and sustained reliability as codebases and requirements evolve. These results demonstrate that hybrid architectures—retrieval and generation—achieve higher accuracy than monolithic LLM prompting. Moreover, empirical outcomes show that RAG enables LLMs to not only match but generally exceed specialized human inspector performance across standard datasets.

From a theoretical standpoint, these findings reinforce the importance of context-aware, semantically-conditioned decoding for software engineering automation. Practically, such pipelines can be deployed to reduce manual review/tuning load, shorten the verification/test/debug loop, and serve as a robust first-line of defense in CI/CD workflows. Runtime overhead and context selection/optimization are salient engineering concerns, meriting research into adaptive, relevance-driven retrieval policies.

The analysis also identifies future research frontiers: context selection under tight latency constraints, robust benchmark development for new defect types, dynamic adaptation to code domain drift, integration of alternative knowledge modalities beyond code (design docs, issue trackers), and formal semantic evaluation metrics (e.g., CodeBLEU).

Conclusion

This research advances the state of automated V&V by demonstrating that RAG-enhanced LLMs significantly improve over baseline LLMs and human inspectors for both code inspection and test case generation tasks. RAG-based frameworks serve as critical infrastructure for trustworthy and scalable LLM deployment in software engineering, with empirical benefits in accuracy, reliability, and (qualified) efficiency. Long-term, this paradigm sets a foundation for adaptive, continuously-learning code analysis agents that can robustly augment software engineers in high-assurance and scale-sensitive environments.