Library-Aware Doubles and Iterative Repair for Large Language Model-Generated Unit Tests in OpenSIL Firmware

Abstract: Validating changes in low-level C firmware is expensive because unit tests (UTs) are fragile under strict build constraints, where missing headers, unresolved symbols, and dependency mismatches frequently prevent compilation and linking. This study introduces an automated UT authoring workflow for the Open-Source Silicon Initialization Library (openSIL) firmware codebase maintained by Advanced Micro Devices (AMD) that reduces manual effort through a LLM guided multi-agent pipeline. The workflow combines automated generation of test scaffolds, library-aware creation or reuse of stubs, mocks, and fakes, and an iterative compile-dispatch repair loop driven by build logs and line-coverage feedback. We evaluate the approach using compilation success, repair iterations, dispatch success, and line coverage, with time, cost, and token usage as secondary measures. Across 76 functions under test, the workflow generated compilable UTs for 73 functions. In a configuration without line coverage guidance or retrieval augmentation, mean line coverage reached 73.9%. On a 48-function subset evaluated under both configurations, mean line coverage reached 98.8% with line-coverage guidance alone and reached 94.7% when combined with vector-database retrieval. Results show that automated generation-and-repair pipelines can substantially improve UT creation efficiency and coverage for constrained firmware environments while reducing manual debugging effort.

• The paper introduces a multi-agent pipeline that iteratively refines generated tests using LCOV-guided feedback and library-aware doubles.
• Empirical evaluations show a 96.1% buildability rate and significant improvements in line coverage compared to direct LLM test generation.
• It emphasizes a compile-dispatch-repair loop to address dependency management and symbol conflicts in complex, build-constrained firmware environments.

Library-Aware Doubles and Iterative Repair for LLM-Generated Unit Tests in OpenSIL Firmware

Introduction

This paper investigates the challenges and proposes a robust methodology for automating unit test (UT) generation for complex low-level firmware code, specifically focusing on the AMD openSIL platform under EDK II build constraints. The salient problem addressed is the fragility of traditional LLM-based UT generation in environments requiring strict compilation and linking discipline, extensive use of hardware abstractions, and sophisticated dependency management. The work's core contribution is an automated, multi-agent pipeline that integrates library-aware test double handling, an iterative compile-dispatch-repair loop, and coverage-guided refinement, augmented by project-specific retrieval via a vector database (VDB).

Methodology

The pipeline is orchestrated as a LangGraph-based state machine encompassing eleven discrete, role-specialized stages, systematically transitioning UTs from initial scaffolds to dispatchable, high-coverage artifacts. The workflow begins with the retrieval of reusable stubs, mocks, and fakes (test doubles) through Chroma-embedded VDB queries that consider project- and module-specific constraints. This retrieval is critical for minimizing symbol conflicts and leveraging existing, buildable helper code.

Drafting, assembly, and subsequent repair operations are performed by distinct LLM agents, each tasked via structured prompts enforcing separation of responsibilities, symbol hygiene, and template conformance. The system incorporates compiler/linker diagnostics, dispatcher outputs, and LCOV-based line coverage analytics at each iteration, directly guiding minimal edits toward buildability and improved coverage.

Notably, the workflow strictly differentiates between "deep doubles"—cross-module dependency replacements with behavioral payload—and "shallow stubs"—minimal, compilation-oriented artifact stubs strictly for sibling call fallout. Guardrails prohibit unsafe symbol redefinitions, and all modifications are restricted to template-approved code regions.

Empirical Evaluation

Experiments comprise 76 distinct functions under test (FUTs), categorized by size and dependency profile, excluding static-only targets without explicit harnesses. All experiments use the openSIL codebase, EDK II, and a set of recent, reasoning-tuned LLMs.

Key findings include:

• Buildability: The workflow achieves a 96.1% compilation/linking success rate (73/76 FUTs), substantially outperforming direct LLM-based UT generation approaches, which are hampered by unresolved dependencies and compilation constraint violations.
• Efficiency and Iterative Repair: Small and medium functions stabilize within two to three iterations, whereas large, pointer-heavy, or XFER/Ip2Ip-involved functions require 4–9.5 iterations, particularly when integrating VDB-retrieved doubles.
• Coverage: In an ablation on a representative 48-FUT subset, LCA-only configuration yields a mean line coverage of 98.8%, with LCA+VDB at 94.7%. Absence of coverage guidance or retrieval yields only 73.9% mean coverage, and direct LLM generation performs markedly worse (down to 12.3% for XFER/Ip2Ip).

  Figure 2: Iterations, time, and cost required for achieving compilation and coverage thresholds, with and without line coverage analysis (LCA).

  

  Figure 4: Resource usage distribution (runtime, cost, and tokens) comparing configurations with and without LCA guidance.

• Resource and Cost Dynamics: LCA materially reduces repair iterations, time, and cost, especially for complex functions. VDB retrieval decreases wall-clock runtime for complicated/test double-heavy FUTs but increases prompt token usage variability.

Category-Specific Analysis

Small and medium, self-contained functions benefit most from coverage-guided repair, generally converging to full coverage rapidly and with consistent resource use. Large, XFER-based, and Ip2Ip-function categories display markedly greater variability, with VDB-enabled retrievals providing notable runtime reductions at the cost of higher token usage.

Figure 6: LCA-only configuration—breakdown of time, cost, and tokens by function category; high predictability for small/medium, but inefficiency for large/XFER without retrieval.

Figure 8: LCA+VDB configuration—amortized runtime gains for large, dependency-heavy categories, with greater token utilization.

Discussion

A critical insight is that most failures result not from test-case logic generation, but from build-time wiring—missing headers, unresolved/duplicated symbols, and incorrect harness assembly dominate error modes. This is consistent with recent literature emphasizing the necessity of compile–run–repair loops in practical LLM-based test generation (Yang et al., 2024, Karanjai et al., 2024, Zhang et al., 27 Jan 2025).

The explicit retrieval and reuse of extant test doubles mitigates linker hazards and consolidates double logic, reducing the likelihood of subtle integration faults and test suite inconsistency. However, aggressive reuse strategies can cap coverage in fixed-iteration regimes, highlighting a trade-off between buildability robustness and coverage maximization.

Coverage-guided iteration, using LCOV feedback, is shown to be essential for achieving high line coverage, especially as codebase complexity increases. This echoes results from recent coverage-guided, LLM-enabled generation systems (Yang et al., 2024, Zhang et al., 18 Jun 2025).

The introduction of retrieval-augmented VDB context amplifies prompt length, resource, and token requirements, but enables the pipeline to reach coverage and buildability targets more efficiently in large-scale, complex environments.

Implications and Future Directions

The methodology formalized herein generalizes to any C-based firmware/system-level codebase with significant module boundaries and nontrivial dependency structures, as long as existing doubles can be indexed and reused. It demonstrates that pure autoregressive generation is inappropriate (and unreliable) for build-constrained targets—direct LLM invocation produces low coverage and fails to integrate with industrial build ecosystems.

From a practical standpoint, such a pipeline can substantially amortize manual debugging and authoring costs of firmware UTs; it may become a foundational approach for automated validation workflows in large silicon and embedded systems environments.

On a theoretical axis, the work clarifies that successful AI enablement in practical software engineering requires not only sophisticated generation, but tightly controlled systems integration, feedback-driven iteration, and project-specific code context injection. The pipeline's modularity admits straightforward extension: better graph-based code dependency analysis, integration of more sophisticated oracles, and automated evaluation infrastructure for test reliability.

Potential next steps include transfer to other codebases, extending static coverage to dynamic/concolic analysis, and benchmarking against industrial developer productivity baselines.

Conclusion

This study offers compelling evidence that LLM-based UT generation in real-world, dependency-heavy firmware settings must be tightly coupled with build- and library-aware iterative repair, structured prompt discipline, coverage-guided feedback, and retrieval-based code context. The result is consistently high buildability and coverage, with strong evidence that project-aware automation can substantially mitigate the traditional fragility of generative code workflows in C/firmware engineering. The outlined workflow serves as a scalable foundation for robust, automated unit test generation in build-constrained, modular systems (2606.19725).