Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development

Abstract: Software documentation is crucial for repository comprehension. While LLMs advance documentation generation from code snippets to entire repositories, existing benchmarks have two key limitations: (1) they lack a holistic, repository-level assessment, and (2) they rely on unreliable evaluation strategies, such as LLM-as-a-judge, which suffers from vague criteria and limited repository-level knowledge. To address these issues, we introduce SWD-Bench, a novel benchmark for evaluating repository-level software documentation. Inspired by documentation-driven development, our strategy evaluates documentation quality by assessing an LLM's ability to understand and implement functionalities using the documentation, rather than by directly scoring it. This is measured through function-driven Question Answering (QA) tasks. SWD-Bench comprises three interconnected QA tasks: (1) Functionality Detection, to determine if a functionality is described; (2) Functionality Localization, to evaluate the accuracy of locating related files; and (3) Functionality Completion, to measure the comprehensiveness of implementation details. We construct the benchmark, containing 4,170 entries, by mining high-quality Pull Requests and enriching them with repository-level context. Experiments reveal limitations in current documentation generation methods and show that source code provides complementary value. Notably, documentation from the best-performing method improves the issue-solving rate of SWE-Agent by 20.00%, which demonstrates the practical value of high-quality documentation in supporting documentation-driven development.

• The paper presents SWD-Bench, a novel benchmark that uses functional QA tasks to evaluate the quality of repository-level software documentation.
• It details a multi-stage pipeline combining data sourcing, contextual enrichment, and manual quality assurance to ensure comprehensive and reliable evaluation.
• The paper shows that integrating detailed documentation with source code improves functionality detection by up to 40% and increases downstream issue resolution success by 20%.

Evaluating Repository-Level Software Documentation through Functional QA and Feature-Driven Development

Motivation and Problem Setting

The proliferation of LLM-powered methods for software documentation has enabled a shift from snippet-level summaries to repository-wide documentation generation. However, the field faces significant challenges in evaluation: traditional benchmarks assess documentation at fragmentary levels, disregarding holistic repository context, and primarily depend on unreliable metrics—like vague LLM-as-a-judge Likert scoring—which often lack precise definitions and are insensitive to practical utility. This creates a mismatch between how documentation assists developers in real scenarios and how its quality is measured.

Documentation-Driven Development and SWD-Bench Strategy

Drawing inspiration from documentation-driven development—in which high-quality, comprehensive documentation is central to repository navigation and comprehension—the authors develop SWD-Bench, a novel evaluation benchmark designed to address both the lack of repository-level perspective and unreliable existing strategies.

The benchmark operationalizes functional evaluation by conceiving QA tasks that mirror a real developer’s workflow: detecting which functionalities are present, localizing their implementation, and completing technical details (i.e., functional parameters, API usage). Critically, SWD-Bench treats LLMs as stand-ins for developers who must answer practical, contextualized questions using provided documentation, thereby evaluating utility rather than superficial textual characteristics.

Figure 1: A typical workflow of documentation-driven development.

Benchmark Construction Pipeline

The SWD-Bench creation pipeline comprises several rigorously-designed stages, ensuring high data quality and repository-level coverage:

1. Data Sourcing and Filtering: The benchmark draws on 12 highly-active open-source repositories, systematically scraping and filtering 177k+ PRs to retain only those that are both functionality-relevant and persistent (contributions that exist in the selected repository snapshot). Filtering ensures PRs represent actual developer efforts and contain sufficient context.
2. Contextual Enrichment: Each PR is augmented with diverse repository-level context, including dependency analysis (via Tree-Sitter for call/callee relationships), associated issues (from GitHub or custom crawlers), web documentation, and commit histories. This step provides global semantic information vital for functional understanding.
3. Task Generation: Functional descriptions are crafted using LLMs guided by chain-of-thought strategies, ensuring intent-alignment across WHAT, WHY, and HOW dimensions. QA pairs are then curated to require multi-file comprehension and avoidance of answer leakage.
4. Manual Quality Assurance: A Kappa coefficient exceeding 90% validates the data's clarity and correctness, underscoring SWD-Bench’s reliability.

   Figure 2: The overview of SWD-Bench's data construction pipeline.

Functional QA Tasks

The benchmark includes three QA categories:

• Functionality Detection: Binary determination of whether a described feature exists in the repository.
• Functionality Localization: Predicting the set of files implementing a specific functionality.
• Functionality Completion: Cloze-style filling of masked technical details, requiring deep contextual retrieval.

Each entry in SWD-Bench is substantial, necessitating localization over multiple files and detail completion spanning up to 23 items per case.

Experimental Setup and Model Evaluation

Six documentation approaches are benchmarked, spanning human-written artifacts, baseline snippet-level LLM methods (e.g., Chat), and repository-level auto-documentation tools (DeepWiki, AutoDoc, DocAgent, RepoAgent). For each, relevant documentation is retrieved using SFR-Embedding-Code-400M_R, across various context window sizes (1024–4096 tokens).

Genuine developer performance is simulated by evaluating documentation with state-of-the-art LLMs (GPT-4.1, Gemini-2.5-pro) in the role of a repository developer, directly answering QA tasks.

Key Results and Analysis

Performance Ceiling and Method Comparison

SWD-Bench is demonstrably challenging: even leading methods like RepoAgent achieve modest absolute scores (e.g., best-case functionality detection MCC ≈ 29.4, functionality completion EM₁.₀ ≈ 28–30), substantiating the gap to practitioner requirements.

Fine-grained, context-rich methods (RepoAgent, DocAgent) consistently outperform intermediate and coarse-grained ones, showing relative improvements of up to 65% in key metrics over less contextualized baselines. This is attributed to their superior ability to encode dependency graphs and present actionable, detailed documentation.

Human-written documentation remains competitive, illustrating developers’ inherent integration of holistic context—hence, models that emulate this integrative approach achieve higher efficacy.

Context Window and Model Robustness

Extensive documentation retrieval (up to 4096 tokens) yields progressive improvements across all tasks, mirroring real-world developer engagement. Model-wise rankings are stable across LLM foundations, demonstrating benchmark robustness under systemic variation.

Evaluation Paradigm Comparison

Through detailed case study analysis (see Figure 3), the authors validate SWD-Bench’s superiority over LLM-as-a-judge methods. Traditional Likert-scale LLM ratings fail to distinguish documentation’s real-world functional utility; SWD-Bench QA-based evaluation exposes documentation that enables functional understanding—even when surface-level assessments are indistinguishable.

Figure 3: A case study on the functionality detail task, including the question, relevant code, documentation retrieved by different methods, and evaluation comparison.

Practical Impact: Documentation and Issue Resolution

Providing high-quality documentation substantially boosts automated issue-solving agents (e.g., SWE-Agent)—with best-performing generated documentation increasing task success rates by up to 20%. There's a clear monotonic relationship: higher documentation quality (per SWD-Bench assessment) directly translates to improved downstream development outcomes.

Figure 4: Performance of SWE-Agent on issue solving when provided with different software documentation.

Source Code and Documentation: Complementary Modalities

Augmenting documentation-based QA with source code retrieval yields further significant gains, particularly in functionality detection and localization (10–40% improvement), while benefits in detail completion are more modest. This demonstrates the necessity of integrating both implementation and descriptive artifacts for comprehensive repository comprehension in LLM-centric workflows.

Implications and Future Directions

For developers, the findings reinforce the importance of combining detailed documentation with targeted source code reading for repository understanding and efficient problem solving.

For researchers, closing the intent-orientation gap in auto-documentation remains critical. Techniques explicitly designed for holistic context integration and cross-modal fusion—spanning code and natural language documentation—are particularly promising. In addition, the SWD-Bench framework facilitates extending evaluation to broader scenarios (e.g., code review, refactoring), providing a foundation for methodical progress in automated documentation.

Potential future work includes expanding SWD-Bench’s repository coverage, refining QA synthesis for richer information needs, and exploring advanced LLM prompting/fusion strategies to further narrow the practical utility gap.

Conclusion

SWD-Bench redefines automated software documentation evaluation by anchoring assessment in functionally-driven, repository-level QA tasks reflective of developer workflows. It systematically addresses the core shortcomings in prior evaluation strategies, provides stable and actionable performance metrics, and illuminates concrete pathways for improving both machine-generated documentation and the downstream applications that rely on it. SWD-Bench thus forms an essential reference for future research and practice in this rapidly advancing intersection of software engineering and AI.