DocFetch: Multi-Artifact Documentation
- DocFetch is a research prototype that generates structured software documentation by integrating information from commits, issues, pull requests, textual files, and code comments.
- It uses a multi-layer, prompt-based LLM pipeline to extract and consolidate API, file, error, license, and project details from varied artifact sources.
- Empirical results highlight strong performance for API- and file-related documentation, though challenges include hallucinated content and context-window truncation.
DocFetch is a research prototype and methodology for generating structured software documentation from multiple software artifacts rather than from source code alone. It is built on the DocMine dataset and uses a multi-layer, prompt-based LLM pipeline to synthesize repository information from commits, issues, pull requests, textual files, and source code comments into documentation organized by type, including API-related, file-related, error/bug-related, license-related, and project-related information (Venigalla et al., 25 Aug 2025).
1. Problem setting and rationale
DocFetch is motivated by the observation that documentation in large open-source software projects is often incomplete, inconsistent, insufficient or incorrect, while the projects themselves evolve rapidly. The reported examples include repositories such as microsoft/vscode and tensorflow/tensorflow, where high commit rates make manual documentation maintenance difficult. The work situates documentation as critical for comprehension, reuse, contributor retention, and project popularity, but also as expensive to maintain if handled manually (Venigalla et al., 25 Aug 2025).
The central claim behind DocFetch is that code-only documentation generation is insufficient. Existing automated approaches are described as focusing mainly on source code summaries, commit message generation from code diffs, and, in some cases, pull-request titles or descriptions. DocFetch instead starts from the premise that information useful for documentation is scattered across co-evolving artifacts such as issues, pull requests, commits, README files, license files, and source code comments. This multi-artifact view is intended to capture usage hints, configuration information, bug discussions, API usage patterns, and project-level practices that do not reside exclusively in code (Venigalla et al., 25 Aug 2025).
This framing is grounded in a prior exploratory study that identified 1.38 million artifacts across 950 GitHub repositories and found documentation-related information dispersed across commits, issues, pull requests, textual files, and source code comments. A plausible implication is that documentation generation becomes less a code summarization problem than a repository-level synthesis problem, where the relevant evidence is distributed across multiple textual channels rather than concentrated in a single artifact class (Venigalla et al., 25 Aug 2025).
2. Artifact base and documentation taxonomy
DocFetch operates over the DocMine dataset, which organizes mined repository artifacts by repository and documentation type. For each repository, the system can access extracted text from five documentation sources: pull requests, issues, commits, textual files, and source code comments. These sources are treated as parallel evidence streams for a selected documentation target (Venigalla et al., 25 Aug 2025).
The documentation targets are organized into a type-based taxonomy. The reported implementation and experiments focus on five types, while the broader background taxonomy also includes architecture-related documentation.
| Documentation type | Intended content |
|---|---|
| API-related | API names, parameters, return types, behaviors, usage context |
| Error/Bug-related | Error messages, contexts, fixes, stack-trace-related discussions |
| File-related | File roles, dependencies, refactorings, file-level changes |
| License-related | License type, permissions, restrictions, notices |
| Project-related | Installation, configuration, supported environments, contribution guidelines |
API-related documentation is described as covering function names, packages or libraries, arguments, return types, usage conditions, and example calls. Error/Bug-related documentation captures recurring error patterns, stack traces, causes, and fixes. File-related documentation addresses file organization, dependencies, and file-level evolution. License-related documentation summarizes legal permissions and restrictions. Project-related documentation gathers installation instructions, environment requirements, contributing guidance, release-oriented information, and general usage instructions (Venigalla et al., 25 Aug 2025).
The prior classification also identified architecture-related documentation. The reported experiments, however, concentrate on the five types above. This suggests a broader conceptual scope than the currently evaluated implementation, with architecture-related synthesis remaining a natural extension of the same multi-artifact design (Venigalla et al., 25 Aug 2025).
3. Multi-layer prompt architecture
DocFetch is implemented as a two-level architecture. The user interacts through a React-based web UI, selects a GitHub repository from DocMine through a text box or dropdown, chooses a documentation type, and receives a downloadable .json document containing the generated structured documentation (Venigalla et al., 25 Aug 2025).
The underlying model is Gemini 2.0 Flash Lite Preview (GFLP2.0), chosen in part because it provides a large context window of about 37K characters. The first level of the architecture performs per-source generation: five parallel LLM instances process pull requests, issues, commits, textual files, and source code comments separately. Each Level 1 instance uses a two-shot prompt with two few-shot input/output pairs and a target repository context. The prompt variables are described as few_shot_ip1, few_shot_ip2, few_shot_op1, few_shot_op2, and context_input, where the output exemplars are manually curated intermediate groundtruth JSON files (Venigalla et al., 25 Aug 2025).
At Level 1, the model is asked to produce source-specific structured JSON for the selected documentation type. The paper gives an example instruction for error-related content from pull requests: “list out all discussions related to errors in <pull request text> and explain the respective discussions.” In practice, this instruction is wrapped inside the few-shot template so that the model learns not only what content to extract but also the target JSON structure (Venigalla et al., 25 Aug 2025).
The second level performs cross-source consolidation. A separate LLM instance receives the five Level 1 outputs for a repository and documentation type, together with a one-shot example showing how five intermediate JSON documents from another repository are merged into a single final JSON document. The output is a consolidated, documentation-type-specific JSON artifact that integrates evidence from all five sources (Venigalla et al., 25 Aug 2025).
The design is explicitly constrained by context length. Very large artifacts are sometimes truncated rather than aggressively chunked, because multi-stage chunk consolidation would itself exceed context limits and increase complexity. The authors also report varying the repositories used in few-shot examples across runs in order to reduce dependence on any single exemplar repository (Venigalla et al., 25 Aug 2025).
4. Groundtruth construction and evaluation protocol
Because DocMine provides documentation-type labels for artifacts but not final structured documentation, the evaluation required a separate manually curated groundtruth. The authors sampled 1,205 artifacts from 18 repositories, noting that this exceeds a statistically significant sample of 664 artifacts for 99% confidence and a 5% margin of error (Venigalla et al., 25 Aug 2025).
Groundtruth construction proceeded in two stages. First, the first author manually analyzed the sampled artifacts and created intermediate groundtruth JSON files for each repository, documentation type, and documentation source. Since there are 5 documentation sources and 5 documentation types, this yields 25 intermediate groundtruths per repository. Second, the intermediate groundtruths for each type were consolidated into 5 final groundtruth JSON files per repository. Across 18 repositories, the result is 450 intermediate groundtruths and 90 final groundtruths, or 30 groundtruth files per repository (Venigalla et al., 25 Aug 2025).
These groundtruths were then reviewed by two volunteer researchers, one a PhD student and one a Master’s student. The paper reports that, after iterative discussion, they reached 100% agreement. This groundtruth corpus is used both for prompt construction and for evaluation (Venigalla et al., 25 Aug 2025).
Evaluation is conducted at both architectural levels. For RQ1, the Level 1 per-source outputs are compared against the intermediate groundtruths. For RQ2, the Level 2 consolidated outputs are compared against the final groundtruths. The reported metrics are BLEU-4 and ROUGE-L, used to measure similarity between generated JSON documentation and the manually curated reference documentation (Venigalla et al., 25 Aug 2025).
5. Empirical results and qualitative behavior
The strongest reported results occur for API-related and file-related documentation generated after consolidating information from all five sources. The paper reports a highest BLEU-4 score of 43.24% and a highest ROUGE-L score of 0.39 for these documentation categories. It also states that other documentation types achieve BLEU-4 scores close to 30%, which it interprets as good performance for the task setting (Venigalla et al., 25 Aug 2025).
At the Level 1 source-specific stage, performance varies substantially by source and documentation type. For ROUGE-L, the reported highlights include 0.60 for error-related information from commits, 0.71 for license-related information from pull requests, 0.66 for license-related information from comments, 0.18 for project-related information from issues, and 0.20 for file-related information from textual files. The same section reports that commits and pull requests are particularly strong for error-related and file-related documentation, whereas issues are comparatively weak for project-related information and textual files are weak for file-related extraction under the adopted schema (Venigalla et al., 25 Aug 2025).
The BLEU-4 results at Level 1 show a similar pattern. Reported values include 39.63% for error-related information from commits, 41.47% for project-related information from commits, 36.42% for file-related information from commits, 35.65% for API-related information from commits, and 35.13% for API-related information from pull requests. These numbers support the broader claim that documentation signal strength depends materially on artifact type, and that different documentation categories draw on different parts of the repository’s textual ecosystem (Venigalla et al., 25 Aug 2025).
The paper also presents qualitative evidence that DocFetch can generate information that is more detailed than the manually curated groundtruth. In an example drawn from the FastJsonParser repository, issue discussions mention the DeviceNotification API, the ServiceAction property, and stack traces involving System.Text.Json.JsonParser. The groundtruth API-related documentation from issues omitted some of this information. DocFetch’s output, by contrast, identified ServiceAction as an enum and described System.Text.Json.JsonParser as a JSON parser in .NET. The authors present this as evidence that the system can infer contextually correct API details from artifact discussions, even when the groundtruth is less complete (Venigalla et al., 25 Aug 2025).
The reported limitations are equally explicit. The system can introduce hallucinated or out-of-context information because of LLM pretraining, and some artifacts must be truncated because of context-window constraints. The paper also notes that BLEU and ROUGE do not directly measure usefulness or correctness; outputs may differ lexically from the groundtruth while still being informative (Venigalla et al., 25 Aug 2025).
6. Position within documentation research, limitations, and prospective extensions
DocFetch is situated against a broader literature in which automated documentation tools have often focused on source code alone (Venigalla et al., 25 Aug 2025). That code-centric lineage includes Autodoc, which treats code-to-comment generation as a neural machine translation problem and integrates docstring generation through AST manipulation, IDE plugins, and Git-hosting web hooks (Zheng et al., 2017). It also contrasts with systems aimed at consistency maintenance, such as CoDAT, which links comments to code blocks in IntelliJ IDEA and uses an LLM only for semantic consistency checking between code and comments (Attie et al., 2024).
The work is also adjacent to newer repository-scale documentation systems. DocAgent frames documentation generation as a multi-agent workflow with a Reader, Searcher, Writer, Verifier, and Orchestrator, coordinated through dependency-aware topological processing of repository structure (Yang et al., 11 Apr 2025). AutoDoc, in a different sense, generates API documentation from Stack Overflow knowledge by combining fine-tuned dense retrieval with GPT-4o summarization and dedicated validation steps to reduce hallucination and redundancy (Kou et al., 12 Jan 2026). Compared with these systems, DocFetch is distinctive in centering multiple repository artifacts as its primary evidence base and in organizing generation around documentation types rather than solely around code entities or external API discussions (Venigalla et al., 25 Aug 2025).
The paper identifies several practical limitations: dependence on LLM behavior, noisy artifact inputs, context-window constraints, and limited evaluation scope across only 18 repositories. It also notes that the current implementation is restricted to repositories already present in DocMine, and that external repositories would require dynamic artifact scraping, with associated API-rate and retrieval constraints (Venigalla et al., 25 Aug 2025).
The stated future work is concrete. The authors propose supporting more repositories beyond DocMine, adding time-bounded documentation generation for user-specified date ranges, expanding and refining the groundtruth with more repositories and annotators, improving artifact alignment at finer granularity, and integrating DocFetch into CI/CD pipelines and IDEs (Venigalla et al., 25 Aug 2025). This suggests a trajectory in which DocFetch evolves from a research prototype for semi-automatic documentation generation into a repository-scale documentation service that continuously synthesizes evolving project knowledge from heterogeneous software artifacts.