KohakuRAG: Hierarchical RAG Framework

• KohakuRAG is a retrieval-augmented generation framework with hierarchical document indexing that improves citation-sensitive technical question answering.
• It employs a four-level tree structure with bottom-up embedding aggregation and LLM-powered query planning with cross-query reranking to enhance retrieval coverage.
• The system stabilizes output via ensemble inference with abstention-aware voting, achieving top leaderboard performance with notable ablation improvements.

Searching arXiv for the specified paper and closely related background papers mentioned in the provided data. Searching for "KohakuRAG: A simple RAG framework with hierarchical document indexing" on arXiv. KohakuRAG is a retrieval-augmented generation (RAG) framework introduced as “a simple RAG framework with hierarchical document indexing.” In the available record, it is presented as a system for question answering over document collections that combines hierarchical document representation, retrieval expansion, and answer stabilization, with the stated objective of improving citation-sensitive technical QA. The primary source characterizes the framework as preserving document structure through a four-level tree representation, improving retrieval coverage through query planning and reranking, and stabilizing outputs through ensemble inference with abstention-aware voting (Yeh et al., 8 Mar 2026).

1. Definition and stated scope

KohakuRAG is defined by two closely related descriptions in the source record. First, its title foregrounds “hierarchical document indexing.” Second, the abstract describes a broader RAG pipeline that includes hierarchical indexing, an LLM-powered query planner with cross-query reranking, and ensemble inference with abstention-aware voting (Yeh et al., 8 Mar 2026). A narrow reading of the title would therefore understate the scope of the system as described in the abstract.

The available material presents KohakuRAG as a framework rather than a single retrieval heuristic or a single generation prompt. Its central theme is the organization of documents into a hierarchy, but the framework is also explicitly associated with retrieval coverage and answer stability. This suggests that the paper’s contribution is framed at the system level: indexing, retrieval, and inference are treated as coupled design problems rather than isolated stages.

Within the provided content, KohakuRAG is also identified as open-source software, with the repository given as https://github.com/KohakuBlueleaf/KohakuRAG (Yeh et al., 8 Mar 2026). No additional implementation metadata is specified in the summary beyond that repository link.

2. Problem formulation in citation-sensitive RAG

The abstract situates KohakuRAG in a specific failure regime for RAG systems: answering questions from document collections when high-precision citations are required. Three compounding difficulties are named. Flat chunking strategies are said to sacrifice document structure; single-query formulations are said to miss relevant passages through vocabulary mismatch; and single-pass inference is said to produce stochastic answers that vary in both content and citation selection (Yeh et al., 8 Mar 2026).

These problem statements establish the framework’s intended operating conditions. The target setting is not generic open-domain QA, but technical question answering in which source attribution and numerical exactness matter. In that setting, retrieval errors and generation variance are not merely nuisances; they affect whether an answer is traceable to a source and whether numerical outputs satisfy evaluation constraints.

A plausible implication is that KohakuRAG is designed for tasks in which retrieval recall, document structure preservation, and inference consistency interact directly with the scoring function. The source record does not, however, provide a formal problem statement, loss formulation, or task-specific notation beyond the brief abstract-level characterization.

3. Architectural description

The most explicit methodological claim in the available material is that KohakuRAG preserves document structure through “a four-level tree representation (document $\rightarrow$ section $\rightarrow$ paragraph $\rightarrow$ sentence) with bottom-up embedding aggregation” (Yeh et al., 8 Mar 2026). This is the core architectural description supplied in the abstract and is the main concrete elaboration of the title’s phrase “hierarchical document indexing.”

The same abstract attributes two additional mechanisms to the framework. Retrieval coverage is said to be improved through “an LLM-powered query planner with cross-query reranking,” and answer stability is said to be improved through “ensemble inference with abstention-aware voting” (Yeh et al., 8 Mar 2026). Taken together, these statements describe a three-part architecture: hierarchical indexing, retrieval expansion and consolidation, and output aggregation.

At the same time, the accompanying summary repeatedly marks many implementation-level details as not specified. The visible record does not provide a formal representation of the tree, an explicit aggregation function, weighting or normalization schemes, prompt templates for the query planner, reranking formulas, or the voting rule for abstention-aware ensembling. The available description is therefore architectural in outline but not algorithmic in full detail.

Component	Stated in the record	Not specified in the record
Hierarchical indexing	Four-level tree; bottom-up embedding aggregation	Formal representation; aggregation equations; weighting
Retrieval	LLM-powered query planner; cross-query reranking	Prompt structure; decision logic; scoring function
Inference stabilization	Ensemble inference; abstention-aware voting	Voting rule; thresholds; tie-breaking; pseudocode

This division between high-level specification and missing low-level detail is important for interpretation. The record supports claims about the system’s named components and their intended roles, but not claims about exact algorithms, hyperparameters, or implementation recipes.

4. Evaluation setting and reported results

KohakuRAG is evaluated on the WattBot 2025 Challenge, which the abstract describes as “a benchmark requiring systems to answer technical questions from 32 documents with $\pm 0.1\%$ numeric tolerance and exact source attribution” (Yeh et al., 8 Mar 2026). That benchmark description is central to understanding the paper’s reported results, because it makes clear that both numerical precision and citation exactness are part of the task definition.

The abstract reports that KohakuRAG “achieves first place on both public and private leaderboards (final score 0.861), as the only team to maintain the top position across both evaluation partitions” (Yeh et al., 8 Mar 2026). This is the principal empirical claim in the available material. It indicates competitive performance on both leaderboard splits and emphasizes consistency across evaluation partitions rather than performance on only one split.

The source also reports several ablation findings. Prompt ordering is said to contribute “+80% relative,” retry mechanisms “+69%,” and ensemble voting with blank filtering “+1.2pp.” It further states that “hierarchical dense retrieval alone matches hybrid sparse-dense approaches,” with “BM25 adds only +3.1pp” (Yeh et al., 8 Mar 2026). These ablation statements are notable because they assign substantial impact to prompt sequencing and retry logic, while characterizing the marginal gain from BM25 as limited in the reported setting.

Because the visible record does not include the experimental section itself, these results are available only at summary granularity. The dataset split protocol, exact metric definition, statistical treatment, and error analysis are not specified in the provided material.

5. System interpretation and relation among components

The source record supports an interpretation of KohakuRAG as a system in which document organization, retrieval diversification, and generation stabilization are deliberately coupled. The hierarchical index is presented as the mechanism for preserving structure; the query planner and cross-query reranking are presented as the mechanism for improving retrieval coverage; and ensemble inference with abstention-aware voting is presented as the mechanism for reducing stochastic variability in answers and citations (Yeh et al., 8 Mar 2026).

This suggests a design philosophy in which failures of citation-sensitive RAG are distributed across multiple stages of the pipeline. Under that reading, flat chunking corresponds to representational loss, single-query retrieval to coverage loss, and single-pass generation to inference variance. KohakuRAG’s architecture mirrors that decomposition by assigning a distinct intervention to each stage. The source does not explicitly formulate this as a general theory, but the correspondence is strongly implied by the abstract’s problem-solution structure.

A common misunderstanding would be to reduce KohakuRAG to hierarchical retrieval alone. The available material does not support that reduction. Although hierarchical indexing is the framework’s titular feature, the abstract explicitly attributes performance and stabilization to query planning, reranking, retries, prompt ordering, and ensemble voting as well (Yeh et al., 8 Mar 2026).

6. Availability, underspecification, and limits of the available record

The paper summary is unusually explicit about what is not specified in the visible content. It states that the precise motivation statements and problem formulation are not specified beyond what is visible; formal representations and aggregation formulas are not specified; retrieval modality details are not specified; and the mechanisms for ensemble inference, retry logic, and prompt ordering are not specified in operational terms. It also states that models, embeddings, chunk sizes, aggregation parameters, index construction details, query planner templates, reranking models, ensemble size, abstention criteria, and inference-time configurations are not specified in the provided content (Yeh et al., 8 Mar 2026).

These omissions matter for scholarly use. The available record is sufficient to establish the paper’s claimed architecture, benchmark setting, headline leaderboard outcome, major ablation trends, and repository location. It is not sufficient to reconstruct the method in implementation detail or to verify the causal pathways behind each ablation without consulting the full paper and code.

The same summary also states that trade-offs, limitations, failure modes, and computational or latency considerations are not specified in the provided material (Yeh et al., 8 Mar 2026). As a result, no definitive claims can be made from the supplied record about efficiency, scaling behavior, or deployment constraints. Any discussion of such properties would be inferential rather than documentary.

In encyclopedic terms, KohakuRAG is therefore best characterized as a hierarchical RAG framework whose publicly stated contributions lie in structure-preserving indexing, multi-query retrieval support, and stabilization of citation-sensitive answer generation, with strong benchmark claims and open-source availability, but with many algorithmic and operational specifics absent from the excerpted record (Yeh et al., 8 Mar 2026).