Generalizability of the proposed GraphRAG framework beyond the code migration domain

Determine whether the proposed GraphRAG framework—comprising dependency parsing-based knowledge graph construction and a hybrid retrieval strategy that fuses one-hop graph traversal with Reciprocal Rank Fusion—generalizes effectively beyond the ABAP custom code migration domain by rigorously evaluating its performance and applicability in other settings, including public question answering benchmarks such as HotpotQA.

Background

The paper introduces a practical GraphRAG framework tailored for enterprise-scale deployment, emphasizing efficient knowledge graph construction via dependency parsing and a hybrid retrieval approach that combines graph traversal with vector similarity using Reciprocal Rank Fusion. The system is evaluated on enterprise datasets related to ABAP custom code migration, demonstrating notable improvements over dense vector baselines while significantly reducing construction costs compared to LLM-based extraction.

Despite strong results within the code migration domain, the authors explicitly note uncertainty about how well the approach transfers to other domains and tasks. They propose future evaluations on broader public benchmarks, such as HotpotQA, to assess whether the dependency-based construction and hybrid retrieval strategies maintain effectiveness outside the tested enterprise context.

References

Second, although our method demonstrates strong performance in code migration domain, its generalizability to other settings remains an open question.

— Towards Practical GraphRAG: Efficient Knowledge Graph Construction and Hybrid Retrieval at Scale  (2507.03226 - Min et al., 4 Jul 2025) in Section: Conclusion, Limitation and Future Work

This rerun will specifically incorporate knowledge-graph-enhanced retrieval, extending AquiLLM's existing hybrid vector/lexical search (Section~\ref{sec:retrievalconfig}) with a graph of relationships between scientific concepts, entities, and their supporting evidence, since Comparative Scientific Analysis, the primary failure surface identified in Section~\ref{sec:results}, centers on cross-source synthesis and reasoning that a graph-structured representation is intended to support more directly than chunk-level retrieval alone.

— AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research  (2609.16519 - Boscoe et al., 15 Sep 2026) in Section 7, “Future Work” (Section \ref{sec:futurework})