BYOKG-RAG: A Retrieval-Augmented KGQA Framework

• BYOKG-RAG is a retrieval-augmented generation framework that bridges natural language questions and custom knowledge graphs.
• It decomposes queries into structured artifacts like entities, paths, and executable queries for precise graph retrieval.
• Empirical evaluations show the framework improves KGQA accuracy and efficiency across diverse domains and benchmarks.

The BYOKG-RAG framework is a retrieval-augmented generation (RAG) approach for knowledge graph question answering (KGQA) that integrates LLMs and specialized graph retrieval tools to address both the semantic and structural variations present in custom ("bring-your-own") knowledge graphs (KGs). The framework systematically decomposes natural language questions, extracts and grounds relevant entities and relations using multiple retrieval strategies, and iteratively refines both graph linking and context retrieval to produce accurate answers. This method demonstrates improved robustness and generalization in KGQA tasks across diverse domains and knowledge graph structures (2507.04127).

1. Architecture and Workflow

BYOKG-RAG operates as a multi-component pipeline designed to bridge the semantic gap between natural language questions and knowledge graph data. The workflow is as follows:

1. Natural Language Query Input: The user submits a question in natural language form.
2. KG-Linker LLM Prompting: An LLM receives a multi-task prompt and generates four essential graph artifacts:
   • Question Entities: Entities detected in the question.
   • Candidate Answers: Entities or node classes hypothesized as possible answers.
   • Reasoning Paths: Sequences of relations hypothesized to connect question entities to answers.
   • Graph Queries: Executable queries (e.g., openCypher) that formalize the retrieval intention.
3. Graph Retrieval Integration: Specialized graph tools process these artifacts to align them with the target KG and retrieve supporting facts, paths, and candidate answer nodes.
4. Iterative Refinement: Retrieved context is fed back into the LLM, which refines its previous artifact generation and linking, resulting in improved answer generation upon subsequent iterations.
5. Final Answer Generation: After iterative refinement, the LLM composes a natural-language answer that is grounded in the retrieved and verified KG context.

2. Graph Artifacts Generation

The first step in BYOKG-RAG centers on structured multi-task LLM prompting to generate artifacts that connect language with a graph's structure. The principal tasks and their corresponding outputs are:

Task	Output Type	Output Notation
Entity Extraction	Set of entities	$\tilde{E}$
Path Identification	Set of relation-paths	$\tilde{P}$
Graph Query Generation	KG queries	$\tilde{Q}$
Answer Drafting	Initial answer	$\tilde{A}$

All outputs are formatted by the LLM in tagged sections (e.g., <entities>, <paths>, <opencypher>, <answer>) as defined by the KG-Linker prompt template. This explicit artifact generation decouples KG schema understanding from implicit LLM reasoning, making the approach more robust to custom or idiosyncratic KGs.

3. Multi-Strategy Graph Retrieval

The retrieved artifacts are operationalized via several graph retrieval modules:

• Entity Linking: LLM-identified entity strings are matched to KG nodes using both fuzzy string matching (e.g., "thefuzz") and embedding-based similarity (e.g., bge-m3 embeddings), resulting in a set of graph node identifiers.
• Path Retrieval: Given relation-paths, the system executes a breadth-first search (BFS) from the linked entities, using Dijkstra's algorithm where shortest paths are required, to enumerate potential reasoning chains in the KG:

$$f = \text{Follow-Paths}(\tilde{P}, S, E), \quad s = \text{Shortest-Paths}(E, \tilde{A}, S), \quad \text{final set} = f \cup s$$

• Graph Query Execution: Generated openCypher queries $\tilde{Q}$ are executed on the KG via an interface (e.g., Neo4j, Memgraph, NebulaDB), producing concrete answer sets $\hat{a} = \text{Execute-Query}(\tilde{Q}, KG)$.
• Triplet Retrieval: Fact-level retrieval is performed either agentically (LLM exploring one-hop neighborhoods, terminating when retrieval yields no new relevant entities) or via scoring-based filtering (ranking (head, relation, tail) triples for semantic similarity with the question).

These retrieval strategies can be employed in parallel or sequence, providing redundancy and coverage for diverse question types and KG schemas.

4. Iterative Refinement Mechanism

A distinctive feature of BYOKG-RAG is its iterative interaction between the LLM and retrieved KG context. After artifact generation and initial retrieval:

• The accumulated graph context (e.g., retrieved entities, paths, and facts) is "verbalized" using templates and supplied as additional input in the next LLM prompt.
• This process corrects prior linking errors, explores alternative relation chains, or suggests modifications to the generated query.
• The iteration proceeds for a predefined number of rounds (typically $T_R=2$), or terminates early if no new entities are extracted, balancing accuracy with efficiency.
• Each iteration leverages the updated KG schema, context, and candidate answers to further ground the LLM's predictions in the KG.

5. Empirical Performance and Generalization

BYOKG-RAG was evaluated on five KGQA benchmarks with KGs of distinct structures, including Freebase (WebQSP, CWQ), Wikidata (CronQuestions), medical domains (MedQA with Disease DrugBank KG), and enterprise-oriented (Northwind, Text2cypher). Results demonstrate:

• Performance Gain: Outperforms the second-best graph retrieval approach by approximately 4.5 percentage points (Hit@K, answer accuracy).
• Generalization: Maintains superior performance and efficiency when applied to enterprise and custom graphs, owing to schema-aware artifact prompts and modular retrieval pipelines.
• Efficiency: The number of LLM calls per query remains competitive, given the system's design for early termination and selective artifact refinement.
• Open Source Availability: The entire pipeline and toolkit are released for reproducible research and community extension at: https://github.com/awslabs/graphrag-toolkit.

6. Adaptability to Bring-Your-Own Knowledge Graphs

The framework is explicitly designed for custom, "bring-your-own" KGs. It requires only access to KG schema metadata (e.g., via a get_schema() utility) and does not necessitate task-specific fine-tuning or retraining.

• Schema Extraction: Schema information (entities, relations, types) is harvested once from the KG and leveraged in prompt construction and entity/path retrieval.
• No Annotated Data Required: All artifact extraction and refinement is performed by the LLM with procedural and natural-language instruction, supporting zero-shot adaptation.
• Executable Query Generation: The system generates openCypher (or other query language) queries on demand, supporting KGs in varied formats.

This extensibility makes BYOKG-RAG applicable across scientific, biomedical, industrial, and enterprise domains with diverse KG representations.

7. Research Significance and Community Impact

BYOKG-RAG advances the state of the art in KGQA by:

• Demonstrating how LLMs can be actively guided, via structured multi-artifact prompting and dedicated retrieval tools, to leverage graph-structured data effectively.
• Providing an empirical foundation and practical toolkit for robust QA over arbitrary knowledge graphs without requiring annotation or domain-specific customization.
• Establishing new performance baselines for both open and enterprise knowledge graph QA tasks, and fostering open-source best practices and reproducibility in KGQA research (2507.04127).

The open-source release of the BYOKG-RAG toolkit further catalyzes research and development in multi-strategy, graph-augmented LLMing for knowledge-intensive applications.