mKGQAgent: Modular Multilingual KGQA Framework

• mKGQAgent is a modular framework for multilingual KGQA that translates natural language questions into accurate SPARQL queries with clear interpretability.
• It employs a human-inspired workflow with discrete planning, entity linking, and query refinement modules enhanced by an experience pool for in-context learning.
• The framework achieves state-of-the-art performance on benchmarks, delivering robust multilingual query handling and significant F1 improvements across diverse languages.

mKGQAgent is a modular, human-inspired LLM agent framework for multilingual question answering over knowledge graphs (KGQA). It is designed to address the challenge of translating natural-language questions in multiple languages into SPARQL queries over large-scale knowledge graphs, such as DBpedia and Wikidata, while maintaining interpretability, flexibility, and robustness across diverse linguistic and script scenarios. mKGQAgent achieves state-of-the-art (SOTA) performance on multilingual KGQA benchmarks, leveraging in-context learning from an experience pool of previously solved examples and eschewing supervised fine-tuning to preserve LLM generality and operational efficiency (Perevalov et al., 22 Jul 2025).

1. Motivation and Challenges in Multilingual KGQA

The core task in text-to-SPARQL conversion is to robustly transform natural-language questions—potentially in low-resource or non-Latin scripts—into executable, correct SPARQL queries. Prior pipelines, such as QAnswer, Platypus, and DeepPavlov 2023, rely on hand-crafted templates, rules, or neural models that often fail to generalize across languages or require expensive multi-task fine-tuning.

End-to-end LLM prompting typically suffers from non-trivial issues: hallucinated query components, lack of transparency, and notably degraded performance in non-English or low-resource linguistics contexts. mKGQAgent addresses these limitations by decomposing the KGQA task into discrete, interpretable subtasks—planning, entity linking, and query refinement—explicitly designed to be agentic and modular. Key challenges resolved by mKGQAgent include:

• Cross-Script Multilinguality: Maintaining SPARQL syntax integrity across disparate scripts (Latin, Cyrillic, etc.)
• Subtask-Level Interpretability: Enabling direct error localization to individual pipeline stages
• Cost-Performance Balance: Optimizing the trade-off between LLM call count (and associated computation cost) and overall answer accuracy

2. Human-Inspired Modular Workflow

mKGQAgent is architecturally composed of three main agentic modules, each orchestrated by an LLM and optionally leveraging in-context learning from the experience pool:

• Planning: For each question $q_i$, generate a multistep plan $p_i$ outlining atomic actions (e.g., entity identification, relation detection, SPARQL construction), optionally conditioned on similar in-domain examples.
  • Algorithmic formulation:

$p_i$3

• Entity Linking (Action): Given LLM-extracted entity and relation mentions, map them to knowledge graph URIs using external, off-the-shelf Named Entity Linking (NEL) APIs (e.g., Wikidata’s WBSearchEntities, Falcon 2.0). No supervised fine-tuning is performed in this step.

$p_i$4 The agent consistently selects the top (highest-confidence) URI returned by the service.

• Query Refinement (Feedback): The agent executes the draft SPARQL $\hat\phi_i$ on the triplestore, retrieves intermediate answers $\mathcal{A}_i$, and re-invokes the LLM with feedback to correct both syntax and intent mismatches. This loop executes a single feedback step per question by design.

$p_i$5 Each atomic step is modular and callable independently, facilitating error analysis and enabling flexible agent configuration.

3. Experience Pool for In-Context Learning

The experience pool is a non-parametric, vectorized memory of previous KGQA episodes, constructed automatically offline. For each $(q_i, \phi_i)$ in the training set, the agent stores:

• $$(q_i, v_{q_i}, p_i, \mathcal{H}_i, \phi_i, \hat\phi_i, F1_i)$$, where $v_{q_i}$ is the embedding (via multilingual e5 large), $p_i$ is the generated plan, $\mathcal{H}_i$ is the chat history, $\hat\phi_i$ the predicted query, and $p_i$0 the answer overlap.

At inference, the agent retrieves the top-N most similar examples in the embedding space, using cosine similarity:

$p_i$1

These retrieved examples are serialized and prepended to LLM prompts for plan/action refinement, but only examples with $p_i$2 are selected to maintain quality. The experience pool enables language-specific prompt augmentation and robust handling of non-English and low-resource questions without task-specific fine-tuning.

4. Implementation and System Architecture

mKGQAgent supports both proprietary and open-source backbones:

• LLMs: GPT-3.5-turbo and GPT-4o (OpenAI SDK); Qwen2.5 72B Instruct and Meta Llama 3.1 70B Instruct (quantized, vLLM deployment).
• Text Embeddings: multilingual e5 large (MTEB SOTA for QA vectorization).
• Frameworks: LangChain (agent orchestration), LangGraph (modular routing), Wikidata SPARQL endpoint and WBSearchEntities API (entity linking), Falcon 2.0 (relation linking).
• Prompting: System and action prompts are available in multiple languages but are English-only in challenge evaluation.
• Operational Metrics: Maximum context size for LLMs is set to 16,384 tokens. Typical cost per 100 questions for full mKGQAgent agentic workflow with GPT-4o is $3.06.

5. Empirical Evaluation and Comparative Performance

A comprehensive evaluation on QALD-9-plus (558 questions, 10 languages) demonstrates mKGQAgent’s SOTA effectiveness on multilingual KGQA benchmarks (Perevalov et al., 22 Jul 2025):

• English QALD-9-plus: mKGQAgent (GPT-4o) achieves 54.83% macro F1, outperforming HQA (GPT-4) at 50.00%, QAnswer at 44.59%, and several neural baselines.
• Cross-Lingual: Notable F1 improvements are observed in German (43.08%), Spanish (38.28%), Belarusian (31.56%), and Bashkir (40.48%) compared to previous bests.
• Machine Translation Study: Using automatically translated questions sometimes improved F1, e.g., a 15.4% gain for Spanish on GPT-4o.

Ablation studies indicate that experience pool augmentation (especially in the action step) delivers the largest accuracy gains. The modular agentic structure incurs up to a 50% increase in LLM call count compared to a simple plan+NEL baseline, but with substantial F1 improvements (full mKGQAgent: +59.5% over SA baseline).

Component	GPT-4o F1	LLM calls	Cost per 100 Q
Simple Agent (Plan + NEL)	34.37%	8.87	$2.08
+ Exp. pool (plan step)	46.48%	9.71	$2.28
+ Exp. pool (action step)	52.68%	9.02	$2.12
+ Feedback step	40.47%	10.93	$2.57
Full mKGQAgent	54.83%	13.03	$3.06

6. Discussion, Limitations, and Future Research

mKGQAgent’s agentic, modular decomposition—together with the experience pool—yields interpretable, SOTA KGQA across English and many non-English languages without supervised LLM fine-tuning. Each pipeline component contributes positively: the experience pool provides robust adaptation to new languages and domains, and the feedback module recovers from minor errors or hallucinations.

However, persistent gaps remain in handling Cyrillic-script languages (performance deficits of up to 20 F1 points versus Latin-script tasks), reflecting low-resource limitations. The agentic workflow, while more accurate, results in a nontrivial increase in computational cost per answer due to additional LLM calls.

Proposed directions include: more sophisticated machine translation pipelines to further boost multilingual F1, improved entity/relation linking integrations, and agentic workflow optimizations (e.g., strategic caching, dynamic subtask skipping, lightweight re-ranking).

The design of mKGQAgent demonstrates that LLM-agent paradigms, when coupled with a curated, high-fidelity example pool, can robustly and transparently handle the text-to-SPARQL semantic parsing task in a multilingual setting, achieving both competitive accuracy and operational modularity in real-world knowledge retrieval systems (Perevalov et al., 22 Jul 2025).