--- title: 'mKGQAgent: Modular Multilingual KGQA Framework' url: https://www.emergentmind.com/topics/mkgqagent type: topic --- # mKGQAgent: Modular Multilingual KGQA Framework mKGQAgent is a modular, human-inspired Large Language Model (LLM) agent framework for multilingual question answering over knowledge graphs (KGQA). It is designed to address the challenge of translating natural-language questions in multiple languages into SPARQL queries over large-scale knowledge graphs, such as DBpedia and Wikidata, while maintaining interpretability, flexibility, and robustness across diverse linguistic and script scenarios. mKGQAgent achieves state-of-the-art (SOTA) performance on multilingual KGQA benchmarks, leveraging in-context learning from an experience pool of previously solved examples and eschewing supervised fine-tuning to preserve LLM generality and operational efficiency [2507.16971]. ## 1. Motivation and Challenges in Multilingual KGQA The core task in text-to-SPARQL conversion is to robustly transform natural-language questions—potentially in low-resource or non-Latin scripts—into executable, correct SPARQL queries. Prior pipelines, such as QAnswer, Platypus, and DeepPavlov 2023, rely on hand-crafted templates, rules, or neural models that often fail to generalize across languages or require expensive multi-task fine-tuning. End-to-end LLM prompting typically suffers from non-trivial issues: hallucinated query components, lack of transparency, and notably degraded performance in non-English or low-resource linguistics contexts. mKGQAgent addresses these limitations by decomposing the KGQA task into discrete, interpretable subtasks—planning, entity linking, and query refinement—explicitly designed to be agentic and modular. Key challenges resolved by mKGQAgent include: - **Cross-Script Multilinguality:** Maintaining SPARQL syntax integrity across disparate scripts (Latin, Cyrillic, etc.) - **Subtask-Level Interpretability:** Enabling direct error localization to individual pipeline stages - **Cost-Performance Balance:** Optimizing the trade-off between LLM call count (and associated computation cost) and overall answer accuracy ## 2. Human-Inspired Modular Workflow mKGQAgent is architecturally composed of three main agentic modules, each orchestrated by an LLM and optionally leveraging in-context learning from the experience pool: - **Planning:** For each question $q_i$, generate a multistep plan $p_i$ outlining atomic actions (e.g., entity identification, relation detection, SPARQL construction), optionally conditioned on similar in-domain examples. - Algorithmic formulation: ```python function PLAN_WITH_EXPERIENCE(q_i, S_plan, LLM, E, EMB): v_q ← EMB(q_i) P ← findTopNPlans(E, v_q) # F1=1.0 only S_exp_plan ← S_plan + serialize(P) p_i ← LLM(prompt=S_exp_plan ▷ includes q_i) return p_i ``` - **Entity Linking (Action):** Given LLM-extracted entity and relation mentions, map them to knowledge graph URIs using external, off-the-shelf Named Entity Linking (NEL) APIs (e.g., Wikidata’s WBSearchEntities, Falcon 2.0). No supervised fine-tuning is performed in this step. ```python function NEL_TOOL(E_candidates, R_candidates, NEL_service): linkedEntities ← {} linkedRelations ← {} for each e in E_candidates: uri ← NEL_service.lookup(e) if uri nonempty: linkedEntities[e] ← uri for each r in R_candidates: uri ← NEL_service.lookup(r) if uri nonempty: linkedRelations[r] ← uri return (linkedEntities, linkedRelations) ``` The agent consistently selects the top (highest-confidence) URI returned by the service. - **Query Refinement (Feedback):** The agent executes the draft SPARQL $\hat\phi_i$ on the triplestore, retrieves intermediate answers $\mathcal{A}_i$, and re-invokes the LLM with feedback to correct both syntax and intent mismatches. This loop executes a single feedback step per question by design. ```python function FEEDBACK_STEP(\phi_i, S_feedback, KG): A_i ← KG.execute(\phi_i) S′_feedback ← S_feedback + serialize(A_i) return S′_feedback ``` Each atomic step is modular and callable independently, facilitating error analysis and enabling flexible agent configuration. ## 3. Experience Pool for In-Context Learning The experience pool is a non-parametric, vectorized memory of previous KGQA episodes, constructed automatically offline. For each $(q_i, \phi_i)$ in the training set, the agent stores: - $(q_i, v_{q_i}, p_i, \mathcal{H}_i, \phi_i, \hat\phi_i, F1_i)$, where $v_{q_i}$ is the embedding (via multilingual e5 large), $p_i$ is the generated plan, $\mathcal{H}_i$ is the chat history, $\hat\phi_i$ the predicted query, and $F1_i$ the answer overlap. At inference, the agent retrieves the top-N most similar examples in the embedding space, using cosine similarity: \[ \mathrm{sim}(v_1, v_2) = \frac{v_1 \cdot v_2}{\|v_1\| \, \|v_2\|} \] These retrieved examples are serialized and prepended to LLM prompts for plan/action refinement, but only examples with $F1=1.0$ are selected to maintain quality. The experience pool enables language-specific prompt augmentation and robust handling of non-English and low-resource questions without task-specific fine-tuning. ## 4. Implementation and System Architecture mKGQAgent supports both proprietary and open-source backbones: - **LLMs:** GPT-3.5-turbo and GPT-4o (OpenAI SDK); Qwen2.5 72B Instruct and Meta Llama 3.1 70B Instruct (quantized, vLLM deployment). - **Text Embeddings:** multilingual e5 large (MTEB SOTA for QA vectorization). - **Frameworks:** LangChain (agent orchestration), LangGraph (modular routing), Wikidata SPARQL endpoint and WBSearchEntities API (entity linking), Falcon 2.0 (relation linking). - **Prompting:** System and action prompts are available in multiple languages but are English-only in challenge evaluation. - **Operational Metrics:** Maximum context size for LLMs is set to 16,384 tokens. Typical cost per 100 questions for full mKGQAgent agentic workflow with GPT-4o is \$3.06. ## 5. Empirical Evaluation and Comparative Performance A comprehensive evaluation on QALD-9-plus (558 questions, 10 languages) demonstrates mKGQAgent’s SOTA effectiveness on multilingual KGQA benchmarks [2507.16971]: - **English QALD-9-plus:** mKGQAgent (GPT-4o) achieves 54.83% macro F1, outperforming HQA (GPT-4) at 50.00%, QAnswer at 44.59%, and several neural baselines. - **Cross-Lingual:** Notable F1 improvements are observed in German (43.08%), Spanish (38.28%), Belarusian (31.56%), and Bashkir (40.48%) compared to previous bests. - **Machine Translation Study:** Using automatically translated questions sometimes improved F1, e.g., a 15.4% gain for Spanish on GPT-4o. Ablation studies indicate that experience pool augmentation (especially in the action step) delivers the largest accuracy gains. The modular agentic structure incurs up to a 50% increase in LLM call count compared to a simple plan+NEL baseline, but with substantial F1 improvements (full mKGQAgent: +59.5% over SA baseline). | Component | GPT-4o F1 | LLM calls | Cost per 100 Q | |-----------------------------|-----------|-----------|---------------| | Simple Agent (Plan + NEL) | 34.37% | 8.87 | \$2.08 | | + Exp. pool (plan step) | 46.48% | 9.71 | \$2.28 | | + Exp. pool (action step) | 52.68% | 9.02 | \$2.12 | | + Feedback step | 40.47% | 10.93 | \$2.57 | | **Full mKGQAgent** | **54.83%**| 13.03 | **\$3.06** | ## 6. Discussion, Limitations, and Future Research mKGQAgent’s agentic, modular decomposition—together with the experience pool—yields interpretable, SOTA KGQA across English and many non-English languages without supervised LLM fine-tuning. Each pipeline component contributes positively: the experience pool provides robust adaptation to new languages and domains, and the feedback module recovers from minor errors or hallucinations. However, persistent gaps remain in handling Cyrillic-script languages (performance deficits of up to 20 F1 points versus Latin-script tasks), reflecting low-resource limitations. The agentic workflow, while more accurate, results in a nontrivial increase in computational cost per answer due to additional LLM calls. Proposed directions include: more sophisticated machine translation pipelines to further boost multilingual F1, improved entity/relation linking integrations, and agentic workflow optimizations (e.g., strategic caching, dynamic subtask skipping, lightweight re-ranking). The design of mKGQAgent demonstrates that LLM-agent paradigms, when coupled with a curated, high-fidelity example pool, can robustly and transparently handle the text-to-SPARQL semantic parsing task in a multilingual setting, achieving both competitive accuracy and operational modularity in real-world knowledge retrieval systems [2507.16971]. Source: https://www.emergentmind.com/topics/mkgqagent