FIRESPARQL: NLQ-to-SPARQL Translation Framework

• FIRESPARQL is a modular framework that converts natural language queries into executable SPARQL over scholarly knowledge graphs using fine-tuning, RAG, and a correction layer.
• It employs a three-stage process—fine-tuned LLM core, optional retrieval-augmented generation, and iterative SPARQL correction—to mitigate structural and semantic errors.
• Empirical results on SciQA show significant gains, with fine-tuned LLaMA-8B achieving 0.90 ROUGE-L and 0.85 RelaxedEM, setting a new state-of-the-art benchmark.

FIRESPARQL is a modular framework for translating natural language questions (NLQs) into SPARQL queries over Scholarly Knowledge Graphs (SKGs), designed to advance question answering (QA) capabilities in complex, schema-rich domains such as the Open Research Knowledge Graph (ORKG). It targets two primary failure modes endemic to LLM-generated SPARQL: structural inconsistencies in the query graph and semantic inaccuracies in entity or property linking. FIRESPARQL integrates specialized LLM fine-tuning, optional retrieval-augmented generation (RAG), and a SPARQL correction layer to systematically address these challenges, achieving new state-of-the-art results on the SciQA benchmark for NLQ-to-SPARQL translation (Pan et al., 14 Aug 2025).

1. Problem Definition and Error Taxonomy

Question answering over SKGs requires robust translation from free-form NLQs to executable SPARQL queries that return the correct result against a triple-based graph. Off-the-shelf LLMs exhibit two main classes of error in this task:

• Structural inconsistencies: These involve missing or redundant triple patterns, incorrect join variables, or misplaced clauses, materially affecting the topology or shape of the graph query.
• Semantic inaccuracies: Here, queries may possess the correct structure but link to inappropriate URIs, entities, or properties, e.g., mapping to an unrelated ORKG predicate.

These errors arise due to LLMs' limited exposure to SKG schema and instance data during pre-training, resulting in out-of-domain mapping and impaired schema awareness (Pan et al., 14 Aug 2025).

2. Modular Framework Architecture

FIRESPARQL is organized as a pipelined, three-module architecture, abstractly depicted as:

[ User NLQ ] ↓ (A) Fine-tuned LLM Core ↓ (B) Optional RAG Context Injection ↓ (C) SPARQL Correction Layer ↓ [ Final SPARQL ] ↓ Execute (QLever) ↓ [ Results ]

• Module A (Fine-tuned LLM Core): Employs LoRA-adapted LLaMA-Instruct (both 3.2B and 8B), fine-tuned on NLQ→SPARQL pairs from SciQA’s curated train set.
• Module B (RAG Context Injection, optional): Supplies context during generation by retrieving SKG-specific property/entity candidates relevant to the NLQ.
• Module C (SPARQL Correction Layer): A lightweight LLM prompt refines the generated SPARQL, correcting syntax and structural defects.

The high-level process, as formalized in the system pseudocode, is:

def FIRESPARQL_Generate(NLQ, mode): if mode includes RAG: ctx = retrieve_context(NLQ) # top-k properties/entities prompt_input = format_prompt(NLQ, ctx) else: prompt_input = format_prompt(NLQ) raw_sparql = LLM_generate(prompt_input) cleaned_sparql = SPARQL_corrector(raw_sparql) return cleaned_sparql

Resultant queries are executed against the ORKG dump using QLever (Pan et al., 14 Aug 2025).

3. Retrieval-Augmented Generation and SPARQL Correction Methods

The RAG mechanism grounds the LLM in SKG-specific surface forms and URIs. Steps include:

• Dense retrieval (DeepSeek-R1-Distill-LLaMA-70B) over a property catalog constructed from ORKG’s distinct predicates.
• Top 5–10 property labels/URIs for the NLQ are selected by cosine similarity in embedding space.
• Prompt augmentation inserts a “Candidates” section listing these properties and labels, biasing token prediction towards grounded entities.

The SPARQL correction layer addresses errors such as unmatched braces, absent GROUP BY in aggregation, extraneous text, or improper sub-SELECT scoping. This correction is invoked iteratively: upon syntax or QLever execution failure, re-correction is attempted to a maximum number of tries, aiming to ensure syntactically valid SPARQL (Pan et al., 14 Aug 2025).

4. Model Variants and Training Paradigms

Variants and training regimes evaluated include:

Mode	LLM Training	Prompt Context
Zero-Shot	Instruction-tuned LLaMA-3.2B/8B, no SKG fine-tuning	None
Zero-Shot + RAG	As above	RAG-injected candidates
One-Shot	In-context demonstration (most similar SciQA pair)	Single example
Fine-Tuning (FT)	LoRA on 1,795 SciQA NLQ/SPARQL pairs, 3–20 epochs	None
FT + RAG	LoRA as above	RAG-injected candidates

Fine-tuning utilized a single NVIDIA H100 GPU, learning rate of $1 \times 10^{-5}$ and LoRA rank 8. All experiments were performed on SciQA’s handcrafted and semi-automatic NLQ/SPARQL sets (Pan et al., 14 Aug 2025).

5. Benchmarking and Metrication

Performance was evaluated on 100 handcrafted NLQs and 2,465 semi-automatic sets from the SciQA benchmark, using:

• BLEU-4: $$BLEU = \exp\left( \sum_{n=1}^4 w_n \cdot \log p_n \right)$$, for $n$-gram overlap (homogenous weights).
• ROUGE-L: $$ROUGE_L = \frac{(1 + \beta^2) \cdot R_L \cdot P_L}{R_L + \beta^2 P_L}$$, leveraging LCS-based recall and precision.
• Relaxed Exact Match (RelaxedEM): Proportion of queries yielding syntactically valid, exact matching results when variable names, duplicate lines, and sorting are standardized. Computed both over valid (RelaxedEM(success)) and all queries (RelaxedEM(all)).

6. Empirical Results and Error Analysis

Major quantitative results (averaged over three runs, $\text{std} < 0.03$) are as follows:

Mode	BLEU-4	ROUGE-L	RelaxedEM(all)
Zero-Shot	≈ 0.03	≈ 0.38	0.00
Zero-Shot + RAG	≈ 0.03	≈ 0.38	0.00
One-Shot (3B)	≈ 0.58	≈ 0.78	0.40
Fine-Tune LLaMA-3.2B	0.70	0.84	0.54
Fine-Tune LLaMA-8B	0.77	0.90	0.85
FT + RAG	≈ 0.58–0.60	–	0.29

Fine-tuned LLaMA-8B (15 epochs) establishes new state of the art (ROUGE-L 0.90, RelaxedEM(all) 0.85), substantially outperforming both baseline and one-shot lineup (+12 percentage points in ROUGE-L and +45 in RelaxedEM relative to best prior) (Pan et al., 14 Aug 2025).

Error profile: Out of 513 queries from the best model, 448 yielded non-empty, successful results, 14 failed due to syntax (predominantly missing GROUP BY or improper sub-SELECT), and 51 returned empty sets (mainly due to property path mislinking). This highlights persistent difficulty in generating proper aggregation clauses and handling nested SPARQL constructs.

7. Foundational Contributions and Prospects

FIRESPARQL delivers the following:

1. Systematic error typology contrasting structural and semantic error modes for LLM-driven SPARQL over SKGs.
2. Modular pipeline integrating LLM fine-tuning, orthogonal RAG, and iterative SPARQL correction.
3. State-of-the-art empirical gains on SciQA, achieved via LoRA-8B fine-tuning (0.90 ROUGE-L, 0.85 RelaxedEM(all)).

All code, data splits, prompt templates, and the 15-epoch LLaMA-3-8B-Instruct LoRA model are openly accessible.

Proposed directions include extending validation to other SKGs (including biomedical graphs), development of graph-aware retrieval and neural reranking for enhanced context, large-scale synthetic data generation via KG traversals, and incorporating SPARQL grammar directly into token decoding for syntax-constrained inference (Pan et al., 14 Aug 2025).