---
title: 'FIRESPARQL: NLQ-to-SPARQL Translation Framework'
url: https://www.emergentmind.com/topics/firesparql
type: topic
---

# FIRESPARQL: NLQ-to-SPARQL Translation Framework

FIRESPARQL is a modular framework for translating natural language questions (NLQs) into SPARQL queries over Scholarly Knowledge Graphs (SKGs), designed to advance question answering (QA) capabilities in complex, schema-rich domains such as the Open Research Knowledge Graph (ORKG). It targets two primary failure modes endemic to Large Language Model (LLM)-generated SPARQL: structural inconsistencies in the query graph and semantic inaccuracies in entity or property linking. FIRESPARQL integrates specialized LLM fine-tuning, optional retrieval-augmented generation (RAG), and a SPARQL correction layer to systematically address these challenges, achieving new state-of-the-art results on the SciQA benchmark for NLQ-to-SPARQL translation [2508.10467].

## 1. Problem Definition and Error Taxonomy

Question answering over SKGs requires robust translation from free-form NLQs to executable SPARQL queries that return the correct result against a triple-based graph. Off-the-shelf LLMs exhibit two main classes of error in this task:

- **Structural inconsistencies**: These involve missing or redundant triple patterns, incorrect join variables, or misplaced clauses, materially affecting the topology or shape of the graph query.
- **Semantic inaccuracies**: Here, queries may possess the correct structure but link to inappropriate URIs, entities, or properties, e.g., mapping to an unrelated ORKG predicate.

These errors arise due to LLMs' limited exposure to SKG schema and instance data during pre-training, resulting in out-of-domain mapping and impaired schema awareness [2508.10467].

## 2. Modular Framework Architecture

FIRESPARQL is organized as a pipelined, three-module architecture, abstractly depicted as:

```
[ User NLQ ]
     ↓
(A) Fine-tuned LLM Core
     ↓
(B) Optional RAG Context Injection
     ↓
(C) SPARQL Correction Layer
     ↓
[ Final SPARQL ]
     ↓
Execute (QLever)
     ↓
[ Results ]
```

- **Module A (Fine-tuned LLM Core):** Employs LoRA-adapted LLaMA-Instruct (both 3.2B and 8B), fine-tuned on NLQ→SPARQL pairs from SciQA’s curated train set.
- **Module B (RAG Context Injection, optional):** Supplies context during generation by retrieving SKG-specific property/entity candidates relevant to the NLQ.
- **Module C (SPARQL Correction Layer):** A lightweight LLM prompt refines the generated SPARQL, correcting syntax and structural defects.

The high-level process, as formalized in the system pseudocode, is:

```python
def FIRESPARQL_Generate(NLQ, mode):
    if mode includes RAG:
        ctx = retrieve_context(NLQ)  # top-k properties/entities
        prompt_input = format_prompt(NLQ, ctx)
    else:
        prompt_input = format_prompt(NLQ)
    raw_sparql = LLM_generate(prompt_input)
    cleaned_sparql = SPARQL_corrector(raw_sparql)
    return cleaned_sparql
```

Resultant queries are executed against the ORKG dump using QLever [2508.10467].

## 3. Retrieval-Augmented Generation and SPARQL Correction Methods

The RAG mechanism grounds the LLM in SKG-specific surface forms and URIs. Steps include:

- Dense retrieval (DeepSeek-R1-Distill-LLaMA-70B) over a property catalog constructed from ORKG’s distinct predicates.
- Top 5–10 property labels/URIs for the NLQ are selected by cosine similarity in embedding space.
- Prompt augmentation inserts a “Candidates” section listing these properties and labels, biasing token prediction towards grounded entities.

The SPARQL correction layer addresses errors such as unmatched braces, absent `GROUP BY` in aggregation, extraneous text, or improper sub-SELECT scoping. This correction is invoked iteratively: upon syntax or QLever execution failure, re-correction is attempted to a maximum number of tries, aiming to ensure syntactically valid SPARQL [2508.10467].

## 4. Model Variants and Training Paradigms

Variants and training regimes evaluated include:

| Mode               | LLM Training                                              | Prompt Context           |
|--------------------|----------------------------------------------------------|--------------------------|
| Zero-Shot          | Instruction-tuned LLaMA-3.2B/8B, no SKG fine-tuning      | None                     |
| Zero-Shot + RAG    | As above                                                 | RAG-injected candidates  |
| One-Shot           | In-context demonstration (most similar SciQA pair)       | Single example           |
| Fine-Tuning (FT)   | LoRA on 1,795 SciQA NLQ/SPARQL pairs, 3–20 epochs        | None                     |
| FT + RAG           | LoRA as above                                            | RAG-injected candidates  |

Fine-tuning utilized a single NVIDIA H100 GPU, learning rate of $1 \times 10^{-5}$ and LoRA rank 8. All experiments were performed on SciQA’s handcrafted and semi-automatic NLQ/SPARQL sets [2508.10467].

## 5. Benchmarking and Metrication

Performance was evaluated on 100 handcrafted NLQs and 2,465 semi-automatic sets from the SciQA benchmark, using:

- **BLEU-4**: $ BLEU = \exp\left( \sum_{n=1}^4 w_n \cdot \log p_n \right) $, for $n$-gram overlap (homogenous weights).
- **ROUGE-L**: $ ROUGE_L = \frac{(1 + \beta^2) \cdot R_L \cdot P_L}{R_L + \beta^2 P_L} $, leveraging LCS-based recall and precision.
- **Relaxed Exact Match (RelaxedEM)**: Proportion of queries yielding syntactically valid, exact matching results when variable names, duplicate lines, and sorting are standardized. Computed both over valid (RelaxedEM(success)) and all queries (RelaxedEM(all)).

## 6. Empirical Results and Error Analysis

Major quantitative results (averaged over three runs, $\text{std} < 0.03$) are as follows:

| Mode                  | BLEU-4      | ROUGE-L    | RelaxedEM(all) |
|-----------------------|-------------|------------|----------------|
| Zero-Shot             | ≈ 0.03      | ≈ 0.38     | 0.00           |
| Zero-Shot + RAG       | ≈ 0.03      | ≈ 0.38     | 0.00           |
| One-Shot (3B)         | ≈ 0.58      | ≈ 0.78     | 0.40           |
| Fine-Tune LLaMA-3.2B  | 0.70        | 0.84       | 0.54           |
| Fine-Tune LLaMA-8B    | 0.77        | 0.90       | 0.85           |
| FT + RAG              | ≈ 0.58–0.60 | –          | 0.29           |

Fine-tuned LLaMA-8B (15 epochs) establishes new state of the art (ROUGE-L 0.90, RelaxedEM(all) 0.85), substantially outperforming both baseline and one-shot lineup (+12 percentage points in ROUGE-L and +45 in RelaxedEM relative to best prior) [2508.10467].

**Error profile**: Out of 513 queries from the best model, 448 yielded non-empty, successful results, 14 failed due to syntax (predominantly missing `GROUP BY` or improper sub-SELECT), and 51 returned empty sets (mainly due to property path mislinking). This highlights persistent difficulty in generating proper aggregation clauses and handling nested SPARQL constructs.

## 7. Foundational Contributions and Prospects

FIRESPARQL delivers the following:

1. **Systematic error typology** contrasting structural and semantic error modes for LLM-driven SPARQL over SKGs.
2. **Modular pipeline** integrating LLM fine-tuning, orthogonal RAG, and iterative SPARQL correction.
3. **State-of-the-art empirical gains** on SciQA, achieved via LoRA-8B fine-tuning (0.90 ROUGE-L, 0.85 RelaxedEM(all)).
   
All code, data splits, prompt templates, and the 15-epoch LLaMA-3-8B-Instruct LoRA model are openly accessible.

**Proposed directions** include extending validation to other SKGs (including biomedical graphs), development of graph-aware retrieval and neural reranking for enhanced context, large-scale synthetic data generation via KG traversals, and incorporating SPARQL grammar directly into token decoding for syntax-constrained inference [2508.10467].

Source: https://www.emergentmind.com/topics/firesparql