QAgent: Efficient Semantic Parsing Tool

• QAgent is a stepwise semantic parsing framework that decomposes questions into atomic tool calls for reliable query construction.
• It employs ERASER, an environmental feedback-based self-correction mechanism, which selectively repairs errors detected during query execution.
• The approach improves efficiency and performance in KBQA, achieving higher F1 scores and reduced runtime compared to traditional beam-search methods.

QAgent, introduced as the shorthand for QueryAgent, is a reasoning framework for semantic parsing that solves a question step-by-step and performs step-wise self-correction through an environmental feedback-based mechanism called ERASER. It was proposed to address two failure modes of LLM pipelines for knowledge-base question answering and related semantic parsing tasks: unreliable end-to-end generation of complex SPARQL or SQL queries under hallucination, and the high cost of few-shot methods that rely on beam search or self-consistency. In QAgent, the model does not generate a full query in one pass; instead, it incrementally constructs the query through atomic tool calls, observes the environment after each step, and selectively repairs only the mistaken step when an error is detected (Huang et al., 2024).

1. Problem formulation and design rationale

The framework is motivated by a specific gap in LLM-based semantic parsing. LLMs achieve strong results on knowledge-base question answering, but they tend to hallucinate when tasked with end-to-end generation of a complex SPARQL or SQL query. Existing few-shot methods address this by exploring large candidate sets with beam search and self-consistency, but the reported cost can reach thousands of candidate queries and minutes of runtime. Agent-based stepwise methods reduce that search burden, yet they introduce a different failure mode: error accumulation. Once an early step is wrong, downstream reasoning is wasted (Huang et al., 2024).

QAgent is designed around two coupled principles. The first is stepwise query construction: the question is decomposed into a multi-turn interaction in which the LLM emits a reasoning trace and an executable action at each turn. The second is selective self-correction: rather than invoking generic reflection after every step, the system monitors environmental feedback and intervenes only when a logical or execution error is detected. This combination targets reliability and efficiency simultaneously.

A recurring misconception is to treat QAgent as a variant of generic self-debugging. The reported ablations distinguish it from such approaches. In the GrailQA and GraphQ experiments, adding zero-shot self-correction to QueryAgent without ERASER reduces performance from 43.7 to 38.5 and from 35.3 to 30.2, respectively, while few-shot self-debug reaches 48.0 and 40.1, and ERASER reaches 60.5 and 50.8. The system therefore depends not merely on “having reflection,” but on the selective and differentiated use of environmental feedback.

2. Stepwise query construction and execution loop

QAgent solves each question through a multi-turn dialogue with an LLM that issues atomic tool calls implemented as PyQL functions. At each turn, the model emits a pair $(\mathrm{Thought}_t,\mathrm{Action}_t)$, where the action is a call such as get_relation(entity), add_fact(...), add_filter(...), set_answer(...), or execute(). The action is executed immediately against the knowledge base or a Python interpreter, and the resulting observation is returned to the model for the next turn (Huang et al., 2024).

The prompt has three parts: a task description together with a document of available PyQL functions, one human-annotated demonstration question with full multi-step reasoning, and the new question $Q$ plus entity linking results. At turn $t$, the prompt is

$$P_t=\{\text{Task}+\text{Tools}+\text{Demo}+\text{New }Q+\text{History of }(\text{Thought},\text{Action},\text{Observation})\text{ up to }t-1\}.$$

The turn-level procedure is then:

1. $$\mathrm{LLM}(P_t)\rightarrow(\mathrm{Thought}_t,\mathrm{Action}_t)$$.
2. Execute $\mathrm{Action}_t$ against the KB or Python interpreter.
3. Capture $(\mathrm{KB\_feedback},\mathrm{Python\_feedback})$.
4. Run ERASER over the environmental feedback and reasoning memory.
5. Append $$(\mathrm{Thought}_t,\mathrm{Action}_t,\mathrm{Observation}_t)$$ to the dialogue history.
6. Repeat until $\mathrm{Action}_t=\texttt{execute()}$.

The simplified main loop is expressed as

$$\begin{aligned} \text{Initialize }RM&\leftarrow\emptyset,\;P\leftarrow\text{(instruction + demo + Q)}\ \text{for }t=1\ldots T:&\ \quad (\text{Thought},\text{Action})&=\mathrm{LLM}(P)\ \quad (\mathrm{kb\_fb},\mathrm{py\_fb})&=\mathrm{Execute}(\text{Action})\ \quad RM&\leftarrow RM\cup\{\text{last step info}\}\ \quad \mathrm{error}&=\mathrm{Detect}(\mathrm{kb\_fb},\mathrm{py\_fb},RM)\ \quad \text{if }error:\quad Obs&=\mathrm{GetGuideline}(error)\ \quad \text{else:}\quad Obs&=\mathrm{kb\_fb}\ \quad P&\leftarrow P + (\text{Thought},\text{Action},Obs)\ \quad \text{if Action}=\texttt{execute()}\ \text{break} \end{aligned}$$

The reported runtime complexity is approximately $Q$0 where $Q$1 is the number of reasoning steps, typically $Q$2. This is central to the framework’s efficiency claims: the reasoning trajectory remains a single top-1 path rather than a beam over many candidate full-query generations.

3. ERASER: environmental feedback-based self-correction

ERASER, expanded as EnviRonmental feedbAck SElf-coRrection, is the mechanism that monitors intermediate feedback and injects corrective guidance only when necessary. The monitored environment includes three information sources: KB execution feedback, Python interpreter feedback, and reasoning memory. KB execution can reveal empty bindings, SPARQL error messages, or unexpected CVT (blank node) results. Python execution can surface invalid function calls or arity mismatches. Reasoning memory stores information such as which variables exist, which relations have been retrieved, and prior results (Huang et al., 2024).

Error detection is formalized by an indicator function

$Q$3

When $Q$4, ERASER classifies the error into a rule-based type such as EmptyResult, InvalidAction, or MissingRelation. For each error type $Q$5, a short Guideline template $Q$6 is defined. The paper’s example is: “Your add_filter produced no results—did you choose the right operator or inadvertently filter out your entity? If not needed, try set_answer().” Guideline selection is written as

$Q$7

where $Q$8 is the rule that detects error type $Q$9.

Two properties distinguish ERASER from broader self-correction schemes. First, it is selective: it triggers only if $t$0, avoiding unnecessary correction overhead. Second, it is differentiated: the inserted Guideline is tailored to the detected error rather than being a generic request to rethink the previous step. In operational terms, if no error is found, the next observation is the ordinary KB feedback; if an error is found, the next observation is the Guideline. The next LLM turn is therefore conditioned not on a full restart or a global replan, but on a local, environment-grounded repair signal.

4. Quantitative results on KBQA and efficiency

In the reported one-shot setting with gpt-3.5-turbo, QueryAgent achieves F1 scores of 60.5 on GrailQA, 50.8 on GraphQ, 63.9 on WebQSP, and 98.5 on MetaQA-3Hop. On GrailQA and GraphQ, the system notably outperforms all previous few-shot methods using only one example by 7.0 and 15.0 F1, respectively. Relative to AgentBench in the same one-shot setting, the gains are much larger: from 30.5 to 60.5 on GrailQA, from 25.1 to 50.8 on GraphQ, and from 26.4 to 63.9 on WebQSP. The 100-shot KB-BINDER baseline is reported at 53.5 on GrailQA, 35.4 on GraphQ, and 48.6 on WebQSP (Huang et al., 2024).

The ERASER ablation clarifies how much of this performance comes from targeted correction. Without ERASER, QueryAgent scores 43.7 on GrailQA and 35.3 on GraphQ. Adding zero-shot generic self-correction reduces those scores to 38.5 and 30.2. Few-shot self-debug raises them to 48.0 and 40.1. The full selective ERASER version reaches 60.5 and 50.8. A common misunderstanding is therefore that any self-correction layer would suffice; the ablation indicates that generic correction can be counterproductive, whereas environment-conditioned correction is beneficial.

Efficiency is measured per question using TPQ for time cost in seconds, QPQ for SPARQL query calls, and CPQ for API invocation cost in USD. QueryAgent reports TPQ $t$1, QPQ $t$2, and CPQ $t$3, compared with AgentBench at TPQ $t$4, QPQ $t$5, and CPQ $t$6, and KB-BINDER at TPQ $t$7, QPQ $t$8, and CPQ $t$9. For gpt-3.5-turbo, the cost is computed as

$$P_t=\{\text{Task}+\text{Tools}+\text{Demo}+\text{New }Q+\text{History of }(\text{Thought},\text{Action},\text{Observation})\text{ up to }t-1\}.$$0

The authors summarize these measurements by stating that QueryAgent reduces QPQ by orders of magnitude versus beam-search in-context-learning methods and halves CPQ and TPQ versus AgentBench, while doubling F1. This situates QAgent as both a performance and systems-efficiency proposal rather than a pure accuracy optimization.

5. Transferability across tasks and agent pipelines

The framework is not confined to the original KBQA setting. On Text2SQL, evaluated on WikiSQL with denotation accuracy, one-shot QueryAgent reaches 72.5, compared with 57.6 for one-shot AgentBench and 67.0 for QueryAgent without ERASER. The same table reports 49.1 for Davinci-003 with 32-shot few-shot prompting, 51.6 for ChatGPT with 32-shot prompting, and 65.6 for StructGPT (ChatGPT) with 32-shot prompting. The gain of QueryAgent over one-shot AgentBench is reported as $$P_t=\{\text{Task}+\text{Tools}+\text{Demo}+\text{New }Q+\text{History of }(\text{Thought},\text{Action},\text{Observation})\text{ up to }t-1\}.$$1 (Huang et al., 2024).

ERASER also transfers to another agent pipeline. When added to AgentBench on KBQA, the performance rises from 30.5 to 38.5 on GrailQA, from 25.1 to 35.6 on GraphQ, and from 26.4 to 32.0 on WebQSP. The paper characterizes this as approximately a 10-point improvement and presents it as evidence of strong transferability.

Several broader generalization claims are also reported. QAgent uses a single demonstration across all domains, whereas in-context-learning methods degrade on compositional and non-i.i.d. splits. On GraphQ, described as 100% non-i.i.d., QAgent’s $$P_t=\{\text{Task}+\text{Tools}+\text{Demo}+\text{New }Q+\text{History of }(\text{Thought},\text{Action},\text{Observation})\text{ up to }t-1\}.$$2 F1 gain far exceeds few-shot in-context learning. This suggests that the stepwise construction plus feedback-triggered repair strategy is less dependent on surface similarity between demonstration and test question than prompt-only query synthesis.

6. Nomenclature, scope, and later reuse of the name

Within the 2024 semantic parsing work, “QAgent” denotes QueryAgent, the stepwise tool-based query construction framework with ERASER (Huang et al., 2024). Later literature reuses the same name for unrelated systems, which makes the term context-dependent rather than field-stable.

In software testing, “QAgent” names an Agentic RAG system for Quality Engineering artifact creation built around a hybrid vector-graph knowledge system and multi-agent orchestration, with roles such as Planner Agent, Case Generator Agent, Tracer Agent, and Aggregator Agent (Hariharan et al., 12 Oct 2025). In quantum programming, “QAgent” denotes an LLM-based multi-agent system for autonomous OpenQASM programming, organized around a Dynamic-few-shot Coder and a Tools-augmented Coder (Fu et al., 26 Aug 2025). In agentic retrieval, “QAgent” identifies a modular search agent for interactive query understanding trained with reinforcement learning through a plan-search-reflect loop (Jiang et al., 9 Oct 2025). The title “Quantum Agents” also uses “QAgent” as a shorthand for a broader framework defined by the tuple $$P_t=\{\text{Task}+\text{Tools}+\text{Demo}+\text{New }Q+\text{History of }(\text{Thought},\text{Action},\text{Observation})\text{ up to }t-1\}.$$3 (Sultanow et al., 2 Jun 2025).

A plausible implication is that “QAgent” should be read as a paper-specific proper name unless the surrounding domain is explicit. In semantic parsing and KBQA, however, the term is most precisely associated with QueryAgent and its ERASER-based self-correction mechanism.