Confidence Estimation for Text-to-SQL

• The paper introduces confidence estimation for Text-to-SQL by assigning a scalar score to SQL queries to predict correctness without gold answers.
• It details methods ranging from white-box probability calibration (using temperature rescaling and isotonic regression) to black-box execution-grounded clustering for reliable output selection.
• Empirical results demonstrate that structured confidence estimators reduce calibration error and enhance execution accuracy, supporting safe abstention and monitoring in production.

Searching arXiv for papers on confidence estimation for text-to-SQL and related reliability/calibration methods. arxiv_search.query({"search_query":"all:(\"confidence estimation\" AND \"text-to-sql\") OR all:(\"text-to-sql calibration\") OR all:(\"reliable text-to-sql\")","start":0,"max_results":10,"sort_by":"submittedDate","sort_order":"descending"}) arxiv_search.query શોધ({"search_query":"all:(\"confidence estimation\" AND \"text-to-sql\") OR all:(\"text-to-sql calibration\") OR all:(\"reliable text-to-sql\")","start":0,"max_results":10,"sort_by":"submittedDate","sort_order":"descending"}) Searching arXiv for the cited papers by identifier to confirm coverage and recency. Confidence estimation for Text-to-SQL is the task of assigning a scalar score to a generated SQL query that predicts whether the query correctly implements a natural-language question without requiring gold answers. In recent work, this problem is formulated either as direct probability calibration of model outputs, as agreement estimation across sampled SQL candidates, as execution-grounded verification, or as parser-independent error detection. The topic has become central to reliability because Text-to-SQL systems are increasingly evaluated არა only by execution accuracy, but also by whether their confidence aligns with true correctness, supports abstention, and enables production monitoring or human intervention (Maleki et al., 8 Aug 2025, Ramachandran et al., 2024, 2305.13683, Li et al., 20 Oct 2025, Arif et al., 30 Apr 2026).

1. Problem formulation and reliability objective

A common formalization defines a confidence estimator as a map

$C:(x,q)\mapsto [0,1],$

where $x$ is a natural-language question and $q$ is a generated SQL query. The intended behavior is that high values of $C(q\mid x)$ coincide with correct queries, while calibration error remains low. One recent treatment explicitly distinguishes black-box estimators $C_{bb}(q\mid x)$, which access only outputs, from white-box estimators $C_{wb}(q\mid x)$, which access per-token logits (Maleki et al., 8 Aug 2025).

This estimation problem is tightly coupled to reliability under selective prediction. TrustSQL defines reliability through a penalty-based score $\phi_c(x)$ that jointly accounts for answerable questions, unanswerable questions, abstention decisions $g(x)$, and a penalty $c>0$ for harmful mistakes. Its overall score is the mean

$\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$

and the paper studies three regimes: $x$0 (“non-critical”), $x$1 (“moderately critical”), and $x$2 (“highly critical”). This framing makes confidence estimation operational: a confidence score is not merely diagnostic, but a control signal for deciding when to predict and when to abstain (Lee et al., 2024).

A related line of work treats confidence as an estimate of semantic correctness for a question–SQL pair. The parser-independent detector of Chen et al. predicts

$x$3

where $x$4 is the question and $x$5 is a candidate SQL. In that formulation, the detector output directly serves as a calibrated confidence estimate, and can be used for error detection, re-ranking, answer-triggering, or interaction-triggering (2305.13683).

2. Probability-based and SQL-aware white-box estimators

White-box confidence estimation begins from the autoregressive likelihood of a generated SQL sequence. Given an NL question $x$6 and a generated SQL hypothesis $x$7, one formulation defines

$x$8

A central empirical observation is that raw sequence probabilities are poorly calibrated for Text-to-SQL: LLMs fine-tuned with RLHF or rejection sampling tend to assign overly peaked token distributions; a single low-probability token can drive the product close to zero; and the average reported $x$9 can diverge substantially from actual execution correctness. A simple correction is temperature rescaling,

$q$0

with $q$1 tuned on held-out data by optimizing a calibration objective such as NLL or ECE; in practice, the scalar transform $q$2 is used as the new confidence (Ramachandran et al., 2024).

Empirically, this simple approach is strong. On Spider with Llama 70B, the raw full-product score has $q$3; after isotonic rescaling of the same score, the paper reports $q$4, $q$5, and $q$6. On BIRD with the same model, prod + isotonic yields $q$7, $q$8, and $q$9. These results outperform or match prompting-based self-check methods while remaining cheaper because no extra API calls are required (Ramachandran et al., 2024).

A more structured white-box line exploits the internal regularities of SQL rather than treating it as ordinary text. One study compares aggregation schemes

$C(q\mid x)$0

combined with token-selection strategies. Full-Token Confidence uses all tokens; Schema-Linked Confidence uses only table names, column names, and literals; SQL-Aware Confidence further applies Token Exclusion, Case Folding, Order Folding, and Synonym Folding before aggregation. Across Spider and BIRD, white-box methods consistently outperform black-box methods, and SAC-Average achieves the highest AUC and lowest ECE. On BIRD, for GPT-3.5, SAC-Average with execution grounding reaches AUC $C(q\mid x)$1 and ECE $C(q\mid x)$2; the same method without grounding has AUC $C(q\mid x)$3 and ECE $C(q\mid x)$4 (Maleki et al., 8 Aug 2025).

Another structural method leverages sub-clause frequencies. Given a primary query $C(q\mid x)$5 and auxiliary samples $C(q\mid x)$6, the sub-clause frequency for clause type $C(q\mid x)$7 is

$C(q\mid x)$8

These clause-level signals are combined with a raw score $C(q\mid x)$9 through multivariate Platt scaling,

$C_{bb}(q\mid x)$0

On SPIDER and BIRD, canonical Platt scaling reduces ECE from uncalibrated levels often exceeding $C_{bb}(q\mid x)$1–$C_{bb}(q\mid x)$2 to approximately $C_{bb}(q\mid x)$3–$C_{bb}(q\mid x)$4, while MPS with SCF features further reduces ECE to approximately $C_{bb}(q\mid x)$5–$C_{bb}(q\mid x)$6 and improves error-detection AUC by $C_{bb}(q\mid x)$7–$C_{bb}(q\mid x)$8 points in reported cases (Liu et al., 27 May 2025).

A distinct white-box uncertainty measure appears in selective-classifier work. There, the sequence-level uncertainty score is the maximum token entropy,

$C_{bb}(q\mid x)$9

or, for proprietary models, the normalized sequence log-probability. This score is later calibrated through MinMax normalization, Platt scaling, or isotonic regression before driving a reject option (Somov et al., 16 Jan 2025).

3. Black-box consistency and execution-grounded confidence

When logits are unavailable, confidence estimation relies on output-side agreement. One formulation samples $C_{wb}(q\mid x)$0 SQL variants $C_{wb}(q\mid x)$1 at temperature $C_{wb}(q\mid x)$2, then scores a candidate by the fraction of samples in its cluster. The clusters can be defined by identical execution outputs, by embedding similarity, or by identical schema links. For execution-based consistency, if $C_{wb}(q\mid x)$3 belongs to result cluster $C_{wb}(q\mid x)$4,

$C_{wb}(q\mid x)$5

Among black-box methods, execution-based consistency is reported as the strongest, while verbalized confidence variants such as vanilla numeric confidence, COT, Augmented COT, and Self-Check Bool underperform; the same study explicitly states that verbalized confidence is poorly calibrated, with ECE $C_{wb}(q\mid x)$6 on BIRD, and does not recommend it for production (Maleki et al., 8 Aug 2025).

DeepEye-SQL instantiates an especially explicit execution-grounded scheme in its “Phase-4” confidence estimator. After candidate revision, all SQLs in $C_{wb}(q\mid x)$7 are executed against the database, and queries are grouped purely by equality of execution outputs: $C_{wb}(q\mid x)$8 when the two result sets are identical modulo ordering. Clusters are equivalence classes

$C_{wb}(q\mid x)$9

ranked by cluster size. There are no tunable hyperparameters in this clustering step beyond the total number of candidates $\phi_c(x)$0, typically $\phi_c(x)$1 generators $\phi_c(x)$2 $\phi_c(x)$3 sample each (Li et al., 20 Oct 2025).

The representative of the $\phi_c(x)$4-th largest cluster receives execution-based confidence

$\phi_c(x)$5

If the top cluster exceeds the threshold $\phi_c(x)$6, DeepEye-SQL takes a high-confidence shortcut and returns $\phi_c(x)$7. Otherwise, it runs unbalanced pairwise adjudication over the top $\phi_c(x)$8 candidates, with $\phi_c(x)$9 by default. Pairwise comparisons are repeated multiple times, majority vote produces $g(x)$0, and each candidate is scored by

$g(x)$1

followed by

$g(x)$2

The paper reports a Pearson correlation of $g(x)$3 between $g(x)$4 and actual execution accuracy on BIRD-Dev, and reports gains over plain self-consistency voting of $g(x)$5, $g(x)$6, and $g(x)$7 on BIRD for Gemma3-27B, Qwen2.5-32B, and Qwen3-30B respectively, with smaller but consistent gains on Spider (Li et al., 20 Oct 2025).

This body of results suggests that black-box confidence becomes substantially more effective when it is grounded in executable semantics rather than only in text-level agreement. That inference is directly supported by the recurring advantage of execution-based clustering in both the dedicated confidence-estimation study and the DeepEye-SQL orchestration framework (Maleki et al., 8 Aug 2025, Li et al., 20 Oct 2025).

4. Calibration metrics and empirical behavior

Calibration work in Text-to-SQL uses the standard probabilistic notion of correctness-conditioned confidence. One benchmark defines perfect calibration for a scoring function $g(x)$8 by

$g(x)$9

In practice, scores are binned and summarized through Expected Calibration Error,

$c>0$0

Adaptive Calibration Error, which uses equal-mass bins, and the Brier Score,

$c>0$1

These metrics quantify different aspects of confidence quality: ECE and ACE measure calibration bias, while the Brier score jointly reflects calibration and resolution (Liu et al., 27 May 2025).

Text-to-SQL calibration studies also report Negative Log-Likelihood and AUC. One formulation defines

$c>0$2

where $c>0$3 is the calibrated confidence, and evaluates AUC as the discrimination of correct versus incorrect queries after ranking by confidence. Practical evaluation commonly uses held-out schema-disjoint 5-fold cross-validation for tuning temperature, isotonic, or Platt parameters, together with uniform or monotonic binning for ECE (Ramachandran et al., 2024).

The empirical behavior of confidence scores is markedly schema-sensitive. One study advises collecting a small held-out set of NL/SQL pairs, preferably schema-disjoint or per schema, and notes that calibration can vary widely across schemas; per-schema tuning yields best F1, whereas very small validation sets risk overfitting isotonic bins, making temperature scaling more robust with limited data (Ramachandran et al., 2024). This schema dependence is important because Text-to-SQL benchmarks are explicitly cross-domain: the dedicated confidence-estimation study evaluates on Spider with $c>0$4 dev examples over $c>0$5 databases and BIRD with $c>0$6 dev examples over $c>0$7 databases (Maleki et al., 8 Aug 2025).

DeepEye-SQL contributes a slightly different calibration signal. The paper does not report a specialized calibration metric such as ECE, but it does perform a threshold sweep for $c>0$8, reports a broad plateau of near-peak performance for $c>0$9, and states that end-to-end execution accuracy peaks at $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$0 EX when $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$1. In that sense, the paper studies threshold robustness rather than post-hoc probability calibration in the narrower sense (Li et al., 20 Oct 2025).

5. Error detection, abstention, and interactive reliability

Confidence estimation is frequently operationalized as error detection. The parser-independent detector of Chen et al. uses CodeBERT on the concatenated question and SQL, then augments the representation with two 3-layer Graph Attention Networks: one over a merged dependency/constituency graph for the question, and one over an ANTLR4-derived AST graph for SQL. The final sigmoid score $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$2 is used directly for thresholding, re-ranking, answer-triggering, and interaction-triggering. On Spider dev, CodeBERT+GAT reaches AUC $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$3 for SmBoP, $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$4 for RESDSQL, and $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$5 for NatSQL, exceeding parser-dependent baselines such as top-prediction probability and MC-dropout variance (2305.13683).

Selective-classifier work makes the abstention trade-off explicit. The system consists of a Text-to-SQL generator $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$6 and a classifier $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$7 that maps an uncertainty score $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$8 to $\phi_c=\frac{1}{N}\sum_{x\in\text{test}}\phi_c(x),$9. The paper defines

$x$00

and

$x$01

It studies thresholding, logistic regression, and Gaussian-mixture classifiers over uncertainty scores. Reported results show that a Gaussian-mixture selective classifier achieves average Recall $x$02 and FDR $x$03, reducing overall execution error by approximately $x$04 at the cost of abstaining on $x$05–$x$06 of queries. Isotonic regression yields the lowest Brier scores in calibration experiments, for example improving T5-3B from $x$07 to $x$08 (Somov et al., 16 Jan 2025).

TrustSQL generalizes abstention beyond answerable questions and incorrect SQL. Its pipeline-based methods combine an answerability detector $x$09, a SQL generator $x$10, and a SQL error detector $x$11; unified methods instead generate SQL and a scalar confidence score $x$12, or directly emit a $x$13 token. The paper reports that in critical regimes, SQL error detectors outperform answerability detectors, that self-consistency voting yields the highest $x$14 and $x$15 across all datasets, and that no method beats the “Abstain-All” baseline when $x$16 (Lee et al., 2024).

Reliable Text-to-SQL with Adaptive Abstention moves confidence upstream into schema linking. Its Branching Point Prediction module trains per-layer classifiers over hidden states, calibrates them with conformal prediction, and computes p-values

$x$17

When a branching point is flagged, the system abstains from further schema linking, optionally applies a surrogate filter, or requests human feedback. On BIRD, using $x$18 layers and $x$19 best sBPP models, mBPP abstention at $x$20 yields table-linking exact match $x$21 if no-abstain and column-linking $x$22; with human feedback, table and column exact match recover to $x$23 and $x$24 respectively. Downstream, RTS-Schema with human loops reaches $x$25 execution accuracy on BIRD-dev, compared with $x$26 under golden schema and $x$27 for the cited prior best without abstention (Chen et al., 18 Jan 2025).

6. Production-native evaluation and system-level orchestration

In production settings, confidence estimation can be attached not only to candidate SQL generation but also to downstream evaluation. STEF is a schema-agnostic Text-to-SQL evaluation framework that takes the user question $x$28, an enriched reformulation $x$29, the generated SQL $x$30, and application-specific rules $x$31. It extracts question-side and SQL-side specifications, aligns filters and other semantic features, obtains an LLM verdict $x$32 together with evaluator confidence $x$33, and computes a composite score. With

$x$34

the confidence multiplier is

$x$35

and the final composite accuracy is

$x$36

The framework recommends storing $x$37 for monitoring dashboards and flagging $x$38 for review or human-in-the-loop fallback (Arif et al., 30 Apr 2026).

DeepEye-SQL places confidence estimation inside a broader software-engineering-inspired Text-to-SQL workflow aligned with the Software Development Life Cycle. Its four stages include grounding ambiguous user intent, N-version SQL generation, deterministic verification through unit tests and targeted revision, and finally confidence-aware selection. The framework’s abstract describes this final stage as clustering execution results to estimate confidence, then either taking a high-confidence shortcut or running unbalanced pairwise adjudication in low-confidence cases to produce a calibrated, quality-gated output. Using approximately $x$39B open-source LLMs without fine-tuning, the full framework reports $x$40 execution accuracy on BIRD-Dev and $x$41 on Spider-Test (Li et al., 20 Oct 2025).

A plausible implication is that confidence estimation in Text-to-SQL is no longer a single post-hoc module. The current literature locates it at multiple points in the pipeline: at token or sequence generation, at clause-structured calibration, at execution-result aggregation, at schema-linking abstention, and at production-time semantic evaluation. Across these settings, the common objective is stable: align a scalar confidence with semantic correctness closely enough that the score can support abstention, triage, adjudication, or continuous monitoring (Maleki et al., 8 Aug 2025, Arif et al., 30 Apr 2026).