R$^3$-SQL: Ranking Reward and Resampling for Text-to-SQL

Abstract: Modern Text-to-SQL systems generate multiple candidate SQL queries and rank them to judge a final prediction. However, existing methods face two limitations. First, they often score functionally equivalent SQL queries inconsistently despite identical execution results. Second, ranking cannot recover when the correct SQL is absent from the candidate pool. We propose R$^3$-SQL, a Text-to-SQL framework that addresses both issues through unified reward for ranking and resampling. R$^3$-SQL first groups candidates by execution result and ranks groups for consistency. To score each group, it combines a pairwise preference across groups with a pointwise utility from the best group rank and size, capturing relative preference, consistency, and candidate quality. To improve candidate recall, R$^3$-SQL introduces agentic resampling, which judges the generated candidate pool and selectively resamples when the correct SQL is likely absent. R$^3$-SQL achieves 75.03 execution accuracy on BIRD-dev, a new state of the art among methods using models with disclosed sizes, with consistent gains across five benchmarks.

• The paper presents a unified Text-to-SQL framework that clusters functionally equivalent queries and applies groupwise ranking to resolve scoring discrepancies.
• The method integrates both pointwise and listwise reward signals with a learned gating agent, effectively enhancing candidate recall and mitigating positional bias.
• Experimental results demonstrate state-of-the-art execution accuracy (75.03 EX on BIRD-dev) while reducing computational overhead and increasing robustness.

R$^3$-SQL: Unified Ranking and Resampling for Robust Text-to-SQL

Overview

This paper introduces R$^3$-SQL, a Text-to-SQL system that establishes a new benchmark for execution accuracy (EX) among approaches using models with disclosed sizes. The key contributions are a unified framework resolving two persistent limitations in conventional LLM-driven Text-to-SQL pipelines: functional inconsistency in ranking and the bounded recall constraint induced by candidate pool generation. R$^3$-SQL strategically integrates groupwise ranking—leveraging both pointwise and listwise reward signals—and agentic resampling driven by a learned LLM gating agent, to offer a systematic upgrade for ranking-based selection and candidate pool recall.

Addressing Functional Inconsistency with Groupwise Ranking

Contemporary Text-to-SQL pipelines use a generate-then-rank approach, where multiple candidate queries are generated and a selection mechanism ranks these candidates. However, functionally equivalent SQL queries—distinct in surface form but producing identical execution outputs—are often assigned divergent scores due to the token-level sensitivity of existing pointwise or listwise rankers.

R$^3$-SQL mitigates this functional inconsistency by grouping candidates according to their execution results: all candidates yielding the same result are clustered together. The scoring paradigm transitions to the group level, aggregating evidence to neutralize noise caused by surface-level variation. The ranking incorporates two complementary signals:

• Cross-group preference (listwise): A pairwise ranker establishes relative preferences across groups by evaluating multiple candidate-pair comparisons, yielding robust ordinal signals.
• Groupwise utility (pointwise): The pointwise signal incorporates both the maximum candidate score within the group and group size, stabilizing the final ordering and resolving ties that may occur due to listwise uncertainty.

This dual-stage ranking is formalized by lexicographically sorting groups first by decisive listwise signals (thresholded to filter unreliable comparisons) and then by pointwise utility.

Figure 1: Comparison of ranking strategies; R$^3$-SQL leverages both group size and ranking signals to correctly identify the best execution group.

Figure 2: Visualization of R$^3$-SQL’s groupwise ranking strategy, combining cross-group pairwise comparisons and intra-group pointwise scoring.

Agentic Resampling: Surpassing Bounded Recall

Most ranking-based Text-to-SQL systems assume that the initial candidate pool generated by the LLM contains the correct SQL. In practice, the model can fail to generate any correct candidate, invoking the bounded recall problem, which strictly limits the upper bound of system accuracy.

R$^3$-SQL introduces an agentic resampling module: a dedicated LLM-based agent (gatekeeper) audits the candidate pool, predicting whether a correct SQL likely exists within. If not, the agent triggers a resampling phase, producing a larger set of candidates, which are then pruned and reranked. This selective resampling, guided by robust gatekeeper decisions, ensures high recall and optimal resource allocation, as resampling is triggered only when beneficial.

Position-Bias Mitigation via Consistency Training

Listwise rankers are susceptible to positional bias; the ordering of input candidates can affect model preference. R$^3$-SQL adopts position-consistency objectives, presenting candidate pairs in swapped positions during training and explicitly rewarding order-invariant correct preferences. This is operationalized via a GRPO framework with an auxiliary reward for consistency, yielding enhanced input-order robustness as empirically quantified.

Numerical and Experimental Results

On the BIRD-dev benchmark, R$^3$-SQL achieves 75.03 EX, the highest among all methods with disclosed model size. The method consistently surpasses baselines across five diverse benchmarks, breaking the 70% average EX barrier. Ablations reveal each module’s quantitative impact: groupwise ranking eliminates surface-level score variance among equivalent SQLs (variance reduced to 0), listwise ranking improves EX by +1.56 pp, and agentic resampling contributes a further +3.92 pp in candidate recall. Stability studies demonstrate that R$^3$-SQL’s multi-seed EX range outperforms the absolute top scores of prior competitive systems, confirming resilience to sampling randomness.

Figure 3: EX stability of R$^3$0-SQL across random seeds, showcasing superior robustness relative to all baselines.

Further, an efficiency analysis reveals that, despite integrating agentic resampling and additional ranking passes, R$^3$1-SQL reduces computational overhead compared to strong listwise baselines due to selective activation and groupwise redundancy reduction.

Figure 4: EX variation with respect to listwise preference threshold $^3$2, indicating optimality at selective, low $^3$3 values.

Theoretical and Practical Implications

By decoupling surface realization from semantic correctness, the groupwise approach enables more consistent reward assignment and robust model selection, a significant theoretical advancement for program synthesis tasks evaluated by execution. The modular architecture allows the integration of stronger or more domain-generalized rankers as they become available, which is likely to improve out-of-domain generalization.

Practically, R$^3$4-SQL’s judicious use of computational budget (via agent-driven resampling) is directly applicable to settings where LLM inference costs or latency are nontrivial. The agentic module’s architecture-agnostic design also suggests applicability to program synthesis domains beyond SQL.

Future Directions

Key avenues for further research include the development of domain-generalized pointwise rankers, which could entirely close the residual out-of-domain performance gap, and the extension of the agentic resampling framework to non-SQL program generation and task-oriented LLM systems. Integrating reward models that reflect semantic entailment beyond execution results could further reinforce robustness against coincidental correctness or spec drift.

Conclusion

R$^3$5-SQL presents a unified, empirically validated resolution to the dual problems of functional inconsistency and bounded recall in Text-to-SQL. Through groupwise reward modeling and agent-guided resampling, it delivers state-of-the-art EX across representative evaluation suites, with strong robustness and computational efficiency. The framework provides a rigorous foundation for future research in LLM-driven program synthesis and robust ranking under execution semantics.

(2604.25325)