FINER-SQL: Boosting Small Language Models for Text-to-SQL

Abstract: LLMs have driven major advances in Text-to-SQL generation. However, they suffer from high computational cost, long latency, and data privacy concerns, which make them impractical for many real-world applications. A natural alternative is to use small LLMs (SLMs), which enable efficient and private on-premise deployment. Yet, SLMs often struggle with weak reasoning and poor instruction following. Conventional reinforcement learning methods based on sparse binary rewards (0/1) provide little learning signal when the generated SQLs are incorrect, leading to unstable or collapsed training. To overcome these issues, we propose FINER-SQL, a scalable and reusable reinforcement learning framework that enhances SLMs through fine-grained execution feedback. Built on group relative policy optimization, FINER-SQL replaces sparse supervision with dense and interpretable rewards that offer continuous feedback even for incorrect SQLs. It introduces two key reward functions: a memory reward, which aligns reasoning with verified traces for semantic stability, and an atomic reward, which measures operation-level overlap to grant partial credit for structurally correct but incomplete SQLs. This approach transforms discrete correctness into continuous learning, enabling stable, critic-free optimization. Experiments on the BIRD and Spider benchmarks show that FINER-SQL achieves up to 67.73\% and 85\% execution accuracy with a 3B model -- matching much larger LLMs while reducing inference latency to 5.57~s/sample. These results highlight a cost-efficient and privacy-preserving path toward high-performance Text-to-SQL generation. Our code is available at https://github.com/thanhdath/finer-sql.

• The paper introduces FINER-SQL, a reinforcement learning framework that leverages dense execution feedback, specifically memory and atomic rewards, to improve small language models in Text-to-SQL tasks.
• It employs a two-stage pipeline of supervised distillation followed by GRPO-based RL, refining policy optimization through interpretable reward signals from multiple reasoning traces.
• FINER-SQL achieves competitive SQL accuracy while reducing inference latency and resource requirements, outperforming larger models on complex, compositional SQL queries.

FINER-SQL: A Reinforcement Learning Framework for Small LLMs in Text-to-SQL

Introduction and Motivation

Recent works have demonstrated that LLMs achieve strong results in Text-to-SQL (NL2SQL) tasks, but their resource demands, privacy risks, and inference latency hinder many practical deployments. This has motivated increased attention toward small LLMs (SLMs, $\leq$3B parameters), which can be deployed efficiently and privately on commodity hardware. However, SLMs are fundamentally constrained by weaker reasoning and poor instruction following, resulting in diminished performance in complex generative tasks. Traditional RL techniques for Text-to-SQL utilize sparse, binary rewards that are not conducive to learning in the sparse-success regime typical for SLMs.

The FINER-SQL framework addresses these limitations by leveraging dense and interpretable execution feedback within a reinforcement learning (RL) paradigm tailored for SLMs. Two principal reward components—memory and atomic—enable continuous, fine-grained feedback to guide efficient and stable policy optimization. FINER-SQL closes the accuracy gap with much larger models while drastically reducing latency and resource requirements.

Figure 1: Execution accuracy (EX%) of FINER-SQL on BIRD dev. The FINER-SQL 3B model outperforms several >14B baselines and proprietary systems, highlighting the efficacy of SLMs with fine-grained feedback.

FINER-SQL Architecture

FINER-SQL consists of a two-stage pipeline: (i) supervised distillation to endow SLMs with explicit reasoning structures, and (ii) GRPO-based RL utilizing dense reward signals for iterative policy refinement. The training pipeline is as follows:

1. Distillation: Multiple LLM teacher models (e.g., GPT-4o, DeepSeek-R1, Qwen-2.5-72B) are prompted with question-schema pairs, producing both reasoning traces and executable SQLs across diverse logical strategies. These outputs are compiled into a “Reasoning Bank.” SLMs undergo supervised fine-tuning exclusively on these completions, promoting robust reasoning and structured SQL generation from the outset.
2. Reinforcement Learning: The distilled model is then trained with group relative policy optimization (GRPO), dispensing with separate critics in favor of efficient, candidate group-based advantage estimation. Dense reward design obviates the problem of reward sparsity, crucial for SLM convergence.

   Figure 2: FINER-SQL’s end-to-end training pipeline—distillation from multiple reasoning teachers followed by RL with dense, fine-grained feedback signals, coordinated by the GRPO objective.

Reward Modeling: Dense and Interpretable Feedback

FINER-SQL overcomes the limitations of sparse binary signals by decomposing the reward into four parts: format, execution, atomic, and memory. The composition and scaling of these rewards are illustrated as predictions gain increasing SQL correctness.

• Format Reward: Enforces adherence to the $<$think$>$-SQL pattern, ensuring tractable downstream parsing and evaluation.
• Execution Reward: Assigns scores $0$ (syntax error), $1$ (executable but wrong), or $2$ (exact match).
• Atomic Reward: Measures Jaccard similarity at operation-level granularity between the candidate and gold SQL's atomic decomposition, granting partial credit for structural correctness.
• Memory Reward: Evaluates semantic alignment of generated reasoning with verified traces stored in a cross-database memory, scoring based on embedding similarity for compositional faithfulness.

  Figure 3: Reward signals across prediction quality levels. Dense, non-binary reward allocation supports interpretable credit assignment for all outputs, not only fully correct SQLs.

Memory Reward: Promoting Reasoning Stability

The memory reward module maintains a vector database of expert-like reasoning traces, indexed by embeddings. For each candidate rollout, the system computes the cosine similarity between the candidate reasoning and the centroid of the top-$k$ similar traces retrieved from other databases. This approach regularizes RL exploration, suppressing degenerate paths and preserving cross-database generalization.

Figure 4: The memory reward flow, showing embedding-based retrieval, centroid calculation, and updates that maintain a diverse set of high-quality reasoning prototypes.

Atomic Reward: Structural Credit Assignment

Each predicted SQL is parsed into atomic operations (projections, joins, predicates, etc.). The reward is computed as the maximal Jaccard overlap between those operations and those from reference SQLs (including augmented, execution-verified variants). A shaped transformation is applied to avoid over-rewarding near-perfect but structurally incomplete solutions, concentrating the signal where the model requires additional refinement.

Figure 5: Atomic reward computation: prediction is aligned with reference SQL operation sets, and final reward is the maximal overlap across semantically equivalent ground-truths.

Empirical Analysis

Overall Performance and Efficiency

FINER-SQL achieves strong execution accuracy (67.73% on BIRD Dev and 85% on Spider Dev with a 3B model), matching or outperforming models with $\geq$14B parameters and even proprietary GPT-4 pipelines. Notably, inference latency and VRAM usage are drastically reduced, with the 3B variant requiring $\sim$10GB, running on a single consumer-grade GPU, and incurring only 5.57s average latency per sample.

Hyperparameter Sensitivity and Ablation

Sampling diversity (number of candidates drawn with stochastic decoding) and temperature are both critical—accuracy saturates at $\sim$30 candidates and $<$0. Larger memory retrieval top-$<$1 and exclusive cross-database retrieval further stabilize RL and improve generalizability.

Figure 6: Pass@K and EX% as a function of candidate count and sampling temperature. Both show monotonic improvement, plateauing beyond moderate values.

Ablations clearly show that both memory and atomic rewards are necessary—for the 3B model, removing memory and atomic rewards drops EX by 2.22% and 3.26% respectively, and their absence notably increases syntax error rates. Memory reward sharply reduces reasoning variance and the number of distinct execution clusters per prompt, thus enhancing majority voting reliability.

Figure 7: Impact of removing reward components on EX% and syntax errors. Both memory and atomic rewards are essential for optimal performance and stability.

Benefits to Complex, Compositional SQL Generation

FINER-SQL’s advantage is most pronounced as SQL complexity increases. The 1.5B and 3B variants consistently outperform both similar-sized and much larger (e.g., 14B, 32B) models on queries requiring deep compositional reasoning. RL with dense rewards robustly closes the complexity gap for challenging database schema scenarios.

Figure 8: Execution accuracy by SQL complexity. FINER-SQL (1.5B, 3B) matches or surpasses much larger LLM-based competitors on the most difficult queries.

Evolution of Model Reasoning

The dense reward regime encourages the transition from verbose, aimless reasoning to concise, coherent, and targeted reasoning trajectories. Post-RL, reasoning traces are significantly shorter and more precise, directly yielding higher SQL correctness.

Figure 9: Reasoning trace length before and after RL. RL yields more compact and effective reasoning strategies, particularly in challenging queries.

Implications and Future Directions

FINER-SQL demonstrates that, when equipped with fine-grained, interpretable feedback, SLMs achieve competitive NL2SQL results compared to LLMs at orders-of-magnitude lower resource consumption. This finding challenges prevailing assumptions regarding parameter scaling in database NLP tasks and opens opportunities for private, low-latency, and sustainable deployment of NL2SQL systems in industry.

Theoretically, the dense memory and atomic rewards serve not only as a practical tool to stabilize RL for SLMs but also as an architectural blueprint for dense credit assignment in other structured reasoning tasks (e.g., program synthesis, step-by-step mathematical derivation). The cross-database memory retrieval could further be exploited for cross-domain generalization and transfer learning. Continued innovation around fine-grained reward design and efficient policy optimization stands to extend SLM applicability to a variety of high-stakes, resource-constrained NLU settings.

Conclusion

FINER-SQL establishes that SLMs, augmented with dense execution feedback and efficient RL training, can rival much larger models in the Text-to-SQL domain. This framework provides concrete advances in inference efficiency, privacy, and deployability, making high-accuracy NL2SQL tractable beyond restricted research infrastructure. The contributions in reward modeling present avenues for the broader adoption of SLMs in structured output tasks where rapid, robust generalization is paramount.

Reference:

"FINER-SQL: Boosting Small LLMs for Text-to-SQL" (2605.03465)