---
title: 'Hexgen-Text2SQL: Scalable Agentic Text-to-SQL Scheduling'
url: https://www.emergentmind.com/topics/hexgen-text2sql
type: topic
---

# Hexgen-Text2SQL: Scalable Agentic Text-to-SQL Scheduling

Hexgen-Text2SQL is a serving framework targeting the efficient scheduling and execution of agentic multi-stage Text-to-SQL workflows leveraging large language models (LLMs) on heterogeneous GPU clusters. The system addresses the unique architectural and operational challenges stemming from multi-tenant, end-to-end Text-to-SQL pipelines in production, characterized by sequential inter-stage dependencies, fine-grained parallelism, stringent latency constraints, and infrastructure heterogeneity. It introduces a hierarchical scheduling methodology, simulation-based hyperparameter tuning, and achieves significant reductions in deadline miss rates and improvements in throughput relative to state-of-the-art serving frameworks [2505.05286].

## 1. System Architecture

Hexgen-Text2SQL orchestrates LLM-based Text-to-SQL workflows in a two-tier structure on GPU clusters. Each user query proceeds through four main stages as informed by the CHESS agentic pipeline:

1. **Schema Linking:** Mapping user question entities to specific database tables and columns.
2. **SQL Candidate Generation:** Initiating multiple parallel LLM invocations with varied prompts to produce SQL candidates.
3. **Self-Correction:** Iteratively re-invoking the LLM (up to 10 iterations) to remediate errors in SQL execution.
4. **Evaluation:** Generating unit-tests and electing the optimal SQL statement.

These stages manifest sequential dependencies, necessitating downstream stages to await predecessor outputs, alongside intra-stage parallelism (e.g., multiple candidate generations and self-correcting passes executed concurrently). 

The framework's core components include:

- **Global Coordinator:** Maintains per-query task dependency graphs and SLO budgets, dispatches sub-tasks across model instances.
- **Model Instances:** Each pinned to a GPU (Triton/vLLM-style endpoints) and equipped with a local, adaptive priority queue governing inference task execution.
- **Monitoring & Feedback:** Continuously updates remaining SLO budgets and activates downstream tasks upon upstream completion.

The lifecycle of a query Q entails construction of tasks $\{q_1,\ldots,q_n\}$, assignment of a global SLO $T^{\text{SLO}}$, sequential triggering of schema-linking, candidate generation, self-correction, and evaluation, with each step managed according to dependency and timing constraints.

## 2. Hierarchical Scheduling Scheme

Hexgen-Text2SQL adopts a two-level scheduling paradigm combining:

- **Global Workload-Balanced Dispatching**
- **Local Urgency-Guided Prioritization**

### 2.1 Global Workload-Balanced Dispatching

The global scheduler aims, for each subtask $q_{i,j}$, to select a model instance $m \in M$ that maximizes the probability the overall query completes within the SLO:

$$
\max_{\phi:\{q\}\to M}\;\Pr\Bigl(\sum_{(i,j)}t_{i,j}^{\phi(q_{i,j})}\le T^{\rm SLO}_i\Bigr)
$$

Due to NP-hardness, Hexgen-Text2SQL implements a heuristic scoring strategy:

- **Predicted Computation Cost:**
  $$
  t_{\text{comp}_{i,j}^m} = t_{\rm prefill}^m (L_{\rm in}(q_{i,j})) + t_{\rm decode}^m (\widehat{L}_{\rm out}(q_{i,j}))
  $$
  where $L_{\rm in}$ and $\widehat{L}_{\rm out}$ denote input and predicted output lengths.
- **Queueing Cost:** 
  $$
  t_{\text{queue}_{i,j}^m} = \sum_{q'\in \Theta^m} t_{\text{comp}^{m}(q')}
  $$
- **Suitability Score for Each $(q, m)$:**
  $$
  \mathrm{Score}(q_{i,j},m) = (1-\alpha)\,{\beta}/{t_{\text{queue}_{i,j}^m}} - \alpha\,t_{\text{comp}_{i,j}^m}
  $$
  with $\alpha \in [0,1]$ trading off “fast execute” and “light load,” and $\beta$ rescaling queue effects.

Subtasks are dispatched to the instance maximizing this suitability score.

### 2.2 Local Adaptive Urgency-Guided Prioritization

Once a subtask is enqueued to model instance $m$, its priority is dynamically determined by an urgency function:

- **Subtask Time Allocation**
  $$
  t^{\text{SLO}}_{i,j} = (T^{\text{SLO}}_i - \tau^i_{\text{elapsed}}) \times \frac{\overline{t}_{\text{comp}_{i,j}}}{\sum_{k=j}^{n_i} \overline{t}_{\text{comp}_{i,k}}}
  $$
- **Urgency Definition**
  $$
  U_{i,j} = t_{\text{comp}_{i,j}^m} - (t^{\rm SLO}_{i,j} - \tau_{i,j})
  $$
  where $\tau_{i,j}$ is the time $q_{i,j}$ has waited in the queue. Higher $U_{i,j}$ signifies higher risk of deadline violation, thus higher execution precedence.

The queue always schedules the subtask with the highest urgency for inference execution on the GPU.

## 3. Simulation-Based Hyperparameter Tuning

The principal dispatch hyperparameter $\alpha$ is optimized online via a low-overhead, trace-driven CPU simulator:

1. **Warm-up:** Collect workflow traces (arrival, queue, durations) for 100 seconds.
2. **Simulation:** Replay the trace under candidate $\alpha$ values.
3. **Selection:** Set
   $$
   \alpha^* = \arg\min_{\alpha \in [0,1]} \frac{1}{N} \sum_{i=1}^N T_i(\alpha)
   $$
   where $T_i(\alpha)$ indicates simulated end-to-end latency.
4. **Sweep:** Coarse grid (step 0.2), followed by fine sweep (0.1) around optimum.
5. **Retuning:** After each sliding window, compare system latency using a one-sided t-test ($p < 0.01$); triggering re-tuning if significant degradation is detected.

Tuning windows incur negligible overhead (115–158 s on CPU) versus workload variabilities operating on the order of hours.

## 4. Empirical Evaluation

### 4.1 Experimental Setup

Key parameters for benchmarking include:

- **Workflow:** CHESS agent pipeline with Llama3.1-70B LLM.
- **Workload Traces:** BIRD-bench (finance, F1, mixed domains).
- **Hardware Topologies:** Heterogeneous clusters—
  - Hetero-1: $2 \times$A100 + $2 \times$A6000
  - Hetero-2: $2 \times$A100 + L40 + A6000
- **Arrival Rates:** Poisson arrivals at 0.5 qps and 1.0 qps.
- **Baseline Comparator:** vLLM framework using round-robin dispatch and FCFS local queue.
- **Metrics:** Attainment of SLO for 95% and 99% of requests by scaled-deadline; sustained throughput (qps).

### 4.2 Performance Results

Hexgen-Text2SQL demonstrates:

- **SLO Attainment:** Up to $1.67\times$ (average $1.41\times$) tighter deadline for 95% attainment, and up to $1.60\times$ (average $1.35\times$) for 99%, relative to vLLM.
- **Throughput:** $1.57\times$–$1.75\times$ improvement (average $1.65\times$) over vLLM.

|                   | Latency SLO (95%) | Throughput         |
|-------------------|------------------|--------------------|
| vLLM              | $T_{\rm base}$   | $X_{\rm base}$     |
| Hexgen-Text2SQL   | $T_{\rm base}/1.41$ | $1.65\,X_{\rm base}$ |

### 4.3 Ablation Analysis

- **Workload-Balanced (WB) vs. Round-Robin (RR) Dispatch:** WB yields up to $1.38\times$ (avg $1.18\times$) better 95% SLO than RR.
- **Urgency Queue vs. FCFS Queue:** Adding local urgency prioritization delivers up to $1.5\times$ (avg $1.2\times$) speed-up in 95% SLO.

## 5. Design Insights and Limitations

Hexgen-Text2SQL’s effectiveness derives from explicit modeling of multi-stage workflow dependencies, allowing for:

- **Elimination of Idle Gaps:** Synchronizing task execution reduces resource underutilization and redundant computations.
- **Heterogeneity-Aware Task Assignment:** Tailoring subtask scheduling to underlying GPU capabilities maximizes computation throughput and system efficiency.
- **Urgency-Driven Local Queues:** Prioritize tasks most at risk of SLO violation, mitigating head-of-line blocking.
- **Simulation-Driven Tuning:** Rapid, workload-responsive optimization of the core dispatch policy parameter $\alpha$.

**Limitations and Prospects:**
- The empirical computation cost model (prefill plus decode time) could be superseded by ML-based predictors for finer granularity.
- Extending to other LLM multi-stage pipelines (e.g., multimodal chains) would require richer and more general dependency graph modeling.
- Reinforcement learning or multi-armed bandit algorithms are suggested as alternatives to simulation sweeps for faster $\alpha$ convergence.
- The potential use of preemption and state-swapping mechanisms to accommodate abrupt high-priority workload arrivals remains an open direction [2505.05286].

## 6. Impact and Future Directions

Hexgen-Text2SQL establishes a methodology for robust, SLO-compliant, and high-throughput serving of complex, multi-stage agentic Text-to-SQL workflows on heterogeneous GPU infrastructure. Its two-level scheduling—the combination of workload-balanced dispatching and urgency-based local priorities—coupled with simulation-based hyperparameter tuning, addresses critical production challenges in LLM-powered database querying. Prospective research avenues include more expressive cost-prediction models, broader generalization to multi-modal or other agentic LLM chains, dynamic adaptation beyond grid-based simulation, and principled integration of preemptive scheduling for extreme deadline sensitivity.

Source: https://www.emergentmind.com/topics/hexgen-text2sql