DBPlanBench: LLM-Driven OLAP Plan Optimization

• DBPlanBench is an open-source framework that enables LLMs to perform localized, semantics-preserving edits on physical plans generated by cost-based optimizers in OLAP engines.
• It serializes compact representations of DataFusion’s physical plans and applies constrained JSON Patch edits, using evolutionary sampling with measured runtime for fitness evaluation.
• The framework demonstrates tangible improvements in query performance—up to 4.78x speedup—by optimizing join orders and leveraging semantic cues from query structures.

Searching arXiv for the specified paper and topic to ground the article in the cited source. DBPlanBench is an open-source harness for LLM–driven physical plan optimization on real OLAP engines. Introduced in “Making Databases Faster with LLM Evolutionary Sampling” (Erol et al., 11 Feb 2026), it was created to expose the physical plans produced by a mature cost-based optimizer, Apache DataFusion, in a compact and editable form so that an LLM can propose localized, semantics-preserving structural edits and those edits can be evaluated end-to-end by executing the patched plan. The system couples this representation with evolutionary sampling, using measured runtime rather than estimated cost as the fitness signal. Its motivating premise is that LLMs can exploit semantic correlations in queries and schemas—such as functional dependencies, time selectivity, and domain conventions—that are difficult for traditional heuristic and statistical models to capture.

1. Purpose, motivation, and problem setting

Traditional query optimization, as framed in the source paper, relies on cost-based optimizers that estimate execution cost using predefined heuristics and statistical models. Improving such heuristics requires substantial engineering effort, and even then they often cannot take into account semantic correlations in queries and schemas that strongly affect intermediate cardinalities and physical plan quality (Erol et al., 11 Feb 2026). The Honda/Accord example is cited to underscore how assuming independence can produce order-of-magnitude cardinality errors.

DBPlanBench is designed to operate precisely in this gap. Rather than replacing DataFusion’s optimizer, it measures gains over DataFusion’s native baseline, which consists of rule-based logical rewrites followed by physical planning. The harness surfaces the physical plan as a compact, serializable object and permits localized edits, termed patches, rather than full rewrites. The key optimization targets emphasized in the paper are join-side selection and join reordering, especially where semantic knowledge can reveal that selective joins should be executed earlier to minimize intermediate results, reduce hash-build footprint, and shorten execution time.

A common misconception would be to interpret DBPlanBench as a generic text-to-plan generator. The reported system is narrower and more constrained: it edits an already valid physical plan produced by DataFusion, enforces strict validation, and accepts only candidates whose outputs are identical to the baseline. This suggests that the contribution is as much about verifiable systems integration and constrained search as about LLM inference.

2. Integration with DataFusion and compact plan serialization

DBPlanBench operates after DataFusion’s normal optimization pipeline, intercepting the in-memory physical plan just before execution (Erol et al., 11 Feb 2026). It traverses DataFusion’s heterogeneous operator graph and serializes plans into JSON through a dedicated serialization layer that maps engine-specific objects, including join strategies and partitioning, into a unified schema. Because raw serialization is extremely verbose, the harness produces a token-efficient “streamlined” representation tailored for LLM consumption.

The serialized plan flattens the DAG into a dictionary keyed by node IDs such as "0", "1", and "2". Each entry stores the operator type and the fields relevant to that operator. Two artifacts are provided to the LLM. The first, structure, captures the operator topology, including node types and input relationships such as left and right for binary operators, while stripping deeply nested duplicates so that control flow and join graphs remain legible. The second, succinct_table_info, provides compact, normalized metadata summarizing table statistics and operator-level semantics without repeating per-partition details or file paths.

Supported operators include the core OLAP physical nodes: ParquetScan, filters, projections/coalesce, HashJoin, aggregates, and other common DataFusion operators. Each node is assigned a unique ID; attributes encode operator-specific parameters; and edges capture dataflow through input references. The paper gives concrete examples of the compressed representation, including streamlined statistics such as "statistics": {"n_rows": 6, "total_bytes": 1243} and abbreviated hashJoin and coalesceBatches fragments. DBPlanBench typically reduces plan prompts by about $10\times$ at the median, allowing complex TPC-DS plans to fit within LLM context windows.

The workflow is specified in six stages: extract and serialize the physical plan; compress it into the streamlined representation; provide the streamlined plan plus succinct table metadata to the LLM; receive localized JSON Patch edits and apply them; re-expand the edited plan into DataFusion’s full physical plan format; then execute the patched plan in a sandbox, validate semantic equivalence, and measure runtime. This architecture makes plan manipulation both engine-aware and execution-grounded.

3. Localized edit model and correctness invariants

DBPlanBench represents edits as JSON Patches conforming to RFC 6902 (Erol et al., 11 Feb 2026). This edit model is central to the harness. Patches are far smaller and less error-prone than emitting full plan text, and they constrain the LLM to localized transformations on a known-valid structure. Typical edits include swapping the left and right inputs of a HashJoin so that the smaller input builds the hash table, and changing join order so that selective joins occur earlier. The paper notes that filter reordering and projection index updates naturally accompany join edits in DBPlanBench’s patch semantics, whereas index and scan hints are not the focus of the work.

The example patch in the source material illustrates a join-side swap. The original join has 'left': 38, 'right': 80, and a key pair {'left': 'item_sk', 'right': 'i_item_sk'}. The patch replaces both child references and the corresponding join-key sides, yielding a semantically aligned swapped join. This exemplifies a broader design principle: structural edits must propagate consistently into all dependent metadata.

Validation is governed by strict safety invariants. DBPlanBench preserves all nodes; it does not allow deletions or dangling references. It maintains a valid DAG, forbidding cycles. It updates join conditions, projection lists, and especially projection indexes consistently when swapping join inputs. If right-side fields move left, projected indexes must be offset by the new left schema’s length. Semantic equivalence is enforced by executing both the baseline and candidate plans and comparing outputs; only identical results are valid. Determinism and schema compatibility are checked through DataFusion deserialization and output comparison.

These invariants distinguish DBPlanBench from unconstrained plan synthesis. A plausible implication is that its search space is intentionally narrow: it excludes arbitrary plan invention in favor of correctness-preserving structural perturbations over an existing optimizer output.

4. Evolutionary sampling, fitness, and optimization dynamics

DBPlanBench implements evolutionary sampling on top of LLM edits and uses measured runtime as the fitness signal (Erol et al., 11 Feb 2026). The optimization begins from the baseline DataFusion plan $p_0$. At each step, the system samples $K$ patch proposals from the LLM for the current plan or plans, applies those patches when they validate, and otherwise retains the parent plan.

The paper evaluates three search regimes. In Parallel Single Threads Evolution (PST-Evol), $K$ threads evolve independently for $T$ steps, and each thread advances only on valid edits. In Best of Last Evolution (BoL-Evol), the system samples $K$ edited candidates per step, keeps the best valid candidate with the lowest runtime, and broadcasts it as the new current plan for the next step. A non-evolutionary Best-of baseline draws $T \times K$ candidates without iterative refinement and reports the best result.

Two metrics formalize progress. The primary metric is per-query speedup relative to the baseline plan:

$$\text{speedup}_{q}(t) = \frac{\tau(p_0)}{\tau(p_t^*)},$$

where $\tau(\cdot)$ is measured execution time and $p_t^*$ is the best valid plan found up to step $p_0$0. The distribution metric is

$p_0$1

which measures the fraction of queries exceeding threshold $p_0$2 by step $p_0$3.

The summarized pseudocode in the paper clarifies the control logic. For each query, DBPlanBench initializes $p_0$4 and per-thread copies $p_0$5. For each step $p_0$6, it samples a patch $p_0$7, applies it to obtain a candidate $p_0$8, validates it by execution and equivalence checking, and records runtime $p_0$9. PST-Evol advances each thread independently on valid edits. BoL-Evol instead selects the valid candidate with minimum runtime and copies it to all threads; if no valid candidate exists, it retains the parent plans.

This measured-runtime objective is important. Traditional optimizers choose plans based on estimated cost, whereas DBPlanBench selects plans based on actual execution behavior in a sandbox. This suggests that the harness is not only an optimizer augmentation mechanism but also a benchmarking framework for studying the divergence between estimated and realized performance.

5. Semantic reasoning and the reported case study

The paper’s central claim is that LLMs can leverage semantic knowledge to identify non-obvious optimizations (Erol et al., 11 Feb 2026). DBPlanBench explicitly instructs the model to estimate intermediate cardinalities using semantic cues. For ParquetScan nodes, it uses row counts from succinct statistics. For Filter nodes, it infers selectivity from domain semantics such as date ranges, status codes, and IDs. For joins, it reasons about foreign-key versus many-to-many relationships and typical join ratios.

The most detailed example is a TPC-DS case study with a maximum reported speedup of $K$0. The query joins a large fact stream formed from a store/web/catalog sales union with item, customer_address, and date_dim, while filtering on d_year = 2001. DataFusion favors joining item first, because its statistics provide limited information about year selectivity. DBPlanBench enables the LLM to reason that a single year is highly selective and to reorder the plan so that date_dim is joined earlier.

The reported quantitative consequences are specific. Intermediate rows shrink by approximately $K$1, from $K$2 million to $K$3 million, before downstream joins. Aggregate hash-build time drops $K$4, from $K$5 seconds to $K$6 seconds. Build memory decreases $K$7, from $K$8 GB to $K$9 MB. End-to-end execution time improves by approximately $K$0, from $K$1 seconds to $K$2 seconds. The actual transformation is realized by a 16-operation JSON Patch that rewires join order, swaps build and probe sides, and updates projections.

This case study is significant because it does not depend on a new join algorithm or a new cardinality estimator inside the database engine. Instead, it exploits semantic priors encoded in the model and translates them into physically executable plan edits whose effect is verified empirically.

6. Benchmarking protocol, transfer, portability, and limitations

The evaluation covers TPC-H and TPC-DS at scale factor 3 for exploration, with TPC-H containing 8 tables and TPC-DS containing 24 tables, including 7 fact and 17 dimension tables (Erol et al., 11 Feb 2026). For each dataset, the query suite contains 120 queries, for 240 in total, generated by GPT-5 at controlled complexities 5–8 and filtered for executability, determinism, and runtime within $K$3.

The runtime protocol is deliberately measurement-heavy. Experiments use Modal containers pinning DataFusion and dependencies, with datasets placed in an in-memory filesystem to minimize I/O noise. Each sandbox is fixed at 4 CPUs and 4 GB RAM. For every candidate plan, the system runs $K$4 independent sandboxes and uses the minimum runtime as the estimator, described as robust against right-skewed noise. External verification on a stable bare-metal machine, an AMD 9900X with 64 GB, confirms that improvements are reproducible within approximately $K$5 variation.

For magnitude of improvements under $K$6 and $K$7, the reported TPC-H results are: BoL-Evol with P50 speedup $K$8, P90 $K$9, maximum $T$0, and $T$1 of $T$2; and Best-of with P25 $T$3 and maximum $T$4, performing generally better at lower percentiles but with a weaker upper tail than evolution. For TPC-DS, PST-Evol reports P75 $T$5, P90 $T$6, maximum $T$7, and $T$8 of $T$9, whereas Best-of reaches a maximum of $K$0. Overall, median speedups are approximately $K$1–$K$2 on TPC-H and approximately $K$3 on TPC-DS, with substantial per-query gains and rare but dramatic $K$4–$K$5 improvements.

The paper also demonstrates a “small-to-large” methodology. Optimizations are discovered at SF3 and then transferred to SF10 through a deterministic, rule-based script that maps scans by normalized signatures consisting of schema, projection, and predicate, and rewrites identifiers while preserving the optimized operator structure. All selected SF3 optimized plans transfer and execute correctly at SF10 with consistent speedup profiles; the speedup scatter lies near $K$6, and one outlier TPC-DS case is reported as $K$7 at SF3 and $K$8 at SF10. The stated assumption is that while operator choices can change at larger scales, many optimization decisions, such as join order, remain beneficial.

Portability is framed in terms of three required capabilities: extracting the engine’s physical plan and serializing it to a compact, engine-neutral schema; reconstructing and executing a patched plan inside the engine or via an equivalent API; and enforcing safety invariants including schema compatibility, determinism, and equivalence checking. The paper notes challenges arising from operator heterogeneity, engine-specific validity constraints such as projection index adjustments, and the possibility that larger engines will require additional compacting to remain within LLM context limits.

The limitations are explicit. Not all queries benefit: for roughly $K$9 of queries, no improvement is found; most speedups stay below $T \times K$0; and gains above $T \times K$1 are rare. Search overhead exceeds the cost of a single execution, though the paper argues that repeated OLAP workloads can amortize exploration. Observed failure modes include invalid patches, deserialization errors, mid-execution errors, and occasional output mismatches described as semantic drift. About $T \times K$2–$T \times K$3 of model generations are non-empty and execute successfully. Prompt length grows with query complexity, and longer prompts show a weak negative association with speedup. Future work includes richer mutation operators, scale-aware transfer policies, tighter integration with engine statistics collection and hybrid cost/measurement objectives, and applying DBPlanBench to other database engines.

Taken together, these results position DBPlanBench as a framework for verifiable, execution-grounded plan optimization rather than an unconstrained LLM planner. Its reported contribution is the combination of compact physical-plan exposure, localized patching, strict correctness enforcement, and evolutionary search guided by real measurements.