Larch: Learned Query Optimization for Semantic Predicates

Abstract: With the advent of LLMs, many database systems introduced semantic operators that enabled analytical queries over unstructured data (e.g. text, images, videos). Semantic operators typically incur high inference costs and latencies making semantic (AI) SQL queries challenging to apply on large scale datasets. At the same time, their semantic nature leads database engines to treat them as black boxes, making AISQL queries difficult to optimize. In this paper, we introduce Larch, a framework for optimizing the execution of semantic filters in AI SQL queries. Larch was inspired by two key observations: i) the high latency of semantic operators leaves significant room for computationally-heavy runtime optimization techniques, ii) unstructured data are typically accompanied by semantic information in the form of embeddings allowing for efficient semantic comparisons between AI_FILTER prompts and data values. Based on these two key observations, we present two Larch variants: Larch-A2C and Larch-Sel. Larch-A2C encodes arbitrary semantic filters expression tree using an embedding-augmented Gated Graph Neural Network and formulates the filter evaluation order as a Markov decision process. In contrast, Larch-Sel leverages a supervised learning model to predict filter selectivities, subsequently applying dynamic programming to find a near-optimal evaluation order for each input row. Evaluated across diverse real-world datasets and comprehensive synthetic workloads, both Larch variants always outperform existing semantic filter optimization techniques in terms of token usage. Our results demonstrate that Larch is robust across diverse workloads, reducing total token cost overhead by 3x-19x compared to Palimpzest and Quest.

• The paper introduces an adaptive framework, Larch, that minimizes costly LLM invocations by dynamically ordering semantic predicates.
• It employs dual techniques: an A2C-based MDP for dynamic predicate ordering and an MLP-driven model leveraging embeddings for selectivity estimation.
• Experimental results demonstrate Larch’s efficiency and scalability, achieving token cost reductions of 3×–19× and robust latency hiding in production workloads.

Larch: Learned Query Optimization for Semantic Predicates

Motivation and Problem Setting

The proliferation of LLM-backed semantic operators in database systems has enabled rich analytics over unstructured data. However, the invocation of LLMs via semantic filters (e.g., AI_FILTER in AI SQL) introduces dominating inference cost and high wall-clock latency. Unlike traditional relational operators—where selectivity and cost properties are amenable to pre-computation—AI_FILTER selectivity is prompt- and data-dependent, often intractable a priori. This severely impedes efficient query planning, particularly when multiple semantic predicates are composed (e.g., via AND/OR trees), as naïve or static ordering can result in orders-of-magnitude unnecessary LLM invocations.

Existing approaches such as Palimpzest and Quest rely on global selectivity estimates from sampled data, producing row-agnostic or per-row static orderings, but these strategies fail to account for local correlations, concept drift, and instance-level predicate behavior. As a result, significant token/cost overheads persist in real workloads.

Larch Framework and Algorithmic Techniques

Larch introduces a latency-hiding, instance-adaptive framework for optimizing semantic predicate execution in AI SQL queries. The approach is underpinned by two key insights: (1) the high latency of LLM inference amortizes expensive online learning and planning, and (2) ubiquitous pre-computed document embeddings can be leveraged for cost-effective selectivity modeling.

Larch System Overview

Figure 1: Larch's architecture situates online learning in the LLM latency window, supporting per-row, per-step adaptive predicate ordering that supersedes static baseline heuristics.

Larch proposes two algorithmic instantiations:

• Larch-A2C: Models the predicate ordering decision process as an MDP, utilizing a Gated Graph Neural Network (GGNN) for dynamic tree encoding and an Advantage Actor-Critic (A2C) RL policy for sequence generation. This technique directly leverages runtime feedback to unify selectivity estimation and plan optimization.
• Larch-Sel: Employs a decomposition: a supervised MLP model estimates per-filter selectivity using document and predicate embeddings as features; a dynamic programming (DP) layer then computes the minimum-cost execution sequence under the independence assumption. This separation explicitly targets the estimation bottleneck.

Both methods exploit an asynchronous, pipelined inference/training protocol in which local model training is hidden behind remote LLM calls.

Semantic Selectivity Estimation

Raw embedding similarity between document and predicate prompt embeddings demonstrates noisy, non-monotonic correlation with predicate truth labels, precluding its use as a sole selectivity signal:

Figure 2: Cosine similarity between document and predicate embeddings is noisy with respect to actual predicate outcomes, motivating learned non-linear selectivity models.

Larch leverages embeddings as high-dimensional nonlinear features for selectivity estimation, enabling robust per-row pass-probability predictions.

Experimental Evaluation

Larch is evaluated on three long-document benchmarks (GovReport, PubMed, BigPatent) using workload patterns characteristic of production AI SQL: pure conjunction, pure disjunction, and mixed predicate trees, with between 2 and 10 semantic predicates per node. LLM inference costs are measured as total tokens and call count.

Token Efficiency and Scalability

Larch consistently outperforms state-of-the-art baselines—PZ and Quest—on all datasets, across selectivity ranges, and for increasing filter counts. Larch-Sel in particular achieves normalized token cost within 2–7% of the per-row optimal lower bound, in stark contrast to 17–32% overheads for PZ/Quest, yielding cost reductions of $3\times$–$19\times$.

Figure 3: On GovReport, both Larch variants substantially reduce token overhead versus PZ and Quest for conjunction, disjunction, and mixed workloads.

This advantage holds as scale increases. On BigPatent, Larch-Sel maintains overhead within 3.6%–3.8% at 67k documents, demonstrating horizon-robust learning dynamics and convergence.

Figure 4: Larch's normalized token cost further decreases with scale, while baseline methods plateau, confirming Larch's superior online adaptation.

Sensitivity Analysis

• Selectivity Range: Larch methods maintain consistently low token overhead across expression selectivity buckets, while PZ and Quest degrade significantly for extreme selectivities due to miscalibrated global estimates.
• Predicate Arity: As the number of semantic filters increases, Larch-Sel sustains flat normalized cost, whereas error accumulation in PZ/Quest leads to nearly linear cost blowup.
• Oracle Comparison: Larch-Sel outperforms "oracle" PZ/Quest methods, which have access to ground-truth global selectivities. This underscores the criticality of per-row adaptation, not just better global estimation.

Latency Hiding and Practicality

All model updates in Larch incur under 10ms latency and are fully hidden behind AI_FILTER inference, with no observable impact on wall-clock query execution. Ablation verifies negligible accuracy costs for the pipelined asynchronous update protocol.

Theoretical Implications

Larch exposes the semantic filter optimization problem as estimation-dominated: with accurate per-row selectivity and cost estimation, optimal plans follow immediately via combinatorial dynamic programming. The MDP approach validates that holistic policy learning is possible, but estimation-based decomposition is vastly more sample efficient in the absence of strong predicate correlation structure.

This architectural insight extends broader lessons to adaptive query processing in AI-integrated data systems, indicating that the amalgamation of learned selectivity, precomputed embeddings, and online plan adaptation forms an efficient regime for managing high-cost semantic operators in production.

Practical Impact and Future Work

Larch's methodology directly translates to lower LLM call volume and reduced token cost in operational data clouds and AI SQL engines, substantially decreasing resource consumption and monetary expenditure. The approach is robust to data drift, workload variation, and scale, with negligible computational or system integration overhead.

Potential directions include:

• Cross-query transfer learning: Enabling selectivity models to warm-start from related queries or workloads, reducing cold-start effects.
• Generalization to broader semantic operator families: Adapting the online learning decomposition for aggregation, join, and pipeline plans.
• Explicit correlation modeling: Integrating conditional dependency estimation for workloads demonstrating strong filter interdependencies, beyond independence-assumption DP.

Conclusion

Larch provides a principled, adaptive, and practical solution to optimizing semantic predicate execution in AI SQL. Via instance-level selectivity learning and latency-hiding online optimization, Larch-Sel in particular achieves token cost within a few percent of the infeasible per-row optimal, outperforming both existing adaptive methods and oracle-enhanced static heuristics. These results suggest that embedding-driven, feedback-based optimization is an essential paradigm for scalable, efficient semantic query processing over unstructured data in LLM-powered data systems (2606.07923).