Semantic Filter (AI.IF): Query Optimization

• Semantic Filter (AI.IF) is a query operator that uses LLM binary judgments to evaluate natural language predicates over text-bearing tuples.
• It relies on formal semantics and advanced strategies like clustering, sampling, and voting to reduce per-tuple LLM calls and improve efficiency.
• The system further optimizes costs and latency with lightweight proxy models and learned predicate ordering, ensuring scalable query execution.

Searching arXiv for papers on AI.IF semantic filtering and related semantic query operators. Semantic Filter (AI.IF) is a semantic query operator that evaluates a natural-language predicate against unstructured or text-bearing tuples by delegating the decision to a LLM. In formal relational-algebra form, given a table $T$ and predicate $e$, the operator returns

$$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$

where each invocation constructs a prompt from a fixed system instruction, the predicate, and tuple content, and asks the model for a binary judgment (Hou et al., 5 Mar 2026). In SQL systems, the same abstraction appears as a Boolean UDF, typically in a clause of the form WHERE AI.IF(PROMPT, column), with a full-fidelity semantics defined by the LLM’s response to the concatenated prompt and row text (Chung et al., 16 Mar 2026).

1. Formal semantics and the linear-scan barrier

The defining property of AI.IF is that its semantics are model-mediated rather than syntax-mediated. A row is selected not because it matches a fixed lexical pattern, but because the model judges that its content satisfies the prompt. In the proxy-analysis formulation, if $x$ is a text column and $q$ is the prompt, then

$$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$

where $q\|x$ denotes string concatenation (Chung et al., 16 Mar 2026).

The naïve execution strategy evaluates the predicate independently for every tuple. In the semantic-query-processing formulation, this is the “reference” algorithm and requires $O(|T|)$ LLM invocations, one per tuple (Hou et al., 5 Mar 2026). The reported consequences are prohibitive latency, massive token consumption, and high financial cost on hosted LLM APIs. In the database-systems formulation, semantic filters are also treated as black boxes because their selectivity depends on latent semantic features only revealed by invoking the LLM at query time; no pre-computed histograms or sketches capture arbitrary natural-language prompts (Zhao et al., 6 Jun 2026).

This execution model becomes even more consequential for compound predicates. For expression trees containing multiple AI_FILTER leaves connected by AND and OR, evaluation order matters because short-circuiting can avoid later LLM calls. The optimization problem is therefore not only whether to approximate an individual semantic filter, but also how to schedule multiple such filters without changing correctness (Zhao et al., 6 Jun 2026).

2. Clustering–Sampling–Voting and sublinear invocation

A recent response to the linear-invocation barrier is the Clustering–Sampling–Voting (CSV) paradigm, which reduces LLM calls to sublinear complexity on average by amortizing decisions across semantically similar tuples (Hou et al., 5 Mar 2026). CSV has an offline clustering phase and an online sampling-voting phase, with recursive re-clustering for ambiguous cases.

In the offline phase, each tuple is embedded with a pretrained sentence embedding model such as E5-Large, and $k$-means is applied to obtain clusters $C_1,\dots,C_k$. The distance metric is Euclidean distance $e$0, optionally combined with BM25 lexical score when the predicate contains explicit keywords. Because this phase is query-agnostic, it can be amortized across many filters (Hou et al., 5 Mar 2026).

In the online phase, each cluster $e$1 is sampled uniformly at random with ratio $e$2 or a minimum sample size. The sampled tuples are labeled by the LLM, and the labels are propagated to the remaining tuples by one of two voting rules. Under Uniform Voting (UniVote), the cluster score is

$e$3

with lower and upper thresholds $e$4 and $e$5. If the score exceeds $e$6, the remaining tuples inherit True; if it is below $e$7, they inherit False; otherwise the cluster is marked undetermined. Under Similarity-Weighted Voting (SimVote), each unsampled tuple receives a personalized score

$e$8

where similarity is cosine or inverse-Euclidean (Hou et al., 5 Mar 2026).

Ambiguous clusters are not forced into a premature decision. CSV collects such clusters into a buffer, re-clusters them into subclusters, and repeats the sampling-to-voting cycle. Recursion depth is capped, and any subcluster still ambiguous at maximum depth falls back to per-tuple LLM calls. The theoretical analysis gives best-case invocation complexity $e$9 and worst-case $$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$0, and provides Bernstein-inequality-based error bounds for UniVote and SimVote under the stated sampling conditions (Hou et al., 5 Mar 2026).

Empirically, CSV was evaluated on IMDB-Review, Codebase, Airdialogue, TC, and Fever. Reported call reductions are $$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$1–$$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$2 versus the per-tuple Reference, $$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$3–$$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$4 versus Lotus, and $$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$5–$$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$6 versus BARGAIN, while maintaining comparable effectiveness in Accuracy and F1. On RV-Q1 over 50K reviews, the Reference makes 50,000 LLM calls, whereas CSV makes 404, a $$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$7 reduction; Lotus makes approximately 144,000 calls, yielding a $$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$8 reduction for CSV; BARGAIN makes approximately 80,000 calls, yielding a $$\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},$$9 reduction (Hou et al., 5 Mar 2026).

3. Proxy approximation over embeddings

A second line of work replaces most LLM invocations with a lightweight classifier over embedding vectors, while retaining the LLM as the semantic reference model (Chung et al., 16 Mar 2026). In this formulation, a cheaper proxy $x$0 is trained so that $x$1 on the row distribution of the table.

The proxy cost model separates full execution from approximation. If $x$2 is the number of rows and $x$3 the sample size, then

$x$4

and

$x$5

where $x$6 is per-row embedding cost, $x$7 is per-row LLM cost, $x$8 is one-time proxy training cost, and $x$9 is per-row proxy inference cost (Chung et al., 16 Mar 2026).

The canonical proxy design is deliberately lightweight. The prompt is embedded once, each row text is embedded, and the classifier is logistic regression:

$q$0

Training minimizes weighted cross-entropy on the LLM-labeled sample, with class_weight="balanced" by default, SMOTE when minority examples are fewer than 100, and default $q$1 penalty $q$2 (Chung et al., 16 Mar 2026). This design is deployed in two architectures: a fully online BigQuery OLAP-friendly mode, where sampling and proxy training happen at query time, and an AlloyDB HTAP-friendly mode, where common prompts are identified offline and the proxy is pre-trained and stored in-database (Chung et al., 16 Mar 2026).

The system also includes an adaptive fallback. If held-out proxy accuracy drops more than a threshold $q$3 below the LLM baseline, the remaining rows are processed by the LLM UDF rather than the proxy. This makes the approximation conditional rather than unconditional (Chung et al., 16 Mar 2026).

On a 10M-row Amazon Polarity “positive review” filter, the full LLM baseline with Gemini 2.5 Flash has latency of approximately 1,320 s and cost of approximately \$q$40.0126, reported as $q$5 faster and $q$6 cheaper. The offline pre-trained proxy further reduces latency to approximately 1.33 s and cost to approximately \$q$7991\times$q$8792\times$ cheaper (Chung et al., 16 Mar 2026).

Accuracy preservation is likewise reported across 11 classification datasets. The proxy achieves Macro-F1(proxy) $q$9 in 9 of 11 tasks, relative accuracy $$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$0 in 7 of 11 tasks, and a ratio greater than 1.0 on six tasks. On Amazon Reviews 10k, the proxy records 0.860 versus 0.739 for the LLM; on BBC News 5-way, 0.830 versus 0.823. On highly nuanced or imbalanced tasks such as Emotion 6 classes and FEVER fact checking, relative accuracy remains high at approximately 0.90 (Chung et al., 16 Mar 2026).

4. Learned optimization of compound semantic predicates

When an AI SQL query contains multiple semantic filters, reducing the cost of each filter is only part of the problem. Larch addresses the complementary problem of choosing the evaluation order of AI_FILTER leaves within a Boolean expression tree so as to minimize total token cost without affecting correctness (Zhao et al., 6 Jun 2026).

Larch is based on two observations: semantic operators have sufficiently high latency to accommodate computationally heavy runtime optimization, and unstructured data are typically accompanied by embeddings that permit efficient semantic comparisons between prompts and data values. The framework provides two variants. Larch-A2C formulates ordering as a Markov decision process and uses an embedding-augmented Gated Graph Neural Network to encode the partially evaluated AND/OR expression tree. The action at each step is the choice of the next unevaluated leaf, and the reward is the normalized negative token cost of the invoked predicate (Zhao et al., 6 Jun 2026).

Larch-Sel decomposes the same problem into selectivity prediction and exact planning. A lightweight MLP predicts row-specific pass probabilities $$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$1 from features of the form

$$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$2

Under an independence assumption, the minimum expected cost of a partially evaluated tree $$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$3 satisfies the Bellman recurrence

$$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$4

with memoized dynamic programming complexity $$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$5 for $$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$6 leaves, reported as practical in milliseconds for $$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$7 (Zhao et al., 6 Jun 2026).

Evaluation uses GovReport, PubMed, and BigPatent, with 45 predicate expressions per dataset for pure conjunctions, pure disjunctions, and mixed trees. Baselines are Simple, Palimpzest, and Quest. Both Larch variants always outperform the baselines in total tokens and calls. Larch-A2C reduces token overhead by roughly $$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$8–$$ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},$$9 relative to Palimpzest and Quest, while Larch-Sel achieves up to $q\|x$0 less token overhead on high-selectivity or low-selectivity pathological cases. Reported local model updates are approximately 10 ms for GGNN backward passes and approximately 7 ms for the MLP, and these costs are hidden behind LLM calls of 100–500 ms, so end-to-end wall-clock time is unchanged (Zhao et al., 6 Jun 2026).

5. Reported trade-offs, tuning regimes, and common misconceptions

Across the recent literature, the central design trade-off is not between semantics and non-semantics, but between full-fidelity LLM evaluation and mechanisms that reduce the number of full-fidelity invocations. CSV, proxy models, and learned predicate ordering all preserve the LLM as the semantic reference point, even when they reduce cost by clustering, prediction, or scheduling (Hou et al., 5 Mar 2026, Chung et al., 16 Mar 2026, Zhao et al., 6 Jun 2026).

A recurring misconception is to equate AI.IF with embedding similarity search. The recent systems do use embeddings, but only as support structures: CSV uses them for clustering and similarity-weighted voting, proxy systems use them as classifier inputs, and Larch uses them for selectivity estimation and graph encoding. The formal semantics remain the binary LLM judgment $q\|x$1 or $q\|x$2 (Hou et al., 5 Mar 2026, Chung et al., 16 Mar 2026, Zhao et al., 6 Jun 2026).

Practical parameter guidance is reported most explicitly for CSV. The number of clusters should start at $q\|x$3–$q\|x$4, with diminishing returns beyond approximately 8; the sampling ratio $q\|x$5 balances cost and error, and in practice $q\|x$6 as low as 0.005 suffices; default thresholds are $q\|x$7 and $q\|x$8; lower $q\|x$9, such as 0.01, is suggested for extremely rare positive classes; and any off-the-shelf sentence embedder such as E5, BGE, or Qwen can be used (Hou et al., 5 Mar 2026). For proxy models, acceleration comes from precomputing embeddings, serial logistic-regression training on a single node, distributed inference, and using class weights rather than expensive stratified queries (Chung et al., 16 Mar 2026). For learned ordering, the main insight is that selectivity estimation dominates end-to-end planning quality, which is why Larch-Sel outperforms the end-to-end RL variant in sample efficiency and proximity to the optimal lower bound (Zhao et al., 6 Jun 2026).

The main reported failure modes are likewise specific rather than generic. CSV can trigger recursive re-clustering when cluster scores fall into the undetermined band, and eventually reverts to per-tuple LLM evaluation at maximum depth (Hou et al., 5 Mar 2026). Proxy systems report that tasks requiring deep multi-sentence reasoning or “needle-in-haystack” rare-event detection may exceed the support of a cheaply sampled proxy, in which case the system falls back to the LLM (Chung et al., 16 Mar 2026). Larch does not approximate individual predicate results, but its gains depend on the quality of selectivity prediction and the availability of short-circuit opportunities (Zhao et al., 6 Jun 2026).

Approach	Core mechanism	Reported effect
CSV (Hou et al., 5 Mar 2026)	Cluster, sample, vote, re-cluster	$O(|T|)$0–$O(|T|)$1 fewer LLM calls vs Reference
Proxy AI.IF (Chung et al., 16 Mar 2026)	Embedding + logistic-regression proxy	$O(|T|)$2–$O(|T|)$3 faster on 10M-row Amazon filter
Larch (Zhao et al., 6 Jun 2026)	Learned ordering of AI_FILTER trees	$O(|T|)$4–$O(|T|)$5 lower token overhead vs state of the art

6. Antecedents, scope, and nomenclature

The semantic-filter idea predates LLM-mediated AI SQL. An earlier lineage appears in semantic content filtering with Wikipedia and ontologies, where documents were represented by Wikipedia concepts, Business-Term Ontology concepts, and bag-of-words features, and a genetic-programming-based Inductive Query-By-Example learner evolved a Boolean rule over topic concepts (Malo et al., 2010). In that framework, semantic relatedness came from Wikipedia link structure and was used for implicit query expansion through document-term relatedness. Evaluated on Reuters RCV1 and TREC-11 filtering topics, the Wiki-SR approach reports average test performance of $O(|T|)$6, $O(|T|)$7, and $O(|T|)$8 on all topics, versus $O(|T|)$9 for the best C4.5 setting and $k$0 for the best LibSVM setting (Malo et al., 2010).

This suggests a historical continuity from knowledge-based semantic content classifiers to modern operator-level semantic predicates. The shift is not merely from symbolic resources to neural models, but from topic-specific classifier construction toward a general-purpose relational operator whose semantics are delegated to an LLM and whose systems problem is large-scale execution (Malo et al., 2010, Hou et al., 5 Mar 2026).

The term “semantic filter” is also polysemous outside database systems. In computer vision it has been used for semantically interpretable filter sets learned from natural images (Prabhushankar et al., 2019) and for a transformer-based Semantic Filter module for few-shot learning (Lai et al., 2022). In the AI.SQL literature, however, “Semantic Filter (AI.IF)” denotes a specific operator: a natural-language predicate evaluated over unstructured data, increasingly accompanied by sublinear invocation strategies, lightweight proxies, and learned query optimization (Hou et al., 5 Mar 2026, Chung et al., 16 Mar 2026, Zhao et al., 6 Jun 2026).