---
title: 'Semantic Filter (AI.IF): Query Optimization'
url: https://www.emergentmind.com/topics/semantic-filter-ai-if
type: topic
---

# Semantic Filter (AI.IF): Query Optimization

Searching arXiv for recent papers on AI.IF semantic filtering and related semantic query operators.
Semantic Filter (AI.IF) is a semantic query operator that evaluates a natural-language predicate against unstructured or text-bearing tuples by delegating the decision to a large language model. In formal relational-algebra form, given a table $T$ and predicate $e$, the operator returns
$$
\sigma_M(e)(T)=\{\,t\in T \mid M(t,e)=\mathrm{True}\,\},
$$
where each invocation constructs a prompt from a fixed system instruction, the predicate, and tuple content, and asks the model for a binary judgment [2603.04799]. In SQL systems, the same abstraction appears as a Boolean UDF, typically in a clause of the form `WHERE AI.IF(PROMPT, column)`, with a full-fidelity semantics defined by the LLM’s response to the concatenated prompt and row text [2603.15970].

## 1. Formal semantics and the linear-scan barrier

The defining property of AI.IF is that its semantics are model-mediated rather than syntax-mediated. A row is selected not because it matches a fixed lexical pattern, but because the model judges that its content satisfies the prompt. In the proxy-analysis formulation, if $x$ is a text column and $q$ is the prompt, then
$$
ML\_IF(q,x)=1 \text{ if } M_{\mathrm{LLM}}(q\|x)\in \text{“Positive”}, \quad 0 \text{ otherwise},
$$
where $q\|x$ denotes string concatenation [2603.15970].

The naïve execution strategy evaluates the predicate independently for every tuple. In the semantic-query-processing formulation, this is the “reference” algorithm and requires $O(|T|)$ LLM invocations, one per tuple [2603.04799]. The reported consequences are prohibitive latency, massive token consumption, and high financial cost on hosted LLM APIs. In the database-systems formulation, semantic filters are also treated as black boxes because their selectivity depends on latent semantic features only revealed by invoking the LLM at query time; no pre-computed histograms or sketches capture arbitrary natural-language prompts [2606.07923].

This execution model becomes even more consequential for compound predicates. For expression trees containing multiple `AI_FILTER` leaves connected by `AND` and `OR`, evaluation order matters because short-circuiting can avoid later LLM calls. The optimization problem is therefore not only whether to approximate an individual semantic filter, but also how to schedule multiple such filters without changing correctness [2606.07923].

## 2. Clustering–Sampling–Voting and sublinear invocation

A recent response to the linear-invocation barrier is the Clustering–Sampling–Voting (CSV) paradigm, which reduces LLM calls to sublinear complexity on average by amortizing decisions across semantically similar tuples [2603.04799]. CSV has an offline clustering phase and an online sampling-voting phase, with recursive re-clustering for ambiguous cases.

In the offline phase, each tuple is embedded with a pretrained sentence embedding model such as E5-Large, and $k$-means is applied to obtain clusters $C_1,\dots,C_k$. The distance metric is Euclidean distance $\|e_i-e_j\|_2$, optionally combined with BM25 lexical score when the predicate contains explicit keywords. Because this phase is query-agnostic, it can be amortized across many filters [2603.04799].

In the online phase, each cluster $C_j$ is sampled uniformly at random with ratio $\xi$ or a minimum sample size. The sampled tuples are labeled by the LLM, and the labels are propagated to the remaining tuples by one of two voting rules. Under Uniform Voting (UniVote), the cluster score is
$$
\mathrm{score}=\frac{|O^+|}{|C'_j|},
$$
with lower and upper thresholds $lb$ and $ub$. If the score exceeds $ub$, the remaining tuples inherit `True`; if it is below $lb$, they inherit `False`; otherwise the cluster is marked undetermined. Under Similarity-Weighted Voting (SimVote), each unsampled tuple receives a personalized score
$$
\mathrm{score}(t)=\sum_{t'\in C'_j}\frac{\mathrm{sim}(e_t,e_{t'})}{\sum_{u\in C'_j}\mathrm{sim}(e_t,e_u)}\;\mathbb{I}\!\bigl(M(t',e)=\mathrm{True}\bigr),
$$
where similarity is cosine or inverse-Euclidean [2603.04799].

Ambiguous clusters are not forced into a premature decision. CSV collects such clusters into a buffer, re-clusters them into subclusters, and repeats the sampling-to-voting cycle. Recursion depth is capped, and any subcluster still ambiguous at maximum depth falls back to per-tuple LLM calls. The theoretical analysis gives best-case invocation complexity $O(\xi\cdot |T|)$ and worst-case $O(|T|)$, and provides Bernstein-inequality-based error bounds for UniVote and SimVote under the stated sampling conditions [2603.04799].

Empirically, CSV was evaluated on IMDB-Review, Codebase, Airdialogue, TC, and Fever. Reported call reductions are $1.28\times$–$200\times$ versus the per-tuple Reference, $1.81\times$–$355\times$ versus Lotus, and $1.68\times$–$200\times$ versus BARGAIN, while maintaining comparable effectiveness in Accuracy and F1. On RV-Q1 over 50K reviews, the Reference makes 50,000 LLM calls, whereas CSV makes 404, a $123\times$ reduction; Lotus makes approximately 144,000 calls, yielding a $356\times$ reduction for CSV; BARGAIN makes approximately 80,000 calls, yielding a $198\times$ reduction [2603.04799].

## 3. Proxy approximation over embeddings

A second line of work replaces most LLM invocations with a lightweight classifier over embedding vectors, while retaining the LLM as the semantic reference model [2603.15970]. In this formulation, a cheaper proxy $M_P:\mathcal{X}\to\{0,1\}$ is trained so that $M_P(q,x)\approx ML\_IF(q,x)$ on the row distribution of the table.

The proxy cost model separates full execution from approximation. If $N$ is the number of rows and $s\ll N$ the sample size, then
$$
Cost_{\mathrm{full}}=N\cdot(c_e+c_{\mathrm{llm}})
$$
and
$$
Cost_{\mathrm{proxy}}=s\cdot(c_e+c_{\mathrm{llm}})+c_{\mathrm{train}}+N\cdot(c_e+c_{\mathrm{pred}}),
$$
where $c_e$ is per-row embedding cost, $c_{\mathrm{llm}}$ is per-row LLM cost, $c_{\mathrm{train}}$ is one-time proxy training cost, and $c_{\mathrm{pred}}$ is per-row proxy inference cost [2603.15970].

The canonical proxy design is deliberately lightweight. The prompt is embedded once, each row text is embedded, and the classifier is logistic regression:
$$
\hat y=\sigma(w^\top e_x+b).
$$
Training minimizes weighted cross-entropy on the LLM-labeled sample, with `class_weight="balanced"` by default, SMOTE when minority examples are fewer than 100, and default $\ell_2$ penalty $\alpha=1.0$ [2603.15970]. This design is deployed in two architectures: a fully online BigQuery OLAP-friendly mode, where sampling and proxy training happen at query time, and an AlloyDB HTAP-friendly mode, where common prompts are identified offline and the proxy is pre-trained and stored in-database [2603.15970].

The system also includes an adaptive fallback. If held-out proxy accuracy drops more than a threshold $\tau$ below the LLM baseline, the remaining rows are processed by the LLM UDF rather than the proxy. This makes the approximation conditional rather than unconditional [2603.15970].

On a 10M-row Amazon Polarity “positive review” filter, the full LLM baseline with Gemini 2.5 Flash has latency of approximately 1,320 s and cost of approximately \$9.15. The online proxy without precomputation reduces this to approximately 4 s and \$0.0126, reported as $329\times$ faster and $728\times$ cheaper. The offline pre-trained proxy further reduces latency to approximately 1.33 s and cost to approximately \$0.0115, reported as $991\times$ faster and $792\times$ cheaper [2603.15970].

Accuracy preservation is likewise reported across 11 classification datasets. The proxy achieves Macro-F1(proxy) $\ge 0.83$ in 9 of 11 tasks, relative accuracy $F1(\mathrm{proxy})/F1(\mathrm{LLM})\ge 0.98$ in 7 of 11 tasks, and a ratio greater than 1.0 on six tasks. On Amazon Reviews 10k, the proxy records 0.860 versus 0.739 for the LLM; on BBC News 5-way, 0.830 versus 0.823. On highly nuanced or imbalanced tasks such as Emotion 6 classes and FEVER fact checking, relative accuracy remains high at approximately 0.90 [2603.15970].

## 4. Learned optimization of compound semantic predicates

When an AI SQL query contains multiple semantic filters, reducing the cost of each filter is only part of the problem. Larch addresses the complementary problem of choosing the evaluation order of `AI_FILTER` leaves within a Boolean expression tree so as to minimize total token cost without affecting correctness [2606.07923].

Larch is based on two observations: semantic operators have sufficiently high latency to accommodate computationally heavy runtime optimization, and unstructured data are typically accompanied by embeddings that permit efficient semantic comparisons between prompts and data values. The framework provides two variants. Larch-A2C formulates ordering as a Markov decision process and uses an embedding-augmented Gated Graph Neural Network to encode the partially evaluated `AND`/`OR` expression tree. The action at each step is the choice of the next unevaluated leaf, and the reward is the normalized negative token cost of the invoked predicate [2606.07923].

Larch-Sel decomposes the same problem into selectivity prediction and exact planning. A lightweight MLP predicts row-specific pass probabilities $\hat s_i(r)=\Pr[f_i(r)=\mathrm{True}]$ from features of the form
$$
\mathbf{x}=[\,\mathbf{d}\|\mathbf{f}\|\mathbf{d}\odot\mathbf{f}\|\cos(\mathbf{d},\mathbf{f})\,].
$$
Under an independence assumption, the minimum expected cost of a partially evaluated tree $T'$ satisfies the Bellman recurrence
$$
\mathrm{OPT}(T')=\min_{f_i\in\mathcal{R}}\Bigl\{c_i+\hat s_i\,\mathrm{OPT}(T'|_{f_i=\mathrm{True}})+(1-\hat s_i)\,\mathrm{OPT}(T'|_{f_i=\mathrm{False}})\Bigr\},
$$
with memoized dynamic programming complexity $O(n\cdot 3^n)$ for $n$ leaves, reported as practical in milliseconds for $n\le 10$ [2606.07923].

Evaluation uses GovReport, PubMed, and BigPatent, with 45 predicate expressions per dataset for pure conjunctions, pure disjunctions, and mixed trees. Baselines are Simple, Palimpzest, and Quest. Both Larch variants always outperform the baselines in total tokens and calls. Larch-A2C reduces token overhead by roughly $1.4\times$–$6\times$ relative to Palimpzest and Quest, while Larch-Sel achieves up to $19\times$ less token overhead on high-selectivity or low-selectivity pathological cases. Reported local model updates are approximately 10 ms for GGNN backward passes and approximately 7 ms for the MLP, and these costs are hidden behind LLM calls of 100–500 ms, so end-to-end wall-clock time is unchanged [2606.07923].

## 5. Reported trade-offs, tuning regimes, and common misconceptions

Across the recent literature, the central design trade-off is not between semantics and non-semantics, but between full-fidelity LLM evaluation and mechanisms that reduce the number of full-fidelity invocations. CSV, proxy models, and learned predicate ordering all preserve the LLM as the semantic reference point, even when they reduce cost by clustering, prediction, or scheduling [2603.04799][2603.15970][2606.07923].

A recurring misconception is to equate AI.IF with embedding similarity search. The recent systems do use embeddings, but only as support structures: CSV uses them for clustering and similarity-weighted voting, proxy systems use them as classifier inputs, and Larch uses them for selectivity estimation and graph encoding. The formal semantics remain the binary LLM judgment $M(t,e)$ or $M_{\mathrm{LLM}}(q\|x)$ [2603.04799][2603.15970][2606.07923].

Practical parameter guidance is reported most explicitly for CSV. The number of clusters should start at $k=4$–$8$, with diminishing returns beyond approximately 8; the sampling ratio $\xi\in[0.005,0.02]$ balances cost and error, and in practice $\xi$ as low as 0.005 suffices; default thresholds are $lb=0.15$ and $ub=0.85$; lower $lb$, such as 0.01, is suggested for extremely rare positive classes; and any off-the-shelf sentence embedder such as E5, BGE, or Qwen can be used [2603.04799]. For proxy models, acceleration comes from precomputing embeddings, serial logistic-regression training on a single node, distributed inference, and using class weights rather than expensive stratified queries [2603.15970]. For learned ordering, the main insight is that selectivity estimation dominates end-to-end planning quality, which is why Larch-Sel outperforms the end-to-end RL variant in sample efficiency and proximity to the optimal lower bound [2606.07923].

The main reported failure modes are likewise specific rather than generic. CSV can trigger recursive re-clustering when cluster scores fall into the undetermined band, and eventually reverts to per-tuple LLM evaluation at maximum depth [2603.04799]. Proxy systems report that tasks requiring deep multi-sentence reasoning or “needle-in-haystack” rare-event detection may exceed the support of a cheaply sampled proxy, in which case the system falls back to the LLM [2603.15970]. Larch does not approximate individual predicate results, but its gains depend on the quality of selectivity prediction and the availability of short-circuit opportunities [2606.07923].

| Approach | Core mechanism | Reported effect |
|---|---|---|
| CSV [2603.04799] | Cluster, sample, vote, re-cluster | $1.28\times$–$200\times$ fewer LLM calls vs Reference |
| Proxy AI.IF [2603.15970] | Embedding + logistic-regression proxy | $329\times$–$991\times$ faster on 10M-row Amazon filter |
| Larch [2606.07923] | Learned ordering of `AI_FILTER` trees | $3\times$–$19\times$ lower token overhead vs state of the art |

## 6. Antecedents, scope, and nomenclature

The semantic-filter idea predates LLM-mediated AI SQL. An earlier lineage appears in semantic content filtering with Wikipedia and ontologies, where documents were represented by Wikipedia concepts, Business-Term Ontology concepts, and bag-of-words features, and a genetic-programming-based Inductive Query-By-Example learner evolved a Boolean rule over topic concepts [1012.0854]. In that framework, semantic relatedness came from Wikipedia link structure and was used for implicit query expansion through document-term relatedness. Evaluated on Reuters RCV1 and TREC-11 filtering topics, the Wiki-SR approach reports average test performance of $F=0.44$, $P=0.47$, and $R=0.54$ on all topics, versus $F=0.31$ for the best C4.5 setting and $F=0.27$ for the best LibSVM setting [1012.0854].

This suggests a historical continuity from knowledge-based semantic content classifiers to modern operator-level semantic predicates. The shift is not merely from symbolic resources to neural models, but from topic-specific classifier construction toward a general-purpose relational operator whose semantics are delegated to an LLM and whose systems problem is large-scale execution [1012.0854][2603.04799].

The term “semantic filter” is also polysemous outside database systems. In computer vision it has been used for semantically interpretable filter sets learned from natural images [1902.06334] and for a transformer-based Semantic Filter module for few-shot learning [2211.00868]. In the AI.SQL literature, however, “Semantic Filter (AI.IF)” denotes a specific operator: a natural-language predicate evaluated over unstructured data, increasingly accompanied by sublinear invocation strategies, lightweight proxies, and learned query optimization [2603.04799][2603.15970][2606.07923].

Source: https://www.emergentmind.com/topics/semantic-filter-ai-if