Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method

Abstract: Evaluating a natural-language yes/no predicate over a document corpus under an accuracy target - the semantic filter - is a cornerstone of LLM-based data processing. Calling the LLM on every document (the oracle) is prohibitive, so cascades pair the oracle with a fast proxy. As deployed today, they leave four limitations on the table. (1) Each cascade family - model-free clustering, prebuilt small-LLM proxies, online-trained proxies - commits to a single representation and pipeline, and wins on only a narrow query regime. (2) The strongest online proxy invests in a custom training scheme on a bi-encoder over dense embeddings, missing the token-level evidence richer predicates require. (3) The proxy is trained against binary yes/no labels, wasting the LLM's per-document confidence at the boundary documents it most needs to learn. (4) Existing calibrations add a uniform safety margin, conflating genuine proxy uncertainty with small-sample noise and inflating cascade cost. We address these by (1) composing families adaptively - model-free clustering first, online proxy only when needed, with oracle calls shared across phases; (2) replacing the cosine bi-encoder with a hybrid of off-the-shelf token-aware models; (3) training the proxy with the oracle's per-document confidence as a soft label; and (4) a calibration that adds the safety margin only where the labeled sample is sparse. We are also the first to use the oracle's per-document confidence for three purposes: a query-level difficulty compass, a lower bound on the minimum oracle calls any proxy-based cascade can make, and the proxy's soft training label. At a 90% accuracy target on three 10K-document corpora, our methods are 1.6-2.0x faster than the best prior method per corpus and meet the target on 95% of queries; the BER-derived lower bound indicates a further ~4-20x of headroom for future work.

• The paper presents a unified framework for LLM semantic filtering using an adaptive two-phase method that minimizes expensive oracle calls while meeting accuracy targets.
• It introduces a hybrid token-level proxy that fuses cross-encoder and ColBERT representations to enhance class-separability and calibration.
• Empirical results on diverse benchmarks show 1.6–2.0× lower latency and reliable SLA adherence with a novel per-bin adaptive calibration strategy.

Fast LLM-Based Semantic Filtering: Unified Framework and Adaptive Two-Phase Execution

Introduction and Motivation

The paper "Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method" (2606.08090) addresses the computational inefficiency of direct LLM-based semantic filtering over large document corpora. The central task is to evaluate natural-language predicates (e.g., yes/no queries) over corpora, achieving a user-specified accuracy target, while minimizing expensive oracle LLM calls. Naive execution, which calls an LLM on every document, is untenable at scale. Earlier work introduced proxy cascades (model-free clustering, small-LLM proxies, and online-trained proxies) but left significant limitations in their representational flexibility, calibration tightening, and adaptability across query regimes.

This paper delivers a unified conceptual framework for semantic filtering cascades, introduces a composition of model-free and proxy-based filtering (the Two-Phase method), proposes a proxy leveraging hybrid token-level modeling, and presents a novel per-bin calibration aligned with practical deployment targets. These contributions not only increase computational efficiency but also tightly bind empirical performance to optimality lower bounds.

Unified Framework for Semantic Filter Cascades

The authors structure the landscape of LLM-based semantic filtering around a six-step pipeline: partition, sample, label, build proxy, calibrate, and deploy. All notable prior methods map to this skeleton, differing only along four axes: representation, training, calibration, and partitioning. This abstraction exposes both synergies and limitations within the traditional CSV (cluster-and-vote), BARGAIN (prebuilt small-LLM proxies), and ScaleDoc (online per-query bi-encoder proxy) methods.

Figure 1: A method-by-step matrix contextualizes existing and novel cascades within the shared six-step framework, highlighting blue cells as the new contributions of this work.

Notably, earlier proxies either focused exclusively on topical similarity, ignored oracle confidence, or imposed calibration overhead unlinked from sample coverage. Additionally, no prior deployment adapts on-the-fly to query difficulty or exploits label reuse across method families.

Representation and Training: Token-Level Hybrid Proxy

Diagnosis of ScaleDoc's bottleneck demonstrates that bi-encoders over dense embeddings lose token-level evidence and cannot support predicates that depend on fine-grained semantic or reasoning attributes. Through visualization of embedding spaces, the authors show that bi-encoders fail to achieve class-separability for such queries.

Figure 2: Visualization highlights the entanglement of yes/no document classes in both raw and bi-encoded embedding spaces, indicating the subtree-level limitations of dense representation models.

To overcome this ceiling, the paper introduces a hybrid proxy architecture combining cross-encoder (CE) and ColBERT (CB) representations, fusing their outputs via a small head network. Crucially, proxy training uses the oracle's per-document confidence (soft labels) aggregated via binary cross-entropy, maintaining uncertainty on ambiguous documents. A primal–dual constraint ties training output directly to the user’s accuracy target, and a coverage regularizer prevents trivial solutions where the proxy simply defers all predictions to the oracle.

Calibration: Per-Score-Range Adaptive Thresholding

Traditional calibrations—such as those in BARGAIN and ScaleDoc—either uniformly add excessive safety margins (thus overspending on cascades) or under-guard in the presence of sparse bins (leading to SLA violations).

This work proposes a calibration blending empirical per-bin proxy error rates with Clopper–Pearson upper bounds, controlled by a mixing weight. Safety margin is increased only in bins with insufficient coverage, ensuring tightness of the threshold while retaining accuracy guarantees. This binwise calibration adapts cost to observed sample uncertainty, rather than imposing global conservative surcharges.

Adaptive Two-Phase Cascade: Model-Free then Proxy

Addressing the empirical observation that no single family wins across all queries, the paper introduces Two-Phase, which begins with lightweight CSV clustering and voting, escalating to the hybrid proxy only if clusters remain ambiguous or coverage grows past a threshold. The transition is costless with respect to training labels, as Phase-1 oracle calls are reused for proxy training. Phase-2 deploys calibrated proxy-based filtering, stratified by query-level difficulty.

Figure 3: End-to-end latency across queries as a function of mean Bayes Error Rate (BER); method costs approach an optimal boundary as query difficulty increases.

Figure 4: The lower-envelope view of per-query cost demonstrates that Two-Phase tracks the lowest deployable latency across all queries.

Oracle Confidence: Difficulty Compass and Optimality Bound

A novel insight in this work is using the oracle's own token-level confidence (Bayes Error Rate, BER) as a difficulty metric predicting cascade efficiency as a function of query. BER can explain which family will be optimal for a particular query and is also used to derive a strict lower bound on the minimum number of oracle calls required for any statistical guarantee. This "BER-LB" serves as a performance benchmark, revealing headroom even for the best current methods.

Figure 5: Bayes Error Rate serves as a compass, with the family boundaries and cross-over points demarcated by empirical BER distribution.

Empirical Results and Ablation Analysis

Empirical evaluation on three 10K-document benchmarks (PubMed, GovReport, BigPatent) under $\alpha=0.9$ demonstrates that the Two-Phase method achieves $1.6$–$2.0\times$ lower latency than previous best methods per dataset and hits the target SLA on $95\%$ of queries. These gains are mainly from a substantial reduction in cascade oracle calls while maintaining small, adaptive labeling budgets.

Ablation studies validate that architecture selection is the dominant factor in proxy effectiveness (with hybrid proxies decisively outperforming bi-encoders) and that calibration tightness is crucial for both efficiency and SLA compliance. The proxy–calibration combination in this work achieves within $10.6\%$ of an omniscient, non-deployable lower bound (using all labels for calibration), and the BER-derived lower bound indicates another $4$–$20\times$ of attainable headroom.

Implications and Future Research

The findings have several theoretical and practical implications:

• Cost-Optimal Retrieval: Exploiting oracle confidence as both a proxy training signal and a driver of per-query plan selection is an efficient unification of semantic filtering and active learning. The method tightly integrates database-inspired cascading with modern token-level NLP proxies.
• Framework Generalization: The six-step framework is extensible across future proxy architectures, calibration procedures, and operator types (joins, aggregation, top-$k$).
• Performance Headroom: There remains robust theoretical headroom, suggesting that future work on adaptive labeling, improved proxies, and finer-grained calibration could further close the gap toward the BER lower bound.
• Operator-Level Routing: The framework supports even richer adaptive execution, such as per-query operator selection and multi-phase deployments, potentially augmented by small LLM proxies in a third-stage or as an auxiliary signal.
• Transferability across Operators: Design innovations in representation and calibration can transfer directly to other document-level LLM operators (semantic group-by, join), informing system-level optimizer strategies for hybrid query plans.

Conclusion

This paper establishes a unified paradigm for LLM-based semantic filtering, introduces architectural and calibration innovations with strong empirical support, and connects proxy performance directly to the oracle's statistical uncertainty. The Two-Phase adaptive method delivers state-of-the-art efficiency, robustness to query difficulty, and transparent headroom diagnosis, establishing technical foundations for future scalable, cost-optimal, semantic data systems (2606.08090).