---
title: 'ScaleDoc: Scalable Semantic Filtering'
url: https://www.emergentmind.com/topics/scaledoc
type: topic
---

# ScaleDoc: Scalable Semantic Filtering

Searching arXiv for the cited ScaleDoc paper and closely related long-document and semantic-filtering work.
ScaleDoc is a system for executing LLM-based semantic predicates over large, unstructured document collections by decoupling expensive LLM computation into a one-time offline representation phase and a lightweight, query-specific online filtering phase that escalates only ambiguous cases to a powerful oracle LLM. In the primary formulation, a workload is defined by a document collection $D$, a natural-language query $q$, and an accuracy target $\alpha$; the task is to assign each document $d \in D$ as positive or negative for the predicate while ensuring $\mathrm{Acc} \ge \alpha$ [2509.12610]. In adjacent work, “ScaleDoc” also functions as a broader scaling objective for long-document understanding, benchmark construction, synthetic data generation, and document representation, connecting semantic filtering to multimodal document systems such as DocSLM, DocsRay, DocGenome, FlexDoc, and S2Doc [2511.11313].

## 1. Problem formulation and system objective

ScaleDoc targets boolean semantic predicates that require contextual understanding beyond keywords, such as “The paper introduces a new drug,” over large document collections [2509.12610]. The central motivation is that direct oracle evaluation of every document with a strong LLM is prohibitively expensive for ad hoc queries. The reported bottlenecks are explicit: one-time offline embedding of 10,000 PubMed documents using NvEmbed costs approximately 24 PFLOPs total, or about 2.4 TFLOPs per document, while a single-pass oracle evaluation at the level of Llama-3-70B requires about $10\times$ more compute than the offline embedding per query. Even a smaller LLM proxy such as Llama-3.2-3B consumes about 13 PFLOPs per query, and those costs recur per query and scale with corpus size [2509.12610].

The system therefore separates computation into reusable document representations and query-time selective inference. This separation is not merely a deployment optimization; it defines the formal execution model. The document collection is embedded once, and subsequent queries operate over those stored semantic representations. A lightweight proxy is then trained per query on a small oracle-labeled sample and used to filter the corpus. Only documents with ambiguous proxy scores are forwarded to the oracle, yielding a cascade constrained by an explicit accuracy target [2509.12610].

A later semantic-filtering paper places ScaleDoc inside a unified cascade taxonomy together with model-free clustering and prebuilt small-LLM proxies. That work treats the semantic filter as a cornerstone of LLM-based data processing and identifies four limitations of existing cascades: fixed commitment to one representation family, insufficient token-level evidence in dense-embedding bi-encoders, wasted oracle confidence near decision boundaries, and conservative calibration that applies uniform safety margins across score regions [2606.08090]. This suggests that ScaleDoc is best understood both as a concrete system and as a design point in a larger family of accuracy-targeted semantic filtering methods.

## 2. Offline representation and online predicate execution

The core ScaleDoc architecture consists of an offline representation phase and an online filtering phase [2509.12610]. In the offline phase, each document $d$ is encoded by a small-scale LLM encoder, NvEmbed derived from Mistral-7B, into a semantic embedding $e_d \in \mathbb{R}^D$. These embeddings are stored once and reused across queries. The paper emphasizes that no chunking or summarization is required in the core algorithm; embeddings are computed at document level.

In the online phase, a query embedding $e_q$ is computed with the same encoder, and a lightweight query-aware proxy is trained on a small sampled subset of documents, typically 5–10% with stratification. Labels for this subset are obtained from a powerful oracle LLM such as GPT-4o. The proxy then assigns a decision score $s(q,d)$ to each document, and a cascade uses score thresholds to partition the corpus into confident negatives, confident positives, and ambiguous documents routed to the oracle [2509.12610].

The proxy maps both query and document embeddings into a shared latent space via a 3-layer MLP encoder $E$:
$$
z_q = E(e_q), \qquad z_d = E(e_d).
$$
The decision score is cosine similarity,
$$
s(q,d)=\mathrm{sim}(z_q,z_d)=\frac{z_q \cdot z_d}{\|z_q\|\|z_d\|},
$$
normalized to $[0,1]$ for cascade computations. Higher scores indicate stronger semantic agreement with the predicate [2509.12610].

The cascade introduces lower and upper thresholds $(lb, rb)$ and partitions the score set into
$$
F^-=\{d:s(q,d)<lb\}, \quad
U=\{d:lb \le s(q,d)\le rb\}, \quad
F^+=\{d:s(q,d)>rb\}.
$$
Documents in $F^-$ and $F^+$ are labeled by the proxy; documents in $U$ are sent to the oracle. The unfiltered rate is
$$
u(lb,rb)=\frac{|\{s_i \in S(W): lb \le s_i \le rb\}|}{|S(W)|},
$$
and the optimization problem is to minimize $u(lb,rb)$ subject to $\mathrm{Acc}(lb,rb)\ge \alpha$, with $0 \le lb \le rb \le 1$ [2509.12610].

A succinct view of the execution pipeline is as follows.

| Component | Function | Stored or computed artifact |
|---|---|---|
| Offline representation | Embed each document once with NvEmbed | Dense vector per document |
| Query-specific proxy | Train 3-layer MLP on sampled oracle-labeled subset | Query-aware latent encoder and scores |
| Adaptive cascade | Select $(lb, rb)$ to meet $\alpha$ | Proxy labels outside ambiguity region, oracle labels inside |

This organization makes the dominant LLM “reading” cost amortizable across many queries while preserving a query-specific decision boundary.

## 3. Contrastive proxy learning and adaptive calibration

ScaleDoc’s main technical contribution is not merely the use of embeddings, but a training and calibration framework intended to make proxy scores reliable for filtering [2509.12610]. The paper argues that a plain MLP classifier on embeddings is inferior and reports that the contrastive proxy yields about a 20% average increase in data reduction, especially on hard cases.

Training proceeds in two phases. Phase 1 enforces semantic monotonicity with a query-anchored InfoNCE-style objective inspired by DPR. For a minibatch containing positives $\{d_i^+\}$ and negatives $\{d_j^-\}$, with temperature $\tau$,
$$
L_{qsim}(q,\{d_i^+\},\{d_j^-\})
=
-\log
\frac{\sum_{i=1}^{m}\exp(\mathrm{sim}(q,d_i^+)/\tau)}
{\sum_{k=1}^{n}\exp(\mathrm{sim}(q,d_k)/\tau)}.
$$
This pulls positives toward the query anchor and pushes negatives away, establishing a monotone relationship between semantic satisfaction and score [2509.12610].

Phase 2 enforces bipolarity through two additional objectives. The supervised contrastive loss clusters same-label documents:
$$
L_{supcon}(\{d_i\})
=
-\sum_{i=1}^{n}
\frac{1}{|U(i)|}
\log
\frac{\sum_{d_p \in U(i)} \exp(\mathrm{sim}(d_i,d_p)/\tau)}
{\sum_{d_k \in A(i)} \exp(\mathrm{sim}(d_i,d_k)/\tau)}.
$$
The polar loss widens separation margins by identifying bellwether anchors near the boundary and pushing positives and negatives toward their respective anchors. The joint Phase 2 objective is
$$
L_2 = \lambda \cdot L_{supcon} + (1-\lambda)\cdot L_{polar},
$$
with $\lambda = 0.2$, reported as empirically robust across datasets [2509.12610].

The cascade is calibrated not through Platt scaling or isotonic regression, but through reconstructed score distributions from a small, stratified, oracle-labeled sample. The calibration procedure combines stratified sampling across score bins, jittering to avoid empty regions, density estimation by linear interpolation, and smoothing by moving average. The outputs are positive and negative PDFs and their CDFs, which are then used for threshold search [2509.12610].

The threshold-selection algorithm exploits a Pareto-frontier observation: the optimal $(lb,rb)$ minimizing the unfiltered rate while satisfying $\mathrm{Acc}\ge \alpha$ lies on the frontier of feasible threshold pairs. Rather than brute-force quadratic search, ScaleDoc traces this frontier in linear time with respect to discretization steps. The paper’s default robust configuration is a training set of 7–10%, a calibration set of about 5%, and 64 discretization bins [2509.12610].

Subsequent work generalizes and critiques this design. “Fast LLM-Based Semantic Filtering” replaces the cosine bi-encoder with a token-aware hybrid of a cross-encoder, a ColBERT-style late-interaction model, and a small hybrid head; it also uses the oracle’s per-document confidence $p_i^\star$ as a soft label and introduces sample-aware calibration that blends empirical error with a one-sided Clopper–Pearson bound only where score bins are sparse [2606.08090]. That paper reports that the hybrid proxy with soft-label training and per-bin calibration is materially cheaper than the original ScaleDoc bi-encoder, while preserving target accuracy on most queries. A plausible implication is that ScaleDoc established the query-aware cascade template, while later work tightened the proxy representation and the calibration layer.

## 4. Empirical results, efficiency, and scaling behavior

ScaleDoc is evaluated on BigPatent, PubMed, and GovReport. Each dataset contains 10,000 documents, and for each dataset 20 ad hoc predicates were manually crafted. Ground-truth labels for evaluation are generated by GPT-4o [2509.12610]. The reported metrics are online end-to-end latency, data reduction rate, accuracy-target compliance, the accuracy-latency tradeoff, and per-stage latency breakdown.

At $\alpha = 0.90$, the system achieves over $2\times$ average online speedup compared to baselines across datasets and reduces oracle invocations by up to 85%, corresponding to about $6.6\times$ cost savings, while meeting the specified accuracy targets [2509.12610]. The paper attributes these gains to low online proxy costs and strong filtering from well-behaved scores. It also reports that relaxing $\alpha$ from 0.96 to 0.80 yields larger latency reductions for ScaleDoc than for baselines because semantic monotonicity and bipolarity enable more aggressive filtering when the target is looser.

The ablation results are especially central to the system’s interpretation. Phase 1 contrastive training is necessary to establish monotonicity but insufficient alone; Phase 2 creates bipolar score distributions and cleaner separation, substantially improving filtering. Calibration is likewise decisive: without reconstruction, naive thresholding on raw calibration samples fails to meet $\alpha = 0.90$ in nearly half the trials, whereas the full calibration pipeline meets the target in more than 80% of trials, with worst-case loss only 0.04 [2509.12610]. The paper also reports that its linear interpolation density estimator yields the lowest Jensen–Shannon distance on average relative to naive stratified sampling, importance sampling, and Beta fitting.

A later benchmark on the same semantic-filtering problem reports concrete comparative timings at $\alpha=0.9$. Its Phase-2 method cuts end-to-end time and oracle calls by about $2\times$ versus ScaleDoc: on PubMed, 264.6 s versus 536.8 s and 1,890 oracle calls versus 3,468; on GovReport, 412.7 s versus 765.0 s and 2,141 versus 3,694; on BigPatent, 116.1 s versus 217.3 s and 1,672 versus 2,292 [2606.08090]. That work also reports that its Two-Phase composition meets $\alpha=0.9$ on 95% of queries and identifies a BER-derived lower bound suggesting approximately $4$–$20\times$ headroom for future reductions in cascade cost. These results position ScaleDoc as a strong first-generation online proxy cascade whose main ideas remain competitive, even as later methods improve token sensitivity and calibration efficiency.

## 5. ScaleDoc in the wider document-understanding ecosystem

In the broader literature, “ScaleDoc” denotes a scaling requirement that recurs in long-document inference, retrieval, benchmarking, synthetic data generation, and representation standards. Several contemporaneous systems address adjacent but non-identical problems.

DocSLM addresses efficient long multimodal document understanding on resource-constrained devices. It compresses each page into exactly 576 visual tokens regardless of OCR length through a Hierarchical Multimodal Compressor and processes arbitrarily long inputs through Streaming Abstention with entropy-calibrated uncertainty [2511.11313]. The paper reports 82% fewer visual tokens, 75% fewer parameters, 71% lower latency, and a $\sim 14$ GB peak GPU memory plateau beyond 10 pages, with documents up to 120 pages processed under that cap. This suggests a complementary scaling axis to ScaleDoc: ScaleDoc minimizes oracle calls over large document collections, whereas DocSLM minimizes per-document memory and latency for very long multimodal inputs.

DocsRay addresses zero-shot understanding of complex multimodal documents through pseudo-TOC-guided hierarchical retrieval-augmented generation [2507.23217]. Its documents average 49.4 pages and 20,971 textual tokens, and its two-stage retrieval reduces complexity from $O(N)$ to $O(S + k_1 \cdot N_s)$. On MMLongBench-Doc, DocsRay-Pro reaches 64.7% accuracy, and pseudo-TOC-guided retrieval reduces query latency from 3.89 s to 2.12 s, a roughly 45% improvement. Here the scaling target is retrieval over a single long, multimodal document rather than predicate execution over a large corpus, but both systems use hierarchical pruning to preserve cost-efficiency.

DocGenome provides the benchmark substrate for large-scale scientific document understanding [2406.11633]. It structures 500K arXiv scientific documents from 153 disciplines, with 6.8M pages, 74.5M component units, 68.5M relationship labels, 13 layout attributes, and 6 logical relationships. Its task suite includes document classification, visual grounding, layout detection, equation-to-LaTeX, table-to-LaTeX, open-ended single-page QA, and multi-page QA. For ScaleDoc-like systems, DocGenome supplies both scale and richly structured supervision for multimodal training and evaluation.

FlexDoc tackles the data bottleneck from another direction by generating multilingual semi-structured synthetic documents with stochastic schemas and parameterized sampling [2510.02133]. It reports absolute F1 improvement of up to 11% on KIE tasks when augmenting real datasets and a reduction in annotation effort of over 90% compared to traditional hard-template methods. This is relevant to ScaleDoc because large-scale semantic filtering and document reasoning systems benefit from training corpora whose diversity, noise, and label richness can be tuned rather than merely collected.

S2Doc provides a formal spatial-semantic document format that combines spatial, logical, semantic, and ontological layers in a single representation [2511.01113]. It defines a document as
$$
D = (P, E, G, K, \Lambda, \Phi, \mathcal{M}),
$$
with page-local coordinate spaces, typed elements, a logical relation graph, and an optional semantic knowledge layer. Its multi-page support and extensibility make it a natural interchange format for scaled document pipelines.

DocPrompt, finally, represents a scaling strategy for document QA rather than corpus filtering. It uses large-scale continued pretraining, with billions of weakly supervised document QA examples in Stage 2 and millions of open-source samples in Stage 3, together with ERNIE-Layout, multi-head span decoding, and an understanding–generation ensemble [2308.10959]. The system reports state-of-the-art performance on four document QA tasks and first-place leaderboard results on WebSRC and English DocVQA. A plausible implication is that ScaleDoc-like deployment pipelines need not be restricted to yes/no filtering; the same scaling pressures apply to extraction-style QA, long-context multimodal reasoning, and layout-aware model training.

## 6. Limitations, misconceptions, and future directions

A common misconception is that ScaleDoc eliminates the need for the oracle. The system does not do so. Its fundamental mechanism is selective oracle invocation: documents in the ambiguous region $U$ are always forwarded for final judgment, and the reported efficiency derives from shrinking $U$ while preserving $\mathrm{Acc} \ge \alpha$ [2509.12610]. Another misconception is that document-level embeddings alone are sufficient. The follow-up literature argues that dense embedding cosine similarity captures topical relatedness but may discard token-level evidence needed for negation, named entities, quantities, and local reasoning [2606.08090].

ScaleDoc’s limitations are therefore tied to representation, sampling, and calibration. If Phase 1 is omitted, the proxy fails to produce semantically monotone scores; if Phase 2 is omitted, score distributions overlap and filtering power deteriorates. Very coarse binning or too-small calibration samples can miscalibrate thresholds, and in about 20% of calibration trials naive selection misses $\alpha = 0.90$ [2509.12610]. The system improves compliance through reconstruction, but the paper does not claim perfect guarantees under extreme sampling constraints.

The broader “ScaleDoc” landscape exposes additional boundary conditions. Long multimodal documents stress memory and latency rather than only oracle cost, which motivates compression-and-streaming architectures such as DocSLM [2511.11313]. Structurally inconsistent documents stress retrieval more than classification, which motivates pseudo-TOC-guided hierarchical RAG as in DocsRay [2507.23217]. Training bottlenecks in enterprise settings stress privacy, annotation cost, and multilingual diversity, which motivates synthetic generation frameworks such as FlexDoc [2510.02133]. Representation heterogeneity across pipelines stresses interoperability, which motivates S2Doc’s spatial-semantic standardization [2511.01113]. These adjacent pressures indicate that large-scale document intelligence is not a single optimization problem but a set of coupled constraints spanning compute, memory, representation fidelity, retrieval precision, annotation economics, and schema compatibility.

Future work in the semantic-filtering line is already well defined in the literature. The adaptive two-phase framework proposes compositional routing across cascade families, soft-label proxy training using oracle confidence, and calibration that concentrates safety margins in sparse score regions [2606.08090]. Other stated directions include adaptive per-query labeling budgets, using BER as an online router, and extending the same skeleton to semantic joins, group-bys, and top-$k$ operators. Within document understanding more broadly, plausible next steps include integrating token-aware semantic filtering with multimodal page compression, richer long-document benchmarks, spatial-semantic interchange formats, and synthetic multilingual pretraining corpora.

Source: https://www.emergentmind.com/topics/scaledoc