Coverage Is Not Redundancy: Maintenance Cost and Exposure of Query-Aware Admission Indexes in Vector Databases Under Workload Drift

Abstract: In a vector database serving production-scale retrieval, a single inserted document can be retrieved for an anomalously large share of the query workload -- a retrieval hub -- and dominate the evidence returned for an entire topic. An emerging defense guards against this at ingest with an admission check: it maintains a set of sentinel queries and admits a document only if its reverse-kNN count against them stays below a threshold tau. Under workload drift this sentinel set is a query-aware auxiliary index that must be maintained online, and we study the cost that maintenance imposes on the ingest path. We identify a structural limit -- coverage is not redundancy: a monitor stops promoting sentinels once a region is covered, but the predicate rejects a hub only once tau sentinels witness it, so exposure has an observation-limited floor that no reduction in update or enforcement latency can close. On real HNSW, IVF-Flat, and IVF-PQ indexes over an 8.8M-vector MS MARCO corpus this floor is only a best case: as index recall falls, exposure and churn rise above it, and below recall ~0.5 the gate stops containing altogether -- worst on the memory-compressed IVF-PQ used at billion scale -- while a recall-aware witness probe restores containment at a fixed O(|S|d) admission cost, under 0.1% of the ANN insert. We validate the law under real (COVID-19) workload drift, implement the gate in PostgreSQL/pgvector at a 0.33% ingest tax, and turn the bound into a provisioning rule that sizes the sentinel budget per emerging region. A count test contains the hub where retrieval-time score normalizers (NNN, QB-Norm) do not, and a pre-registered causal suite isolates the missing-coverage mechanism from retrieval fragmentation across two embedding families (BGE-1024, E5-768).

• The paper proves an observation-limited exposure floor, E ≥ p_cover(m_safe + ⌈(τ−c₀)_+/r⌉), showing that faster updates cannot compensate for missing query witnesses during workload drift.
• Experiments find that coverage stopping fails to satisfy threshold-based containment: exposure scales linearly with the witness deficit, while approximate indexes below about 0.5 recall can stop detecting hubs entirely.
• The paper shows the admission check adds under 0.33% to insert CPU time, but recommends high-recall witness probes because they restore containment and avoid the churn and blind spots caused by serving-index approximation.

This paper analyzes the ingest-path cost of maintaining a query-aware admission index—a sentinel-query reverse-$k$NN gate that rejects retrieval hubs at insert time—in a vector database whose query workload drifts. Its central result is a structural lower bound: exposure of an inserted hub is observation-limited, independent of update and enforcement latency, and degrades sharply on approximate ANN indexes. The paper validates the bound with controlled experiments, real drift replay, production-scale indexes, and a pgvector prototype.

System model and the coverage–redundancy mismatch

The setting is a vector store serving top-$k$ retrieval (HNSW or IVF) over a corpus, with an admission predicate evaluated before commit. A sentinel set $S$ is sampled from the live workload; a document $d$ is rejected when its reverse-$k$NN count $\kappa_S(d)$ reaches threshold $\tau$, calibrated to a benign false-positive rate. Under workload drift, $S$ becomes an incrementally maintained auxiliary index: a monitor detects emerging query regions after $m_{\mathrm{safe}}$ qualifying detections and promotes sentinels at observed query locations.

The paper's titular claim is a disagreement between two maintenance policies over "enough." The monitor is novelty-driven—it stops promoting once a region is covered by one sentinel—while the predicate requires $\tau$ witnesses to reject. When an off-axis region emerges, an inserted hub has initial support $k$0 against pre-existing sentinels, so the monitor marks the region covered long before the predicate can reject. The gap, the missing coverage mass $k$1, is both the residual exposure and the maintenance churn required to close it.

The witness-deficit lower bound

Under three assumptions—containment requires $k$2; sentinels may be promoted only at observed query locations; each promotion raises $k$3 by at most $k$4—the paper proves an observation-limited floor:

$k$5

which is tight ($k$6) in the emergence regime via a covering lemma: if the cover radius does not exceed the retrieval radius, every promotion that raises $k$7 is seeded by a query that also retrieves $k$8. Two consequences follow directly. First, latency-independence: even instantaneous detection, update, and enforcement cannot close the floor, because witnesses cannot be synthesized faster than in-region queries arrive. Second, cost–exposure coupling: churn-to-close equals exposure-to-close while the region is sparse, since covering a hub and retrieving it are the same $k$9NN event. This yields budget-parametric infeasibility—for any churn budget below the floor, no novelty-only policy meets any exposure target. The bound rests on one geometric hypothesis ($S$0), which the paper verifies empirically rather than assuming.

Experimental validation of the mechanism

The evaluation uses a deterministic control structure on BEIR-derived corpora encoded with BGE-large (1024-d) and E5-base (768-d), operating at $S$1 / $S$2 (FPR $S$3), $S$4. Key findings:

• Deficit, not fragmentation, causes exposure. In the sparse regime, measured $S$5 across all regional dispersions ($S$6–$S$7); injecting fragmentation as a free parameter leaves the deficit slope unchanged at 1.0.
• The scaling law is exact. Sweeping $S$8 gives exposure linear in the deficit with unit slope and $S$9; varying witnesses-per-observation $d$0 moves median exposure inversely and exactly ($d$1).
• Coverage-stopping fails categorically. The three-model contrast gives exposure 0 (oracle) versus 248 (novelty-stopping, unbounded growth) on synthetic data, and 0 / 17 / 1000 on real BGE geometry, where continued promotion contains at exactly $d$2.
• Cross-domain prediction. Across four emergent BEIR domains, an independently measured $d$3 predicts the observed floor to within 0.5 in every case (e.g., TREC-COVID: predicted 17.0, observed $d$4)—a genuine out-of-sample test with no parameter fit to exposure.

A dense-region control locates a sharp regime boundary: as legitimate competitor documents accumulate, $d$5 collapses from 1.0 to 0.09, churn decouples from exposure, and retrieval competition—not the gate—suppresses the hub. The paper is explicit that the clean law holds only in the emergence window ($d$6 density), which is precisely where the worst-case insertion operates.

Placement contrast: count versus normalization

Against fairly configured, FPR-matched online NNN and QB-Norm normalizers, the count test contains at exposure 17 at both ingest and serve placements, while neither normalizer contains at either (exposure 42–43). The distinguishing axis is mechanism, not placement: a query-bank normalizer down-weights the hub and its in-region neighbours together, preserving the hub's rank regardless of bias strength. This reframes retrieval-time hubness normalization as the wrong tool for adversarial hub containment, and identifies serve-time count gating—an unexplored design point—as a placement choice differing only in re-evaluation overhead $d$7.

The floor on real approximate indexes

On an 8.8M-vector MS MARCO store (BGE embeddings, emergent TREC-COVID region, analytical floor 19), exact-retrieval churn already exceeds the floor (24.5) due to natural background competitors ($d$8). Approximation inflates it monotonically:

Index	Recall	True exposure	Churn	Contains
Exact	1.000	17.5	24.5	yes
HNSW efS=64	0.838	21.0	31.5	yes
HNSW efS=8	0.474	35.0	50.0	25% of orders
IVF-Flat nprobe=1	0.258	35.0	50.0	never
IVF-PQ nprobe=64	0.308	23.5	34.5	yes

Two results stand out. Below recall ≈ 0.5 the monitor stops containing entirely—at IVF-Flat nprobe = 1 the index returns the hub for only 2 in-region queries while exact retrieval returns 35, converting a contained threat into an invisible one. IVF-PQ, recall-capped by quantization at ≈ 0.31, pays a permanent ~1.5× churn penalty yet still contains, because its recall loss is spread across queries rather than concentrated. The fix is cheap: counting witnesses on a high-recall probe decoupled from the serving index (exact scoring of only the $d$9 sentinels) restores containment to 1.0 and pins true exposure to the exact floor at every operating point. The blind spot recurs across four embedding families—on GTE it is more severe (IVF-Flat fails up to recall 0.78; IVF-PQ never contains).

Real drift, prototype, and provisioning

Replaying the Wikipedia clickstream across the COVID-19 emergence (Nov–Dec 2019 → Mar–Apr 2020), five organically emerging regions obey $k$0 exactly (slope 1.0, $k$1). Steady-state recurrence is confirmed over 24 sub-regions, all opening windows (median deficit 11), with cumulative churn growing linearly in regions spawned (~15 promotions/region).

The gate's CPU cost is negligible: the $k$2 check is $k$3 and corpus-independent—0.04 ms against a 46 ms faiss insert (0.08%) and 0.038 ms against an 11.6 ms pgvector insert (0.33%), shrinking relatively as the corpus grows. Read as a design equation, the bound provisions per-region promotion rates that meet exposure targets on all five real COVID regions, at up to 24% less churn than worst-case fixed-rate sizing.

Limitations and open questions

The paper concedes several scoping constraints plainly. Calibration uses reconstructed budgets: $k$4 is not locked to a separate stationary study, and $k$5, $k$6 are placeholders, so whether the floor crosses a deployment budget remains deliberately unresolved. The clean law and coupling are emergence-window results; past the measured density boundary, fragmentation co-governs and the gate is not the controlling factor. Real-embedding phase results are BGE-only, since E5 places these domains illumination-safe ($k$7). Closed-loop adversaries—parking in under-covered zones, shifting regions after each promotion, weaponizing a monitor's churn budget—are left open, though the present bound assumes continued promotion and is therefore generous to the defender.

Conclusion

The paper establishes that maintaining a sentinel-based admission gate sound under workload drift carries an observation-limited exposure floor, $k$8, that no reduction in update or enforcement latency can close, and that this floor is a best case on approximate indexes—degrading monotonically with recall and collapsing into silent non-containment below recall ≈ 0.5 until witnesses are counted on a higher-recall probe. The check itself costs under 0.33% of ingest; the cost that scales is exposure, paid per emerging region, and the same bound that predicts it also provisions against it. The general lesson—that locally sufficient coverage-driven maintenance composes into globally unaccounted cost when feeding a threshold predicate—applies beyond vector databases wherever such layers coexist.