TEngineDB-V: An OLAP-Native Vector Search System for Large-$k$ Workloads at Tencent

Abstract: Vector search systems are essential infrastructure for modern data-driven applications. Large-$k$ analytical vector search, which retrieves $k=10^3$--$10^5$ results for analytics (e.g., aggregation, filtering, joins), is increasingly important for emerging workloads, including LLM data management and advertising analysis at Tencent. Existing systems remain inadequate: specialized vector databases often cap $k$ (e.g., $k \leq 10^4$) to satisfy tail-latency constraints and offer limited analytical support, while OLAP systems typically embed per-segment vector indexes as black boxes, causing severe read/compute amplification and preventing native query optimization. This paper presents TEngineDB-V, an OLAP-native vector search system for large-$k$ workloads. TEngineDB-V makes vector search a first-class analytical primitive in Tencent's OLAP engine through a global segment-decoupled index materialized as relational tables, eliminating scatter-gather execution, reducing amplification, and enabling native storage optimizations. It decomposes IVFPQ-based search into relational operators, integrates OLAP optimizations, and introduces DPPQ, which combines direction-aware quantization with hierarchical residual refinement to improve recall while preserving relational efficiency. TEngineDB-V further incorporates index-aware query rewriting and a distributed-aware cost model for efficient distributed execution. Experiments show that TEngineDB-V achieves up to a $145\times$ speedup over competitive systems such as StarRocks, and up to a $52\times$ improvement in 10-billion-scale production deployments.

• The paper introduces an OLAP-native vector search system that re-architects large-k workloads by decoupling vector indexing from data segmentation.
• It presents novel indexing and quantization techniques, including DPPQ, to boost recall and reduce latency, achieving significant speedups over conventional systems.
• Empirical results demonstrate robust scalability and practical performance gains for billion-scale datasets, validated through production deployments at Tencent.

TEngineDB-V: OLAP-Native Vector Search for Large-$k$ Workloads

Motivation and Problem Statement

TEngineDB-V addresses the system bottlenecks inherent in supporting large-$k$ ($k = 10^3$–$10^5$) vector search workloads within OLAP infrastructures, as observed in emerging use cases at Tencent (e.g., LLM data curation, advertising analytics, multimodal data analytics). Existing vector systems are largely optimized for latency-critical small-$k$ retrieval; they typically couple vector indexes per segment and treat these as black boxes within OLAP systems, leading to severe read/compute amplification and preventing native query optimization. The challenge is thus to re-architect vector search as a first-class analytical primitive, natively integrated into OLAP processing pipelines, while maintaining scalability and efficiency at billion-scale.

Architectural Innovations

TEngineDB-V implements a novel OLAP-native design rooted in two principles: segment-decoupled global indexing and table-centric index materialization.

A global, segment-decoupled IVFPQ-based index is built across the entire dataset, with centroids, codebooks, and PQ codes expressed as sharded, columnar relational tables. This eliminates the scatter–gather execution and $N \times k$ result amplification seen in segment-coupled systems, unlocking direct access to only relevant shards during query processing. All index structures participate in native storage optimizations—columnar compression, multi-tier block caching (memory/SSD/remote), and parallel I/O—allowing for high throughput even under high cardinality requests.

Figure 1: System architecture of TEngineDB-V highlighting control, computation, and storage layers.

Compared to traditional segment-coupled paradigms, the architectural separation between data and global index allows efficient cache management, fine-grained access, and robust support for both analytical and vector workloads.

Figure 2: Comparison between segment-coupled and segment-decoupled storage architectures.

TEngineDB-V exposes a declarative SQL interface for data, index, and hybrid analytical/vector workloads. Index refresh is handled asynchronously and atomically at the global level, ensuring OLAP consistency guarantees suitable for analytical queries.

Relational Execution Paradigm

TEngineDB-V decomposes IVFPQ vector search into relational operators, mapped over the global index tables. This enables:

• IVF Prune: SCAN–PROJECT–TopK–JOIN to select and access relevant clusters.
• LUT Compute: Scan the codebook, compute per-query LUTs, and aggregate to minimize memory footprint.
• Distance Estimate: Cross-join LUTs and candidate vectors, with fused operator implementations utilizing columnar FastScan and SIMD parallelization.

  Figure 3: Relational execution pipeline for decomposed IVFPQ search.

Such composability allows seamless fusion with analytical operations (filters, joins, and aggregates), benefits from vectorized execution and caching, and enables late materialization, all within the standard OLAP processing engine.

Enhanced Quantization: DPPQ

To address the recall limitations of standard PQ-based ANN at large $k$, TEngineDB-V introduces DPPQ (Direction-aware Progressive PQ):

• Direction-aware quantization: Prioritizes angular preservation via explicit decomposition of each subvector into a norm and a unit direction, encoding directions using PQ and reconstructing using cosine law.
• Hierarchical residual refinement: Multi-epoch refinement recursively quantizes residuals to iteratively reduce error at each layer.

This design improves large-$k$ recall without incurring high cost from raw vector refinement.

Figure 4: Recall-bit budget tradeoffs for PQ, RaBitQ, and DPPQ across datasets.

With equivalent bit budgets, DPPQ consistently outperforms both PQ and RaBitQ on recall at target $k$, with minimal incremental storage overhead.

Distributed and Cost-Based Query Optimization

TEngineDB-V extends the Cascades optimizer to be index- and distribution-aware. For hybrid queries (FANNS), the optimizer enumerates eight candidate plans determined by the axes of predicate pushdown (pre- vs post-filter), join order (build/probe side), and data exchange (broadcast/shuffle). A weighted cost model jointly estimates CPU, memory, and network costs. Adaptive tuning of candidate pool sizes ($K_{ann}$) and join/build placement ensures plan selection reflects workload selectivity and cluster resource constraints.

Figure 5: Alternative distributed execution plans for filtered ANN queries (FANNS).

Figure 6: Latency of eight execution plans and the cost-based plan for FANNS under varying selectivity.

This mechanism guarantees that no single execution plan is universally adopted; instead, the optimizer dynamically selects the most efficient plan per query and data characteristics, particularly critical as large-$k$0 queries shift network and memory bottlenecks.

Numerical Results

Experiments on datasets spanning 1M to 10B vectors demonstrate strong scalability and numerical advantage:

• Latency–recall trade-off: For $k$1, TEngineDB-V achieves 332 ms (Wikipedia, 0.86 recall), representing a $k$2–$k$3 speedup versus baselines (Milvus, StarRocks, PGVector, DiskANN).
• Scalability with $k$4: As $k$5 increases to $k$6, TEngineDB-V outpaces all competitors, unaffected by the severe I/O and merge amplification that cripple segment-coupled or graph-based approaches.
• FANNS: Under both low- and high-selectivity filtering, cost-based planning yields $k$7–$k$8 improved latency at given recall.
• Production-scale deployment: For Tencent-Image (10 billion entries), at $k$9, TEngineDB-V achieves a $k = 10^3$0–$k = 10^3$1 latency reduction for recall 0.88–0.8, with stable operation in clusters spanning 100TB and <5s query latency.

  Figure 7: QPS comparison of mainstream vector indexes as a function of $k = 10^3$2.

  

  Figure 8: Latency–recall trade-off for ANN workloads across representative vector search systems.

  

  Figure 9: Latency scaling as $k = 10^3$3 increases across candidate systems.

  

  Figure 10: FANNS latency–recall performance under different selectivities.

  

  Figure 11: Production performance results at 10-billion scale in Tencent deployments.

Theoretical and Practical Implications

TEngineDB-V's architecture demonstrates that decoupling vector index organization from physical data layout—combined with relational expression of vector operations—enables OLAP-grade scalability for vector analytics. The composability of vector search with analytical processing permits unified optimization, paving the way for hybrid, similarity-driven, and multi-modal workloads at internet-scale. The DPPQ quantization scheme provides a practical balance between relational executability and state-of-the-art recall at constrained bit budgets.

Practically, the work suggests that future large-scale multi-modal analytics engines should move away from legacy segment-coupled and black-box index paradigms, towards native integration with analytical execution and storage. This is further supported by the observed trends: increasing complexity in similarity search (joins, aggregations) and growing demand for unified analytical and vector pipelines in modern enterprise data settings.

Conclusion

TEngineDB-V establishes a new execution and storage paradigm for large-$k = 10^3$4 vector search, treating vector similarity as a first-class analytical computation. The system’s architectural decisions—global segment-decoupled indexing, table-centric materialization, OLAP-native execution, and distributed cost-based optimization—enable robust, scalable, and efficient support for billion-scale workloads. Strong empirical and production results substantiate the claims of up to $k = 10^3$5 speedup over competitive systems and sustainable operation at 10-billion vector scale. The design principles and methods introduced in TEngineDB-V are likely to inform future developments in OLAP- and cloud-native vector data management architectures, especially in use cases combining structured and unstructured analytics at scale.