To GPU or Not to GPU: Vector Search in Relational Engines

Abstract: Vector search (VS) is now available in most database engines. However, while vector search is a common feature in AI/ML/LLMs where the dominant computing platforms are GPUs, existing database engines operate on CPUs even when implementing vector search. This raises the question of whether integrating vector processing on GPUs as part of the engine would be a better design. In this paper, we explore this question in detail. First, we extend the TPC-H benchmark with vector data (from text and images) and propose a number of representative SQL+VS queries. Second, we develop a modular execution engine that can run SQL+VS queries across CPU and GPU. Third, we perform extensive experiments on a number of deployments: running the SQL+VS queries across CPU and/or GPU, with data residing in CPU or GPU memory, with existing indices and novel, optimized versions, as well as across different GPUs and interconnects (PCIe, NVLink). The results provide actionable and counter-intuitive insights on how to run such queries over CPUs and GPUs. For instance, the relational components benefit much more from running on the GPU than the vector search part. In addition, when the vector search involves moving data and indexes, using the GPU is not the best option, even with fast interconnects. Thus, we develop an alternative organization of vector index and embeddings that reduces the size of the index, making GPU-based vector search more competitive. With these improvements, the final result is that both the relational and vector search components are faster on the GPU, particularly on fast interconnects, in contrast with the architecture used in existing engines.

• The paper demonstrates that GPU acceleration yields significant speedups for relational operators over vector search components.
• The authors introduce the Vec-H Benchmark and MaxVec Execution Engine to evaluate heterogeneous operator placement and index architectures.
• The study reveals that non-data-owning indices drastically reduce data movement overhead, emphasizing the need for tailored system optimizations.

Vector Search Acceleration in Relational Engines: Assessing Heterogeneous Execution

Problem Statement and Motivation

The integration of vector search (VS) into general-purpose relational database systems has emerged as a direct response to the adoption of semantic representations in AI/ML/LLMs. While ML-centric VS engines heavily leverage GPUs for query processing, mainstream relational engines largely execute all query components—including VS—on the CPU. This implicit hardware bifurcation raises a fundamental systems question: does extending GPU acceleration to support both relational and vector operators within a unified query plan significantly improve end-to-end analytical workloads? The work systematically addresses not only the potential performance impact but also the design space for index organization and data movement in heterogeneous computing environments (2605.15957).

Methodology: Benchmark and System Contributions

To provide empirical rigor, the authors introduce two primary artifacts:

• Vec-H Benchmark: An extension of TPC-H, Vec-H incorporates realistically scaled vector data (semantic embeddings for text and images) into two new tables, and defines eight SQL+VS queries covering canonical patterns—inner, lateral, anti/semi, etc.—that capture the full expressiveness of analytic workflows involving both relational and VS operations.
• MaxVec Execution Engine: Building on the modular Maximus engine, MaxVec supports “per-operator” heterogeneous placement for both relational and VS operators with native, separable support for CPU and GPU, as well as nuanced index- and data-movement orchestration. Critically, MaxVec enables the comparison of all permutations: CPU-only, GPU-only, and hybrid plans with varying index organizations (data-owning vs non-owning).

Experimental Analysis: Findings and Numerical Results

Heterogeneous Execution—Operator Placement

A primary empirical finding is counterintuitive: GPU acceleration yields greater performance benefits for relational operators than for vector search operators themselves. For ANN searches with state-of-the-art (SOTA) graph indices (e.g., CAGRA or IVF), the majority of the CPU-to-GPU speedup (median 77–87%) is attributable to relational operator offloading, with the vector search component benefiting relatively less. This represents a critical departure from prevailing assumptions that VS would be the primary GPU beneficiary.

• MaxVec on GPU achieves 2x–69x speedup over CPU-only plans, with the highest gains in queries where relational joins/aggregations dominate the critical path.
• In exhaustive (ENN) VS queries, GPU speedup is also substantial (within ~1.2–1.6x of fastest graph-based ANN kernels), but vector search cost can still dominate unless index organization is carefully optimized.

Data Movement and Index Organization

A second core result is that index and embedding data movement, not computation, is the main limitation for GPU-accelerated VS, even with high-bandwidth interconnects (e.g., NVLink-C2C).

• For “data-owning” indices (where the VS index contains full embedding payloads), transferring indexes can account for up to 95% of query wall time, especially for ANN indices with high partition counts.
• NVLink interconnects mitigate but do not eliminate the penalty: e.g., moving a ~10 GB IVF index can take >1s, far below hardware bandwidth limits due to small-chunk memcpy inefficiencies and index-structure transformation overheads.

The authors demonstrate that decoupling embeddings from the index (non-data-owning design), enabled by hardware support for host-resident GPU memory access (ATS on NVLink-C2C), reduces index movement by over two orders of magnitude. Transferring only the compact index metadata (centroids or graph) lowers transfer time to single-digit milliseconds per query.

Execution Strategies

The evaluation encompasses six strategies, summarized below:

Strategy	Rel. Op Placement	VS Placement	Index Res.	Emb. Res.	Data Movement	Notable Impact
cpu	CPU	CPU	Host	Host	None	Baseline, universally supported
gpu	GPU	GPU	Device	Device	None (preloaded)	Fastest but GPU-memory limited
copy-di	GPU	GPU	Host → Device	Host → Device	Full index+data	Often dominated by transfer cost
copy-i	GPU	GPU	Host → Device	Host	Index only, on-demand embeddings	Trade-off at batch size, fast queries only
gpu-i	GPU	GPU	Device	Host	On-demand embeddings	Efficient if index fits on GPU
hybrid	GPU	CPU	Host	Host	Relational data only	Effective compromise

Strong numerical findings:

• MaxVec’s non-data-owning index with host-resident memory: using the gpu-i or copy-i strategy achieves 1–21x improvement over CPU baseline.
• For large batch (VS join) queries, the index transfer overhead is quickly amortized—on IVF1024, gpu-i outperforms cpu for batch sizes as low as 10–100.
• For all query types and index organizations, analytical SQL+VS query latencies improve by 2x–69x when relational and VS operators are both executed on GPU with non-data-owning indices (provided GPU memory budget is sufficient for index metadata and working set).

Hybrid Execution and Architecture Implications

Hybrid execution (CPU VS, GPU relationals) provides a pragmatic path when GPU memory is insufficient and substantial index transfer latency (for VS on device) is present. In particular:

• Hybrid is within 1.3–5.7x of the best GPU-only configuration, but avoids the need to keep the full embedding set resident on the device.
• For SOTA ANN indices, hybrid execution allows simultaneous relational acceleration while VS remains on CPU where index structures are too large or amortization is insufficient.

On single-tier unified memory systems (e.g., DGX-Spark), where CPU and GPU share memory with no inter-device bottleneck, the GPU always outperforms the CPU—even with LPDDR5x rates (~273 GB/s) that are 15x slower than HBM3—yielding speedups of 2.5–15x for ENN and 1.5–9x for ANN per query.

Design Principles and Best Practices

The results of this investigation directly inform system and query optimization decisions:

• Relational operators benefit more from GPU acceleration than VS; allocation of GPU resources should favor fully offloading relationals where possible.
• Non-data-owning index structures with host-side embedding storage and GPU-accessible unified memory vastly reduce index transfer overhead; this organization is more future-proof given continuing advances in hardware support for coherent host/device memory.
• Query planners/optimizers must estimate index and data transfer costs, batch sizes, GPU memory availability, and recall requirements to select among cpu, hybrid, gpu-i, and copy-i strategies dynamically.

Implications and Future Directions

From a practical engineering perspective, enabling efficient, hardware-adaptive, analytical SQL+VS queries in relational engines necessitates (a) native support for per-operator hardware placement, (b) flexible index design decoupled from embedding storage, and (c) system-level optimizations for data and index movement. These requirements surpass what current DBMS+VS platforms and vector DBMSes typically offer.

Theoretically, the findings challenge the assumption that VS is inherently GPU-bound in analytic contexts. Instead, for real-world SQL+VS workloads with moderate query batching, hardware acceleration should primarily target the relational backbone, with VS operator acceleration realized only when data movement is minimized by system design and hardware features align.

As hardware architectures increasingly move towards larger memory pools, tighter coupling (e.g., single-chip CPU/GPU), and higher interconnect bandwidths, the practical overheads associated with data and index movement will further diminish, making GPU-accelerated execution of both relationals and VS operators feasible even for large-scale analytical workloads. These results suggest that DBMS vendors and engine designers should prioritize non-data-owning index designs, co-evolution with unified memory architectures, and the development of sophisticated, cost-based query optimizers aware of heterogeneous resources.

Conclusion

This paper delivers an exhaustive empirical analysis of GPU adoption for analytical SQL+VS in modern relational engines, establishing that most overall performance gain comes from accelerating relational operators, not vector search itself. Further, it demonstrates that index organization and data-movement architecture are critical: non-data-owning (host-resident) indices combined with advanced hardware features (e.g., NVLink, unified memory) are pivotal for removing bandwidth bottlenecks. In practice, the optimal execution strategy is highly context-dependent, requiring careful trade-off analysis by query optimizers and system architects. As memory and interconnect hierarchies evolve, leveraging the full potential of GPUs in this domain will depend on the widespread adoption of these architectural and system-level principles.