When More Cores Hurts: The Vector Database Scaling Paradox in HPC

Abstract: Vector databases have been designed and optimized for cloud environments; however, emerging scientific AI workloads (e.g., molecular search, meteorological trajectory detection, and literature-driven hypothesis generation) demand efficient, scalable execution on HPC systems. We present a large-scale evaluation of three state-of-the-art vector databases -- Qdrant, Milvus, and Weaviate -- on two production supercomputers, scaling to 256 distributed workers across 64 compute nodes. We evaluate representative workload patterns -- mixed read/write and write-then-read -- using popular benchmarks, multimodal embeddings, and a novel real-world scientific dataset. Our results reveal that workload characteristics can limit latency reduction, additional cores can reduce query throughput by up to 30.67%, and scaling from 16 to 256 workers (16x) only yields a 5.46x improvement. This scaling paradox exposes the fundamental mismatch between cloud-oriented designs and HPC systems, highlighting the need for new, HPC-aware vector database designs.

• The paper demonstrates that adding more cores on HPC can paradoxically reduce query throughput due to fundamental architectural mismatches.
• It introduces VECHINI, an HPC-native benchmarking suite that evaluates vector database performance using large-scale, real-world embedding datasets.
• The findings highlight the need for HPC-aware design changes such as decoupled write paths and adaptive indexing to overcome scaling limitations.

Vector Database Scaling Paradox in HPC: An Expert Synthesis

Motivation and Problem Framing

The significant acceleration of unstructured data generation in scientific applications—spanning molecular search, meteorological trajectory detection, and hypothesis generation—demands highly scalable semantic search. Vector databases (VDBs) have achieved strong traction in cloud environments through cloud-oriented optimizations, enabling efficient ANN queries over embeddings for foundation model and recommendation system workloads. However, the architectural divergence between clouds and HPC platforms—multi-tier storage hierarchies, many-core CPUs, high-bandwidth interconnects—introduce latent inefficiencies when deploying VDBs at supercomputing scale. This work provides the first comprehensive, empirical analysis of three leading open distributed VDB architectures (Milvus, Qdrant, Weaviate) on two state-of-the-art production supercomputers, exploring their performance, scalability, and workload lifecycle characteristics up to 256 distributed workers and highlighting fundamental architectural mismatches and the emergence of scaling paradoxes.

Experimental Framework, Datasets, and Platforms

A critical contribution is the introduction of VECHINI, a schema-driven, HPC-native benchmarking suite for VDBs, enabling systematic, reproducible evaluation down to configuration layer details. The workload is grounded in real-world, large-scale embeddings (e.g., the 843.56 GB, 88M-chunk Pes2o-VE corpus, and 1B-row Yandex T2I) and canonical benchmarks. Experimental design sweeps intra/inter-node parallelism, storage media (RAM, SSD, DAOS, Lustre), ingestion/load/query batch size, and both read-dominated and hybrid read/write access patterns.

Aurora (Intels PVC, 104-core nodes) and Polaris (NVIDIA A100, 32-core nodes) are leveraged, mapping to cloud VDB deployments for contrast. All VDBs are containerized with Apptainer to meet supercomputing security constraints, and critical system-level bottlenecks are instrumented.

From Cloud to HPC: Nonuniform and Limited Acceleration

Deployments on Aurora exhibit heterogeneous performance gains (1.33×–5.2× QPS, up to 13.53× latency reduction) compared to cloud, but low-latency cloud queries remain largely unimproved. The greatest absolute reduction is observed in index construction time (up to 6.83×) rather than query serving, confirming index construction as the maximally bandwidth- and compute-bound workload stage where high core count and memory bandwidth are most effective. Surprisingly, workloads with high intrinsic dimensionality (ID) in embedding space, such as GIST, benefit disproportionately—consistent with large-k deep graph traversals being more sensitive to the enhanced node and interconnect characteristics of HPC platforms.

Vector Database Lifecycle Analysis

Figure 1: Life cycle of a vector database, illustrating ingestion, indexing, and querying phases, and their critical bottlenecks.

Ingestion: Architectural Bottlenecks and Storage Backends

Insertion rates are strongly attenuated by persistent storage latency. For all VDBs, in-memory storage yields peak throughput, but transition to high-latency parallel filesystems (Lustre, DAOS) erodes performance. Notably, DAOS achieves rates nearly commensurate with DRAM (8758 vs 6432 vectors/s for Milvus). Qdrant is especially penalized on Lustre due to its pipeline, which serializes insert completion on WAL and queue draining. Write amplification in Milvus is particularly severe, with a 2.8× increase in physical storage usage for large datasets, highlighting acute scaling limitations in bulk ingest scenarios.

Figure 2: Impact of different storage mediums on insertion rate for Pes2o-VE on Polaris and Aurora.

Insertion throughput saturates rapidly (32–64 concurrent clients per worker), bottlenecked by WAL or shared resource contention. Increasing partitioning (shard count) can realize higher rates; with 32 clients and WALs, Qdrant and Milvus reach 223,843 and 138,860 vectors/s, respectively.

Figure 3: Single-worker, multi-client insertion throughput on Aurora, exposing the WAL bottleneck at high concurrency.

Scaling to multiple workers yields sublinear performance; the best system (Qdrant) achieves 489,213 vectors/s at 32 workers, well below potential interconnect bandwidth, dramatically confirming that per-insert consistency guarantees are mismatched for bulk ingestion on HPC.

Figure 4: Insertion scaling across distributed workers on Aurora; throughput improvements are sublinear due to architectural constraints and resource underutilization.

Indexing: Many-core/Multi-GPU and the Diminishing Returns Regime

Indexing cost is primarily a function of dataset size and embedding dimensionality. High-dim datasets accentuate the cost of HNSW construction, but GPU acceleration shows substantial multiplicative benefit (10.04× speedup for Milvus on 10M Pes2o-VE), in line with compute/memory-bandwidth-limited index build.

CPU-based scaling (by virtual core count) quickly hits diminishing returns: the majority of the benefit accrues by 32–64 vCPUs, after which additional parallelism yields negligible or even negative improvements due to memory stalls and architectural inefficiencies. Weaviate is especially limited due to single-shard, sequential index construction, while Qdrant dynamically modulates segment count and threads, revealing nontrivial effects on scaling.

Figure 5: Indexing time versus core count for 10M Pes2o-VE and Yandex-T2I on Aurora; the white line indicates initial index build completion for Milvus.

Query: Implicit Cost-Recall Trade-offs and Scaling Paradox

The querying phase exposes further architectural mismatches. Milvus and Qdrant partition shards into multiple segments, constructing independent HNSW indices which are searched in parallel, resulting in higher recall (Milvus achieves 0.982 on Pes2o-VE-10M vs Weaviate’s 0.918), but at an increased per-query computational cost. The number of segments is now an implicit, under-researched axis of the recall-cost trade-off, supplementing traditional HNSW parameters.

Scaling query throughput with core count plateaus (or degrades) beyond 32–64 vCPUs across VDBs, often due to increased memory stall cycles and contention. Qdrant’s internal resource scheduler, segmentation, and thread assignment complicate monotonic scaling expectations. Weaviate is strictly limited by its single-shard graph design.

Figure 6: Query throughput as a function of core count for 10M-vector Yandex-T2I and Pes2o-VE; throughput often plateaus or decreases with excessive parallelism due to contention and architectural limitations.

Distributed, multi-node querying (up to 64 nodes) shows that broadcast-gather patterns cause a single aggregating worker to bottleneck performance. Qdrant scaling is sublinear—16× node scaling produces only 5.46× throughput—and performance in Weaviate often degrades at higher node counts due to merging and coordination overhead. The standard distributed query execution paradigm is ill-suited for exascale environments and non-homogeneous work partitioning, motivating research into new distributed query strategies.

Figure 7: Distributed query performance for the full Pes2o-VE dataset on Aurora; throughput increases sublinearly with worker count, and head-of-line blocking and imbalanced workload distribution emerge as critical challenges.

Simultaneous read-write workloads (mixed workloads) result in sharp increases in P99 latency (34.9%–344.77% increases depending on system/dataset), especially when queries contend with insertions for segment/index state. This has direct implications for SLA-constrained, latency-sensitive workflows.

Contradictory Claims and Quantitative Highlights

• Monotonic resource scaling fails: adding workers or cores can reduce query throughput by as much as 30.67%. This directly contradicts the primary design premise of distributed cloud-native architectures.
• Sublinear parallel speedup: 16× increase in workers yields only 5.46× throughput increase—an Amdahlian ceiling arising mainly from architectural bottlenecks in network coordination, merging, and synchronization.
• Implicit, system-driven recall/cost trade-off through segmentation: Increasing segment count and distributed partitions increases recall (e.g., for Pes2o-VE, 1 to 8 segments increased recall from 0.962 to 0.9854) but at a 5.4× increase in search time.
• Write amplification is highly nontrivial: Milvus incurs 2.8× raw data storage costs, which is unsustainable at scale.

Practical and Theoretical Implications

For domain scientists, these empirical results indicate that naive deployments of current VDBs on HPC will severely underutilize system resources except under very narrow workload conditions (high-dimensional indexing, intensive index build phases). Optimal exploitation of hardware requires careful co-tuning of ingestion parallelism, batch size, indexing and partition strategy, and storage backend selection.

From a systems perspective, architectural principles behind current VDBs (per-insert strong consistency, strict worker-centric segmentation, single-worker aggregation, and broadcast-gather query plans) require foundational revision to be compatible with the characteristics of next-generation supercomputing workloads and resource profiles. There is a strong need for relaxed-consistency ingestion APIs, write-path restructuring, and hierarchical merging/query execution strategies. Segment-based cost-recall trade-offs must become first-class, explicitly configurable controls.

Directions for Future AI System Development

Next-generation VDB platforms for scientific, multimodal, and agentic workloads must be re-architected for HPC scale. Necessary directions include:

• Write-path decoupling: Bulk import APIs with deferred/differentiated consistency for efficient high-throughput initialization.
• Query plan decentralization: Multi-stage aggregation, hierarchical merging, and P2P reduction trees to avoid single-aggregator bottlenecks.
• Resource-adaptive segmentation: Dynamic segment creation, auto-tuning for cost-recall trade-off explicitness, and NUMA-aware partitioning.
• Profiling-driven threading models: Adaptive parallelism, fused CPU-GPU indexing, and concurrency controls tailored to both the hardware and data modality.
• System-aware index structures: Architectures and index layouts that match the access and memory hierarchy of exascale supercomputers.

Conclusion

This work concretely demonstrates that contemporary vector databases, while performant on cloud hardware, exhibit substantial architectural and scaling limitations on state-of-the-art supercomputers. Adding cores or workers can reduce query throughput, and ingestion is bottlenecked by unnecessary consistency and write amplification. Sublinear and sometimes negative scaling is a direct result of misaligned design assumptions, necessitating a new generation of HPC-aware VDB systems. For practitioners, system- and workload-level tuning is essential to leveraging available resources. For systems researchers, this study illuminates crucial design axes—execution models, index parallelism, storage interaction, and relaxed consistency—that must be reconsidered to achieve scalable, robust scientific AI infrastructure.

Reference: "When More Cores Hurts: The Vector Database Scaling Paradox in HPC" (2606.08950)