Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives
Abstract: GPU collective communication is typically optimized for bandwidth, yet many emerging workloads are increasingly limited by latency. Long-context decode-heavy LLM inference is a prime example, where serving large models requires multiple GPUs, and many small collectives lie directly on the critical path of token generation. Therefore, even microsecond of overhead can impact performance and cost. In this work, we study how to approach the hardware Speed-of-Light (SoL) lower bound for GPU collectives within a scale-up network. We identify key principles for near-optimal designs, including barrier-free synchronization and efficient use of symmetric memory and multicast. Building on NCCL's device-side API, we develop low-latency interfaces for constructing custom collective kernels and use them to implement new symmetric collectives in NCCL. Microbenchmarks show substantial latency reductions for small and medium messages, reducing overhead to within 7% of the absolute SoL lower bound. When integrated into real applications, these kernels improve inter-token latency and throughput in LLM inference and accelerate cuSOLVERMp, demonstrating benefits for both AI inference and traditional HPC workloads.
Summary
- The paper introduces novel barrier-free synchronization protocols that achieve near speed-of-light latency in intra-node GPU collectives.
- It presents a two-shot LL128 atomic AllReduce method and a buffer-centric API that minimize overhead and enhance performance.
- Empirical evaluations show up to 13% ITL reductions in LLM inference and consistent HPC speedups, indicating tangible cost savings.
Near Speed-of-Light Latency in GPU Collectives: Principles, APIs, and Impact
Motivation and Problem Statement
The paper "Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives" (2607.16100) addresses the increasing significance of latency in GPU collective communication, especially within scale-up domains such as NVLink/NVSwitch interconnects. Traditional approaches prioritize bandwidth, yet emerging workloads—most notably long-context, small-batch tensor-parallel LLM inference—are pushing collective latencies to the critical path of token generation. For such workloads, frequent invocation of small-message collectives (AllReduce, ReduceScatter, etc.) directly constrains both performance and inference cost, with even a few microseconds of overhead compounding substantially at trillion-token production scales.
Figure 1: NCCL low-latency kernel reduces small-message AllReduce latency, approaching the SoL bound; this yields lower inter-token latency (ITL) and cost savings for Llama-3.1-70B inference.
Architectural and Algorithmic Analysis
Collective Communication Models
The study reviews GPU communication libraries (NCCL, NVSHMEM, RCCL) and emphasizes device-initiated communication and symmetric memory abstractions enabling efficient intra-node collectives.
Figure 2: Device-initiated communication and symmetric memory in NCCL and NVLink domains.
With device APIs exposing direct memory operations (load/store and atomics across peers), designers can replace host-driven orchestration and barriers with fine-grained, GPU-local protocols. The focus is on optimizing collectives such as AllReduce, which scales poorly in ring-based or barrier-laden forms for latency-sensitive contexts.
Barrier-Free Synchronization Techniques
Global memory barriers are identified as the dominant source of latency overhead—benchmarks show multiple microseconds per barrier, incurring up to 40% of the total latency in short AllReduce kernels.
Figure 3: Barrier latency increases with GPU count, regardless of signaling method.
The paper introduces several techniques to eliminate global memory barriers:
- LL (Low-Latency) Protocol: Combines data and flag in a single transaction via atomic stores, allowing polling for data readiness.
- Sentinel Synchronization: Initializes buffers with sentinel values; arrival detected by value change, avoiding additional signaling.
- Bidirectional Communication with Double Buffering: Partitions buffer space into independent regions, ensuring writes do not overwrite unread data by exchanging chunks in alternately active buffers.
Figure 4: Double buffering allows bidirectional chunk exchange without memory barriers.
Novel Two-Shot LL128 Atomic AllReduce
A new two-shot variant extends the LL128 protocol: atomic additions synchronized at the L2 cache enable reductions distributed across cache-lines, allowing scalable and deterministic collective operations (for addition). This approach sharply reduces scratch buffer requirements and wastes little bandwidth relative to LL-based designs.
Figure 5: Two-shot LL128 atomic AllReduce algorithm—distributed atomic addition over cache-line units.
API Abstraction
To generalize and encapsulate these techniques, the authors develop a buffer-centric API layered atop NCCL’s device interface. The API (ncclLLBuffer and related primitives) allows fine-grained thread-level control, multi-buffered execution, and choice of synchronization mode (LL or sentinel).
Figure 6: Overview of the proposed low-latency API design, with both device and host functions.
APIs enable succinct, efficient custom collective kernels that minimize synchronization and maximize device-side communication parallelism.
Figure 7: Construction and layout of ncclLLBuffer with alternating epochs and sub-buffers.
Figure 8: send() and recv() operations of ncclLLBuffer visualize peer communication for a single CTA.
Figure 9: Example: one-shot AllReduce using low-latency NCCL API with three highlighted stages.
Quantifying the Speed-of-Light Latency Bound
The absolute hardware lower bound ("speed-of-light") for small-message AllReduce is formalized based on minimal L2 cache and remote-store round-trip times.
Figure 10: Minimal data movement for AllReduce: load from local L2, simultaneous remote stores, reduction at SM.
Empirical measurements on GB200 hardware confirm tight correspondence: observed lower bounds are ~1.4 μs, invariant to rank count when using one-shot push protocols.
Empirical Evaluation and Microbenchmarks
Extensive microbenchmarks on NVL72 (GB200) and Alps (GH200) platforms demonstrate substantial latency reductions:
- One-shot LLBuffer kernels approach within 7% of the SoL lower bound for small messages; two-shot LL128 atomic shows superior scalability as GPU count increases.
- Hardware multicast (NVLink SHARP) further improves scalability for large rank counts, although incurs slight overhead for small groups.
Figure 11: Out-of-place AllReduce latency as a function of message size and GPU count; shaded regions show kernel wins.
Scratch buffer sizing analysis reveals that two-shot kernels require buffer space to match the full message size for single-iteration efficiency, while one-shot kernels benefit from moderate buffer sizing to avoid bandwidth saturation.
Figure 12: AllReduce latency as scratch buffer size varies—plateaus at single-iteration min for two-shot/atomic kernels.
Application Case Studies
LLM Inference
Integration of optimized kernels into vLLM shows measurable acceleration in real-world long-context LLMs:
- ITL reductions of up to 13% (4 GPUs) and 11% (8 GPUs) across Llama, DeepSeek, Qwen3 models.
- Throughput improvements yield estimated cost savings exceeding $11 per 1M output tokens in large-model decode-heavy configurations.
Figure 13: ITL, throughput, and cost savings for vLLM inference across kernel and memory configurations.
Traditional HPC Workloads
Low-latency collectives yield consistent speedups in cuSOLVERMp symmetric-definite eigensolvers, with gains most pronounced in communication-heavy configurations.
Figure 14: cuSOLVERMp performance: GFLOPS per GPU with/without low-latency NCCL kernels; improvements up to 8%.
Discussion, Implications, and Future Directions
This work demonstrates that meticulously engineered device-side synchronization, memory layout, and atomic communication primitives can push GPU collective latency to within a small multiplicative factor of the hardware minimum. The benefits are tangible in both advanced LLM inference (where microseconds propagate into production-scale cost savings) and traditional HPC solvers (where latency accrues over deep time-stepping). These results underscore the necessity for API-level abstractions that prioritize latency, expose device-side control, and permit the rapid construction of custom kernels adapted to application requirements.
Theoretical implications include the realization that global memory barriers, prevalent in legacy and ring-based designs, are not intrinsic to correctness and can be replaced with protocols leveraging buffer alternation and atomic signaling. This can be generalized further to other collective and point-to-point communication patterns. Practically, as large-scale AI systems increasingly operate within scale-up NVLink/NVSwitch domains, these techniques will define the performance envelope for inference and scientific computing.
Possible future work includes developing accurate architectural performance models for automated kernel selection, extending API functionality to block- and warp-level grain for improved usability, and generalizing synchronization protocols beyond addition by leveraging new atomic instructions or specialized hardware primitives.
Conclusion
The study systematically analyzes and composes synchronization and communication primitives to enable barrier-free, device-initiated, and memory-efficient collective kernels that approach the speed-of-light latency bound in intra-node scale-up networks. Through API abstraction, algorithmic innovations, and rigorous benchmarking across diverse workloads, the paper evidences significant reductions in latency and cost in both AI inference and HPC. The techniques and APIs introduced provide a foundation for constructing latency-optimal collective operations as GPU communication evolves toward device-centric paradigms.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about
This paper is about making groups of GPUs talk to each other much faster when they need to share and combine small pieces of data. The authors focus on a common group operation called “AllReduce,” which you can think of as “everyone adds their numbers together, and everyone gets the final total.” They show how to cut the wait time (latency) of these operations to nearly the best possible time the hardware allows, which they call the “speed‑of‑light” (SoL) limit.
Why this matters: When big AI models (like LLMs, or LLMs) run across several GPUs, they do lots of tiny, frequent group communications while generating each new word or token. Even a few microseconds of delay can add up and make responses slower and more expensive. The same is true for many science and engineering simulations that need quick, repeated reductions.
The main goals and questions
The authors set out to:
- Figure out why existing GPU group communications take longer than the hardware’s best‑case limit.
- Design new ways to coordinate GPUs so they don’t waste time waiting for each other.
- Build simple, reusable tools (APIs) so developers can create their own ultra‑low‑latency collective operations.
- Show, with tests and real apps, that these ideas make AI inference and scientific codes faster and cheaper.
In simpler terms: How can we remove the little pauses that slow everything down, and get close to the fastest time the hardware can possibly do?
How they approached it (in everyday language)
Think of several students (GPUs) who need to combine their answers and make sure everyone has the final result. The delays often come from “are you ready?” checks and hand‑raises (synchronization barriers). The authors’ main idea is to avoid those slow, everyone‑waits checkpoints by using smarter ways to signal “your data is here” and by organizing the traffic carefully.
Key ideas explained simply:
- Latency vs. bandwidth: Latency is the time to start and finish a tiny message (like the start‑up delay when a call connects). Bandwidth is how fast you can move large amounts of data once you’ve started (like the speed of the conversation). For many LLM tasks, lots of tiny messages matter most, so latency is the bottleneck.
- AllReduce in one shot vs. two shots:
- One‑shot: Everyone sends their data once, combines locally, and they’re done. Fast when messages are small.
- Two‑shot: First, everyone reduces different pieces (ReduceScatter), then they swap the pieces so everyone has the full result (AllGather). Better when messages are a bit bigger because it cuts total traffic.
- Avoiding “are you ready?” barriers:
- LL (low‑latency “flag with data”): Each piece of data carries a tiny tag that says “I’m valid now,” so the receiver can just look at the tag to know when it can use the data—no separate “ready?” message needed.
- Sentinel values: The receiving spot starts with a special “impossible” value (like a red card). When the value changes, the receiver knows valid data arrived. This keeps full data speed without extra tags.
- Double buffering with two‑way flow: Each pair of GPUs uses two “inboxes.” While one inbox is being read, the next data goes into the other one. Think of two trays: you never dump new papers onto the tray someone is still reading. Combined with sending and receiving in both directions, this removes the need for global “everyone stop and wait” pauses.
- A new “LL128 atomic” two‑shot algorithm: This clever design uses the hardware’s ability to add incoming data together at the cache line level (like a shared bin that sums contributions automatically). A small, embedded counter acts as a built‑in “everyone has contributed” signal. It’s very lean on extra space/bandwidth and scales well, though it’s tailored to NVLink hardware and addition.
- Tools for developers: They built experimental, low‑latency APIs on top of NCCL (a widely used GPU communication library). These APIs wrap the ideas above (LL, sentinel, double buffering, multicast) so others can write custom fast collectives more easily.
- Measuring the “speed‑of‑light” bound: They define SoL as the absolute minimum time the hardware needs to move one tiny chunk of data through the caches and across the GPU links—ignoring all other overhead. This gives a fair yardstick to see how close their methods get to perfect.
What they found and why it matters
- Barriers are expensive: A typical barrier (an all‑GPU “ready?” check) costs over 1 microsecond. For small messages that finish in a few microseconds, this is a big chunk of the total time. Removing barriers is a big win.
- Near‑hardware best: Their new kernels (implementations) cut small‑message AllReduce latency dramatically and get within about 7% of the SoL lower bound in microbenchmarks. For example, on 4 high‑end GPUs, they reduced small‑message latency from about 11 microseconds to about 2.4 microseconds.
- Real apps got faster:
- LLM inference: Lower “inter‑token latency” (the wait time between generating tokens) and higher throughput, which translates into noticeable cost savings. They estimate roughly 0.9% lower cost per 1 million output tokens for each microsecond removed from AllReduce.
- Scientific/HPC codes: Faster solvers (like cuSOLVERMp) and better scaling when many small reductions are needed.
- Practical and reusable: The new low‑latency APIs make it easier to build more custom, barrier‑free collectives, not just for this paper’s AllReduce variants.
What this could mean going forward
- Faster, cheaper AI serving: As LLMs grow, more GPUs must cooperate. Cutting microseconds from many small communications speeds up responses and reduces cloud bills at large scale.
- Better science and engineering performance: Many simulations need frequent small reductions. Lower latency helps them run faster and scale to more GPUs.
- A blueprint for others: The barrier‑free techniques (LL, sentinel, double buffering, two‑way flow) and the new APIs show a path to near‑SoL performance on today’s hardware. While this work focuses on NVIDIA GPUs and single‑node “scale‑up” networks (like NVLink domains), the ideas can inspire similar designs on other platforms.
In short: The paper shows that by removing unnecessary waits and using smart signaling and buffering, GPU groups can communicate almost as fast as the hardware physically allows. That’s a big deal for both AI inference and traditional high‑performance computing.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a focused list of what remains missing, uncertain, or unexplored in the paper, phrased to guide concrete follow-up work:
- Scope limited to single-node scale-up: no design or evaluation for multi-node (scale-out) hierarchies, GPUDirect RDMA/GIN across NICs, or integration with hierarchical collectives beyond the NVLink domain.
- Vendor/hardware dependence: techniques and measurements rely on NVIDIA-specific features (NVLink/NVSwitch/NVLS, L2 atomicity, multimem, CUDA VMM); portability to AMD/Intel GPUs and non-NVLink interconnects is untested.
- LL128 atomic constraints: only supports addition on FP16/FP32 with vectorized atomics and NVLink cache-line atomicity; no path for other ops (min/max, logical, non-commutative) or integer types.
- Determinism and reproducibility: LL128 atomic is non-deterministic (order of floating-point atomics); no deterministic variant or reproducibility mode is provided or evaluated.
- Numerical accuracy impact: error bounds are stated analytically, but there is no empirical evaluation of numerical drift in end-to-end applications (e.g., LLM quality, HPC solver convergence) versus standard reductions.
- Sentinel safety: correctness relies on values never matching the sentinel (e.g., −NaN for FP); no generic, collision-proof protocol, automated detection, or mitigation strategy is provided for all data types (incl. integer, BF16/FP8).
- Sentinel overheads: the cost and policies for sentinel-buffer initialization/reset (frequency, granularity, fences for visibility) are not quantified; trade-offs with LL for varying message sizes are not fully mapped.
- Double/multi-buffering policies: only double buffering is illustrated; optimal k-buffering, epoch sizing, and adaptive buffer management across diverse message sizes and GPU counts remain unexplored.
- Flow control and deadlock analysis: bidirectional credit-like progression is assumed safe, but there is no formal proof or stress testing under asymmetric progress, skew, or unexpected stalls across N>2 ranks.
- In-place collectives: constraints around in-place AllReduce are only briefly mentioned (two-shot requires symmetric LSA output); correctness and performance trade-offs for in-place semantics are not characterized.
- Concurrency and interference: impact of busy-wait polling and aggressive device-initiated communication on SM scheduling, power, thermal throttling, and interference with concurrent kernels is not evaluated.
- Scratch buffer sizing in real systems: microbenchmarks sweep size, but no automatic tuning heuristics, runtime adaptation, or memory pressure management across multiple concurrent collectives/applications is provided.
- Autotuning and algorithm selection: no runtime policy to select among one-shot, two-shot, LL, sentinel, LL128 atomic, and multicast based on hardware, N, message size, contention, or topology.
- Topology sensitivity: results are from GB200 NVL72; behavior under different NVSwitch generations, partial-connectivity topologies, multiple NVLink hops, and heterogeneous clock domains is not studied.
- SoL model fidelity: the SoL bound assumes simultaneous stores to all peers, L2 hits, and ignores instruction/serialization/queuing effects; validation against microarchitectural counters and contention models is missing.
- Barriers revisited: barrier cost is measured for a specific NCCL session; alternative device-side barrier designs (e.g., tree-based, LL/epoch barriers, warp-synchronous variants, hardware-assisted barriers) are not explored as a baseline.
- API granularity: device API exposes only thread-level primitives; lack of warp- and block-level variants may leave performance on the table for common patterns and complicate safe composition.
- Robustness under faults: no discussion of error handling, retry semantics, or recovery when a GPU drops out or when remote stores time out; implications for fault-tolerant collectives are unaddressed.
- Multicast/NVLS variability: multicast variants are shown, but selection criteria, fallback behavior when NVLS is unavailable, and sensitivity to NVLS firmware/driver versions are not detailed.
- Integration with existing frameworks: how the new kernels interact with NCCL’s autotuner, stream semantics, priorities, and framework-level schedulers (e.g., PyTorch/vLLM runtime overlap policies) is not described.
- Workload breadth: application evaluation focuses on LLM inference (decode-heavy) and cuSOLVERMp; training workloads, MoE all-to-all, sparse/gather-heavy collectives, and broader HPC applications are largely unevaluated.
- Energy and cost models: the conversion from μs latency to serving cost is shown for one configuration; no sensitivity analysis across hardware SKUs, pricing models, batch shapes, or long-context distributions.
- Formal memory-model guarantees: correctness relies on assumed memory ordering/visibility (e.g., L2 as PoC, fence requirements) across GPUs; a formalized model and minimal fencing rules for each synchronization mode are not provided.
- Security/isolation in multi-tenant settings: implications of exposing symmetric memory and device-initiated RMA across MIG/virtualization boundaries are not addressed.
- Compiler/tooling support: reliance on specific CUDA/PTX features (vectorized atomics, multimem) poses maintenance risks; fallback code paths and testing strategies for future toolchains are unspecified.
- Extendability to other collectives: while additional collectives are mentioned, detailed designs and evaluations for latency-critical All-to-All, ReduceScatter, and AllGather in diverse regimes are limited.
- NIC-driven paths: adapting barrier-free techniques to GPU-initiated networking (GIN) and NIC atomics for inter-node phases (and their latency/consistency implications) remains an open engineering and research task.
Practical Applications
Immediate Applications
The following applications can be deployed now by updating collective libraries and runtime configurations, especially on NVIDIA NVLink/NVSwitch systems running NCCL 2.28+ with device-side APIs and symmetric memory.
- Sector: Software/AI infrastructure (cloud inference, model serving)
- Action: Reduce inter-token latency and cost for long-context, small-batch tensor-parallel LLM inference by switching to NCCL’s new low-latency symmetric kernels.
- Steps:
- Upgrade to NCCL ≥2.28 with device-side APIs and symmetric memory enabled; ensure topology confines tensor-parallel groups within a single NVLink/NVSwitch domain.
- Select low-latency kernels and sync mode via environment variables:
1 2 3 4 5 6 7 8 9 10 11
# One-shot / Two-shot / LL128 Atomic variants export NCCL_SYM_KERNEL=AllReduce_LLBuffer # one-shot # or export NCCL_SYM_KERNEL=AllReduce_LLBuffer_Twoshot # two-shot # or export NCCL_SYM_KERNEL=AllReduce_LL128_Atomic # non-deterministic add, FP16/FP32 only # Synchronization mode for LLBuffer-based kernels: LL or Sentinel export NCCL_SYM_LLBUFFER_SYNC=LL # very small messages # or export NCCL_SYM_LLBUFFER_SYNC=Sentinel # better for moderate small messages
- Size and pre-initialize symmetric scratch buffers (use ncclCalcScratchBufferSize and ncclLLBufferInitSentinel if using Sentinel).
- For two-shot kernels, use symmetric (LSA) output buffers; for one-shot, ensure symmetric scratch is available.
- Expected benefits:
- Near speed-of-light latency for small messages; the paper reports average AllReduce latency down to ~2.37 μs on 4× GB200 and up to 8.7% inter-token latency reduction for Llama-3.1-70B decoding.
- Cost impact: on 4× GB200, ~0.9% lower cost per 1M output tokens for every 1 μs removed from AllReduce latency.
- Dependencies/assumptions:
- NVIDIA NVLink/NVSwitch domain; NCCL symmetric memory (LSA); sufficient scratch buffer capacity; inference group confined to one scale-up island.
- LL128 Atomic variant: non-deterministic, addition only, FP16/FP32, requires NVLink cache-line atomicity.
- Sentinel mode: ensure sentinel values (e.g., -NaN) cannot appear in valid data; must reset buffers before reuse.
- Sector: End-user AI products across verticals (finance, healthcare, education, customer service, code assistants)
- Action: Deliver lower latency, smoother streaming, and higher throughput for long responses by enabling the above kernels in serving stacks (vLLM, SGLang, TensorRT-LLM).
- Use cases: code assistants generating long sequences; clinical note summarization; tutoring systems with long-context; customer support chat with multi-agent chains.
- Dependencies/assumptions:
- Model parallelism spans only a single NVSwitch island; small message collectives dominate decode path.
- Frameworks must opt into NCCL’s symmetric kernels or expose a toggle.
- Sector: HPC/scientific computing (academia and industry)
- Action: Improve strong scaling in time-stepping solvers and synchronization-heavy phases by upgrading collectives in cuSOLVERMp- and cuBLASMp-based workflows.
- Use cases: MILC/LULESH-like kernels, CFD, FEM, particle simulations with frequent small reductions.
- Steps: link with updated NCCL, select two-shot kernels for moderate message sizes; tune scratch buffer and CTA counts; prefer sentinel mode for better payload efficiency beyond the tiniest messages.
- Dependencies/assumptions:
- Workloads run within an NVLink domain; reductions fit in L2 for best SoL proximity; multicast (NVLS) accelerates bcast where available.
- Sector: Systems/DevOps (cluster orchestration and placement)
- Action: Topology-aware scheduling to pin tensor-parallel and latency-sensitive collectives within the same NVSwitch island and enable hierarchical (local-first) collectives for multi-node jobs.
- Steps: Integrate NVLink-domain awareness in SLURM/Kubernetes; avoid cross-island collectives for small-message paths; enable NCCL’s device-side collectives for local phases.
- Dependencies/assumptions:
- Accurate GPU topology discovery; coordination with job schedulers and container runtimes.
- Sector: Library and framework developers (PyTorch Distributed, custom runtimes)
- Action: Rapidly prototype domain-specific low-latency collectives using the provided ncclLLBuffer-based API (send/recv/recvReduce/bcast/reset, double buffering, LL or sentinel).
- Use cases: specialized ReduceScatter/AllGather for attention blocks; small-message all-to-all variants; latency-aware fused compute+collective kernels.
- Dependencies/assumptions:
- Developers adopt device-side NCCL APIs; ensure bidirectional patterns and double buffering to avoid barriers.
- Sector: Observability/Performance engineering
- Action: Establish latency SLOs and regression tests using the paper’s SoL estimation methodology (L2 RTT via threadfence, remote-store RTT via ping-pong) to set realistic lower bounds and diagnose barrier overheads.
- Tools/workflows: incorporate SoL baselines in CI perf tests; monitor barrier frequency; auto-select one-shot vs two-shot based on message-size histograms.
- Dependencies/assumptions:
- Microbench infrastructure with access to device-side timing and peer-to-peer paths; representative small-message distributions.
- Sector: Energy/sustainability in data centers
- Action: Reduce energy per token and per time-step by minimizing idle/wait time in small collectives with barrier-free kernels.
- Steps: deploy low-latency kernels on NVSwitch clusters running long-context inference or synchronization-heavy HPC.
- Dependencies/assumptions:
- Realized energy savings depend on platform power scaling and utilization; benefits are largest when latency dominates.
Long-Term Applications
These applications require additional research, hardware support, or ecosystem integration beyond single-node NVLink/NVSwitch domains.
- Sector: Distributed systems (multi-node AI/HPC)
- Opportunity: Extend barrier-free, push-based, double-buffered designs to scale-out (InfiniBand/RoCE) via hierarchical collectives and GPU-initiated networking (GIN).
- Potential product: “Latency-optimized hierarchical collectives” in NCCL/MSCCL++ that combine local near-SoL phases with ultra-lean inter-node synchronization.
- Dependencies/assumptions:
- NIC and runtime support for GPU-initiated multicast/reduction and low-latency completion; careful credit/flow-control design across nodes.
- Sector: Semiconductor/hardware co-design
- Opportunity: Hardware features that make near-SoL latency routine and deterministic: cross-chassis multicast/reduction, deterministic cache-line atomics, vectorized atomics for more ops (min/max), and broader type support (BF16/TF32/FP64).
- Potential products: Next-gen NVSwitch/NIC firmware enabling multicast/reduce across racks; standardized vendor-agnostic atomic semantics.
- Dependencies/assumptions:
- ISA and fabric changes; cross-vendor alignment (AMD ROCm, Intel oneCCL); backward compatibility.
- Sector: HPC numerics and reproducibility
- Opportunity: Deterministic, near-SoL reductions for scientific workloads; new algorithms that preserve bitwise reproducibility while avoiding global barriers.
- Potential tools: deterministic “two-shot” reductions with bounded ordering variance; compiler/runtime scheduling to enforce reproducibility without large stalls.
- Dependencies/assumptions:
- New atomic primitives or collectives with deterministic accumulation order; potential trade-offs in throughput.
- Sector: Compilers/runtime systems (XLA, TorchInductor, Triton)
- Opportunity: Automatic selection and fusion of one-shot/two-shot/LL/sentinel based on live telemetry (message sizes, scratch availability, multicast presence) and kernel fusion that overlaps compute with barrier-free communication.
- Potential products: latency-aware codegen passes that insert ncclLLBuffer primitives; dynamic autotuners selecting algorithms per layer/token.
- Dependencies/assumptions:
- Stable device-side APIs; robust profiling hooks; accurate topology detection.
- Sector: Edge/robotics and on-prem multi-accelerator boxes
- Opportunity: Bring near-SoL collectives to small edge systems with multiple accelerators for real-time perception+LLMs or VLMs that jointly run across GPUs.
- Potential products: low-latency inference kits for 2–4 GPU edge servers; robotics stacks with multi-GPU planning and reasoning.
- Dependencies/assumptions:
- Availability of NVLink-like interconnects in edge; thermal and power constraints; software support for device-side collectives.
- Sector: AI training systems (MoE, all-to-all, pipeline parallelism)
- Opportunity: Adapt barrier-free techniques to small-message heavy MoE gating, hybrid tensor/pipeline-parallel steps, and speculative decoding coordination.
- Potential tools: micro-collectives libraries tuned for gate/route exchanges; pipeline schedulers that choose near-SoL collectives on critical paths.
- Dependencies/assumptions:
- Algorithms for irregular patterns; generalization beyond reductions (e.g., gather/scatter) with minimal synchronization.
- Sector: Cloud policy, procurement, and SLAs
- Opportunity: Define and adopt latency-centric SLAs and procurement guidelines emphasizing NVSwitch island size, multicast capability, and device-initiated networking for AI/HPC.
- Potential outcomes: standardized “latency class” tiers for AI instances; incentives for energy-efficient low-latency deployments.
- Dependencies/assumptions:
- Broad provider buy-in; measurement standards (e.g., publish SoL and achieved % of SoL for small collectives).
- Sector: Datacenter schedulers and cost optimizers
- Opportunity: Latency-aware bin packing that co-locates TP groups and small-collective-heavy stages within the same NVSwitch island; cost models that factor “μs per collective” into placement and autoscaling.
- Potential products: Kubernetes/SLURM plugins that minimize cross-island hops; per-model policies selecting one-shot vs two-shot based on observed traffic.
- Dependencies/assumptions:
- Topology-aware resource discovery; integration with framework-level profilers; acceptance of small batch trade-offs.
- Sector: Cross-vendor portability
- Opportunity: Implement LL, sentinel, and double-buffered bidirectional protocols on other accelerators (AMD/Intel) exposing GPU-initiated peer memory access and fences.
- Potential products: low-latency collectives in RCCL/rocSHMEM/oneCCL with symmetric memory analogs.
- Dependencies/assumptions:
- Peer LSA, remote stores/loads visible to device polling, and memory ordering primitives comparable to NVIDIA’s stack.
Notes on feasibility across applications:
- The largest gains occur when small-message collectives are on the critical path (e.g., decode-heavy LLMs, tightly synchronized solver steps).
- Benefits rely on keeping groups within a single scale-up domain; crossing nodes typically requires hierarchical designs.
- Choosing between LL and sentinel is workload dependent: LL for tiny messages with minimal flags, sentinel for better effective payload on slightly larger small messages; both require careful buffer management and double buffering to avoid barriers.
- LL128 Atomic offers excellent scalability and low overhead but is non-deterministic and operation-type limited; use only when numerical determinism is not required.
Glossary
- AllGather: A collective operation that distributes different segments of data so every rank receives the complete reduced result. "Moreover, many implementations decompose AllReduce into ReduceScatter and AllGather or Reduce and Broadcast, so techniques that minimize AllReduce latency often apply directly to these building blocks."
- AllReduce: A collective operation that reduces data across ranks and returns the result to all ranks. "In long-context, small-batch tensor-parallel (TP) LLM inference, many small AllReduce operations lie on the critical path, making collective latency a crucial bottleneck."
- BF16: Bfloat16, a 16-bit floating-point format with 8-bit exponent and 7-bit mantissa used for efficient ML training/inference. "The bound is larger for FP16 and BF16."
- Bidirectional communication: A communication pattern where data flows in both directions between peers each iteration to enable safe, barrier-free progress. "We can eliminate them by using bidirectional communication and double buffering."
- Broadcast: A collective operation where one rank sends the same data to all other ranks. "The released artifact also includes low-latency Broadcast, Reduce, ReduceScatter, and AllGather kernels constructed from the same LL and sentinel primitives."
- Cache line: The fundamental unit of data transfer between cache and memory; on NVIDIA GPUs commonly 128 bytes. "Together, the 8 threads operate on 128 bytes, which matches the size of a cache line."
- CTA (Cooperative Thread Array): A CUDA thread block that executes cooperatively on an SM with shared resources. "Threads within a CTA are divided into two groups: 496 regular threads and 16 extra threads, which handle displaced elements."
- CUDA-aware MPI: MPI implementations that can directly handle GPU memory, enabling GPU-resident communication. "In contrast, traditional frameworks like MPI, originally designed for CPU-based systems, have been extended to support GPUs (e.g., CUDA-aware MPI~\cite{kraus2025cudaawarempi})"
- Device-Initiated Communication: Communication triggered directly by GPU kernels without CPU intervention. "Device-Initiated Communication and Symmetric Memory"
- Double buffering: Using two buffers in alternation to overlap communication and computation and avoid overwrites. "We can eliminate them by using bidirectional communication and double buffering."
- Forward-error bound: A theoretical bound on the numerical error accumulated during floating-point operations. "In terms of numerical stability, the algorithm obeys the standard forward-error bound for floating-point summation."
- GPUDirect Async Kernel-Initiated (GDAKI): A mechanism allowing GPUs to directly drive NIC operations without CPU involvement. "and GPUDirect Async Kernel-Initiated (GDAKI), which allows GPUs to directly interact with network interfaces without CPU intervention."
- GPU Virtual Memory Management (VMM): Hardware/runtime support providing a unified virtual address space across GPUs. "These capabilities are enabled by underlying hardware and runtime support, including GPU Virtual Memory Management (VMM), which provides a unified virtual address space across GPUs"
- GPU-initiated networking (GIN): GPU-driven networking path enabling direct network operations from the GPU. "For inter-node communication, GPU-initiated networking (GIN) supports GDAKI and proxy-assisted data transfers over InfiniBand and RoCE"
- Inter-token latency (ITL): The latency between consecutive tokens during autoregressive generation in LLM inference. "This translates into lower inter-token latency (ITL) and higher cost savings for Llama-3.1-70B inference."
- KV-cache: Key-value cache used to store past attention states for efficient LLM decoding. "In addition, for long-context inference, the KV-cache memory grows with sequence length, and the batch size is often reduced to fit within device memory."
- L2 round-trip time (L2 RTT): The latency for a transaction to reach and be acknowledged by the L2 cache. "This instruction enforces ordering and visibility at the L2 level, forcing the SM to wait until outstanding memory transactions reach the cache. Hence, its latency provides a good approximation of the L2 RTT."
- Load-store accessible (LSA): Memory regions that can be directly accessed via loads/stores across GPUs. "Using NCCL’s ncclLsaBarrierSession, which implements a memory barrier for load-store accessible (LSA) devices, we measure the overhead of such synchronization"
- LL (Low Latency): A protocol that co-transmits data with a small flag using atomic stores to avoid explicit signaling. "The first technique we introduce is LL, short for low latency, which originates from the LL protocol in NCCL"
- LL128: A low-latency NCCL protocol operating on 128-byte units (cache lines) with specific thread grouping. "Since it resembles NCCL’s LL128 protocol and relies on atomic additions, we refer to it as the two-shot LL128 atomic algorithm."
- Memory barrier: A synchronization primitive enforcing ordering/visibility of memory operations across threads/devices. "we begin by identifying global memory barriers across participating GPUs as a key source of latency in existing collective implementations."
- Multicast: Hardware capability to send data from one source to multiple destinations in a single operation. "When multicast support is enabled, the implementation uses hardware multicast to distribute data in a single operation."
- NCCL: NVIDIA Collective Communications Library for optimized multi-GPU communication. "Building on NCCL's device-side API, we develop low-latency interfaces for constructing custom collective kernels"
- NVLink: NVIDIA’s high-bandwidth interconnect enabling fast GPU-to-GPU communication. "On GPUs connected through PCIe or NVLink and supported by CUDA Virtual Memory Management (VMM), symmetric memory regions can be mapped into a unified virtual address space"
- NVLink SHARP (NVLS): In-network acceleration for multicast/reduction over NVLink/NVSwitch fabrics. "On systems with NVSwitch and NVLink SHARP (NVLS), multimem load/store instructions can further accelerate communication by enabling multicast and in-network reduction."
- NVSwitch: A switch fabric connecting many GPUs with NVLink to form large NVLink domains. "On systems with NVSwitch and NVLink SHARP (NVLS), multimem load/store instructions can further accelerate communication by enabling multicast and in-network reduction."
- NVSHMEM: NVIDIA’s PGAS library enabling one-sided GPU communication and synchronization. "NVSHMEM adopts a partitioned global address space (PGAS) model that enables one-sided communication and exposes finer device-side control"
- One-shot AllReduce: An AllReduce variant that completes the reduction in a single communication phase. "In a one-shot AllReduce, the entire reduction is completed in a single communication phase, where each GPU fetches data from all peers, performs the reduction locally, and writes the result to the output."
- Partitioned global address space (PGAS): A programming model exposing a global address space partitioned across PEs for one-sided operations. "NVSHMEM adopts a partitioned global address space (PGAS) model that enables one-sided communication and exposes finer device-side control"
- Processing Element (PE): An execution unit (here, a GPU) in PGAS/SHMEM contexts. "Remotely accessible data objects, called symmetric objects, have identical type, size, and layout on each processing element (PE), which corresponds to a GPU in this case."
- Reduce: A collective that aggregates values (e.g., sum) from all ranks to one destination. "The released artifact also includes low-latency Broadcast, Reduce, ReduceScatter, and AllGather kernels"
- ReduceScatter: A collective that reduces data across ranks and scatters disjoint reduced chunks to each rank. "In the ReduceScatter phase, each GPU exchanges partitions of its input with peers and performs the reduction on its assigned chunk, producing a partial result."
- Ring AllReduce: An AllReduce algorithm passing chunks around a ring; optimal for bandwidth but high in synchronization steps. "In contrast, algorithms with inter-step dependencies and without bidirectional communication, such as ring AllReduce, still require explicit memory barriers."
- Scale-up network: A single-node (or single NVLink domain) high-bandwidth interconnect as opposed to multi-node scale-out. "we study how to approach the hardware Speed-of-Light (SoL) lower bound for GPU collectives within a scale-up network."
- Sentinel: A synchronization technique using a special preinitialized value to signal data readiness without explicit flags. "Instead of embedding a signal in the transmitted data, the receiving scratch buffer can be initialized with a sentinel value that is unlikely to appear in valid computations"
- SHMEM: A family of PGAS libraries providing symmetric memory and one-sided operations. "Symmetric memory originates from the SHMEM family of PGAS models."
- Speed-of-Light (SoL) bound: The hardware lower bound on latency determined by minimal required data movement. "The microbenchmark shows that our NCCL low-latency kernel reduces small-message AllReduce latency relative to other implementations, approaching the speed-of-light (SoL) bound."
- Symmetric heap: The memory region holding symmetric objects identically allocated across PEs in SHMEM/PGAS. "These objects reside in the symmetric heap, which supports one-sided operations such as get, put, and atomics."
- Symmetric memory: Memory allocated identically across ranks/PEs, enabling direct addressing and one-sided operations. "We identify key principles for near-optimal designs, including barrier-free synchronization and efficient use of symmetric memory and multicast."
- Tensor-parallel (TP): A parallelization strategy partitioning model tensors across multiple devices. "In long-context, small-batch tensor-parallel (TP) LLM inference, many small AllReduce operations lie on the critical path"
- Two-shot AllReduce: An AllReduce variant split into ReduceScatter and AllGather phases, reducing communication volume. "A two-shot AllReduce is decomposed into two phases."
- Unified virtual address space: A single address space across devices allowing direct cross-GPU pointers/accesses. "which provides a unified virtual address space across GPUs"
- Vectorized atomics: Atomic operations applied to vectors (e.g., 128-bit) enabling efficient updates of multiple elements. "It requires NVLink to guarantee cache-line–level atomic addition and supports only single- and half-precision types due to the availability of vectorized atomics in CUDA"
- vLLM: An inference framework emphasizing high-throughput and low-latency LLM serving. "We also integrate our low-latency kernels into real workloads, including vLLM and cuSOLVERMp."
Collections
Sign up for free to add this paper to one or more collections.