Analyzing Reverse Address Translation Overheads in Multi-GPU Scale-Up Pods
Abstract: Distributed ML workloads rely heavily on collective communication across multi-GPU, multi-node systems. Emerging scale-up fabrics, such as NVLink and UALink, enable direct memory access across nodes but introduce a critical destination-side translation step: translating Network Physical Addresses (NPAs) to System Physical Addresses (SPAs), which we term Reverse Translation (Reverse Address Translation). Despite its importance, the performance impact of Reverse Address Translation remains poorly understood. In this work, we present the first systematic study of Reverse Address Translation in large-scale GPU clusters. Using an extended ASTRA-sim framework with Omnet++ as the network backend, we model Link MMUs and Link TLBs and evaluate their effect on All-to-All collective communication across varying input sizes and GPU counts. Our analysis shows that cold TLB misses dominate latency for small, latency-sensitive collectives, causing up to 1.4x performance degradation, while larger collectives benefit from warmed caches and experience diminishing returns from over sized TLBs. Based on these observations, we propose two avenues for optimization: fused pre-translation kernels that overlap Reverse Address Translation with computation and software-guided TLB prefetching to proactively populate likely-needed entries. These techniques aim to hide translation latency, particularly for small collectives, improving throughput and scalability for inference workloads. Our study establishes a foundation for designing efficient destination-side translation mechanisms in large-scale multi-GPU systems.
Summary
- The paper demonstrates that reverse address translation can incur up to 1.4× performance degradation for 1MB collectives due to cold TLB misses.
- It uses simulations of a UALink-connected 16-GPU pod to analyze translation latencies across hierarchical TLB systems and page walker incursions.
- The study advocates fused pre-translation kernels and software-guided TLB prefetching to mitigate latency in ML inference and training workloads.
Reverse Address Translation Overheads in Multi-GPU Scale-Up Pods
Introduction: Context and Motivation
The ongoing expansion of deep learning models, exemplified by the transition from millions to trillions of parameters, has directly catalyzed the deployment of large-scale GPU clusters interconnected via high-bandwidth, memory-semantic fabrics such as NVIDIA NVLink and UALink. These infrastructures enable direct inter-GPU communication across both intra-node and inter-node boundaries, critical for distributed execution paradigms like tensor, model, and data parallelism. However, sophisticated address-mapping mechanisms become necessary to correctly resolve remote GPU memory accesses, introducing Network Physical Addresses (NPAs) that require translation to System Physical Addresses (SPAs) at the memory target side. This process, termed "Reverse Address Translation," imposes a destination-side translation bottleneck that is not yet well-characterized.
The work under review provides a systematic analysis of Reverse Address Translation in contemporary multi-node, multi-GPU scale-up pods, quantifying its performance implications and evaluating the efficacy of hardware translation modules (Link MMU, L1/L2 Link TLBs) for collective communication workloads, specifically the AlltoAll pattern that dominates ML training and inference. The paper further outlines architectural and software strategies for latency mitigation.
Figure 1: A multi-node multi-GPU system connected over a UALink network used as the baseline in this analysis.
System Model and Baseline Translation Architecture
The evaluation adopts a scale-up pod constructed with a UALink-connected GPU cluster. UALink enables intra- and inter-node GPU memory transactions via NPAs, which require destination translation to SPAs at the remote GPU. Each GPU incorporates a Link MMU to perform this translation, with private L1 TLBs per UALink station and a shared L2 TLB per GPU. Unresolved translations trigger page walker traversal, as instantiated in the simulation's translation hierarchy.
Figure 2: Reverse Address Translation: mapping an NPA to an SPA at the target GPU, enabled via a Link MMU and hierarchical TLB infrastructure.
Figure 3: The evaluated baseline Reverse Address Translation hierarchy with per-station L1 TLB, a shared L2 TLB, and shared page walker.
The simulation stack utilizes Astra-sim with OMNeT++ for network modeling, MSCCLang for collective workload generation, and a parameterization aligned with contemporary UALink hardware resources.
Reverse Address Translation: Quantitative Overheads
The principal experimental finding is that reverse translation imposes severe latency penalties for small AlltoAll collectives in cold TLB states—up to 1.4× performance degradation for 1MB collectives, diminishing to ~1.1× for 16MB collectives. This penalty emerges from cold-miss-dominated address translations, as individual requests frequently trigger high-latency page table walks before TLBs are warmed.
Figure 4: Reverse Address Translation performance overhead vs. ideal zero-overhead translation, across increasing GPU pod and collective sizes.
Empirical analysis of per-request translation latency corroborates this result: small collectives (1MB) sustain high average translation latency until the working set is paged in, while larger collectives efficiently amortize translation costs across repeated TLB hits.
Figure 5: Average Reverse Address Translation latency per request as a function of pod and collective size.
For a typical 16-GPU pod and 1MB collectives, up to 30% of total round-trip request latency is attributable solely to reverse translation.
Figure 6: Fraction of total round trip latency contributed by Reverse Address Translation for various collective sizes.
Translation Hierarchy Behavior
Breakdown studies further reveal that while over 90% of requests hit the L1-MSHR interface, latency is nonetheless dictated by cold-miss scenarios at the L2- or page-walk level during system warm-up and initial access.
Figure 7: Distribution of hits and misses in the Reverse Address Translation hierarchy under variable AlltoAll sizes, 16-GPU system.
A more granular decomposition shows that as collective size increases, L1-TLB hits dominate and the frequency of costly lower-level translation events decreases. Conversely, small collectives are disproportionately susceptible to L2-TLB misses and hit-under-miss events, which magnify the performance penalty.
Figure 8: Detailed breakdown of hit-under-miss and miss scenarios in translation hierarchy, highlighting role of cold page walks for small collectives.
Analysis of request-level latency traces substantiates these aggregate effects: cold miss spikes are prominent for small collectives, but are rapidly damped as access patterns instantiate TLB and cache residency for larger transfers.
Figure 9: Per-request translation latency trace for 1MB collectives; high-latency cold misses dominate during warm-up.
Figure 10: Latency trace for a 256MB collective; cold-miss spikes at page boundaries, with diminishing impact as streaming pattern proceeds.
Architectural Insights: L2-TLB Provisioning and Locality
A salient architectural conclusion is that L2-TLB capacity is not a limiting factor for these ML collective workloads. The custom AlltoAll streaming pattern presents minimal temporal locality—the active translation working set per destination is bounded by one page per participating source GPU. Hence, once the L2 TLB capacity matches the pod size (number of participating GPUs), further provisioning yields negligible performance improvement.
Figure 11: Performance sensitivity to L2-TLB size for 32-GPU, 16MB collectives—no improvement is observed beyond the pod size.
Implications and Optimization Directions
The principal implication is that, in latency-sensitive workloads such as inference (where small collectives are prevalent), Reverse Address Translation overheads comprise a substantial fraction of overall system latency and are not adequately mitigated by naive TLB overprovisioning. The limited temporal locality of ML collective access patterns further constrains hardware approaches.
The paper proposes two complementary software/hardware strategies to address this bottleneck:
- Fused Pre-Translation Kernels: Embedding NPA-to-SPA translation operations into computation kernels ahead of future data movement can warm TLBs, effectively overlapping translation with computation and mitigating cold-miss penalties.
- Software-Guided TLB Prefetching: Leveraging static or profiled access pattern information to prefetch required TLB entries, thus reducing first-access latency for new memory pages during small or irregular collectives.
These directions exploit the determinism and regularity of communication patterns in distributed ML workloads to proactively hide translation latency, reducing the translation component of inference-time latency/cost—an increasingly critical metric in cloud-scale deployments.
Theoretical and Practical Ramifications
This analysis fills a gap in the address translation literature by establishing the performance characteristics and working set properties of destination-side, NPA-to-SPA translation in scale-up multi-GPU fabrics—a scenario unaddressed by source-side translation optimization efforts.
On a practical level, cluster architects and runtime developers must target translation latency reduction via co-designed translation path management and translation-aware collective scheduling, especially as inference becomes the dominant deployment scenario in production environments. Hardware designers can avoid costly overprovisioning of TLB resources, focusing instead on hierarchical organization and effective software interface mechanisms for translation prefetching.
Theoretically, this work opens further questions around dynamic translation working set size estimation, more aggressive hardware/software fusion for translation-aware communication scheduling, and cross-layer translation policy specification in ML middleware.
Conclusion
This work provides the first systematic quantification of Reverse Address Translation overheads in UALink- and NVLink-style multi-GPU scale-up pods. The finding that cold TLB misses are the principal determinant of communication latency for small, latency-critical collectives is robust, with up to 1.4× performance penalty for 1MB collectives. The minimal temporal locality in streaming collective access fundamentally limits the benefit of bulk TLB overprovisioning for ML communication patterns. Addressing reverse translation bottlenecks through fused pre-translation and software-guided TLB prefetching offers a promising avenue for future architectural and system-level optimization, and is directly relevant for scalable ML inference and training system design.
(2604.02473)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper looks at a hidden slowdown in big AI computer systems that use lots of GPUs working together. When one GPU on one machine wants to read or write memory on a GPU in another machine, the destination machine has to “translate” the incoming memory address into its own local address before it can do anything. The authors call this step Reverse Address Translation. They show that this translation can quietly add delay, especially for small data exchanges that are common in AI inference.
The key questions
- How much does Reverse Address Translation slow down communication between GPUs in different machines?
- When is this slowdown worst?
- Can we design simple hardware or software tricks to hide or reduce that delay?
How did they study it?
Think of memory addresses like house addresses. Inside one city, everyone uses the same map. But when a package comes from another city, the destination post office must convert the “other city” address into a local address before delivery. That conversion takes time.
- In GPU terms:
- Network Physical Address (NPA) = “other city” address coming over the network.
- System Physical Address (SPA) = local address on the destination machine.
- Reverse Address Translation = converting NPA to SPA at the destination.
- Link TLB and Link MMU = small “notebooks” and “clerks” that remember recent address conversions and look up new ones when needed. If the notebook doesn’t already have the answer (a “cold miss”), a slow lookup is required.
Because it’s hard to measure this directly on real clusters, the authors built a detailed simulation:
- They extended a simulator called ASTRA-sim and used a network simulator called OMNeT++ to model how messages move between GPUs.
- They added models of the destination-side translation hardware (Link MMU and Link TLBs).
- They tested a common “everyone talks to everyone” pattern called All-to-All, used in large AI models and Mixture-of-Experts, across 8 to 64 GPUs and with different message sizes (from 1 MB to 4 GB).
What did they find?
Here are the main takeaways presented in simple terms:
- Small messages get hit the hardest: For small All-to-All exchanges (like 1 MB), first-time address lookups at the destination are slow and happen a lot, causing up to about 1.4× slowdown compared to a world with “instant translation.”
- Big messages feel it less: For larger data exchanges, the “notebooks” (TLBs) at the destination warm up and remember translations, so later lookups are fast. The slowdown shrinks to around 1.1× or less.
- “First time” is the main problem: The biggest delays come from cold misses—when the translation notebook is empty and must do a full, slow lookup.
- Bigger isn’t always better: Making the translation notebook (especially the bigger, shared L2-TLB) huge doesn’t keep helping. That’s because these All-to-All patterns mostly “stream” through memory pages and don’t come back to the same pages often. A modest notebook that can cover about one active page per participating GPU is usually enough.
- The number of GPUs matters: As more GPUs join the All-to-All, the destination must handle more “first-time” pages at once, but once those are learned, later requests remain fast.
Why this is important: Inference (running a trained model) often uses small, latency-sensitive messages. That means these translation delays can matter in real systems and waste time and money at large scale.
What could this change?
If we want faster AI systems, especially for inference, we should reduce or hide the destination’s first-time translation costs. The paper suggests two practical ideas:
- Fused pre-translation during compute: While the GPU is busy computing, start the address translations early for the memory you will send soon, so the notebooks are warmed up when communication starts.
- Software-guided prefetching: Use software to “guess” which pages will be needed next and pre-load their translations into the notebook ahead of time.
These ideas aim to keep small messages from getting stuck on slow, first-time lookups.
Simple conclusion
Reverse Address Translation—the step where a destination GPU maps an incoming network address to its own local address—can quietly slow down multi-GPU communication, especially for small, latency-sensitive messages common in modern AI inference. The slowdown mainly comes from “first-time” lookups. Once the translation caches are warmed up, things speed up. Oversizing the caches doesn’t help much; instead, warming them up early with pre-translation and smart prefetching can deliver better performance and scalability for large AI systems.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a focused list of what remains uncertain or unexplored in the paper, framed as concrete, actionable items for follow-on research.
- Empirical validation gap: No measurements on real NVLink/UALink hardware exist to calibrate or validate the simulator’s reverse translation latencies (e.g., 50 ns L1, 100 ns L2, 300 ns link/switch). Collect baseline microbenchmarks to anchor model parameters and verify trends.
- Evolving-spec uncertainty: UALink parameters are still in flux; the study assumes specific bandwidths/latencies and a single-level Clos. Conduct sensitivity analyses across plausible spec ranges and alternative topologies (multi-stage Clos, fat-tree, dragonfly) to assess robustness.
- Single collective focus: Only All-to-All (all-pairs/direct) is evaluated. Quantify reverse-translation overheads for other collectives and algorithms (AllReduce, AllGather, ReduceScatter; ring/tree/recursive-doubling/SCCL-synthesized plans) and mixed-collective workloads.
- Remote access semantics: Experiments assume two-sided, remote-store-only collectives. Evaluate remote loads, atomics, and mixed RMA patterns where ordering/backpressure and read-return paths could interact differently with reverse translation.
- Fixed page size: Only 2 MB pages are modeled. Explore 4 KB/64 KB/1 GB pages, huge-page promotion, and TLB coalescing to quantify trade-offs between miss rate, fragmentation, and memory management overheads.
- Incomplete design-space coverage: Sensitivity is shown mainly for L2-TLB capacity. Systematically sweep L1/L2 associativity, L1 capacity, MSHR depth, PTW parallelism, page walk cache sizes/levels, and replacement policies to identify true bottlenecks and cost-effective design points.
- No data-cache effects: The study assumes all memory accesses miss in all cache levels and uses a fixed 120 ns CU→NoC latency. Incorporate realistic cache behavior, NoC contention, DMA engines, and copy-path specifics to quantify end-to-end impact under realistic on-GPU traffic.
- Contention and interference: The Link MMU/TLB and PTW are modeled in isolation from other GPU/IO activities. Analyze interference between reverse translation and (i) local GPU MMU activity, (ii) concurrent collectives, (iii) NIC/PCIe traffic, and (iv) HBM bandwidth consumed by page walks.
- Workload representativeness: Traffic is generated from MSCCLang all-pairs schedules. Validate with traces from production libraries (NCCL/RCCL/oneCCL) and real MoE/TP/PP/FSDP training and inference jobs, including micro-batch variability and runtime scheduling effects.
- Scaling beyond 64 GPUs: Experiments stop at 64 GPUs; UALink targets up to 1,024. Characterize how translation working sets, PTW pressure, TLB thrashing, and switch/link contention evolve at 128–1,024 GPUs and under multi-rail configurations.
- Multi-tenant and virtualization scenarios: No evaluation of SR-IOV/MIG, IOMMU isolation, QoS, or TLB partitioning. Study fairness, DoS resilience, and security implications (e.g., prefetch-based side channels) in shared pods.
- Dynamic memory management: Page faults, demand paging/UVM, page migration, and TLB shootdowns/invalidation are not modeled. Quantify their frequency and impact on reverse translation, especially for elastic or memory-oversubscribed workloads.
- Placement and mapping policies: The paper assumes specific NPA→SPA mapping but does not study effects of page coloring, contiguity, or NUMA placement on translation locality. Evaluate OS/driver policies (pinning, preregistration, batching) to reduce cold-miss bursts.
- Network-level translation caching: Only target-GPU Link MMUs/TLBs are modeled; potential translation caching in switches or cooperative caching across accelerators is unexplored. Assess feasibility, correctness, and performance/area trade-offs.
- Proposed optimizations unquantified: Fused pre-translation kernels and software TLB prefetching are proposed but not implemented or evaluated. Open questions include: prefetch accuracy/coverage, pacing, pollution, abort semantics, API/ISA hooks, correctness under re-mapping, and compute-communication overlap costs.
- Security and correctness of speculative translation: What protections prevent unauthorized or stale translations (e.g., post-unmap accesses) when prefetching? Define coherence/invalidation protocols and proof-of-correctness for pre-translation.
- Power/area implications: No analysis of area, power, timing, or verification cost for larger TLBs, deeper PTWs, or additional caches. Produce a PPA study to justify hardware allocations relative to performance gains.
- Congestion-control interactions: How do translation stalls interact with credit-based flow control, head-of-line blocking, and retransmissions? Analyze feedback loops where TLB/PTW delays throttle or destabilize network performance.
- Message segmentation and chunking: The impact of chunk sizes, pipeline depth, and per-destination message scheduling on translation working sets and cold-miss bursts is not explored. Co-design collective scheduling with translation-aware chunking.
- Hot-spot and skewed access patterns: The study focuses on uniform streaming per-page access. Investigate skew (hot experts, popular shards) that cause concentrated translation pressure and evaluate mitigation (e.g., replication, caching policies).
- Heterogeneous systems: Cross-vendor pods and mixed accelerators/NPUs may have different Link MMU/TLB designs. Assess portability and tuning portability of reverse-translation optimizations across heterogeneous hardware.
- Reliability and error handling: Effects of ECC errors in page tables, walker retries, and recovery behavior (timeouts/abort) on collectives are not discussed. Define resilience mechanisms and their latency budget.
- Reproducibility and artifacts: The ASTRA-sim/OMNeT++ extensions and parameter sets are not stated as publicly released. Provide artifacts to enable independent reproduction and extension of the study.
Practical Applications
Immediate Applications
The following applications can be deployed on today’s multi-GPU pods using NVLink/UALink or similar memory-semantic fabrics, and focus on reducing latency for small collectives and avoiding wasteful overprovisioning identified in the paper.
- Right-size Link TLBs and PTW resources in new pods (hardware design and procurement)
- Sectors: semiconductor, cloud/IT infrastructure, HPC
- Action: Size per-GPU L2 Link TLB capacity to approximately the number of concurrently active remote pages (≈ number of participating GPUs per collective), rather than large, power-hungry caches; provision PTW parallelism and page-walk caches to minimize cold-starts.
- Tools/Products: “Pod configurator” spreadsheets or DSE scripts that map expected collective sizes and GPU counts to TLB and PTW targets.
- Assumptions/Dependencies: Applicable mainly to streaming/collective-heavy ML workloads; requires hardware configurability or next-gen silicon.
- Software-guided Reverse-Translation (Link TLB) prefetch in communication libraries
- Sectors: software/AI infrastructure (NCCL/RCCL/oneCCL/MSCCL), cloud serving
- Action: Insert software-driven prefetches to populate destination Link TLBs before remote stores in All-to-All and related collectives; prefetch next-page NPAs based on buffer layout and stride.
- Tools/Products: Extended NCCL/RCCL/MSCCL kernels with “prefetch_npa()” driver hooks; library-level prefetch knobs.
- Assumptions/Dependencies: Driver or ISA support for prefetch or equivalent doorbells; careful prefetch pacing to avoid pollution or bandwidth waste.
- Fused pre-translation within compute kernels
- Sectors: software/AI infrastructure, compiler toolchains (CUDA/HIP), LLM/MoE frameworks
- Action: Add a “pre-translation” phase to compute kernels (e.g., MoE gating, attention blocks) to overlap NPA→SPA translation with compute before communication begins.
- Tools/Products: Compiler passes that auto-insert pre-translation intrinsics; framework primitives (PyTorch/XLA) that fuse compute and pre-translation.
- Assumptions/Dependencies: Requires kernel/compiler changes and correctness guarantees on ordering; benefits strongest for small, latency-sensitive collectives.
- TLB warm-up workflows for inference services
- Sectors: cloud/IT, online inference (healthcare diagnostics, finance risk scoring, e-commerce recsys)
- Action: At model load/scale-up, issue low-overhead micro-collectives (or prefetch-only passes) that touch one page per destination to warm Link TLBs.
- Tools/Products: “Warm-up” hooks in inference servers (Triton, TorchServe); SRE runbooks for rolling warm-ups.
- Assumptions/Dependencies: Short warm-up allowed within SLOs; isolation to avoid interfering with co-located tenants.
- Collective-aware micro-batching to amortize translation
- Sectors: cloud inference across verticals (healthcare triage, ad ranking, conversational AI)
- Action: Slightly increase collective message sizes (e.g., token bucket coalescing) to amortize cold misses without violating latency SLOs.
- Tools/Products: Autoscaling/micro-batching controllers that adapt batch size to observed reverse-translation latency.
- Assumptions/Dependencies: Requires tight SLO monitoring; not suitable where single-request latency dominates.
- Job placement and pod residency policies that preserve warmed translations
- Sectors: cloud orchestration (Kubernetes + NCCL/RCCL), HPC schedulers (SLURM)
- Action: Keep long-running inference jobs resident on the same pod/GPU set to retain warm Link TLB state; schedule traffic to minimize cold-starts after resharding.
- Tools/Products: Scheduler plugins that use translation-latency telemetry (below).
- Assumptions/Dependencies: Requires telemetry and job/pod affinity controls; may trade off utilization.
- Reverse-translation telemetry and profiling
- Sectors: cloud/IT ops, performance engineering
- Action: Expose counters for Link TLB hit rates, page-walk latency, and per-collective cold-start penalties; integrate into observability stacks.
- Tools/Products: “Reverse Translation Profiler” CLI/SDK; Grafana dashboards; NCCL/RCCL tracing extensions.
- Assumptions/Dependencies: Driver/hardware support to surface counters; minimal overhead instrumentation.
- Design-space exploration with the extended ASTRA-sim + OMNeT++ model
- Sectors: semiconductor R&D, academic labs, cloud architecture teams
- Action: Use the paper’s simulator extensions to evaluate different Link TLB/PTW designs, page sizes, and collective patterns before hardware tape-out or procurement.
- Tools/Products: Prepackaged simulation configs for common pod sizes and collectives; CI-integrated DSE pipelines.
- Assumptions/Dependencies: Calibrated parameters for specific vendors; fidelity depends on accurate link and MMU latencies.
- Procurement/spec updates for AI pods
- Sectors: enterprise/cloud buyers, public-sector HPC
- Action: Include requirements for link-translation telemetry, software prefetch support, and right-sized Link TLBs in RFPs; avoid paying for ineffective TLB overprovisioning.
- Tools/Products: RFP templates; evaluation checklists tied to translation overhead metrics.
- Assumptions/Dependencies: Vendor cooperation; standards clarity (UALink/NVLink capabilities differ by generation).
Long-Term Applications
These applications require further hardware support, standardization, or substantial software ecosystem changes, but are guided directly by the paper’s findings on cold-miss dominance and minimal temporal locality.
- Hardware prefetch engines for Reverse Address Translation
- Sectors: semiconductor, interconnect standards (UALink/NVLink)
- Concept: Dedicated engines that predict and prefetch destination TLB entries based on collective descriptors (stride, participants), with QoS and backpressure.
- Potential Products: “Link-TLB Prefetcher” IP blocks; driver APIs to program prefetch streams.
- Assumptions/Dependencies: ISA/driver extensions; avoidance of cache pollution and security pitfalls.
- TLB entry multicast/broadcast across pods after first miss
- Sectors: semiconductor, standards bodies
- Concept: After an initial NPA→SPA resolve at a destination, multicast the translation to other destinations that will soon access adjacent pages in the same collective.
- Potential Products: Switch-assisted TLB distribution; UALink spec extensions for secure translation sharing.
- Assumptions/Dependencies: Strong isolation and authentication; coherence/invalidation semantics for revocation.
- Shadow/replicated page-table segments for collective buffers
- Sectors: OS/vendor firmware, data center platforms
- Concept: Proactively replicate page-table entries for hot collective buffers across target nodes to eliminate cold page walks at use time.
- Potential Products: “Collective buffer registration” API that seeds destination Link MMUs.
- Assumptions/Dependencies: Memory overhead and coherence management; compatibility with memory protection and multi-tenancy.
- Huge-page strategies for inter-node collective buffers
- Sectors: OS/runtime, AI frameworks
- Concept: Use 1 GB/512 MB/2 MB pages for cross-node buffers to reduce translations per transfer and cold-miss frequency.
- Potential Products: Allocators with huge-page pinning for collective regions; policy knobs in frameworks.
- Assumptions/Dependencies: Fragmentation and NUMA constraints; TLB support for large pages in Link MMU.
- Translation-aware collective algorithm synthesis and runtimes
- Sectors: software/AI infrastructure, academia
- Concept: Extend SCCL/MSCCL to co-optimize for translation working-set size (e.g., early warm-up routes, phased token routing) in addition to bandwidth/latency.
- Potential Products: “TLB-aware” collective planners that adapt to pod size and current translation state.
- Assumptions/Dependencies: Runtime access to translation telemetry; manageable synthesis complexity.
- Compiler/IR intrinsics for reverse-translation control
- Sectors: compiler toolchains (CUDA/HIP/LLVM), frameworks
- Concept: First-class IR ops to issue NPA pre-translation, query link-TLB state, and hint page strides; auto-scheduling to overlap with compute.
- Potential Products: LLVM passes, HIP/CUDA intrinsics, PyTorch graph transforms.
- Assumptions/Dependencies: Vendor ISA support; portability across hardware generations.
- TLB-aware cluster scheduling and autoscaling
- Sectors: cloud orchestration, MLOps
- Concept: Schedulers that co-locate complementary jobs to maintain warm translations, or throttle/phase small collectives to reduce cold-start collisions.
- Potential Products: Kubernetes/SLURM plugins with translation-latency signals; autoscalers that pre-warm pods before scale-out.
- Assumptions/Dependencies: Fine-grained metrics; potential utilization trade-offs.
- Security and isolation policies for destination-side translation
- Sectors: policy/standards, multi-tenant clouds
- Concept: Rate limiting, per-tenant quotas on page walks, and attestation for NPA→SPA translation to mitigate DoS and side channels.
- Potential Products: UALink/NVLink security profiles; compliance test suites including reverse-translation stress tests.
- Assumptions/Dependencies: Coordination between hardware vendors and standards bodies; measurable overhead of security checks.
- Translation-optimized pod SKUs and SLAs
- Sectors: cloud providers, managed AI services
- Concept: Offer pods with tuned Link TLB/PTW designs and software prefetch stacks, marketed for low-latency inference (e.g., MoE routing).
- Potential Products: “Low-latency collective” instance classes with SLAs on collective response time.
- Assumptions/Dependencies: Differentiated hardware and software stacks; market demand for specialized SLAs.
- Cross-fabric generalization (NVLink network, CXL, UCIe)
- Sectors: heterogeneous compute, disaggregated memory
- Concept: Apply reverse-translation insights to other memory-semantic fabrics where destination-side translations exist; unify APIs and telemetry.
- Potential Products: Fabric-agnostic “reverse translation” libraries and simulators.
- Assumptions/Dependencies: Fabric-specific semantics and page-table formats; standardization timelines.
Notes on Feasibility and Scope
- The gains are most pronounced for small, latency-sensitive collectives (e.g., MoE token exchanges), as the paper shows cold Link TLB misses can cause up to 1.4× degradation; bandwidth-bound large collectives see diminishing sensitivity.
- Findings assume streaming, stride-based access with minimal temporal locality and a typical page size of 2 MB; workloads with different access patterns may need revalidation.
- Immediate software techniques (prefetch, fused pre-translation, warm-ups) require driver/ISA hooks and careful performance tuning to prevent pollution or extra traffic.
- Hardware right-sizing and new prefetch engines apply primarily to future generations; existing systems may rely on software-only mitigations and scheduling.
Glossary
- All-to-All: A collective communication pattern where each participant sends data to every other participant. "evaluate their effect on All-to-All collective communication across varying input sizes and GPU counts."
- AllGather: A collective operation that gathers data from all processes and distributes the combined data to all of them. "collective operations like AllReduce, AllGather, AllToAll, implemented in high-performance libraries such as NCCL"
- AllReduce: A collective operation that reduces values from all processes and broadcasts the result back to all. "collective operations like AllReduce, AllGather, AllToAll, implemented in high-performance libraries such as NCCL"
- AMD Infinity Fabric Link: AMD’s high-speed interconnect for linking GPUs/CPUs, enabling low-latency, high-bandwidth communication. "AMD Infinity Fabric Link~\cite{IF24_Schieffer}"
- ASTRA-sim: A simulator for modeling large-scale training and network hierarchies, used to study collective communication performance. "Using an extended ASTRA-sim framework with Omnet++ as the network backend"
- Clos network: A multistage, non-blocking switching topology used to build scalable, high-bandwidth interconnects. "a railed, single-level Clos backbone network of high-radix Ultra Accelerator Link (UAL) switches."
- cold TLB miss: A translation lookaside buffer miss that occurs when the TLB has not yet been populated (warmed) with the needed entries. "cold TLB misses dominate latency"
- collective communication: Communication operations that involve a group of processes/GPUs coordinating data movement (e.g., AllReduce, AllGather). "Distributed ML workloads rely heavily on collective communication across multi-GPU, multi-node systems."
- Compute Unit (CU): A GPU execution block containing SIMD lanes and other execution resources. "from a CU to the NoC fabric."
- die-to-die latency: The latency incurred when signals traverse between chips/chiplets within a package or across dies. "die-to-die latency"
- FSDP (Fully Sharded Data Parallelism): A training approach that shards model parameters, gradients, and optimizer states across devices to reduce memory use. "fully sharded data parallelism (FSDP)"
- GPU pod: A group of accelerators interconnected by a scale-up fabric and managed as a unified device. "These fabrics treat entire multi-GPU pods as unified devices,"
- high-radix: Describes a switch with a large number of ports, enabling high connectivity and bandwidth. "high-radix Ultra Accelerator Link (UAL) switches"
- hit-under-miss: A cache/TLB behavior where hits can be serviced while outstanding misses are being resolved. "L2-TLB-hit-under-miss scenarios dominate,"
- HBM (High Bandwidth Memory): Stacked DRAM with very high bandwidth used as on-package memory for accelerators. "HBM & 150ns access latency"
- IOMMU: An Input-Output Memory Management Unit that translates device-issued addresses and enforces protection for DMA. "prior GPU IOMMU designs~\cite{TransFW23_Li}"
- L1 Link TLB: A small, low-latency translation cache at each UALink station on the target GPU for reverse translations. "Each UALink station has a private L1 Link TLB."
- L2 Link TLB: A larger, shared translation cache on the target GPU that serves all UALink stations. "Accesses that miss in the L1 TLB are forwarded to a shared L2 Link TLB,"
- Link MMU: The destination-side memory management unit that translates NPAs to SPAs for incoming remote requests. "the Link MMU on the target GPU"
- memory-semantic interconnect: An interconnect that supports direct load/store and atomic operations between devices, not just messages. "high-bandwidth, memory-semantic interconnects that allow accelerators to directly load, store, and perform atomic operations across nodes."
- Mixture-of-Experts (MoE): A model architecture that routes inputs to specialized expert sub-networks, often requiring All-to-All for token dispatch and gather. "In MoEs, All-to-All collectives occur twice per layer:"
- Miss Status Handling Register (MSHR): Hardware structure that tracks outstanding misses and merges new requests for the same line/page. "256-entry MSHR"
- NCCL: NVIDIA’s library for high-performance multi-GPU collective communication. "NCCL, RCCL, and oneCCL"
- Network-on-Chip (NoC): The on-chip interconnect fabric linking compute units, caches, and I/O within a GPU/SoC. "NoC fabric"
- Network Physical Address (NPA): An address that identifies memory in a remote OS domain, used for inter-node accesses. "the Network Physical Address (NPA)"
- NVLink: NVIDIA’s high-speed interconnect providing high bandwidth and memory semantics between GPUs. "NVLink network"
- NVSwitch: NVIDIA’s switch that connects multiple NVLink endpoints to build larger GPU fabrics. "NVLink/NVSwitch"
- Omnet++: A discrete-event simulation framework used to model detailed network behavior. "Omnet++ is a discrete-event, component-based simulation framework"
- oneCCL: Intel’s library for collective communication optimized for their platforms. "NCCL, RCCL, and oneCCL"
- page table walker (PTW): Hardware engine that traverses page tables to resolve address translations on TLB misses. "page table walker (PTW), which is also shared across all UALink traffic at the target GPU."
- page walk cache: A cache that stores intermediate page table entries to speed up subsequent page walks. "page walk caches"
- RDMA: Remote Direct Memory Access, enabling direct memory reads/writes across the network with minimal CPU involvement. "using RDMA~\cite{DNNOverRDMA19_Xue,OneSidedRDMA13_Mitchell,AcceleratingSecureML24_Ren,understandingRDMAMicro23_kong,SurveyRdmaNIC24_Hu}"
- RCCL: AMD’s library for high-performance multi-GPU collective communication. "NCCL, RCCL, and oneCCL"
- remote store: A memory store operation that targets memory on a remote node over the interconnect. "All schedules for MSCCLang are two-sided and use remote store instructions."
- Reverse Address Translation: Destination-side translation of NPAs to SPAs required to service remote memory operations. "We call this process Reverse Address Translation,"
- Station (UALink Station): A group of lanes forming a logical UALink interface that can be bifurcated into multiple links. "A UALink Station (or simply Station) comprises four lanes"
- System Physical Address (SPA): The physical address within a local OS domain used for servicing memory operations. "System Physical Address (SPA)"
- translation working set: The set of distinct pages/addresses whose translations are needed over a time window. "performance is primarily dictated by the translation working set of a collective,"
- two-sided (collective): A communication model requiring explicit participation from both sender and receiver for each operation. "All schedules for MSCCLang are two-sided"
- UAL (Ultra Accelerator Link): The underlying link/switch technology referenced by UALink for accelerator interconnects. "Ultra Accelerator Link (UAL) switches"
- UALink: An open, memory-semantic accelerator-to-accelerator interconnect and switching standard. "UALink supports direct load/store and atomic operations between accelerators without requiring host mediation."
- UALink Switch (ULS): The switch component in UALink networks used to build large accelerator pods. "Dedicated UALink Switches (ULS) enable the formation of âpodsâ containing up to 1,024 accelerators."
Collections
Sign up for free to add this paper to one or more collections.