Coalesced DMA Optimization

• Coalesced DMA is a technique that merges small, contiguous data transfers into a single request, reducing per-transfer latency and boosting effective throughput.
• It is applied in GPU clusters and near-memory systems using methods like FiCCO and AXI4MLIR to optimize data movement and overlap compute-communication tasks.
• The approach improves hardware utilization and performance while balancing trade-offs such as alignment constraints, burst size limits, and potential contention.

Coalesced Direct Memory Access (DMA) is a system and hardware architecture design pattern focused on efficiently grouping small, logically contiguous or compatible data transfers into larger, single DMA requests. By reducing the number of independent transfers, coalesced DMA amortizes startup latency, improves sustained bandwidth utilization, and alleviates contention between compute and data-movement operations. Modern distributed and heterogeneous computing environments—ranging from deep learning clusters leveraging GPU peer-to-peer communication to near-memory accelerators and high-throughput persistent memory systems—employ this pattern to approach hardware throughput limits otherwise unattainable with fine-grained transfer patterns.

1. Formal Definition and Core Performance Model

Coalesced DMA refers to merging multiple logically adjacent, same-destination (or compatible) data fragments into a single hardware DMA transaction. In GPU-based multicompute systems, this is implemented (e.g., using hipMemcpyDtoDAsync) such that, instead of launching $n$ DMA requests for $n$ fragments, a single coalesced DMA copy is issued for a larger, contiguous region (Pal et al., 11 Dec 2025).

The time to transfer $m$ bytes via a DMA is modeled by a Hockney-style equation: $$T_\text{DMA}(m) = \alpha_\text{DMA} + \frac{m}{B_\text{DMA}}$$ where $\alpha_\text{DMA}$ is the per-transfer launch or latency overhead, and $B_\text{DMA}$ is the sustainable bandwidth (e.g., 64 GB/s for AMD Infinity Fabric).

Fragmentation into $n$ transfers of size $\frac{M}{n}$ causes a total transfer time: $$T_\text{total\_fragmented} = n \cdot \alpha_\text{DMA} + \frac{M}{B_\text{DMA}}$$ relative to the ideal single transfer, coalescing recovers the inefficiency term: $$\Delta T_\text{fragment} = (n - 1) \alpha_\text{DMA}$$ Thus, coalescing drives $n$0, minimizing latency and maximizing link utilization.

2. Architectural and Algorithmic Implementations

GPU Compute/Communication Overlap (FiCCO)

In ML parallelization, FiCCO (Finer-Grain Compute-Communication Overlap) schedules exploit coalesced DMA to offload fine-grained peer-to-peer sharded communication onto DMA engines, merging same-destination sub-shards wherever possible. The architecture supports various communication patterns (1D/2D sharding) and fuses compute kernels where beneficial. Four key schedules control the decomposition and overlapping strategy, with coalesced DMA central to minimizing startup costs (Pal et al., 11 Dec 2025).

Accelerator–CPU Transfers (AXI4MLIR)

For custom hardware accelerator driver stacks, coalesced DMA is implemented by a compiler-level pass (e.g., in AXI4MLIR) that merges adjacent accel.send operations (targeting consecutive regions) into single, variadic send calls. The backend emits a single dma_submit covering the contiguous span, bounded by alignment and maximum burst size constraints (Haris et al., 2024).

$$T_\text{DMA}(m) = \alpha_\text{DMA} + \frac{m}{B_\text{DMA}}$$1

Near-Memory Indirect Access (AXI-Pack/SpMV)

In irregular streaming workloads such as SpMV, near-memory hardware units buffer and coalesce narrow requests (e.g., 64b words) into wide DRAM bursts (e.g., 512b lines), using a windowed scheduler to merge all accesses within a DRAM block before issuance. The AXI-Pack protocol facilitates this by defining "indirect burst" descriptors; the Request Coalescer block issues composite reads, sustaining high memory bandwidth even for non-contiguous application-level accesses (Zhang et al., 2023).

3. Quantitative Impact and Performance Results

Distributed ML Systems

FiCCO evaluation on 8× AMD MI300X systems shows:

• Non-coalesced peer-to-peer sharding can slow down performance by up to $n$1 vs. ideal.
• Non-coalesced DMA with FiCCO yields geometric mean $n$2 speedup (up to $n$3).
• Emulated 2D coalesced DMA achieves $n$4 geomean speedup (up to $n$5).
• DMA-offload (hardware-engine driven) surpasses GPU-core-driven communication by up to $n$6 (Pal et al., 11 Dec 2025).

Custom Accelerator Data Movement

In the AXI4MLIR MatMul accelerator case study, baseline core utilization is $n$7. Incorporating coalescing (alongside DMA allocation and pipelining) increases compute-core utilization up to $n$8, and standalone coalescing reduces per-tile DMA setup overhead by $n$9 (Haris et al., 2024).

Sparse-Memory Vector Multiplication (SpMV)

With a 256-window coalescing buffer, the near-memory adapter achieves $m$0 bandwidth (over $m$1 improvement relative to no coalescer), attaining $m$2 of DRAM peak. System-level SpMV speedup is $m$3, with off-chip traffic and utilization approaching ideal levels (Zhang et al., 2023).

RDMA and Persistent Memory Hashing

Continuity hashing groups all lookup positions into one segment, enabling every lookup to complete in exactly one RDMA read regardless of load factor. This leads to $m$4 throughput speedup and $m$5 latency reduction compared to level hashing or P-FaRM-KV, with fewer persistent memory writes per operation (Liu et al., 2021).

4. Design Trade-offs, Inefficiencies, and Bounds

Coalesced DMA introduces several design trade-offs:

• Alignment and Burst Size Constraints: Coalesced blocks must typically align to DMA/DRAM controller requirements (e.g., 64B or 512B). Exceeding the hardware max ($m$6) causes further splitting (Haris et al., 2024, Zhang et al., 2023).
• Latency vs. Concurrency: Over-coalescing (very large bursts) may block access to other flows or increase latency; hardware schedulers balance window size $m$7 and maximum in-flight requests (MLP).
• Decomposition and Contention Loss: In FiCCO, decomposing large kernel operations (both compute and communication) introduces decomposition inefficiency loss (DIL) and contention inefficiency loss (CIL). The optimal schedule—whether to coalesce for larger kernels (favoring low DIL) or to stagger for lower CIL—depends on operation intensity, data movement, and available hardware parallelism. Heuristic selection based on op-to-byte ratio (OTB) and predicted memory traffic selects among four FiCCO schedules to best exploit coalesced DMA (Pal et al., 11 Dec 2025).

Schedule	DIL Degree	CIL Degree	Coalesced DMA Role
uniform-fused-1D	low	high	All-to-all, largest fusable regions
hetero-fused-1D	medium	medium	Mixed: some early, some staggered
hetero-unfused-1D	high	low	Fine granularity, minimal overlap
uniform-fused-2D	lowest	medium	Largest 2D patches, high utilization

5. Protocols, System Integration, and Hardware Considerations

Implementation of coalesced DMA is dependent on integration at several architectural levels:

• Software/Compiler: Compiler passes (e.g., MLIR) generate coalesced transfer requests automatically during code emission.
• Bus Protocols: Standards like AXI4MLIR and AXI-Pack provide protocol-level support for variadic and indirect bursts, with sideband metadata for mapping logical to physical accesses (Haris et al., 2024, Zhang et al., 2023).
• Hardware Schedulers: Request Coalescers implement multi-entry windows and metadata queues, with tight design cost trade-offs (e.g., 27 kB SRAM and $m$8 mm² area for a 256-entry coalescer at 1 GHz) (Zhang et al., 2023).
• Atomic Transactionality: Coalesced RDMA designs for consistency (e.g., continuity hashing) place atomic indicator words at the head of each coalesced region to provide log-free, crash-consistent state transitions with a single atomic operation per transfer (Liu et al., 2021).

6. Broader Applications and Measured System-Level Effects

Coalesced DMA is a central optimization across a range of system domains:

• In distributed model parallelism, coalescing communication shards unlocks performance on multi-GPU/accelerator clusters by enabling higher compute-communication overlap and improved steady-state kernel throughput (Pal et al., 11 Dec 2025).
• For sparse, irregular applications, a hardware coalescer with effective bandwidth utilization ($m$9) recovers near-theoretical DRAM throughput, directly accelerating algorithms such as SpMV by $$T_\text{DMA}(m) = \alpha_\text{DMA} + \frac{m}{B_\text{DMA}}$$0 end-to-end (Zhang et al., 2023).
• In persistent memory systems, one-round-trip, one-store coalesced protocols eliminate the RDMA access amplification and logging overhead prevalent in naive designs, improving both latency and memory endurance (Liu et al., 2021).
• Compiler-generated coalesced DMA in host drivers provides seamless scalability as custom accelerator workloads expand, directly coupling with pipelining and staging to maximize arithmetic resource occupancy (Haris et al., 2024).

7. Summary and Future Outlook

The coalesced DMA pattern—grouping fine-grained, logically adjacent transfers into hardware-efficient bursts—is now widely recognized as a prerequisite for bandwidth-saturating operation in high-performance computing, storage, and accelerator systems. It eliminates per-transfer startup overhead, reduces control-path contention, and enables principled trade-offs between kernel launch efficiency and memory system utilization. As systems and workloads continue to fragment both spatially (across devices) and temporally (into finer compute/communication tiles), further advances in coalescing algorithms, hardware schedulers, and protocol-level integration will remain essential to closing the gap between effective and peak attainable performance (Pal et al., 11 Dec 2025, Liu et al., 2021, Haris et al., 2024, Zhang et al., 2023).