---
title: 'SDDMM: Sampled Dense-Dense Matrix Multiplication'
url: https://www.emergentmind.com/topics/sampled-dense-dense-matrix-multiplication-sddmm
type: topic
---

# SDDMM: Sampled Dense-Dense Matrix Multiplication

Sampled Dense-Dense Matrix Multiplication (SDDMM) is a fundamental sparse linear algebra primitive critical for high-performance graph analytics, recommendation systems, document clustering, and deep learning workloads—particularly within the computational graphs of graph neural networks (GNNs) and collaborative filtering tasks. SDDMM computes edge-specific (i.e., nonzero-pattern-preserving) products between dense matrices, resulting in a sparse output aligned with a prescribed mask or pattern. Efficient SDDMM implementations are essential due to its recurring role as a performance bottleneck in real-world data-parallel and accelerator-based ML systems. Recent advancements have focused on hardware/software co-design, distributed memory scaling, communication minimization, and fusion with SpMM (Sparse × Dense Matrix Multiplication) for further efficiency.

## 1. Mathematical Formalism

Let $A \in \mathbb{R}^{m \times r}$ and $B \in \mathbb{R}^{n \times r}$ be input dense matrices, and $S \in \{0,1\}^{m \times n}$ (or, more generally, $S$ sparse) denote a binary/sparse mask encoding desired nonzero locations. The SDDMM computes $R \in \mathbb{R}^{m \times n}$ such that:
\[
R_{ij} = S_{ij} \cdot \langle A_{i,:}, B_{j,:} \rangle = S_{ij} \cdot \sum_{k=1}^r A_{ik} B_{jk}
\]
for all $(i,j) \in \mathrm{nnz}(S)$, and $R_{ij}=0$ elsewhere. In matrix notation:
\[
R = S \odot (A B^\top)
\]
where $\odot$ is the Hadamard (element-wise) product. In practical SDDMM implementations, explicit computation is restricted to $(i,j)$ pairs where $S_{ij} \ne 0$, avoiding unnecessary dense operations. The operator generalizes semiring-based binary and user-defined “edge functions,” encompassing inner products, attention kernels, and more [2203.07673][2011.06391][2404.19638].

## 2. Distributed-Memory and 3D Parallel Implementations

Modern SDDMM demands strong scalability across distributed-memory systems. Canonical layouts are adapted directly from communication-optimal SpMM schemes: 1.5D and 2.5D data distributions admit straightforward role reversal to SDDMM, preserving communication cost and data layout.

- **Processor Grid and Data Partitioning**: Using a $P=X\times Y\times Z$ logical grid, $S$ is block-partitioned into $X\times Y$ subblocks, each further nonzero-wise divided among $Z$ partitions. $A$ and $B$ are likewise decomposed and distributed.
- **Communication Scheme**: Sparsity-aware approaches (e.g., SpComm3D) only transmit required $A$-rows to processors (point-to-point in $Y$) and $B$-rows (in $X$), using directed zero-copy or bufferless strategies (MPI\_Type\_indexed), in contrast to dense 3D (sparsity-agnostic) approaches, which waste bandwidth and memory broadcasting all partitions regardless of utility.
- **Computation**: Each processor computes its assigned block by evaluating the dot products only for its local set of nonzeros, sharing minimal results where output reduction is necessary.
- **Performance**: For $K=120$ and $Z=4$, SpComm3D achieves geometric mean speedup of $3.3\times$ to $4.7\times$ in SDDMM runtime compared to dense 3D across various graph matrices, with $4-6\times$ network volume reduction and $2.5-10\times$ lower memory per rank. Near-ideal strong scaling is observed up to $1800$ ranks, exceeding prior layouts [2404.19638][2203.07673].

## 3. Hardware-Accelerated and Vectorized Implementations

Efficient SDDMM execution requires tightly-optimized kernels matching hardware capabilities:

- **FlashSparse on NVIDIA Tensor Cores**: Leverages swap-and-transpose MMA—by algebraic manipulation, vector length on the sparse side is halved (from 16 to 8), reducing zero padding and computation by 50%. With ME-BCRS storage (memory-efficient Blocked CSR), redundant memory and compute patterns are minimized. On H100 and RTX 4090 GPUs, FlashSparse achieves up to $3.2\times$ superior SDDMM throughput over prior state-of-the-art, with $2-5\times$ measured end-to-end application speedups [2412.11007].
- **ESIMD/SYCL on Intel GPUs**: By explicitly programming vector operations and prefetching for gather-scatter patterns, SDDMM kernels reach $80\%$ of theoretical peak for the device, up to $10$ TFLOP/s, and $4.5\times$ faster than CUDA V100 implementations on select benchmarks. Critical optimizations include register-blocking, chunked simd, occupancy balancing, and indirect load prefetching [2311.00368].
- **Cerebras CS-3 Accelerator**: Uses a 2D PE grid with dataflow routes; SDDMM is mapped as a streaming operation where each worker PE computes only needed nonzero entries, with host-to-device and device-to-host bandwidth directly proportional to dense and sparse blocks, respectively. For densities $0.01\!-\!0.1$, this approach yields $15-20\times$ CPU speedup; at extreme sparsity ($<0.001$), bandwidth overhead from padding becomes dominant [2604.27985].

## 4. Algorithmic Fusion: SDDMM+SpMM (FusedMM)

SDDMM often directly feeds into a SpMM, e.g., in message passing for GNNs or factor updates in collaborative filtering. Materializing the SDDMM output before SpMM incurs avoidable DRAM pressure and communication. The “FusedMM” kernel performs both without intermediate writes:

- **Design**: FusedMM unifies per-edge SDDMM computation and per-vertex SpMM reduction inside a single kernel, using user-supplied vector/scalar/message operations matching the embedding or neural model semantics [2011.06391].
- **Benefits**: Eliminates large temporary buffers ($O(d\,\mathrm{nnz})$), dramatically reduces memory traffic, and enables larger mini-batches and embedding sizes.
- **Distributed Fusion**: In large-scale distributed-memory settings, communication-eliding strategies—either by reusing block replication across both stages or fusing the local cyclic-shift loops into one—achieve $30\%$ reduction in communication time over naive kernel sequencing, with total runtime $1.6\times$ faster and up to $10\times$ faster than PETSc SpMM in real-world scenarios [2203.07673].
- **Portability and Platform-Specifics**: FusedMM approaches generalize efficiently across Intel, AMD, and ARM, and can be fused with sparse-communication strategies (SpComm3D) in distributed 3D layouts [2404.19638][2011.06391].

## 5. Complexity, Communication, and Memory Footprint

Key performance metrics and formulas include:

| Model / Method     | Comm Volume per Rank              | Memory per Rank                | Peak Speedup    |
|--------------------|-----------------------------------|-------------------------------|-----------------|
| Dense3D            | $\bigl(|A|+|B|\bigr)\frac{\sqrt{P/Z}-1}{P/Z}$ | $(|A|+|B|)/\sqrt{P/Z}$        | (Baseline)      |
| SpComm3D (SA)      | $\sum_i(\lambda_i-1)\frac{K}{Z} + \sum_j(\lambda'_j-1)\frac{K}{Z}$ | $\frac{\mathrm{nnz}(S)}{P} + \frac{K}{Z}(|\mathcal{I}|+|\mathcal{J}|)$ | $3-5\times$ (runtime) |
| 1.5D/2.5D RepReuse | $nr[2/c + (c-1)/p]$               | block-cyclic by design         | $30\%$ comm. reduction |
| FlashSparse (GPU)  | N/A (device-level)                | $50\%$ lower index/value storage | $2-5\times$    |

All numerical figures appear directly in the referenced papers [2404.19638][2203.07673][2412.11007][2311.00368]. Work per nonzero is typically $2K$ flop (dot product: $K$ multiply–add pairs).

## 6. Workload Integration and Empirical Impact

SDDMM is a linchpin operation in GNN pipelines (attention/inference), graph embedding (ForceAtlas2, VERSE, Force2Vec), and collaborative filtering (alternating least squares).

- **Graph Processing**: Enables edge-wise attention and message construction in GATs, AGNN, and classical layouts [2011.06391][2203.07673].
- **Performance**: On large matrices (hundreds of millions to billions of edges), state-of-the-art distributed SDDMM achieves $10\times$–$20\times$ speedup over established CPU and library baselines, both at the kernel and end-to-end application level.
- **Bandwidth and Memory Pressure**: Sparsity-aware and fused approaches alleviate I/O and DRAM bottlenecks, enabling real-time, large-scale training that is otherwise infeasible with dense or naive sparse kernels [2404.19638][2203.07673][2011.06391].

## 7. Implementation Guidelines and Future Considerations

Implementation and adoption of SDDMM must consider:

- **Data Decomposition**: 3D sparsity-aware partitioning is advised for maximal scalability; choose $(X,Y,Z)$ and owner assignment to minimize overlapping communication.
- **Kernel Design**: Explicit vectorization, prefetching for index indirection, and chunk-based register management are essential for leveraging full hardware throughput for both sparse and dense operands [2311.00368][2412.11007].
- **Pattern Regularity**: The benefit of sparsity-aware methods grows as sparsity and pattern irregularity increase; at extreme sparsity, padding and underutilization become limiting factors [2604.27985].
- **Fused Pipelines**: Whenever SDDMM output directly feeds into SpMM or similar operations, fusion is optimal to reduce memory and orchestration overhead.
- **Trade-offs**: Minimum block size (e.g., $8\times1$ tiles for FlashSparse), static pattern constraints, and preprocessing/storage format transformations balance maximum theoretical throughput with practical complexity and resource limits [2412.11007].

SDDMM remains an area of active development, with ongoing research pursuing finer-grained fusion, sparser-friendly dataflow orchestration on novel accelerators, and portable high-level APIs matched to modern hardware [2203.07673][2404.19638][2011.06391][2412.11007][2311.00368][2604.27985].

Source: https://www.emergentmind.com/topics/sampled-dense-dense-matrix-multiplication-sddmm