Matrix Two-Pass Forward Scan Algorithm

• Matrix Two-Pass Forward Scan Algorithm is a method that reformulates the prefix-sum computation as matrix multiplications on fixed-size tiles, enabling efficient local scanning.
• It employs a two-pass structure with an initial local matrix-based inclusive scan followed by a forward propagation stage to correct and merge partial results.
• The approach adapts to various hardware architectures, leveraging tensor cores and optimized memory hierarchies to achieve high throughput and energy efficiency.

“Matrix Two-Pass Forward Scan Algorithm” is an Editor’s term for a class of scan methods in which prefix-sum computation is reformulated as matrix multiplication on fixed-size tiles and then extended to larger inputs by a second forward propagation stage over tile, block, or grid summaries. In the relevant literature, the phrase is not used as a formal algorithm title. Its closest explicit realizations are the tensor-core scan formulation in “Accelerating Reduction and Scan Using Tensor Core Units” (Dakkak et al., 2018), the Ascend cube-unit scan schemes in “Parallel Scan on Ascend AI Accelerators” (Wróblewski et al., 21 May 2025), and the segmented-scan construction in “Segmented Operations using Matrix Multiplications” (Sobczyk et al., 30 Jun 2025). Across these works, the common structure is a local matrix-based inclusive scan followed by a forward carry, correction, or fix-up stage, which is the conventional substance of a two-pass or multi-pass forward scan.

1. Definition and scope

The underlying problem is the classic prefix-sum problem. One formulation defines scan on a vector $A = [a_1, a_2, \ldots, a_n]$ as

$[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$

This is an inclusive scan. In segmented form, the input consists of a value vector $\mathbf{x}$ and a boolean flag vector $\mathbf{f}$, where $f(i)=1$ means that position $i$ starts a segment and $f(0)=1$. The segmented inclusive scan is defined by

$$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$

In both cases, “forward” refers to the ordinary left-to-right dependency structure: each output depends on all prior inputs in the linear order (Dakkak et al., 2018, Sobczyk et al., 30 Jun 2025).

The phrase “matrix two-pass forward scan” most naturally refers to prefix-sum rather than to a literal two-dimensional raster scan. In the tensor-core and Ascend papers, the scan is a one-dimensional inclusive prefix sum over a linear array, but the array is packed into square matrices so that local prefixes can be computed by matrix multiplications. At larger scopes, the algorithm becomes hierarchical: local scans are computed first, partial totals are scanned second, and the resulting offsets are propagated back into local results. This suggests that the term denotes a family of matrix-engine scan schemes rather than a single standardized algorithm (Dakkak et al., 2018, Wróblewski et al., 21 May 2025).

A distinct usage appears in connected component labeling, where a two-pass forward scan over a 2-D binary image assigns provisional labels on a first raster pass and canonical labels on a second pass. That usage is domain-specific and tied to 8-connectivity, union-find equivalence maintenance, and final relabeling, rather than to the prefix-sum primitive (Gupta et al., 2016).

2. Matrix encoding of forward scan on a tile

The most explicit matrix formulation appears in the NVIDIA tensor-core work. For exposition it assumes a $16\times16$ tile, so a vector $V$ of 256 elements is stored in a matrix $[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$0 by

$[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$1

Two structured matrices then encode the scan. The upper-triangular matrix $[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$2, with ones on and above the diagonal, computes row-wise inclusive scans: $[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$3 The strict lower-triangular matrix $[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$4, with ones below the diagonal and zeros on the diagonal, computes an exclusive column scan: $[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$5 From this, the inter-row carry matrix $[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$6 is formed by reducing the rows of $[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$7 and replicating the result across columns: $[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$8 The complete tile scan is then

$[a_1,\; a_1+a_2,\; \ldots,\; \sum_{i=1}^n a_i].$9

where $\mathbf{x}$0 is the all-ones matrix. Read back in linear order, this yields the inclusive prefix sums of the original 256-element vector (Dakkak et al., 2018).

This decomposition separates two dependency types. Right multiplication by $\mathbf{x}$1 handles intra-row forward dependencies, while left multiplication by $\mathbf{x}$2, followed by multiplication with the all-ones matrix, handles inter-row carry propagation. Exclusive scan appears internally even though the externally visible result is inclusive segmented scan. The same conceptual structure reappears in the Ascend work, which defines $\mathbf{x}$3 as the upper-triangular all-ones matrix, $\mathbf{x}$4 as the strictly lower-triangular all-ones matrix, and $\mathbf{x}$5 as the all-ones square matrix, and writes

$\mathbf{x}$6

Its implementation decomposes this identity as

$\mathbf{x}$7

$\mathbf{x}$8

$\mathbf{x}$9

Here $\mathbf{f}$0 is the row-major $\mathbf{f}$1 view of a tile $\mathbf{f}$2 of length $\mathbf{f}$3, padded with zeroes if necessary (Wróblewski et al., 21 May 2025).

3. Two-pass structure across warp, block, and grid levels

At the warp level, the tensor-core formulation is a local matrix realization of scan rather than a canonical two-pass algorithm. For $\mathbf{f}$4, the pseudocode is

$\mathbf{f}$5

$\mathbf{f}$6

$\mathbf{f}$7

$\mathbf{f}$8

For a single 256-element tile, this is a three-matrix-multiply tile scan. For multiple tiles in one warp, it becomes a forward streaming scan over 256-element chunks: each tile consumes the prefix total from the previous tile via the broadcast matrix $\mathbf{f}$9 and emits a new carry from the last output element $f(i)=1$0 (Dakkak et al., 2018).

The block level is the clearest instance of a matrix two-pass forward scan in the conventional hierarchical sense. The tensor-core paper states that the algorithm “first computes the segmented scan using the warp primitives … stores the reduced values into a partials list … performs a scan on the partial list … and adds the values to the intermediate results to get the output.” The pseudocode gathers the last-row elements of per-warp tiles into a matrix

$f(i)=1$1

scans those totals by

$f(i)=1$2

and applies the fix-up by

$f(i)=1$3

This is the classic decomposition into local scan, scan of block totals, and uniform add or fix-up (Dakkak et al., 2018).

The grid level is explicitly multi-pass. The same paper describes a device-wide “scan-then-propagate strategy” with three kernel calls: a first kernel computing segmented scans and block partials, a second kernel scanning the partial values, and a third kernel uniformly adding the scanned block offsets to corresponding segments. The Ascend paper presents an analogous hierarchy in MCScan, which “consists of two phases separated by a global synchronization barrier across all cores (blocks).” In Phase I, cube units compute tile-local scans and vector units compute block reductions. In Phase II, vector units compute or use scanned block totals and propagate the carries through the local tile scans. The paper states that MCScan is similar to the Scan-Scan-Add paradigm, but differs in that it performs partial re-computation of block-level reductions during its first phase (Wróblewski et al., 21 May 2025).

4. Segmented scan and speculative correction

The MMV-RAM paper generalizes the matrix-scan idea into a theoretical framework with a scalar unit, a vector computing unit, and a matrix multiplication unit. Its segmented scan algorithm is not named as a “two-pass forward scan,” but each recursion level has a structure that is naturally interpreted as a speculative local forward pass followed by a correction or propagation pass. The main algorithm SegScan is composed of BlockSegScan, Recurse, and UpdateFirstSegment (Sobczyk et al., 30 Jun 2025).

BlockSegScan begins with speculative unsegmented scans inside each block of size $f(i)=1$4: $f(i)=1$5

$f(i)=1$6

$f(i)=1$7

The matrix multiplication with $f(i)=1$8 computes blockwise inclusive scans of values and of flags, while revertspecs removes contributions that crossed true segment boundaries. The recursive stage then contracts the problem by gathering block-end values and block-level flag summaries, recursively segmented-scans those summaries, scatters the resulting carries back, and invokes UpdateFirstSegment so that only the first segment of each block receives inter-block correction. This suggests a two-pass forward interpretation at each level: first compute speculative local scans and summaries, then propagate recursively derived prefix information back into the blocks (Sobczyk et al., 30 Jun 2025).

The paper gives both asymptotic upper and lower bounds. Its segmented scan runs in

$f(i)=1$9

steps, with work

$i$0

and it proves that any polynomial-work algorithm that uses only the vector unit requires

$i$1

steps. In this framing, matrix multiplication is not merely an implementation trick; it changes the depth complexity of segmented scan under the MMV-RAM model (Sobczyk et al., 30 Jun 2025).

5. Hardware mapping and implementation constraints

The tensor-core realization is tightly coupled to the NVIDIA WMMA programming model. NVIDIA Tensor Cores on V100 expose a warp-level API in which each Tensor Core performs

$i$2

on $i$3 internal arrays, while programmer-visible WMMA shapes are $i$4, $i$5, and $i$6. The scan paper uses the $i$7 shape. Inputs $i$8 and $i$9 must be half precision, while accumulators $f(0)=1$0 and outputs $f(0)=1$1 can be half or single precision. One warp cooperatively performs one WMMA operation on one $f(0)=1$2 tile; a thread block contains multiple warps; shared memory holds intermediate tile outputs and per-warp partial totals; and grid-level scan is orchestrated by multiple kernels (Dakkak et al., 2018).

The implementation is strongly shaped by memory-layout and API constraints. Tiles are loaded into WMMA fragments using LoadTile, structured matrices $f(0)=1$3, $f(0)=1$4, and $f(0)=1$5 must be materialized as fragments, and block-level fix-up requires gathering the last row of each tile in shared memory by a strided load into the matrix $f(0)=1$6. The paper also lists three practical API limitations: loads and stores only at fragment granularity; fragments can only be loaded from or stored to global or shared memory, not constant memory; and matrix_a, matrix_b, and accumulator fragments have different opaque internal layouts, with no cast API. The authors mitigate these constraints by exploiting knowledge of fragment layout and by constructing matrices such as the upper-triangular matrix directly in fragment registers (Dakkak et al., 2018).

The Ascend implementation is similarly hardware-specific, though with different units and memory hierarchy. It uses the cube unit for matrix multiplication, the vector unit for vector operations, and scratchpads L0A, L0B, L0C, L1, and UB. In the single-core algorithms ScanU and ScanUL1, the cube and vector units are pipelined: the cube unit computes local tile scans, and the vector unit applies the running carry partial. In MCScan, a global synchronization point SyncAll separates the local-scan phase from the propagation phase. Experimental settings include $f(0)=1$7, with the paper noting that larger $f(0)=1$8 improves performance and that $f(0)=1$9 maximizes utilization of the level-0 scratchpad memories of the cube unit (Wróblewski et al., 21 May 2025).

6. Performance regimes, applications, and limitations

The tensor-core paper reports that its scan algorithm achieves $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$0 of peak memory copy bandwidth, is up to $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$1 faster than state-of-the-art methods for small segment sizes, and decreases power consumption by up to $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$2 for scan. For segmented scan specifically, it reports a theoretical peak of $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$3 billion half-precision elements per second and measured throughput of $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$4 of ideal throughput for segment sizes between 16 and $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$5. Compared with Thrust’s inclusive_scan_by_key, the TCU implementation is up to $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$6 faster for small segment sizes; for full scan, the unoptimized grid-level implementation is comparable to CUB and faster than Thrust (Dakkak et al., 2018).

The Ascend paper reports single-core speedups ranging from $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$7 to $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$8 compared to vector-only implementations for sufficiently large input lengths, a multi-core scan reaching up to $$y(i)= \begin{cases} x(i), & \text{if } i=0 \text{ or } f(i)=1,\ y(i-1)+x(i), & \text{otherwise}. \end{cases}$$9 of theoretical memory bandwidth, and a radix sort offering up to $16\times16$0 speedup over the baseline. It also uses scan as a building block for sorting, tensor masking, top-$16\times16$1, top-$16\times16$2 sampling, weighted sampling, split, and compress. For top-$16\times16$3, it states that the algorithm executes 17 scans for each batch: 16 scans for radix sort and one additional scan required by the algorithm (Wróblewski et al., 21 May 2025).

The strongest performance regime is small regular segment sizes and tile sizes that match the native matrix engine. The tensor-core paper states that regular segmented scan is the natural fit, and that irregular segmented scan would need padding or masking. It also notes precision limitations, since Tensor Core inputs are half precision, and potential underutilization for very large segment sizes if only a few blocks are launched. The Ascend work similarly assumes row-major $16\times16$4 tiles and zero padding of incomplete final tiles. More generally, these algorithms are formulated for summation, not for an arbitrary monoid interface; conceptually they rely on associativity, but operationally they are implemented as floating-point or integer sums supported by the matrix engines (Dakkak et al., 2018, Wróblewski et al., 21 May 2025, Sobczyk et al., 30 Jun 2025).