---
title: Sawtooth Wavefront Reordering on GPUs
url: https://www.emergentmind.com/topics/sawtooth-wavefront-reordering
type: topic
---

# Sawtooth Wavefront Reordering on GPUs

Sawtooth wavefront reordering is an algorithmic and scheduling technique developed to mitigate L2 cache miss rates in memory-bound GPU kernels with strong wavefront synchrony, particularly in large-tiled attention mechanisms such as FlashAttention operating on NVIDIA GB10 ("Grace Blackwell") GPUs. By modifying the access pattern of streaming data tiles during parallel execution, sawtooth wavefront reordering substantially reduces non-compulsory L2 misses and increases throughput, especially in transformer inference workloads with long sequence lengths [2601.16032].

## 1. Architectural and Algorithmic Context

The NVIDIA GB10 GPU integrates a large monolithic L2 cache (24 MiB) shared across Streaming Multiprocessors (SMs), each with a private L1 region ("L1Tex" or scratchpad). In high-performance attention kernels—such as split-Q FlashAttention—the memory access pattern exhibits sequential scans over key (K) and value (V) tiles for each query (Q) tile. For large sequence lengths (S ≳ 10⁵), the working set of K+V tiles exceeds L2 capacity, and both L1 and L2 act as pass-throughs for global (HBM/LPDDR5X) accesses, with L1 hit rate measured at <0.03% for FlashAttention [2601.16032]. This context makes improving L2 cache reuse central to scaling transformer throughput.

## 2. L2 Miss Phenomena in Cyclic Tile Scans

Standard cyclic tile scans, in which each Cooperative Thread Array (CTA) processes Q blocks and iterates through K,V tiles in identical forward order, induce high L2 miss rates due to large reuse distances and synchronized wavefronts. The access distance ($D_{\text{cyc}}$) between two uses of the same cache line scales with the tile count $T_c = \lfloor S/T \rfloor$, such that:

$$ \text{Reuse distance}~D_{\text{cyc}} = T_c $$

As all CTAs progress synchronously, simultaneous requests evict useful cache lines from L2, causing nearly all accesses to become non-compulsory misses once the per-CTA working set $W$ (bytes) sits above the effective L2/SM budget. Empirically, L2 hit rate exhibits:

$$ \text{HitRate} \approx 1 - \frac{1}{N_{\text{SM}}} $$

where $N_\text{SM}$ is the number of active SMs [2601.16032]. This synchrony also exacerbates L2 bank conflicts.

## 3. Sawtooth Wavefront Reordering Mechanism

Sawtooth wavefront reordering modifies the access trajectory across K and V tiles: it alternates the scan direction for each neighboring Q tile (per CTA). On even Q tiles, the inner scan is forward; on odd Q tiles, backward. This "sawtooth" pattern (Editor's term) spatially and temporally interleaves accesses, reducing the contention and, crucially, halving the average reuse distance relative to cyclic scan:

$$ D_{\text{saw}} \approx D_{\text{cyc}}/2 $$

Consequently, the upper bound for non-compulsory miss probability decreases:

$$ P_{\text{miss}} \leq \min(1, D/C_{\text{L2}}) $$

With $D_{\text{saw}} \approx D_{\text{cyc}}/2$, the resulting miss probability is reduced by up to 2×, directly improving effective memory bandwidth and kernel throughput [2601.16032].

### Key Implementation

Below is the schematic pseudocode for applying sawtooth reordering within each CTA:

```cpp
// i_local: Q tile index within CTA
bool forward = (i_local & 1) == 0;
int j0 = forward ? 0 : T_c−1;
int j1 = forward ? T_c : -1;
int step = forward ? 1 : -1;
for (int j = j0; j != j1; j += step) {
    load K_j, V_j;
    compute_attention(q, K_j, V_j);
}
```

In CuTile, the transformation uses "parfor 2" and reversed range idioms to generate the desired alternation:

```python
tile_seq = range(0, T_c) if i%2==0 else range(T_c-1, -1, -1)
for kv_tile in tile_seq:
    compute_tile(kv_tile)
```

## 4. Quantitative Impact on Memory Behavior and Throughput

Systematic evaluation on GB10 with split-Q FlashAttention (sequence $S=131072$, tile $T=64$, batch=8) demonstrates the impact:

| Variant            | L2 Misses (×10⁶) | Throughput Non-Causal (TFLOPS) | Throughput Causal (TFLOPS) |
|--------------------|------------------|-------------------------------|----------------------------|
| Cyclic-Static      | 370              | 61.0                          | 41.0                       |
| Sawtooth-Static    | 120              | 69.0 (+13%)                   | 66.0 (+61%)                |
| Cyclic-Tile        | 350              | 62.5                          | 42.0                       |
| Sawtooth-Tile      | 115              | 70.8 (+13%)                   | 65.6 (+56%)                |

Measured with Nsight Compute counters, sawtooth reordering achieves a 50%–67% reduction in L2 sector misses and throughput improvements up to 60% (causal mask), with a consistent ≈13% boost even for non-causal patterns [2601.16032].

## 5. Implementation Tuning and Trade-offs

Optimal deployment of sawtooth wavefront reordering requires balancing tile size $T$ and thread block layout:

- Smaller $T$ improves L2 cache reuse but increases arithmetic and scheduling overhead.
- Larger $T$ may exhaust shared memory or register file, forcing kernel splitting and diminishing both reuse and performance.
- Thread block occupancy, typically 128 threads per CTA arranged in an 8×16 layout, is tuned to match $T$ and maximize SM pipeline occupancy.

For best results, tile sizing and sawtooth directionality should correspond to on-chip SRAM and L2 cache availability, as well as workload synchrony.

## 6. Generalization, Applicability, and Limitations

Sawtooth wavefront reordering is applicable beyond GB10. Its benefits extend to other NVIDIA GPU generations (e.g., A100, H100) with similar coordinated SM wavefront execution and symmetric memory hierarchies. It is also effective in non-attention streaming kernels such as matrix multiplication and convolution, given sufficiently regular CTA progress [2601.16032].

Limitations include:

- Reduced benefit when CTA runtime variability disrupts inter-CTA synchrony.
- In CuTile, very large tiles may trigger kernel splitting (due to register/shared-memory limits), potentially interfering with intended access ordering.
- The approach is less effective if the per-CTA tile working set is significantly larger than the cache or in architectures with non-uniform memory access domains.

### Future Directions

Potential avenues for enhancement include run-time adaptation of the sawtooth pattern based on cache miss counters, dynamic tile sizing, and analytic integration with cache-reuse models to predict optimal scan orders at compile time.

## 7. Relationship to GB10 Microarchitecture and Broader Implications

Sawtooth wavefront reordering leverages the structural features and constraints of the GB10 memory system. With L2 hit latency of ~358 cycles and DRAM hit latency of ≈876 cycles, optimizing for L2 cache reuse is critical for high-performance ML workloads [2507.10789]. The technique interacts with higher-level kernel tuning, such as tile decomposition and memory staging, as discussed in microbenchmark analyses of Blackwell’s tensor core and memory subsystems [2512.02189].

A plausible implication is that as LLM and foundation model inference pushes sequence lengths higher, systematic scan-pattern reordering becomes indispensable in maximizing effective on-chip memory bandwidth and cache utility, especially given the trend toward larger, shared L2 architectures.

---
**References**  
- Sawtooth Wavefront Reordering: Enhanced CuTile FlashAttention on NVIDIA GB10 [2601.16032]  
- Microbenchmarking NVIDIA's Blackwell Architecture: An In-depth Architectural Analysis [2512.02189]  
- Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks [2507.10789]

Source: https://www.emergentmind.com/topics/sawtooth-wavefront-reordering