NVIDIA Grace Blackwell (GB10)

• NVIDIA Grace Blackwell (GB10) is a GPU-accelerated system combining advanced Grace CPU cores with Blackwell GPU architecture for deep learning and scientific computing.
• It features a unified 24 MiB L2 cache, optimized INT8/INT4 Tensor Cores, and expanded 80 GB HBM3 memory per GPU to mitigate bottlenecks.
• Innovations like Sawtooth Wavefront Reordering and fixed-point emulation achieve significant throughput gains and L2 miss reductions in mixed-precision workloads.

NVIDIA Grace Blackwell (GB10) refers to a generation of GPU-accelerated systems combining advanced Grace CPU cores with Blackwell GPU architecture. Key innovations in GB10 include large unified L2 cache, highly optimized INT8/INT4 Tensor Cores, expanded HBM3 memory subsystem (80 GB/GPU), and NVLink-4 interconnect. These features target workloads in large-scale deep learning (e.g., FlashAttention in LLMs), high-performance tensor network contractions, and mixed-precision quantum chemistry. The architectural advances of GB10 facilitate substantial increases in throughput for attention mechanisms and enable end-to-end mixed-precision computational workflows for scientific and engineering tasks.

1. Memory Architecture and L2 Reuse in GB10

GB10 implements a unified L2 cache of 24 MiB shared across 48 Streaming Multiprocessors (SMs), with critical implications for memory-bound workloads. The cache is organized in 256 B lines and 32 B sectors, providing a working set window for high-throughput access from parallel CTAs (Cooperative Thread Arrays). Each CTA can process a single tile of dimension $T \times d$ in float16, occupying $T \cdot d \cdot 2$ bytes. The tile-to-L2 mapping, along with the streaming nature of Q/K/V/O tensor access patterns, is particularly sensitive to cache size and access order. On GB10, the L1 cache is effectively bypassed for Q/K/V/O streaming, with L2 servicing nearly all requests—Nsight metrics report L1Tex hits consistently below $0.03\%$ of L2 sector traffic for typical sequence lengths ($S=32\text{K}$ to $128\text{K}$), confirming that L2 is the relevant memory bottleneck for large-scale attention operations (Zhu et al., 22 Jan 2026).

2. CuTile FlashAttention Kernel Design and Analysis

The GB10 architecture supports high-performance split-Q, fused multi-head attention kernels, both in CUDA and via the high-level CuTile abstraction. Each CTA handles a “Q” tile, loading it into shared memory, and then streams through all $N_{KV}$ K and V tiles. For each K/V tile, the kernel loads the data, computes attention scores $S_{ij}$ via WMMA, updates online softmax statistics, performs the value-weighted accumulation $P_{ij}V_j$ via WMMA, and accumulates into the output $O_i$. At the completion of processing over all K/V, $O_i$ is written back to global memory. When the working set $T \cdot d \cdot 2$0 exceeds the L2 cache capacity, repeated cold misses dominate and constrain attainable throughput. The precise L2 access model for non-causal masking is analytically described as $T \cdot d \cdot 2$1 for float16, $T \cdot d \cdot 2$2, $T \cdot d \cdot 2$3, $T \cdot d \cdot 2$4 configurations, with empirical results matching within $T \cdot d \cdot 2$5–$T \cdot d \cdot 2$6 MAPE (Zhu et al., 22 Jan 2026).

A key empirical result is that L2 “cold-miss” behavior (16S sectors for 4 tensors accessed once) persists up to a threshold ($T \cdot d \cdot 2$7), above which measured L2 misses diverge due to cache capacity and trashing effects.

3. Sawtooth Wavefront Reordering: Reducing L2 Misses

Sawtooth Wavefront Reordering is a loop schedule transformation introduced for GB10 to mitigate capacity and trashing misses in attention kernel memory behavior. The conventional cyclic K/V scanning pattern results in reuse distances as large as $T \cdot d \cdot 2$8 tiles, which can significantly exceed L2 capacity and result in severe thrashing. Sawtooth Wavefront Reordering alternates the scan direction between ascending and descending on each Q tile iteration—thus folding the reuse window for most accesses from $T \cdot d \cdot 2$9 down to $0.03\%$0. This reduction translates analytically to $0.03\%$1, offering an estimated halving of L2 miss counts for regimes where $0.03\%$2.

The approach is illustrated as follows: for Q index $0.03\%$3, incrementally alternate between

• $0.03\%$4 (“ascending”) for even $0.03\%$5
• $0.03\%$6 (“descending”) for odd $0.03\%$7

This reordering is applied transparently in both CUDA and CuTile kernels. The primary effect is to halve reuse distance, thereby doubling the chance that relevant K/V blocks remain in cache when reused by other CTAs progressing in synchronized wavefront fashion.

4. Quantitative Performance and Benchmarking

The impact of Sawtooth Wavefront Reordering is evident across multiple experimental regimes:

• For CUDA-only kernels, L2 non-compulsory misses are reduced by approximately $0.03\%$8 across batch sizes of $0.03\%$9, with throughput boosted from $S=32\text{K}$0 TFLOPS to $S=32\text{K}$1 TFLOPS (+$S=32\text{K}$2).
• For high-level CuTile implementations ($S=32\text{K}$3, $S=32\text{K}$4, $S=32\text{K}$5, $S=32\text{K}$6), the static cyclic miss count ($S=32\text{K}$7370M sectors) is reduced to $S=32\text{K}$8120M ($S=32\text{K}$9) in the sawtooth schedule.
  • Throughput rises from $128\text{K}$0 to $128\text{K}$1 TFLOPS ($128\text{K}$2) without causal masking.
  • With causal masking, throughput increases from $128\text{K}$3 to $128\text{K}$4 TFLOPS ($128\text{K}$5).
• Scalability studies show similar $128\text{K}$6–$128\text{K}$7 miss reduction and $128\text{K}$8–$128\text{K}$9 throughput gain for $N_{KV}$0 and SM counts in $N_{KV}$1.

A summary of quantitative improvements is shown below:

Kernel/Config	L2 Miss Reduction	Throughput Gain
CUDA, batch=1–8	~50%	+85% (1.3→2.4 TFLOPS)
CuTile, no masking	67%	+13% (61→69 TFLOPS)
CuTile, masking	–	+60% (41→66 TFLOPS)

These gains exploit both the shared L2 size and the wavefront scheduling of GB10, which allows CTAs to reuse cache-resident K/V blocks synchronously (Zhu et al., 22 Jan 2026).

5. Fixed-Point Emulation and Mixed-Precision Tensor Algebra

GB10’s advanced INT8/INT4 Tensor Cores enable high-throughput, mixed-precision algebraic approaches, prominently the Ozaki fixed-point emulation scheme for FP64. In this framework, each IEEE FP64 number $N_{KV}$2 is decomposed into

$N_{KV}$3

where $N_{KV}$4 are INT8 slices, $N_{KV}$5 is a shared exponent, and $N_{KV}$6 is determined by desired mantissa fidelity ($N_{KV}$7 for full FP64, $N_{KV}$8 or $N_{KV}$9 suffice for chemical accuracy). Tensor contractions, Davidson/Lanczos solvers, and SVD steps in DMRG workflows are mapped to GB10’s INT8 tensor cores via this emulation, with per-slice reduction and accumulation in INT32, final scaling, and casting to FP64 (Brower et al., 6 Oct 2025).

Practical kernel strategies include:

• Tiling for $S_{ij}$0 or $S_{ij}$1 blocks aligned with Tensor Core warp sizes.
• cuBLAS-X API enabling FP64 emulation, with controls for slice count or automatic PERFORMANT mode (adapts slicing to compute-bound subproblems).
• Overriding all main DGEMM calls with INT8-emulated kernels in DMRG workflows.

6. Application to Quantum Chemistry and Best-Practice Guidelines

Mixed-precision DMRG on GB10 achieves chemical accuracy on challenging benchmarks including FeMoco [CAS(113,76)] and CYP [CAS(63,58)], with wall-clock times and arithmetic throughput in line with or surpassing native FP64. Key results include:

• CYP: Native FP64, 3.2 h (DGX B200, 6×GB10); Emulated $S_{ij}$2, 3.0 h ($S_{ij}$3); $S_{ij}$4, 3.1 h ($S_{ij}$5)
• FeMoco: Native, 7.8 h; $S_{ij}$6, 7.5 h ($S_{ij}$7); $S_{ij}$8, 7.7 h ($S_{ij}$9)
• Arithmetic throughput in DGEMM peaks: $P_{ij}V_j$0 TFLOPS (H100), $P_{ij}V_j$1 TFLOPS (GB10 native), $P_{ij}V_j$2 TFLOPS ($P_{ij}V_j$3 emu.), $P_{ij}V_j$4 TFLOPS ($P_{ij}V_j$5 emu.)
• Accuracy: $P_{ij}V_j$6 slicing delivers $P_{ij}V_j$7 Ha for all bond dimensions, saturating variance. $P_{ij}V_j$8 achieves $P_{ij}V_j$9 Ha, stable for practical chemical applications.

Best-practice guidelines (Brower et al., 6 Oct 2025):

• Use $O_i$0 or $O_i$1 slices for tensor contractions and Krylov solvers.
• For SVD and density matrix diagonalization, prefer CPU FP64 for $O_i$2, GPU emulated $O_i$3.
• Set residual thresholds $O_i$4 for $O_i$5; relax to $O_i$6 at $O_i$7.
• Dynamic block-state selection (DBSS) for bond dimension.
• Leverage GB10’s $O_i$8 TB aggregate HBM3 memory for large active spaces; slice packing reduces memory footprint by $O_i$9.

Scaling improves for larger active spaces and higher bond dimensions, though, at very large $O_i$0, SVD and inter-GPU communication reduce the speedup from fixed-point emulation. Anticipated optimizations include INT16 Tensor Cores for SVD, hierarchical slicing (e.g., $O_i$1 for DGEMM, $O_i$2 for correction), asynchronous exponent/slice operations, and AI-driven auto-tuning of $O_i$3 and $O_i$4 during sweeps.

7. Architectural Implications and Limitations

The effectiveness of cache and tensor contraction optimizations on GB10 is fundamentally tied to its unified L2 cache and highly synchronized SM wavefront scheduling. Large, cache-centric reordering like Sawtooth Wavefront Reordering leverages this determinism for near-optimal data reuse, achieving significant reductions in L2 misses and corresponding throughput gains without hardware modification (Zhu et al., 22 Jan 2026). No hardware changes are required; the improvements come entirely from software-level scheduling. However, should tile sizes increase or architectures shift towards larger or NUMA-distributed L2, cache reuse patterns and the benefits of current loop-reordering schemes may change. Greater L1 prefetching or less predictable SM scheduling in future GPU generations will likely require adaptation of these strategies.

For tensor network workloads, the Ozaki emulation’s gain is bounded by the proportion of workload dominated by DGEMM; as memory or communication bottlenecks increase with problem scale, overall speedup is limited. Energy error plateaus at the sum of block-state truncation and fixed-point rounding; achieving micro-Hartree precision on intractably large active spaces could require additional hardware support or a fallback to full FP64 in precision-sensitive routines (Brower et al., 6 Oct 2025).

• "Sawtooth Wavefront Reordering: Enhanced CuTile FlashAttention on NVIDIA GB10" (Zhu et al., 22 Jan 2026)
• "Mixed-precision ab initio tensor network state methods adapted for NVIDIA Blackwell technology via emulated FP64 arithmetic" (Brower et al., 6 Oct 2025)