Multicore Wavefront Diamond Blocking

• Multicore Wavefront Diamond Blocking (MWD) is an advanced temporal-blocking technique for stencil computations that fuses diamond tiling and synchronized wavefront updates.
• It significantly reduces code balance and DRAM bandwidth usage by enhancing cache reuse and increasing arithmetic intensity, leading to substantial energy savings.
• MWD employs multi-dimensional intra-tile parallelization with cooperative thread groups, enabling efficient scaling on multicore and distributed-memory architectures.

Multicore Wavefront Diamond Blocking (MWD) is an advanced temporal-blocking technique for stencil-based numerical computations targeting multicore architectures with multilevel shared caches. MWD fuses diamond tiling for spatial-temporal locality and multicore-aware synchronized wavefront updates, enabling multi-threaded execution with shared cache block residency. This method has proven effective in addressing the memory bandwidth bottleneck fundamental to low-arithmetic-intensity stencil codes, such as those encountered in finite-difference electromagnetics, Jacobi, and variable-coefficient operators. MWD achieves significant reductions in code balance, enhances computational intensity, and offers substantial energy savings through DRAM bandwidth minimization.

1. Diamond-Wavefront Tile Geometry

MWD partitions the computational domain $$\Omega = \{0 \leq x < N_x\} \times \{0 \leq y < N_y\} \times \{0 \leq z < N_z\}$$ into extruded diamond tiles and advances updates via synchronized wavefront sweeps. Each diamond is defined by its width $D_w$ in the $y$–$z$ plane and wavefront depth $B_z$ ($N_F$ in some formulations) in the $z$ (or time) direction, with wavefront width $W_w = D_w + B_z - 1$ for radius-$R=1$ stencils. The cross-section at fixed $x$ is

$D_w$0

yielding a half-diamond extruded over $D_w$1 layers. The total $D_w$2–$D_w$3 points in one tile is

$D_w$4

(for $D_w$5), so each tile covers $D_w$6 grid-point updates.

The sweep proceeds such that dependencies from points $D_w$7 and $D_w$8 always reside “behind” the advancing wavefront, enabling correctness in concurrent updates. This geometric construction underpins both spatial and temporal reuse (Malas et al., 2015, Malas et al., 2014).

2. Multi-Dimensional Intra-Tile Parallelization and Thread Groups

Traditional single-thread diamond blocking assigns an individual tile to each thread, but MWD introduces thread groups (TGs), typically sized to match cache and core constraints, that cooperatively update one tile. Three orthogonal forms of intra-tile parallelism are exploited:

• Division along x: $D_w$9 threads handle contiguous $y$0-slabs, optimizing cacheline stride and hardware prefetch efficacy.
• Division across field components: For multi-component stencils (e.g. $y$1 and $y$2 fields in THIIM FDFD (Malas et al., 2015)), $y$3 threads are assigned subsets of these.
• Optional division in y: Partitioning the $y$4-range among $y$5 threads can alleviate load imbalance.

All multi-threaded updates within a tile synchronize on each $y$6-layer via barriers. The end-to-end process is managed by a global FIFO of ready tiles, with dependencies dynamically scheduled:

$z$9 Cooperation within TGs reduces per-thread cache requirements, increases arithmetic intensity, and enables scalable wavefront concurrency (Malas et al., 2015, Malas et al., 2014).

3. Memory Traffic, Code Balance, and Cache-Footprint Models

A key metric in high-performance stencil codes is code balance $y$7 (bytes per lattice update, LUP), governing the achievable memory-bound performance $y$8 with $y$9 as the available memory bandwidth.

MWD cache footprint models are formulated as:

• Cache size for a single extruded diamond:

$z$0

for $z$1 stencils, where $z$2 is the $z$3-tile size, $z$4 the number of domain-sized arrays, and $z$5 (Malas et al., 2014, Malas et al., 2015).

• Code balance:

$z$6

which asymptotically decreases as $z$7 increases, with a lower bound of $z$8.

MWD drives $z$9 towards its theoretical minimum, attaining up to an order-of-magnitude reduction in bytes/LUP compared to spatial blocking (e.g., $B_z$0–$B_z$1 bytes/LUP vs. $B_z$2 bytes/LUP for FDFD spatial block (Malas et al., 2015)) and similarly drastic reductions for other stencil schemes (Malas et al., 2014, Malas et al., 2014). DRAM power consumption correlates nearly linearly with code balance (Malas et al., 2014).

4. End-to-End Parallel Workflow and Distributed Implementation

MWD accommodates hybrid MPI+OpenMP parallelism by decomposing the domain along $B_z$3 (or $B_z$4), assigning sub-$B_z$5-slabs and halo layers to MPI ranks. Within each rank, OpenMP manages tiled sweeps:

$W_w = D_w + B_z - 1$0 Dynamic scheduling of diamonds via FIFO lists, local barriers within TGs, and distributed halo exchanges enable scalable distributed-memory MWD (Malas et al., 2015, Malas et al., 2014).

5. Practical Auto-Tuning for Parameter Selection

Efficient deployment of MWD requires judicious selection of tiling and thread group parameters. The practical auto-tuning strategy proceeds by:

1. Cache-fit filter: Discard $B_z$6 combinations where the predicted $B_z$7 exceeds half of L3 cache capacity.
2. Parameter grid search: For remaining options, benchmark a small set of TG shapes (e.g., $B_z$8) for update rate.
3. Cost model-guided refinement: Finalize $B_z$9 maximizing performance per memory access (minimizing $N_F$0); prioritize larger $N_F$1 where in-cache reuse can be maximized.

This three-step process identifies near-optimal configurations with only a few dozen trials, avoiding exhaustive full-space searches (Malas et al., 2015). Recommendations include maximizing $N_F$2 as feasible, setting $N_F$3 ($N_F$4) at or above the group size for concurrency, dynamic tuning of $N_F$5 to structure leading dimension access, and exploring TG sizes from one up to all available cores for balanced cache and parallel efficiency (Malas et al., 2014, Malas et al., 2014).

6. Performance, Energy, and Scalability Results

Experimental studies on Intel Haswell (18-core) and Ivy Bridge (10-core) platforms provide quantitative benchmarks:

• Pure spatial blocking saturates memory bandwidth at 6–8 cores ($N_F$6, $N_F$7 MLUP/s).
• Single-threaded wavefront-diamond (SWD) shifts saturation but thrashes L3 cache and deteriorates beyond 12 cores.
• MWD with shared cache-blocks never hits memory bandwidth limits, maintains code balance (200–400 bytes/LUP), and scales to all cores at $N_F$8 parallel efficiency.
• MWD achieves $N_F$9–$z$0 speedup versus best spatial-blocked kernels on 18 cores; DRAM bandwidth savings of 38–80% (Malas et al., 2015).
• Roofline analysis: MWD can approach cache-bound ceilings (e.g., $z$1 GLUP/s in-L3 for 7-point stencil) when $z$2 fits in cache; at large $z$3, memory traffic is no longer limiting (Malas et al., 2014).
• Energy measurements show that DRAM consumption drops near linearly with code balance; optimal energy-to-solution may not coincide with peak speed (e.g., 2WD fastest execution, 10WD lowest energy on the 7-point variable-coefficient stencil (Malas et al., 2014)).

Key empirical highlights on Intel Ivy Bridge:

Variant	GLUP/s (7pt const)	GLUP/s (7pt var)	DRAM BW savings	DRAM energy reduction
Spatial Block	1.6	1.0	Baseline	Baseline
1WD	4.3	2.55	30–68%	57→22 pJ/LUP
10WD	3.8	1.15	up to 80%	77.5→70 pJ/LUP

MWD scalability has also been demonstrated in distributed-memory setups (e.g., near-ideal scaling to 16 sockets for 2WD/5WD/10WD until surface-to-volume ratio effects intervene) (Malas et al., 2014).

7. Methodological Advantages and Limitations

MWD delivers a blend of spatial and temporal blocking, exploiting shared-cache residency and multi-threaded cooperation to decouple memory bandwidth from execution performance in bandwidth-bound stencil codes. The technique markedly reduces code balance (e.g., $z$4–$z$5 B/LUP vs. $z$6–$z$7 B/LUP baseline), saves DRAM power, and accommodates fine-grained concurrency essential for strong scaling under MPI.

However, overall speedup is ultimately capped by “in-cache” performance, barrier overhead in large TGs, and cache associativity/capacity conflicts at “edge” sizes. For small domains or many-core nodes with limited diamond concurrency, TG size must be chosen accordingly. Auto-tuning remains crucial for adapting $z$8 to specific stencil types and architectures (Malas et al., 2014, Malas et al., 2014, Malas et al., 2015).

A plausible implication is further acceleration and energy gains on future architectures with more bandwidth-starved memory subsystems, since MWD effectiveness scales with cache/memory hierarchy granularity.

---

References:

• "Optimization of an electromagnetics code with multicore wavefront diamond blocking and multi-dimensional intra-tile parallelization" (Malas et al., 2015)
• "Towards energy efficiency and maximum computational intensity for stencil algorithms using wavefront diamond temporal blocking" (Malas et al., 2014)
• "Multicore-optimized wavefront diamond blocking for optimizing stencil updates" (Malas et al., 2014)