Concurrency-Hierarchy-Guided Tiling for GPUs

• The paper introduces concurrency-hierarchy-guided tiling, a method that aligns warp-sized overlapped tiling, fusion, and hybrid storage with the GPU's execution hierarchy.
• It employs warp-overlapped tiling and hybrid storage (across registers and shared memory) to reduce synchronization latency and boost occupancy, achieving measurable speedups.
• The approach integrates a GPU-aware cost model with dynamic programming to optimize global memory transactions, occupancy, and redundancy, outperforming traditional methods.

Concurrency-Hierarchy-Guided Tiling denotes a GPU execution and scheduling methodology for image-processing pipelines in which loop fusion, overlapped tiling, hybrid storage, and a GPU-aware cost model are organized around the hardware concurrency hierarchy. In the formulation reported for PolyMage-GPU, the method combines warp-overlapped tiling (OTPW), hybrid tiling across registers and shared memory, and a dynamic-programming fusion scheduler so that fused overlapped tiles are mapped to a single warp, synchronization is performed with __syncwarp, and legal fusion and tiling choices are constrained by occupancy, register pressure, shared-memory capacity, and global-memory transaction behavior (Jangda et al., 2019). The method is positioned against GPU pipeline compilation strategies used in domain-specific languages such as Halide and Forma, which divide images into overlapped tiles and fuse loops for locality, but may incur non-trivial intra-thread-block synchronization cost, a small-tile versus shared-memory trade-off, and simplified autoscheduling models that produce inefficient global-memory accesses.

1. Problem setting and motivating trade-offs

The methodology addresses three limitations identified for prior GPU execution of image-processing pipelines. First, conventional overlapped tiling and fusion strategies may require intra thread block synchronization, and the synchronization cost is described as non-trivial. Second, they must choose between small tiles, which require more overlapped computations, and large tiles, which increase shared memory access and lower occupancy. Third, autoscheduling algorithms may use simplified GPU models that can result in inefficient global memory accesses (Jangda et al., 2019).

The core design principle is to align tiling and fusion with the GPU’s concurrency hierarchy rather than treating tiles only as locality units. In this construction, a fused tile is sized so that it fits in a single warp, and the legal search space is then governed by hardware parameters such as WarpSize, maximum registers per thread, maximum shared memory per block, maximum warps per SM, maximum blocks per SM, NSM, cores/SM, and global bandwidth. This suggests that the methodology is not merely a tiling transformation; it is a scheduling discipline in which tile shape, warp decomposition, fusion partitioning, and storage placement are co-optimized under explicit resource constraints.

A recurring misconception is that image-pipeline tiling on GPUs is reducible to maximizing locality alone. The reported formulation rejects that view. It treats locality, occupancy, synchronization granularity, and memory-transaction structure as coupled variables, and it further reports that pure register blocking under-performs because of excessive recomputation, while pure shared-memory OTPW can force tile sizes so small that halo regions dominate shared-memory usage (Jangda et al., 2019).

2. Warp-overlapped tiling and warp-level synchronization

Warp-Overlapped Tiling, denoted OTPW, is the first principal component. Its goal is to fuse successive stages of an image-processing pipeline so that each fused tile, including overlap, is owned by exactly one warp. Loop fusion is performed in the polyhedral model so that producer and consumer kernels become a single loop nest; standard overlapped tiling is then applied, but tile dimensions are chosen so that the total number of threads per tile equals one warp. The overlapped tile is mapped to a warp rather than to a whole thread block, and the threads within that warp cooperatively scan the tile and exchange halo values with __syncwarp (Jangda et al., 2019).

The distinction from block-scoped synchronization is operationally significant. Warp synchronization stalls only the threads within the warp, whereas other warps on the SM can proceed. The reported motivation is therefore not only lower synchronization latency, but also reduced issue stalls at the SM level. In the experimental summary, OTPW is reported to eliminate all __syncthreads-induced issue stalls, with Unsharp Mask exhibiting approximately 21% less warp-stall due to syncs (Jangda et al., 2019).

This organization reframes overlap handling. In conventional overlapped tiling, halo exchange often implies block-wide coordination or conservative staging in shared memory. In OTPW, halo management remains explicit, but synchronization scope is reduced to the warp. A plausible implication is that the concurrency-hierarchy aspect of the method lies precisely in this narrowing of synchronization scope: the tiling unit is chosen so that the GPU’s fundamental lockstep execution unit becomes the natural synchronization domain.

3. Formal parametrization of overlapped warp tiles

The formal description is given for $D$ dimensions, with $D=2$ or $3$, indexed by $i \in \{x,y,(z)\}$. Let $T_i$ denote the core tile width in dimension $i$, and let $\Delta_i$ denote the halo width in that dimension, typically determined by the maximum producer–consumer dependence. Let WarpSize denote the hardware warp width, for example $32$, and choose a decomposition

$$W_x \times W_y \times W_z = \mathrm{WarpSize}, \qquad W_i \mid \mathrm{WarpSize}.$$

Under this decomposition, each warp-overlapped tile has total extents

$$(T_x+\Delta_x)W_x,\quad (T_y+\Delta_y)W_y,\quad (T_z+\Delta_z)W_z.$$

The number of useful points per tile is

$D=2$0

whereas the total number of points including halo is

$D=2$1

The extra, or redundant, computations per tile are therefore

$D=2$2

and the fractional overlap is

$D=2$3

These quantities are not merely descriptive. They enter directly into the scheduler’s cost function, so overlap is modeled as an explicit optimization penalty rather than as an implicit side effect of locality-oriented tiling (Jangda et al., 2019).

The parametrization also clarifies how warp geometry and tile geometry interact. The variables $D=2$4 encode a factorization of warp lanes across spatial dimensions, while $D=2$5 encode the per-lane core coverage. As a result, the methodology distinguishes between the logical shape of a warp’s covered patch and the overlap required by pipeline dependences. This separation is important because it lets the scheduler explore distinct trade-offs among halo overhead, coalescing behavior, and occupancy without collapsing them into a single tile-size parameter.

4. Hybrid tiling across registers and shared memory

Hybrid tiling is introduced to resolve the limitations of pure shared-memory OTPW and pure register blocking. The reported design splits each warp tile along one split dimension $D=2$6 into a left region of width $D=2$7, stored in shared memory, and a right region of width $D=2$8, stored in per-thread registers and communicated via warp-shuffle. The parameter $D=2$9 is tunable. For the register-stored and shared-memory subtiles, the notation is

$3$0

Shared memory used per warp is

$3$1

typically with one buffer per stage. Registers used per thread are

$3$2

plus scratch registers for intermediate temporaries (Jangda et al., 2019).

The central trade-off is explicit. Increasing $3$3 reduces shared memory and can increase occupancy, but it also reduces the register savings of recomputation versus pure shared-memory staging. The methodology therefore treats $3$4 as a schedulable parameter and sweeps it from $3$5 to $3$6. This makes hybrid tiling a continuum between shared-memory-dominant and register-dominant realizations rather than a binary choice.

The experimental summary reports that Hybrid Tiling, denoted +HT, reduces global loads by $3$7–$3$8 and raises achieved occupancy by up to approximately $3$9 relative to OTPW+Shared alone, while register-only OTPW+RT under-performs because of excessive recompute (Jangda et al., 2019). A common misconception is that moving more data into registers is uniformly beneficial on GPUs. The reported results contradict that simplification: the hybrid construction is valuable precisely because it mediates between occupancy loss from shared memory and recomputation growth from register-only blocking.

5. Cost model, DP-Fusion, and hierarchy-driven legality

The scheduler embeds fusion and tiling choices into a weighted cost function minimized by DP-Fusion, described as a dynamic-programming enumerator over fusion choices. For a fused group $i \in \{x,y,(z)\}$0, a tile $i \in \{x,y,(z)\}$1, a block shape decomposed into warps by $i \in \{x,y,(z)\}$2, and a fraction $i \in \{x,y,(z)\}$3, the model includes the following factors: $i \in \{x,y,(z)\}$4, the total number of global memory transactions for input loads, computed by coalescing all loads of one warp over the tile; theoretical occupancy $i \in \{x,y,(z)\}$5; effective bandwidth $i \in \{x,y,(z)\}$6 defined from GlobalMemBW, #[SMs](https://www.emergentmind.com/topics/streaming-multiprocessors-sms), Cores/SM, and WarpSize; memory time $i \in \{x,y,(z)\}$7; compute time

$i \in \{x,y,(z)\}$8

where $i \in \{x,y,(z)\}$9 is the measured per-point cost of stage $T_i$0; and a load-imbalance penalty $T_i$1, representing how many thread blocks cannot fill all SMs exactly (Jangda et al., 2019).

The weighted objective is reported as

$T_i$2

The constraints include

$T_i$3

Within this framework, DP-Fusion explores all valid fusion partitions of the pipeline DAG and, for each group $T_i$4, searches by brute force over small integer grids for $T_i$5, $T_i$6, and $T_i$7, then chooses the minimum-cost schedule (Jangda et al., 2019).

The “concurrency hierarchy” aspect becomes explicit in the derivation of warp decomposition from block shape. For each candidate block shape $T_i$8, DP-Fusion computes an induced warp decomposition $T_i$9 by greedily filling a 3D factorization of WarpSize across the three dimensions so that threads in one warp map to a contiguous 3D patch of the tile. This induced decomposition sets occupancy constraints through shared memory and registers, affects the cost model, and restricts legal tiling choices to those that keep $i$0 constant after scaling and fusion. This suggests that legality and profitability are jointly hierarchy-dependent rather than sequentially determined.

6. PolyMage-GPU realization and reported performance

The methodology is implemented in PolyMage-GPU, a GPU backend for PolyMage. The implementation measures each pipeline stage’s per-point time $i$1 and register usage $i$2 once by micro-benchmarking a unit tile, re-tools DP-Fusion with the GPU cost model, and searches a discrete space in which tile widths satisfy $i$3, block shapes are multiples of warp size, $i$4, and global TxSz belongs to $i$5. The weights $i$6 are obtained by leave-one-out cross-validation over six image benchmarks. Once a schedule is selected, code generation emits CUDA with warp loops for each $i$7, __syncwarp barriers, and, for hybrid tiling, unrolled register-tiling loops plus __shfl_sync calls to fetch halo pixels from neighbor threads’ registers (Jangda et al., 2019).

The experimental evaluation is reported on six standard image pipelines with high-resolution inputs: Unsharp Mask, Harris Corner, Bilateral Grid, Multi-scale Interp, Camera Pipeline, and Pyramid Blend. Relative to Halide manual schedules, PolyMage-GPU is reported as having a geometric-mean speedup of $i$8 on an NVIDIA GTX 1080Ti and $i$9 on an NVIDIA Tesla V100. Compared to other autotuners—Halide-GradientGPU, Rawat et al., and PolyMage-A—the method is reported as $\Delta_i$0–$\Delta_i$1 faster and to generate its schedule in seconds rather than hours or days (Jangda et al., 2019).

In summary terms, the methodology combines warp-sized overlapped tiles, warp-level synchronization, partial register blocking, and a GPU-aware cost model to automate fusion and tiling of image-processing pipelines. The reported outcomes are higher occupancy, fewer global transactions, and speedups over expert-tuned Halide code. A plausible implication is that the significance of Concurrency-Hierarchy-Guided Tiling lies less in any single primitive than in the coupling of synchronization scope, storage placement, and fusion search to the GPU’s native hierarchy of warp, thread block, and SM.