HilbertA: GPU-Efficient Sparse Attention

• HilbertA is a 2D-aware sparse attention method that reorders image tokens along Hilbert curves to preserve spatial locality while ensuring GPU-efficient memory access.
• It partitions the reordered sequence into contiguous tiles with a sliding schedule and a shared central region to enhance local and cross-tile communication.
• Evaluated on Flux.1-dev, HilbertA achieves notable attention and end-to-end speedups while maintaining image quality comparable to or surpassing dense baselines.

HilbertA is a sparse attention mechanism for diffusion transformers that targets the specific systems problem of reconciling two-dimensional spatial locality in images with GPU-efficient memory access. It reorders image tokens along Hilbert curves to obtain a contiguous memory layout while preserving spatial neighborhoods, partitions the reordered sequence into contiguous tiles for local attention, slides those tiles across layers to propagate information globally, and augments each tile with a small central shared region to improve cross-tile communication and positional awareness. The method is implemented in Triton and evaluated on Flux.1-dev for high-resolution image generation, where it is reported to deliver substantial attention speedups while maintaining image quality comparable to or surpassing baselines (Zheng et al., 30 Sep 2025).

1. Problem setting and design objective

HilbertA addresses a recurrent limitation in sparse attention for diffusion image generation: methods that enforce two-dimensional spatial locality often incur uncoalesced memory access on GPUs. The paper frames this as a hardware-alignment problem rather than a purely algorithmic sparsity problem. HilbertA is therefore designed as a 2D-aware and GPU-efficient sparse attention mechanism whose primary objective is to preserve image-local neighborhoods without sacrificing contiguous, coalesced memory access (Zheng et al., 30 Sep 2025).

The method is introduced in the context of diffusion transformers operating on image tokens derived from a two-dimensional grid. In that setting, the central systems question is how to linearize the grid into a sequence suitable for transformer kernels while preserving locality strongly enough for image generation. HilbertA answers that question by combining a Hilbert-curve token order with a layerwise sliding tile schedule and a shared central region. A plausible implication is that the method should be understood as a co-design of traversal order, sparsity pattern, and kernel layout rather than as a sparsity mask alone.

The paper identifies several concrete contributions: Hilbert-curve-based 2D-local sparse attention with GPU alignment, a sliding tile schedule for long-range structured propagation, a static parameter-free central shared region, and a lightweight Triton-kernel-friendly implementation consisting mainly of token reordering operations and efficient indexing (Zheng et al., 30 Sep 2025).

2. Hilbert-curve token ordering

The first component of HilbertA is the reordering of image tokens along a Hilbert curve. Transformers consume sequences, whereas images are natively two-dimensional. The paper contrasts Hilbert ordering with naive linearizations such as row-major and Z-order, which either scatter spatial neighbors or fragment neighborhoods. Hilbert curves are described as space-filling fractals that map 2D grids to 1D sequences while preserving locality (Zheng et al., 30 Sep 2025).

The locality claim is supported through two metrics:

$$EAS(\pi) = \frac{1}{|E|} \sum_{(u,v)\in E} |\pi^{-1}(u) - \pi^{-1}(v)|$$

where Edge Average Stretch measures how far adjacent pixels separate in sequence order, and Geometric Distortion Error, which the paper states matches distances across scales. The reported result is that Hilbert curves have the lowest GDE and near-optimal EAS versus all tested traversals, making them suitable for preserving both spatial locality and hierarchical structure (Zheng et al., 30 Sep 2025).

In HilbertA, this ordering is not an auxiliary preprocessing step but the basis of the sparse attention layout. After reordering, neighboring image tokens tend to occupy nearby positions in memory, which enables contiguous tiles in sequence space to correspond to coherent local regions in image space. This is the key link between geometric locality and GPU efficiency.

For non-square or non-power-of-two images, the paper states that the Hilbert bijection is computed over the smallest enclosing square and masked as needed, thereby maintaining coalesced access. This extends the ordering strategy beyond the canonical square-image case without changing the fundamental systems rationale (Zheng et al., 30 Sep 2025).

3. Tile-and-slide attention structure

After Hilbert ordering, HilbertA partitions the sequence into fixed-size contiguous tiles. Within each tile, full attention is computed. The paper gives the example of 16 tiles for $1024\times 1024$ images. This local tiling is the sparse component of the method: each attention computation remains dense inside a tile, but the global pattern is sparse because interactions are limited to tile-local neighborhoods plus the shared region (Zheng et al., 30 Sep 2025).

Long-range propagation is handled by a sliding schedule across layers. If the sequence length is $N$, the tile size is $N_T$, and the sliding cycle is $L$, the per-layer tile assignment is written as

$$\mathcal{A}_i^{(\ell)} = \text{tile}\left( q_i(\ell) \right), \quad q_i(\ell) = \left\lfloor \frac{i + \ell \Delta}{N_T} \right\rfloor \mod T .$$

Under this schedule, tiles are shifted by a fixed offset $\Delta$ at every transformer layer. The stated effect is systematic receptive-field growth without expensive inter-tile data movement. After $T$ layers, the effective receptive field covers the full image:

$$|\text{ERF}_i(t)| = N_T \cdot \min\{T, t\}, \qquad C_i(t) = \min\left\{ 1, \frac{t}{T} \right\}.$$

The paper emphasizes that sliding requires only pointer or index updates and avoids repeated or uncoalesced memory access. Compared to overlapping windows or runtime block regrouping, it is described as faster and cleaner on GPU, and it is specifically intended to reduce boundary artifacts (Zheng et al., 30 Sep 2025).

A central shared region complements the tile-and-slide mechanism. A small fixed region at the image center, such as $16\times 16$ at $1024\times 1024$0, is included in every attention tile. This region acts as a relay for global context, facilitates cross-tile information flow with near-negligible compute overhead, and provides a stable anchor for RoPE-based positional encoding. The region remains fixed and parameter-free, so it does not require extra learned state or expensive reallocation (Zheng et al., 30 Sep 2025).

4. Kernel implementation and systems properties

HilbertA is implemented in Triton. The paper characterizes the implementation as lightweight: only two gathers are required, one to transform tokens from raster order to Hilbert order and one to transform them back at the output. Attention over local-tile tokens and shared-region tokens is executed in a single kernel, which avoids extra launches and supports block-wise contiguous access patterns (Zheng et al., 30 Sep 2025).

The reported kernel design computes per-batch and per-head block indices, selects contiguous query, key, and value slices for both local and shared blocks, accumulates results in place, and writes outputs efficiently. The emphasis is not on novel algebraic attention reduction but on memory layout and launch efficiency. The paper explicitly states that the real-world speedups exceed what sparsity alone would predict, which it attributes to the hardware-aligned layout (Zheng et al., 30 Sep 2025).

The overhead of token reordering is reported as less than $1024\times 1024$1 of attention time at large sizes. Because the reordering is performed only once up and once down per inference, regardless of transformer depth or generation steps, that overhead is described as amortized. This detail is central to the method’s systems profile: the permutation cost is bounded, while the benefits compound across layers (Zheng et al., 30 Sep 2025).

The paper also states that HilbertA can be inserted into DiT-style backbones by replacing dense attention with the HilbertA module, requiring only pre- and post-token reordering. Text tokens and the shared anchor region are processed simultaneously with image tiles for cross-modal generation. From-scratch training is reported as stable, with convergence and FID nearly matching the baseline (Zheng et al., 30 Sep 2025).

5. Empirical evaluation on high-resolution image generation

The empirical evaluation is reported on Flux.1-dev and compares HilbertA against Dense attention, CLEAR, and SpargeAttention at $1024\times 1024$2 and $1024\times 1024$3. The paper’s central empirical claim is that HilbertA achieves strong attention and end-to-end speedups while remaining competitive in image quality metrics (Zheng et al., 30 Sep 2025).

Resolution	Method	Selected reported metrics
$1024\times 1024$4	HilbertA (16 tiles)	94% sparsity, FID 31.3, LPIPS 56.3, CLIP-I 87.6, attention speedup $1024\times 1024$5, end-to-end $1024\times 1024$6
$1024\times 1024$7	HilbertA (16 tiles)	94% sparsity, FID 47.5, LPIPS 57.1, CLIP-I 78.2, attention speedup $1024\times 1024$8, end-to-end $1024\times 1024$9
$N$0	Dense (Flux.1-dev)	FID 30.6, LPIPS 0.0, CLIP-I 100.0, attention speedup $N$1, end-to-end $N$2

At $N$3, the table in the paper reports Dense at FID 30.6 and HilbertA at FID 31.3, with HilbertA achieving $N$4 attention speedup and $N$5 end-to-end speedup. CLEAR is reported with best attention speedup $N$6 and best end-to-end speedup $N$7, while SpargeAttention is reported as slower than Dense in both attention and end-to-end terms at this resolution (Zheng et al., 30 Sep 2025).

At $N$8, HilbertA is reported to reach $N$9 attention speedup and $N_T$0 end-to-end speedup. CLEAR is reported at $N_T$1 attention and $N_T$2 end-to-end speedup with 99% sparsity, while SpargeAttention remains below Dense in both speedup categories. The paper summarizes this by stating that HilbertA is the only method to achieve both high attention and end-to-end speedup, even at lower sparsity, due to its hardware-efficient memory layout (Zheng et al., 30 Sep 2025).

The quality interpretation is nuanced. The abstract states that HilbertA achieves image quality comparable to or surpassing baselines, and the detailed summary says quality remains very close to, or sometimes better than, CLEAR and the dense model. The tabulated FID, LPIPS, and CLIP-I values vary across methods and resolutions, so the most precise description is that the paper reports a speed-quality trade-off it considers competitive rather than uniformly dominant (Zheng et al., 30 Sep 2025).

6. Ablations, robustness, and scope

The ablation results reported in the paper focus on the sliding cycle length, shared-region size, and the role of the shared anchor in reducing tile-boundary artifacts. For sliding cycle length $N_T$3, LPIPS and CLIP-I are reported as almost unchanged for $N_T$4 or $N_T$5, while sliding itself is described as crucial for reducing tile-boundary artifacts. This ties the sliding schedule to image quality rather than only to receptive-field expansion (Zheng et al., 30 Sep 2025).

For the shared region, the reported optimal sizes are $N_T$6 for $N_T$7 resolution and $N_T$8 for $N_T$9 resolution. The paper states that a bigger shared region helps global coherence. Because the region is static and parameter-free, this improvement is obtained without adding learned routing or dynamic token selection (Zheng et al., 30 Sep 2025).

The method’s scope is broader than the specific Flux.1-dev experiments. The paper states that HilbertA can naturally extend to video diffusion models using a 3D Hilbert order integrating spatial and temporal attention. This is presented as a natural extension rather than a reported benchmarked result. The same distinction applies to arbitrary-shape support: the paper provides an implementation route through masking on the smallest enclosing square, but the principal empirical evidence remains concentrated on high-resolution image generation (Zheng et al., 30 Sep 2025).

A common misconception in sparse attention is that higher sparsity automatically yields higher speed. The paper explicitly counters this by reporting that HilbertA can outperform methods with greater nominal sparsity because its memory access pattern remains contiguous and coalesced. This places HilbertA within a systems-oriented strand of sparse attention research in which memory layout and kernel behavior are first-order concerns (Zheng et al., 30 Sep 2025).