TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout

Abstract: LLM inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe, offer higher capacity or bandwidth than HBM, but require a minimum access granularity of kilobytes. We show that these Large-Granularity Memory Systems (LGMS) can degrade the performance of tiled matrix-multiplication, which is the dominant operation in LLM inference, by up to an order of magnitude. The root cause of the slowdown is read amplification, where memory requests fetch far more data than the tile actually needs. This waste stems from a fundamental mismatch between the two-dimensional nature of compute tiles and the one-dimensional memory layout, leading to each request spilling well beyond the tile boundaries. To mitigate read amplification, we propose to use tile-major layout for LGMS. Rather than storing data as a one-dimensional strip, tile-major layout reshapes each contiguous memory block into a two-dimensional rectangle, aligning memory granularity with tile boundaries. To ease the adoption of tile-major layout on GPUs, we propose TileLens, lightweight software and hardware extensions that collectively cover major classes of GPU kernels. TileLens-SW extends GPU DSLs so that DSL-based kernels can adopt tile-major in global memory by changing only the layout descriptor. TileLens-HW extends the Tensor Memory Accelerator (TMA) for transparent tile-major support in TMA-based kernels without code changes. We evaluate TileLens on a cycle-level simulator using matrix-multiplication kernels from Qwen-3 30B and Llama-3.1 70B. Combining a tile-major layout with an adaptive hardware prefetcher, TileLens achieves near-HBM performance on HBF-augmented GPUs with a 5us HBF NAND read latency, reducing the geomean slowdown from 1.61-6.49x with conventional layouts to within 1% of an HBM-only baseline.

• The paper proposes a novel tile-major 2D memory layout that eliminates read amplification by aligning 4 KB memory blocks with 2D compute tiles.
• It integrates software and hardware extensions, enabling transparent adoption without kernel rewrites while achieving near-HBM performance.
• Empirical results demonstrate 1.61×–6.49× performance improvements and 70%–98% bandwidth utilization on large-scale models like Qwen-3 and Llama-3.

Efficient Utilization of Large-Granularity Memory Systems on GPUs via TileLens

Introduction

The increasing size of LLMs, now requiring up to terabytes for both weights and per-user state, has exposed the memory subsystem—especially HBM capacity and bandwidth—as the primary bottleneck during inference. Emerging architectures such as High-Bandwidth Flash (HBF) and RoMe augment or replace HBM with higher-capacity alternatives but enforce a large minimum access granularity (typically 4 KB), in stark contrast to the 32 B of HBM. This paper, "TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout" (2607.04031), systematically analyzes the penalty incurred by this mismatch—read amplification—on matrix multiplication workloads, then introduces TileLens, a software-hardware solution built on a two-dimensional (2D) memory tile-major layout that eliminates amplification.

Read Amplification in Tiled Matrix Multiplication

Analysis of Baseline Layout Mismatch

Matrix multiplications—matmul—is the fundamental kernel in LLM inference, especially during the decode phase when generated tokens must be mapped using the model’s weights. On GPUs, these kernels partition matrices into 2D compute tiles, each processed by a CTA. Traditional global memory layouts are row-major or column-major; under LGMS (e.g., HBF or RoMe), when each memory access fetches a 4 KB contiguous region, a significant fraction of the fetched data lies outside the 2D compute tile’s boundaries. Consequently, memory bandwidth is wasted, and CTAs are stalled awaiting oversized memory requests.

Figure 1: (a) HBF/RoMe require 4 KB granularity, vastly coarser than HBM. (b) Standard layouts overfetch, pulling mostly unusable data. (c) Tile-major layout geometrically matches compute and memory tiles, ensuring all accessed bytes are utilized.

Quantitative Impact

Microarchitectural simulation on representative LLM kernels (Qwen-3 30B, Llama-3.1 70B) reveals that conventional layouts degrade kernel execution time by up to $11\times$ (column-major) or $3.3\times$ (row-major) relative to HBM, especially under HBF’s longer read tail latency. Read amplification wastes bandwidth and, for shared data (row-major), induces a "straggler" effect where CTAs are serialized by the slowest outstanding request (Figure 2, Figure 3, Figure 4).

Figure 2: HBF access latencies have a long tail; CTA execution blocks on straggler requests.

Figure 4: Read amplification on hybrid HBM/HBF GPUs leads to severe underutilization and stalls, squandering effective bandwidth.

Tile-Major Layout: Geometrically Coupling Access Granularity

To address the geometric mismatch, the authors formalize a tile-major global memory layout, parameterized by explicit 2D memory tile shapes. Each 4 KB memory block is mapped to a rectangle that fits within (or divides) the compute tile boundary. This transformation is shown to eliminate read amplification when properly aligned. Moreover, the memory tile and compute tile can be decoupled; a wisely chosen memory tile shape (e.g., $64\times64$ for FP8) divides evenly into many common compute tile dimensions (Figure 5).

Figure 5: In tile-major layout, 4 KB regions are shaped as rectangles aligned with compute tiles. Offsets and index decomposition are efficiently computed in hardware.

Address translation from $(i,j)$ logical coordinates to physical addresses in tile-major layout requires only bit shifts and masks, ensuring minimal critical-path impact.

TileLens: Transparent Adoption via Software and Hardware Extensions

While tile-major addresses amplification, its parameterized nature would otherwise necessitate kernel rewrites per tile size. TileLens bridges this gap, enabling transparent tile-major adoption via two mechanisms:

• TileLens-SW: Extends GPU DSL frameworks (e.g., CUDA CUTLASS/CuTe, FlashAttention) to support arbitrary tile-major layouts in global memory by reinterpreting matrices as 4D tensors whose contiguous tiles align with physical memory blocks. Only the layout descriptor requires modification; compute logic is left unchanged.
• TileLens-HW: Extends the Tensor Memory Accelerator (TMA)—ubiquitous in NVIDIA Hopper and similar architectures—by augmenting the tile descriptor with the memory tile shape and the global matrix’s leading stride. Logical-to-physical mapping is performed on-the-fly inside the DMA engine, making most kernels, including closed-source binaries, layout-agnostic at runtime.

  Figure 6: TileLens extends TMA descriptors for tile-major mapping; software passes memory tile shape at runtime.

This design achieves comprehensive coverage: TMA-based (e.g., cuBLAS, DeepGEMM), DSL-based, and even hand-written or binary-only kernels can utilize tile-major layouts, often with zero source code change.

Figure 7: TileLens-HW modifies TMA source address logic. Most logic is reused and only bit shifts/masks are added for tiling.

System Integration for HBF and LGMS

Beyond layout issues, practical HBF adoption must resolve microarchitectural concerns:

• Data Placement: Store weights (read-only) in HBF; keep more dynamic per-user data (KV cache, activations) in HBM due to endurance constraints.
• Heterogeneous L2/Cache Management: Mixed-granularity L2—with 4 KB blocks for HBF—prevents capacity pollution and leverages intra-tile locality.
• Microsecond Latency Hiding: A dedicated stride prefetcher, aware of regular tile traversal patterns, maintains sufficient concurrency to saturate HBF’s bandwidth despite latency ($>5~\mu$s typical). Prefetched data targets both L2 and optionally an SRAM buffer.

  Figure 8: Evaluated GPU system configurations feature both HBM and HBF, along with mechanisms such as SRAM buffers for latency hiding.

Evaluation

TileLens was evaluated on cycle-accurate GPU simulation using real kernel traces from NVIDIA H200-class GPUs. Three layouts (column-major, row-major, tile-major) and several hardware options (prefetching, SRAM buffers) were compared under varying HBF/ROMe parameters.

Performance

Tile-major with adaptive prefetching achieves HBF-augmented kernel latency within $1\%$ of the HBM-only baseline (see Figure 9), a $1.61\times$–$6.49\times$ improvement over row/column-major layouts. SRAM buffering yields negligible additional benefit once a stride prefetcher is active.

Figure 9: Tile-major layout with prefetching virtually eliminates LGMS-induced slowdown across Qwen-3 and Llama-3.

Bandwidth Utilization

Tile-major consistently achieves $70\%$-$98\%$ of theoretical maximum bandwidth on HBF (Figure 10), while column-major suffers from bandwidth waste and row-major is bottlenecked by synchronization.

Figure 10: Tile-major layout restores near-peak bandwidth; row-major leaves substantial bandwidth unused due to straggler penalties.

Sensitivity Analysis

Execution time remains robust up to $3.3\times$0s HBF read latency with prefetching, but degrades at higher latencies. Crucially, the optimal 4 KB tile shape is the largest rectangle fitting within the compute tile boundary—a wider tile shape yields increased amplification and more straggler-prone outstanding requests (Figure 11).

Figure 11: Performance is a strong function of memory tile shape. Tiles fitting the compute tile boundary minimize amplification.

Application to RoMe

Although RoMe does not exhibit long access latency (DRAM-class), its 4 KB granularity still benefits from tile-major: bandwidth waste in column-major is substantial, but the straggler problem seen in HBF is absent due to shorter DRAM latency.

Figure 12: On RoMe, tile/row-major both eliminate bandwidth waste; straggler effects vanish at DRAM-class latency.

Implications, Limitations, and Future Prospects

TileLens conclusively demonstrates that the geometric mismatch between 1D layouts and 2D compute tiles is the main obstacle to leveraging LGMS for LLM inference. By introducing a minimally invasive, parameterized 2D memory layout—adoptable via both software abstraction and hardware augmentation—TileLens allows large capacity and high bandwidth memory subsystems to be efficiently harnessed without kernel reengineering.

This insight is directly applicable to any future memory technology that enforces a large access granularity or significantly coarser page size. The approach is robust to all compute kernels that operate over regular 2D domains, especially multi-dimensional tensor contraction in both dense and sparse settings.

Potential directions include: generalizing prefetching for irregular workloads, optimizing layout adaptation in the presence of multiple layers/operators, and extending transparent layout management to cross-node and distributed memory scenarios.

Conclusion

TileLens provides a systematic pathway to eliminate the performance-pathological read amplification and straggler synchronization endemic to large-granularity memory systems in GPU-accelerated LLM inference. By unifying software and hardware abstraction for tile-major layouts, it ensures both capacity and bandwidth scalability for emerging GPU accelerators without sacrificing kernel composability or legacy code compatibility. Empirical results confirm that with an appropriate microarchitectural stack, HBF and similar technologies can deliver near-HBM throughput to modern matmul workloads.