TileLens: Optimizing LGMS Memory for LLM Inference

• TileLens is a framework of software and hardware extensions that realigns memory layout to match two-dimensional compute tiles in large-granularity memory systems.
• It introduces a tile-major layout that minimizes read amplification by reshaping 4 KB memory granules into data blocks that align with GPU GEMM kernels.
• By combining DSL-based kernels, TMA remapping, and an adaptive prefetcher, TileLens achieves near-HBM performance despite high NAND read latencies.

Searching arXiv for the cited TileLens paper and closely related tile-based systems for accurate citations. arxiv_search.query({"search_query":"id:(Ju et al., 4 Jul 2026) OR ti:TileLens Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout","start":0,"max_results":5}) arxiv_search({"query":"id:(Ju et al., 4 Jul 2026)","max_results":5}) TileLens is a set of lightweight software and hardware extensions for efficiently using Large-Granularity Memory Systems (LGMS) in GPU-based LLM inference. It addresses a mismatch between two-dimensional GEMM compute tiles and one-dimensional kilobyte-granularity memory layouts in systems such as High-Bandwidth Flash (HBF) and RoMe, where conventional row-major or column-major layouts can degrade tiled matrix-multiplication performance by up to an order of magnitude through read amplification. TileLens proposes tile-major layout, TileLens-SW for DSL-based kernels, and TileLens-HW for transparent TMA-based kernels; combined with an adaptive hardware prefetcher, the design reduces the geomean slowdown from 1.61–6.49x with conventional layouts to within 1% of an HBM-only baseline at a 5us HBF NAND read latency (Ju et al., 4 Jul 2026).

1. Problem setting and motivation

TileLens is motivated by the observation that LGMS expose a minimum transferable unit on the external data bus at kilobyte scale, such as 4 KB, whereas HBM operates at 32–64 B sector granularity. In the paper’s terminology, HBF offers bandwidth comparable to HBM and capacity that is 8–16× higher, but its access granularity is in kilobytes and its page read latency is microseconds; RoMe similarly increases the effective granularity of HBM from 32 B to 4 KB to raise bandwidth by approximately 12.5%. These properties are favorable for capacity or bandwidth, but they interact poorly with tiled matrix multiplication, which dominates LLM decode (Ju et al., 4 Jul 2026).

The central difficulty is that GPU GEMM kernels fetch rectangular tiles from global memory, while LGMS return aligned one-dimensional strips. The data notes that compute tiles are two-dimensional rectangles with typical contiguous strip widths of 32–512 B, whereas an LGMS request returns a 1D 4 KB-aligned strip. Under conventional layouts, each request therefore spills well beyond the tile boundary, and most fetched bytes are not consumed by the current tile. The paper identifies this effect as read amplification.

The consequences differ by layout. With FP16 and a row slice width of 64 elements, the useful slice is 128 B, so a 4 KB fetch amplifies reads by approximately $4096/128=32\times$ per row. When weights are column-major, tiles fetched by CTAs are disjoint, so amplified bytes are promptly discarded and effective bandwidth can collapse by up to an order of magnitude. With row-major, the amplified data may be reused across CTAs, but shared outstanding requests induce straggler stalls because CTAs wait for the slowest 4 KB request before progressing.

2. Read amplification analysis

The paper formalizes the mismatch using a dense matrix with element size $s$ bytes, leading dimension $ld$ elements, and LGMS granularity $G$ bytes. For a rectangular compute tile spanning rows $i \in [i_0, i_0+H-1]$ and columns $j \in [j_0, j_0+W-1]$, the conventional row-major address function is

$$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$

If the first element of a row slice has byte address $\mathrm{off}(i)$, and $\Delta_i = \mathrm{off}(i) \bmod G$, then the number of $G$-sized granules touched by that row slice is

$s$0

The total fetched bytes over $s$1 rows is $s$2, while useful bytes are $s$3. The read-amplification ratio is therefore

$s$4

Several asymptotic regimes are explicit in the paper. When $s$5 and each row slice lies within a single granule, $s$6. With random alignment, the expected amplification becomes approximately $s$7. For larger slices with $s$8, $s$9, the expected number of blocks is $ld$0, and the amplification correspondingly approaches 1 only when slices are very wide. The analysis also notes that if $ld$1, the alignment pattern is constant across rows; otherwise, boundary crossings vary row by row. The same reasoning applies to column-major layout after swapping the roles of rows and columns (Ju et al., 4 Jul 2026).

This analytical framing is important because it attributes slowdown to a structural incompatibility between memory granularity and the geometry of compute tiles, rather than to a deficiency in tiling itself. A plausible implication is that remedies confined to scheduling or prefetch timing cannot eliminate the underlying overfetch if the physical layout remains misaligned with tile boundaries.

3. Tile-major layout

TileLens addresses the mismatch by replacing conventional one-dimensional layout with tile-major layout. In this scheme, each contiguous memory block of size $ld$2 is reshaped into a two-dimensional rectangle aligned to compute tile boundaries. A memory tile of dimensions $ld$3 elements stores $ld$4 bytes contiguously, and the matrix is laid out as a grid of such tiles in global memory (Ju et al., 4 Jul 2026).

For a matrix of size $ld$5, with memory-tile dimensions $ld$6, the tile indices are $ld$7 and $ld$8, while the intra-tile indices are $ld$9 and $G$0. If $G$1 is the number of tiles per row, the tile-major address mapping is

$G$2

The design objective is to choose $G$3, or an integer multiple or fraction that still fits naturally within the compute tile. If compute tiles are larger than one granule, the paper stores them as multiple granule-sized subtiles; for example, a FP16 $G$4 compute tile occupies 8 KB and is stored as two $G$5 memory tiles. If the natural tile is smaller than one granule, adjacent memory tiles can be coalesced as long as the coalesced region remains within the compute-tile footprint.

The paper’s worked example makes the effect explicit. For a $G$6 tile with $G$7 B and $G$8 KB, conventional layouts fetch $G$9 granule requests, totaling $i \in [i_0, i_0+H-1]$0 KB, to use only $i \in [i_0, i_0+H-1]$1 KB, so $i \in [i_0, i_0+H-1]$2. Under tile-major, the same compute tile is represented as two $i \in [i_0, i_0+H-1]$3 memory tiles, requiring exactly two granule requests, fetching $i \in [i_0, i_0+H-1]$4 KB and using $i \in [i_0, i_0+H-1]$5 KB, so $i \in [i_0, i_0+H-1]$6.

4. TileLens architecture

TileLens comprises a software path for DSL-based kernels, a hardware path for TMA-based kernels, and an adaptive stride prefetcher. Together, these mechanisms are intended to make tile-major practical without forcing wholesale kernel rewrites (Ju et al., 4 Jul 2026).

Component	Mechanism	Kernel coverage
TileLens-SW	Changes only the layout descriptor to expose tile-major through a 4D tensor view	CuTe/CUTLASS, FlashAttention
TileLens-HW	Extends TMA source address computation for transparent tile-major remapping	cuBLAS, DeepGEMM
Adaptive prefetcher	Issues K-stride prefetches to hide microsecond-scale HBF latency	HBF-augmented tiled GEMM

TileLens-SW extends GPU DSLs so that a tiled 2D matrix is represented as a 4D tensor separating intra-tile and inter-tile coordinates. For column-major weights of size $i \in [i_0, i_0+H-1]$7, element size $i \in [i_0, i_0+H-1]$8, and memory tile $i \in [i_0, i_0+H-1]$9 with $j \in [j_0, j_0+W-1]$0, the descriptor is given as

$j \in [j_0, j_0+W-1]$1

Runtime mapping converts $j \in [j_0, j_0+W-1]$2 to $j \in [j_0, j_0+W-1]$3 via $j \in [j_0, j_0+W-1]$4, $j \in [j_0, j_0+W-1]$5, $j \in [j_0, j_0+W-1]$6, and $j \in [j_0, j_0+W-1]$7. The paper states that CUTLASS/CuTe kernels and FlashAttention can then fetch tiles in tile-major without changing compute code.

TileLens-HW targets kernels that rely on NVIDIA’s Tensor Memory Accelerator. The extension adds memory-tile dimensions $j \in [j_0, j_0+W-1]$8 and the leading stride $j \in [j_0, j_0+W-1]$9 to the descriptor, then remaps ordinary logical coordinates to tile-major addresses internally. For aligned matrices, the tile base address is

$$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$0

Within a TMA tile, the dimension that normally steps by stride $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$1 is split into nested counters, replacing $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$2 with $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$3 and replacing the tile extent $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$4 with $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$5. When $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$6, meaning that the memory tile is wider than the TMA tile in the contiguous dimension, TileLens-HW applies a bit permutation on the 12-bit LGMS-granule offset. The hardware overhead is reported as approximately 3–4 K gates, with an address-generation overhead of approximately 5–7 cycles that is fully hidden in overlapped compute.

The adaptive prefetcher is orthogonal to layout transformation. Even when $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$7, HBF’s microsecond page latency requires enough in-flight requests to saturate the bus. The prefetcher therefore targets the deterministic $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$8-stride of tiled GEMM. With total HBF bandwidth $$\mathrm{addr}(i,j) = \mathrm{base} + (i\cdot ld + j)\cdot s.$$9, page-read latency $\mathrm{off}(i)$0, $\mathrm{off}(i)$1 SMs, $\mathrm{off}(i)$2 concurrent CTAs per SM, and $\mathrm{off}(i)$3 LGMS requests per compute tile, the number of demand requests in flight per $\mathrm{off}(i)$4-iteration wave is

$\mathrm{off}(i)$5

The prefetch degree is then chosen as

$\mathrm{off}(i)$6

with $\mathrm{off}(i)$7 used to compensate NAND plane collisions.

5. Evaluation and empirical behavior

The evaluation uses Macsim, extended with DRAM, RoMe, and HBF models, together with traces from an NVIDIA H200 obtained through an NVBit SASS tracer. The simulated GPU has 132 SMs at 2.0 GHz, 256 KB L1 per SM, and 50 MB L2; the HBM-only baseline uses 6 HBM3e stacks with 4.915 TB/s total bandwidth. HBF is modeled with 16 channels per stack, 64 pins per channel at 3200 MHz, 4 KB granularity, and NAND page read latency swept from 1–20 µs. The workloads are GEMM kernels from Qwen-3 30B and Llama-3.1 70B, using fixed BF16 compute tiles of $\mathrm{off}(i)$8 for Qwen and $\mathrm{off}(i)$9 for Llama, at batch sizes 16, 64, and 256 (Ju et al., 4 Jul 2026).

Under a 5 µs HBF NAND latency in the daisy-chain configuration, conventional layouts behave as predicted by the read-amplification model. Column-major suffers 3–10× slowdown due to bandwidth waste, while row-major reduces the waste but still slows by geomean 1.61× because of straggler stalls. By contrast, tile-major combined with the adaptive prefetcher reduces geomean slowdown to approximately 1% from HBM-only: 1.01× without the SRAM buffer and 1.00× with it. Effective bandwidth utilization reaches 70–98% of peak and in some cases matches or exceeds the HBM-only baseline, which does not use a prefetcher.

The straggler analysis clarifies why row-major remains suboptimal on HBF. For Qwen’s $\Delta_i = \mathrm{off}(i) \bmod G$0 BF16 tile, row-major requires 128 outstanding 4 KB requests per tile, whereas tile-major requires 16. Median per-request latency is similar, but the row-major tail extends to approximately 15 µs, stretching tile arrival time by approximately 1.5×. The paper therefore treats read amplification not only as wasted bandwidth but also as a mechanism that worsens tail latency and serializes CTAs through shared requests.

Sensitivity analyses further delimit the method’s operating regime. Tile-major with prefetching maintains near-HBM performance up to 10 µs NAND latency, whereas slowdown rises to 3–4× at 20 µs. Tile shape also matters: for Qwen’s $\Delta_i = \mathrm{off}(i) \bmod G$1 tile and a 4 KB memory granule, wider memory tiles such as $\Delta_i = \mathrm{off}(i) \bmod G$2 BF16, which fit entirely inside the compute tile, yield the best performance. Narrower or taller shapes that spill beyond the compute tile raise both read amplification and the number of outstanding requests. In RoMe, where latency remains DRAM-scale despite 4 KB granularity, column-major still suffers up to 10× slowdown from bandwidth waste, but row-major and tile-major both perform near the HBM-only baseline because straggler effects are negligible.

6. Limitations, scope, and broader tile-centric context

TileLens is not presented as a universal replacement for conventional memory layout. The paper states explicitly that its benefits diminish when granularity $\Delta_i = \mathrm{off}(i) \bmod G$3 is small enough that the contiguous slice width $\Delta_i = \mathrm{off}(i) \bmod G$4 already yields $\Delta_i = \mathrm{off}(i) \bmod G$5, when access patterns are irregular rather than rectangular, or when compute tiles are not multiples of the memory-tile dimensions and therefore require padding or partial-tile handling. It also notes that tile-major helps training or non-GEMM kernels only when access patterns remain tile-like (Ju et al., 4 Jul 2026).

The design is positioned against several alternatives. Pure HBM avoids the amplification problem through fine granularity but does not address capacity constraints for large models or long-context inference. Software prefetching or paging-based approaches can overlap latency, but they do not eliminate bandwidth waste when the fetched page is misaligned with the tile footprint. Compiler tiling improves compute reuse, yet it does not change the physical global-memory layout, so 4 KB strips still spill beyond tile boundaries. TileLens’s contribution is therefore specifically to align physical granularity with the two-dimensional geometry of compute tiles.

A broader context is that tile-based decomposition also appears in several distinct arXiv systems. TL compiles tile-based programs such as Triton kernels onto spatial dataflow architectures by distributing tile instances across cores and planning on-chip communication and buffering (Li et al., 17 Dec 2025). Tetris decomposes stationary video into a tile-based polyomino data model for fine-grained spatiotemporal pruning under an accuracy constraint (Kittivorawong et al., 25 May 2026). In medical imaging, one framework classifies each fixed-size $\Delta_i = \mathrm{off}(i) \bmod G$6 histopathology tile as the segmentation unit (Shakarami et al., 16 Jul 2025), while another aggregates local tile predictions for microscopy-based classification of paclitaxel exposure (Fletcher et al., 27 Oct 2025). This suggests a broader tile-centric design pattern in which the tile serves as the unit of layout, scheduling, transport, sampling, or decision aggregation.

Within that broader pattern, TileLens is distinguished by its treatment of tiles as a memory-layout primitive rather than as an application-level segmentation or sampling primitive. Its central claim is not merely that tile-wise structure is useful, but that when external memory transfer granularity becomes kilobyte-scale, the physical memory system itself must be reshaped to match the geometry of the compute tile.