Adaptive Tile Grouping Strategies

• Adaptive tile grouping is a technique that dynamically adjusts tile boundaries, membership, and execution policies based on content characteristics, predicted utility, and hardware constraints.
• It unifies methods across image processing, 360° streaming, accelerator dataflows, and numerical linear algebra to improve efficiency, reduce latency, and enhance resource utilization.
• Key implementations include dynamic tiling for enhanced object detection, viewport-aware grouping for QoE in video streaming, and adaptive low-rank tiling methods for scalable matrix factorization.

Adaptive tile grouping denotes a family of strategies in which tiles are not treated as a static partition with fixed computational semantics; instead, tile membership, overlap, priority, factorization rank, scheduling policy, or dataflow role is adapted to content, predicted utility, numerical structure, or hardware constraints. In the recent literature, this idea appears in perceptual grouping over image patches, viewport- and cybersickness-aware 360° streaming, inference-time retessellation for small-object detection, tile-level watermark decoding with resource-aware scheduling, tile-granular accelerator dataflows, rendering pipelines that decouple sorting and rasterization granularities, and tile low-rank factorizations whose ranks vary across blocks and frequencies (Deng et al., 2023, Pang, 2023, Nguyen et al., 2023, Zhong et al., 2 Sep 2025, Li et al., 25 Mar 2025, Jo et al., 31 Aug 2025, Boukaram et al., 2021, Chen et al., 2024).

1. Conceptual scope and recurring design axes

A central distinction in this literature is between adaptive tile geometry and adaptive tile policy. Some methods change which tiles exist or overlap at inference time. Dynamic Tiling begins with non-overlapping tiles, identifies potentially fragmented objects near boundaries, and constructs dynamic overlapping tiles $D(i,j)$ only where boundary detections indicate fragmentation; overlap can extend up to $50\%$ into the neighboring tile in the one-sided common-edge case, and corner cases use a tile-size region centered on the corner (Nguyen et al., 2023). By contrast, QRMark keeps a regular grid of non-overlapping $l\times l$ tiles after normalizing images to $256\times256$, but adapts which tile is decoded and how tile workloads are grouped and sharded across CUDA streams and CPU threads (Zhong et al., 2 Sep 2025).

A second axis is what signal drives grouping. In 360° streaming, VATP360 groups ERP tiles by importance derived from predicted viewport tiles $T_v$ and object-related tiles $T_{\text{obj}}$, yielding four priority groups—Top, Top-Mid, Mid-low, and Low—summarized by a per-chunk weight vector $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$ (Pang, 2023). In TSCC, the grouping signal is viewport probability together with cybersickness-related variables: the selected set is $V=\{v\mid p_{v,t+1}>0\}$, and low-probability boundary rows or columns are trimmed iteratively (Lin et al., 2021). In tile low-rank factorization, grouping is driven by local numerical rank; H2OPUS-TLR uses adaptive randomized approximation to determine per-tile ranks $k_{ij}$, while the reciprocity-aware MDD method first computes a benchmark solution and then estimates an optimal tile size and numerical rank for each frequency (Boukaram et al., 2021, Chen et al., 2024).

A third axis is whether adaptivity changes semantics or only execution. TAS keeps GEMM tiles fixed by PE-array dimensions but adaptively selects input-stationary plus output-stationary or weight-stationary plus output-stationary depending on whether $M<K$ or $50\%$0 (Li et al., 25 Mar 2025). FlatAttention groups accelerator tiles into logical $50\%$1 super-tiles so that larger attention blocks can be staged across local memories and coordinated by row/column collectives, reducing HBM traffic without changing attention semantics (Zhang et al., 2 Apr 2026). GS-TG makes the same separation explicit in rendering: sorting is performed as if large tiles were used, while rasterization proceeds on the original small tiles through per-Gaussian bitmasks, and the method is described as lossless (Jo et al., 31 Aug 2025). A common misconception is therefore that adaptive tile grouping must mean variable-size tiles; in the cited work it also means adaptive membership, adaptive priority, adaptive batching, adaptive stationary choice, or adaptive reuse.

2. Iterative perceptual grouping over image tiles

In visual representation learning, adaptive tile grouping is instantiated as iterative grouping of patch tokens into latent group tokens. Perceptual Group Tokenizer starts from an image $50\%$2, splits it into small patches, embeds each patch into $50\%$3, and then repeatedly samples $50\%$4 group tokens $50\%$5, binds input tokens to these groups through attention, and uses the resulting groups as contexts to refine the input tokens. Within one grouping operation, group tokens are initialized from either a Gaussian distribution or a one-step normalizing flow, updated through attention-based aggregation from keys and values, passed through a GRU cell, and refined by grouped LayerNorm plus grouped MLP. The resulting grouping behaves as a set of soft clusters in feature space, and the model’s own summary representation is derived by attending from a learnable summary token over all group tokens, trained with a DINO-style student–teacher self-distillation objective (Deng et al., 2023).

The same framework is explicitly presented as transferable to “adaptive tile grouping” in a ViT-like backbone. In that formulation, patch embeddings are the input tiles, while group tokens act as latent adaptive tiles. Merging occurs when adjacent or semantically related tiles concentrate into one or a few group tokens; splitting occurs when one tile contributes to multiple groups, allowing later layers to separate mixed content. Iteration count $50\%$6, number of grouping heads, group-token dimension, and group count become the main control parameters. The reported ablations make the grouping aspect concrete: more grouping iterations improve linear probe performance, and 6-head grouping versus 1-head with the same total tokens yields $50\%$7 linear-probe accuracy. The model also exposes budget adaptivity at inference because the number of group tokens can be changed without retraining; for example, a base model trained with 256 group tokens reaches $50\%$8 top-1 when inferred with 256 groups, $50\%$9 with 512, and $l\times l$0 with 1024, while dropping to $l\times l$1 with only 16 groups. On ImageNet-1K linear probe, PGT$l\times l$2-B-1024 reaches $l\times l$3, and on ADE20k semantic segmentation PGT$l\times l$4-B with 1024 inference groups reaches $l\times l$5 mIoU compared with $l\times l$6 for a DINO + ViT-B/16 pipeline. This usage makes “tile grouping” an object-centric, content-adaptive alternative to quadratic self-attention.

3. Viewport-, QoE-, and cybersickness-driven tile grouping in 360° streaming

In 360° video streaming, adaptive tile grouping is closely tied to predicted utility. VATP360 uses ERP frames split into $l\times l$7 tiles, predicts future viewports through an LSTM driven by head movement trajectory and 3DCNN features from saliency maps, refines the viewport with YOLOv3 object detections, and then classifies each tile into one of four priority groups. Tiles inside the refined viewport are Top; tiles in object boxes overlapping the viewport are Top-Mid; tiles in object boxes disjoint from the viewport are Mid-low; all remaining tiles are Low. The resulting vector $l\times l$8 becomes the group-level state representation for an A3C bitrate allocator, so RL acts over four priority groups rather than 64 individual tiles. This reduces action-space and state-space complexity while preserving a content-aware distinction between critical viewport tiles, nearby context, other object regions, and background. The paper reports viewport prediction accuracies of $l\times l$9, $256\times256$0, and $256\times256$1 on its three content groups, and under 4G bandwidth VATP360 reaches a combined QoE of $256\times256$2, compared with $256\times256$3 for PARMIA and $256\times256$4 for Cluster. An ablation further reports a normalized QoE increase of $256\times256$5 for adding tile priority classification relative to the VP-s + ABR variant (Pang, 2023).

TSCC uses a different, but related, grouping logic. Each chunk is tiled into $256\times256$6 tiles with 5 quality levels, and the system predicts tile viewing probabilities $256\times256$7 from a Gaussian viewport model driven by head movement. The selected tile set is $256\times256$8, after which boundary rows or columns whose tiles all satisfy $256\times256$9 are removed iteratively. The method then performs joint quality assignment over the group $T_v$0 by dynamic programming under a bandwidth budget and augments this with cybersickness control through the Cybersickness Indicator (CI), Video Loss Indicator (VLI), and Sickness Migration Indicator (SMI). CI depends on optical flow, viewing probability, FoV shrink factor $T_v$1, and DoF activation $T_v$2; VLI uses distortion in the same weighted form; SMI adds temporal accumulation through the sickness queue $T_v$3. This turns adaptive tile grouping into a coupled spatial-temporal policy: membership in the viewport group changes with predicted head motion, while tile qualities inside the group are jointly adjusted as cybersickness accumulates (Lin et al., 2021).

Taken together, these streaming systems illustrate two closely related interpretations. VATP360 is an explicit importance-grouping framework in which tiles of equal priority share bitrate policy. TSCC is an implicit viewport-group optimization framework in which a predicted rectangular tile set is treated as the optimization domain and then internally reweighted by CI, VLI, and SMI. Both replace per-tile myopic decisions with group-level control variables that are updated every chunk.

4. Inference-time regrouping, selective overlap, and resource-aware tile scheduling

Dynamic Tiling treats adaptive tile grouping as an inference-data-centric mechanism for repairing fragmentation while controlling forward-pass count. The pipeline begins with a regular non-overlapping partition, classifies detections as Central, Near-Edge, or Near-Corner using a boundary threshold of $T_v$4 of tile width or height, and retains Central detections immediately. Boundary-adjacent detections become candidates for dynamic overlapping tiles $T_v$5: if two tiles share an edge, the dynamic tile straddles that edge and is bounded by the extent of the boundary detections, with maximum size no larger than one standard tile; if detections appear only on one side, overlap into the neighbor can reach $T_v$6 of the neighboring width or height; and if tiles meet at a corner, the dynamic tile has the same size as a standard tile and is centered symmetrically on the corner. A tile minimizer then packs multiple dynamic tiles into fewer composite inputs of base-tile size, and a large–small filtering step keeps only large detections from full-image inference and only small detections from tiles. On VisDrone with YOLOv8m, “Dynamic Tiling + FI + Tile Minimizer + TTA” reports $T_v$7 [email protected], $T_v$8 [email protected]:0.95, $T_v$9 [email protected], and $T_{\text{obj}}$0 s average inference time, compared with $T_{\text{obj}}$1, $T_{\text{obj}}$2, $T_{\text{obj}}$3, and $T_{\text{obj}}$4 s for “YOLOv8m + SAHI + FI + TTA” (Nguyen et al., 2023).

QRMark moves the same idea to a tiled watermark-detection system, but here the adaptive element is primarily execution policy rather than tile geometry. Images are cropped and resized to $T_{\text{obj}}$5, partitioned into non-overlapping tiles of size $T_{\text{obj}}$6, and a tile-trained extractor decodes a Reed–Solomon-coded signature from a selected tile; the default sampling strategy is random_grid. The paper emphasizes that only a single tile is decoded per image in the detection setup, and robustness is recovered through RS coding rather than multi-tile voting. Resource adaptivity enters through warm-up profiling of per-stage latency $T_{\text{obj}}$7 and memory $T_{\text{obj}}$8, adaptive stream allocation across preprocess, tiling, decoding, and RS correction, and a resource-aware scheduler that assigns tile tasks to streams by longest-processing-time while respecting a memory cap $T_{\text{obj}}$9, a balance slack $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$0, and a minimum mini-batch size $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$1. Tasks that would overload a stream are sharded and reassigned. End-to-end, QRMark reports an average $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$2 inference speedup over the sequential baseline and an increase in GPU utilization to $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$3 (Zhong et al., 2 Sep 2025).

These two systems capture a broader methodological split. Dynamic Tiling uses detection outputs to decide where new overlapping tiles should exist. QRMark assumes fixed tiles but adapts which tile tasks are grouped, sharded, and scheduled. Both are properly described as adaptive tile grouping, but one is geometry-sensitive and the other is throughput-sensitive.

5. Tile grouping as dataflow control in accelerators and rendering

In accelerator design, adaptive tile grouping often means choosing what a tile holds stationary or which neighboring tiles share communication and state. TAS applies this to GEMMs underlying transformer linear projections. For $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$4, $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$5, and $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$6, tiles are fixed by the PE array, but the stationary scheme is chosen adaptively from the problem dimensions: if $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$7, the controller uses input-stationary plus output-stationary; if $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$8, it uses weight-stationary plus output-stationary. The decision follows from comparing $$W_m = \{w_{\text{top}}, w_{\text{top-mid}}, w_{\text{mid-low}}, w_{\text{low}}\}$$9 and $V=\{v\mid p_{v,t+1}>0\}$0, or equivalently the sign of $V=\{v\mid p_{v,t+1}>0\}$1. The paper reports that TAS can reduce external memory access by more than $V=\{v\mid p_{v,t+1}>0\}$2 compared to traditional stationary schemes while remaining compatible with various attention optimization techniques and hardware accelerators (Li et al., 25 Mar 2025).

FlatAttention generalizes grouping to a tile-based many-PE fabric. A $V=\{v\mid p_{v,t+1}>0\}$3 set of accelerator tiles is treated as a logical super-tile that collectively stores larger $V=\{v\mid p_{v,t+1}>0\}$4, $V=\{v\mid p_{v,t+1}>0\}$5, $V=\{v\mid p_{v,t+1}>0\}$6, and $V=\{v\mid p_{v,t+1}>0\}$7 blocks than a single tile could hold in its 384 KiB local scratchpad. Group rows align with query rows and group columns align with key columns. Diagonal tiles touch HBM, loading $V=\{v\mid p_{v,t+1}>0\}$8, $V=\{v\mid p_{v,t+1}>0\}$9, and $k_{ij}$0 slices and then using hardware row/column multicast primitives to distribute them; row-wise reductions compute row maxima, softmax denominators, and final output sums. This grouping reduces HBM traffic from $k_{ij}$1 to $k_{ij}$2 when an $k_{ij}$3 group expands the effective block scale. The paper reports up to $k_{ij}$4 utilization, $k_{ij}$5 speedup over FlashAttention-3, and $k_{ij}$6 lower HBM traffic, with an average $k_{ij}$7 speedup over GH200 attention implementations on a GH200-comparable 32×32 tile configuration (Zhang et al., 2 Apr 2026). A notable caveat is over-flattening: for short sequences, very large groups shrink per-tile slices so much that matrix-engine utilization falls.

GS-TG applies yet another interpretation in 3D Gaussian Splatting. Here the problem is that small rasterization tiles reduce useless per-pixel Gaussian evaluation but cause heavy redundancy in tile-wise sorting, since neighboring tiles share most Gaussians. GS-TG therefore groups small tiles into larger groups only during sorting, producing one depth-sorted Gaussian list per group, while rasterization still operates on the original small tiles. A 64×64 group built from sixteen 16×16 tiles uses a 16-bit bitmask per Gaussian to indicate which small tiles inside the group are actually influenced. Rasterization filters the shared depth order by bitmask and then performs the same $k_{ij}$8-computation and $k_{ij}$9-blending as the baseline, which is why the method is described as lossless. The GPU and hardware results show the intended decoupling: sorting behaves like large-tile rendering, rasterization behaves like small-tile rendering, and the reported average speed-up over state-of-the-art 3D-GS accelerators is $M<K$0 (Jo et al., 31 Aug 2025). This case is instructive because grouping itself is static or predetermined; what is adaptive is the choice to decouple sorting granularity from rasterization granularity.

6. Adaptive low-rank tiling in matrix factorization and inverse problems

In numerical linear algebra, adaptive tile grouping appears as blockwise low-rank modeling with per-tile ranks and dynamic batching. H2OPUS-TLR partitions an $M<K$1 matrix into $M<K$2 tiles of size $M<K$3, stores diagonal tiles densely, and approximates each off-diagonal tile as $M<K$4 with adaptive rank $M<K$5. Adaptive randomized approximation increases rank until a tile-specific error tolerance is met, but this creates irregular workloads because different tiles converge at different iterations. The implementation addresses that by sorting tiles in a column by estimated rank, maintaining an active subset ri, removing converged tiles via getConvergedTiles, and refilling the batch through updateSubset. Sampling and projection are then executed through large non-uniform batched GEMMs whose size is governed by available workspace and the number of parallel sample buffers. This dynamic batching strategy yields over $M<K$6 TFLOP/s in double precision on the V100 GPU, and the abstract reports that a covariance matrix of size $M<K$7 can be factored to accuracy $M<K$8 in just a few seconds (Boukaram et al., 2021).

The reciprocity-aware adaptive tile low-rank MDD method extends the idea to a structured inverse problem in which the unknown Green’s function $M<K$9 must satisfy $50\%$00. Receivers are first reordered by a Hilbert space-filling curve, $50\%$01, to increase locality. For each frequency, a benchmark preconditioned solver produces an approximate dense $50\%$02, candidate tile sizes are tested, and every tile undergoes SVD-based numerical-rank estimation using the criterion $50\%$03 with $50\%$04, where $50\%$05 is the maximum singular value over all tiles at that frequency. The chosen tile size is the one that minimizes the total number of unknown parameters, or equivalently maximizes the entry count ratio. Symmetry is enforced structurally: diagonal tiles are represented either as $50\%$06 when rank-deficient or as dense preconditioned forms; sub-diagonal tiles are parameterized as $50\%$07; and super-diagonal tiles are set to their transposes. The empirical frequency dependence is pronounced: with natural ordering, at frequency index 10 the entry count ratio is below 60 and the optimal tile size is 3186, whereas with Hilbert ordering the ratio is above 80 and the optimal tile size is 295; at frequency index 190 the corresponding values are about 5 and 30 for natural ordering, and about 10 and 45 for Hilbert ordering (Chen et al., 2024).

This line of work shows a final, domain-specific interpretation of adaptive tile grouping. Tiles are not grouped by attention, viewport, or scheduling load, but by latent numerical structure revealed through geometry-aware reordering and frequency-specific benchmark analysis. The method’s stated limitation is equally revealing: because optimal tile sizes and local ranks are inferred from a benchmark solver, robustness depends on the quality of that benchmark, especially under noise and sparse sampling. That limitation suggests a broader research direction already implicit across the literature: adaptive tile grouping is most powerful when the grouping signal is informative, but the source of that signal—predictions, detections, workload profiles, sequence dimensions, or approximate dense solutions—also defines the method’s failure mode.