Hardware-Aware Tile Quantization

• The paper presents hardware-aware tile quantization schemes that tailor tile size, bit allocation, and metadata placement to specific hardware constraints.
• It details methodologies including per-tile, per-tap, and pair-wise quantization with adaptive bit allocation and outlier suppression to reduce error and overhead.
• Empirical evaluations show improved convergence, throughput, and energy efficiency across diverse hardware platforms such as NPUs, AI engines, and tensor cores.

Searching arXiv for recent work on hardware-aware tile quantization and related tile-centric mixed-precision/quantization schemes. Hardware-aware tile quantization refers, across recent work, to quantization schemes in which the basic numerical unit is a tile, group, tap, or other small block, and in which tile size, bit allocation, metadata placement, transforms, or execution flow are chosen to match hardware constraints such as memory bandwidth, on-chip storage, SIMD behavior, DMA limits, compiler tiling, or array utilization. The topic spans activation quantization for pipeline parallelism, AWQ-style weight-only inference on NPUs, outlier-aware pair encoding, Winograd-domain tap-wise quantization, low-rank 2D tiling for Mixture-of-Experts, and joint quantization-hardware search. A useful boundary case is the tile-centric mixed-precision GEMM of Zhang et al., which supports user-specified FP64/FP32 tile mixes in PaRSEC but does not provide automatic hardware-aware tile precision selection, quantization formulas, or predictive accuracy/performance models (Zhang et al., 20 Aug 2025, He et al., 2 Jun 2025, Pang et al., 9 Jun 2026, Guo et al., 2023, Andri et al., 2022, Gu et al., 10 May 2026, Wang et al., 9 Jan 2025).

1. Scope and representative forms

The literature does not present a single canonical scheme under this name; rather, it presents several families of tile-based quantization and mixed-precision methods whose hardware awareness is expressed in different ways. In some cases, the tile is a communication unit, as in tile-wise activation quantization for pipeline parallel training. In other cases, it is a microkernel or DMA unit, as in tiled weight layouts for NPUs. Elsewhere, the tile is a pair of adjacent values, a Winograd-domain tap, or a 2D block used to share low-rank factors across experts. This diversity is important because the phrase “hardware-aware” does not always mean adaptive runtime selection; it can also mean that the quantized representation is shaped to fit a specific memory system, dataflow, or execution primitive.

Work	Tile unit	Hardware-facing objective
TAH-Quant	tile-wise groups of size $p\times p$ with $G=p^2$	tune $p$, $b_{\text{high}}/b_{\text{low}}$, and $\tau$ to memory bandwidth, on-chip storage, and compute units / SIMD
TileFuse	$T_k\times T_n$ weight tiles built from 8×8 sub-blocks	eliminate large-stride DMA, co-pack metadata, and utilize the full 4×8 AIE array
OliVe	1×2 tiles	memory-aligned one-byte packing and low-overhead decode on tensor-core or systolic-array input paths
Winograd $F_4$	per-tap quantization on 6×6 Winograd-domain taps	enable integer-only inference with shift-add transform units
TileQ	2D tiles in an $M\times N$ expert grid	replace many small expert computations with two large GEMMs plus regular gather/scatter
JAQ	compiler tiles $(t_{oc},t_{ic},t_{oh},t_{ow})$ and channel subset $\Omega^l$	accelerate joint hardware/compiler search and reduce search-time memory overhead

A recurring distinction is between tile-aware execution and tile-aware quantization. Zhang et al. implement a tile-centric mixed-precision GEMM in which the user specifies a global “D:S” percentage such as 80D:20S, 50D:50S, or 20D:80S, and tiles are randomly assigned to DP or SP to meet that fraction. The PTG graph in PaRSEC is annotated so that each tile-multiply task knows whether it is a DP or SP kernel, and datatype conversion is handled at the receiving end. However, there is no runtime-driven per-tile precision decision based on error-bound estimators, hardware performance counters, or bandwidth/compute ratios, and no dynamic re-quantization or hardware probing (Zhang et al., 20 Aug 2025).

2. Quantization at tile, tap, and pair granularity

A central design choice is the granularity at which scales, zero-points, or alternate numeric formats are assigned. TAH-Quant quantizes activation tiles independently. For an activation tensor $G=p^2$0, each token’s $G=p^2$1-vector is split into non-overlapping tiles of size $G=p^2$2, or $G=p^2$3 elements, and each tile $G=p^2$4 is quantized with its own scale $G=p^2$5 and zero-point $G=p^2$6. With a $G=p^2$7-bit budget, the asymmetric uniform quantizer is

$G=p^2$8

with

$G=p^2$9

Smaller $p$0 reduces the dynamic range within each tile and therefore reduces quantization error, but increases the number of $p$1 pairs and the per-tile overhead (He et al., 2 Jun 2025).

TAH-Quant also adds token-level adaptive bit allocation. For each token, it computes the normalized magnitude histogram

$p$2

and entropy

$p$3

The allocation rule is

$p$4

All tiles of token $p$5 share the same $p$6. This construction makes the tile-level quantizer depend not only on local range but also on token-wise signal dispersion (He et al., 2 Jun 2025).

Other works define the tile differently. OliVe uses 1×2 tiles $p$7, marks entries above a threshold $p$8 as outliers, and converts the adjacent normal entry into a “victim” set to zero. Normal values are quantized with

$p$9

while outliers use a 4-bit E2M1 “abfloat” format

$b_{\text{high}}/b_{\text{low}}$0

Each 1×2 tile occupies exactly one byte, preserving 8-bit alignment and avoiding sparse indexing or variable-length records (Guo et al., 2023).

Winograd $b_{\text{high}}/b_{\text{low}}$1 quantization goes one level deeper and assigns scales per tap in the Winograd domain. For each tap index $b_{\text{high}}/b_{\text{low}}$2, the scale is chosen as

$b_{\text{high}}/b_{\text{low}}$3

and the quantized tap is

$b_{\text{high}}/b_{\text{low}}$4

After INT8 multiplication and INT32 accumulation, the result is rescaled by $b_{\text{high}}/b_{\text{low}}$5. The motivation is that different taps of $b_{\text{high}}/b_{\text{low}}$6 and $b_{\text{high}}/b_{\text{low}}$7 have very different dynamic ranges, so a single global scale is inadequate for $b_{\text{high}}/b_{\text{low}}$8 (Andri et al., 2022).

3. Hardware-aware transforms, metadata, and layouts

In many schemes, hardware awareness is carried not by an adaptive controller but by the representation itself. TAH-Quant uses a Hadamard-based outlier suppression stage with pivot swap. For a tile $b_{\text{high}}/b_{\text{low}}$9, it detects an outlier using the ratio

$\tau$0

and if $\tau$1, it swaps the dominant entry into the pivot position, applies a normalized Hadamard transform,

$\tau$2

then quantizes $\tau$3. On dequantization, it applies $\tau$4 and the inverse permutation. The transform is described as a “butterfly” network of $\tau$5 and $\tau$6 additions/subtractions, so the decision to apply it can be coupled to device support for lightweight SIMD-friendly transforms (He et al., 2 Jun 2025).

TileFuse is a more explicit example of hardware-specific design. It targets AMD XDNA2, whose compute array consists of 32 AI Engine cores arranged in a 4×8 grid, with 8 shim cores, 8 memory cores, and hardware-accelerated BF16 8×8 multiply-accumulate at up to $\tau$7 TFLOP/s per array. AWQ-style W4A16 and W8A16 use per-group scales $\tau$8 and zero-points $\tau$9, but XDNA2 natively supports only simple symmetric per-tensor or per-channel quantization with limited metadata streams. TileFuse therefore reshapes the $T_k\times T_n$0 weight matrix into $T_k\times T_n$1 tiles with

$T_k\times T_n$2

where each $T_k\times T_n$3 tile is built from 16 sub-blocks of 8×8 INT4 or INT8 elements. A column-major interleaving stores all of a column’s tiles consecutively, eliminating the large stride that would exceed the $T_k\times T_n$4-bit DMA limit when $T_k\times T_n$5, and supporting $T_k\times T_n$6. Inside each tile payload, packed INT4 data come first, then the per-column BF16 scales, then duplicated per-column INT8 zero-points for 128-byte BD alignment, so that a single BD fetch brings in both weights and $T_k\times T_n$7 (Pang et al., 9 Jun 2026).

TileFuse also fuses unpacking, dequantization, and GEMM/GEMV execution. For each 8×8 sub-block, it unpacks INT4 to INT8, converts INT8 to BF16, and applies

$T_k\times T_n$8

buffering the dequantized block locally before repeated BF16 8×8 multiply-accumulate. This avoids extra global-memory writes of dequantized BF16 weights and amortizes unpack/dequant overhead across reuse of each weight tile (Pang et al., 9 Jun 2026).

4. Scheduling, search, and dataflow

Hardware-aware tile quantization is often coupled to scheduling or search, but the degree of adaptivity varies sharply. In the PaRSEC mixed-precision GEMM of Zhang et al., integration is static. The tile-centric SUMMA-style algorithm is

$T_k\times T_n$9

where “$F_4$0”, “$F_4$1”, and “$F_4$2” denote the stored precisions of $F_4$3, $F_4$4, and $F_4$5 tiles. The receiving task converts data on the fly if it needs a different internal precision, but there is no adaptive scheduling routine that maps hardware metrics to tile precisions, no dynamic load-balancing re-quantization, and no mathematical model for accuracy loss or performance gain as a function of tile precision (Zhang et al., 20 Aug 2025).

TAH-Quant sketches a more explicitly hardware-aware control loop. For each micro-batch, it computes token entropies, selects $F_4$6, checks whether $F_4$7 and whether the device supports Hadamard, computes $F_4$8, quantizes, and sends $F_4$9 with metadata. The design variables $M\times N$0, the bit-ratio, and $M\times N$1 are then swept against measured network throughput, cache sizes, and transform latencies to find a Pareto-optimal operating point. The paper states that one can sweep $M\times N$2, bit-ratios, and $M\times N$3 for a given hardware target (He et al., 2 Jun 2025).

JAQ makes the hardware-search dimension explicit. Its BatchTile component encodes operator attributes, accelerator attributes, and tile extents $M\times N$4, then uses a hardware-generation network of 5 residual blocks to predict a discrete distribution over accelerator parameters using Gumbel-Softmax. Instead of enumerating tile mappings serially, BatchTile packs all $M\times N$5 candidates into one batched call to an “Energy/Latency Estimator,” computes

$M\times N$6

and reduces accelerator search time per iteration to $M\times N$7 seconds. On the quantization side, JAQ’s channel-wise sparse quantization selects a top-$M\times N$8 subset of activation channels using batch-norm scales $M\times N$9, with $(t_{oc},t_{ic},t_{oh},t_{ow})$0, so that only the selected channels are quantized during search. This is not tile quantization in the same sense as TAH-Quant or TileFuse, but it is a hardware-software co-exploration mechanism that couples quantization sparsity to compiler tiling and accelerator search (Wang et al., 9 Jan 2025).

TileQ moves the hardware/dataflow interface to the expert level. It embeds experts into a tiled block matrix $(t_{oc},t_{ic},t_{oh},t_{ow})$1, factorizes it as $(t_{oc},t_{ic},t_{oh},t_{ow})$2, then replaces $(t_{oc},t_{ic},t_{oh},t_{ow})$3 small expert computations with a single input projection $(t_{oc},t_{ic},t_{oh},t_{ow})$4, a vectorized gather/scatter with gate weights, and a single output reconstruction $(t_{oc},t_{ic},t_{oh},t_{ow})$5. The paper characterizes this as two large GEMMs plus two very regular gather/scatters, with contiguous row-major layouts for $(t_{oc},t_{ic},t_{oh},t_{ow})$6 and $(t_{oc},t_{ic},t_{oh},t_{ow})$7, and a fused custom kernel for scatter-add (Gu et al., 10 May 2026).

5. Accuracy, convergence, and reported trade-offs

The empirical and theoretical literature evaluates tile-aware quantization along several axes: numerical fidelity, convergence, throughput, energy, and memory overhead. TAH-Quant provides a convergence guarantee under standard assumptions together with a contractive quantization error assumption,

$(t_{oc},t_{ic},t_{oh},t_{ow})$8

and concludes that

$(t_{oc},t_{ic},t_{oh},t_{ow})$9

matching the convergence rate of vanilla stochastic gradient descent. Empirically, its ablations report that $\Omega^l$0 yields the lowest quantization error but higher metadata overhead, $\Omega^l$1 maintains accuracy on Qwen-3B SFT with MMLU $\Omega^l$2 while reducing scale storage by $\Omega^l$3, and $\Omega^l$4 begins to harm final accuracy with MMLU dropping $\Omega^l$5 points. On a 10 Gbps $\Omega^l$6 100 Mbps throttled network, full FP16 achieves $\Omega^l$7K tokens/s and TAH-Quant at 3–4 bits reaches $\Omega^l$8K tokens/s, a $\Omega^l$9 speed-up, while matching AQ-SGD accuracy without extra storage and avoiding AQ-SGD’s $G=p^2$00 memory overhead (He et al., 2 Jun 2025).

TileFuse reports kernel speedups over a full-precision BF16 baseline on XDNA2 of up to $G=p^2$01 for GEMM and $G=p^2$02 for GEMV. In end-to-end LLM experiments on Ryzen AI laptops, AWQ W4A16 on Llama3-8B reduces prefilling latency by up to $G=p^2$03 for long prompts of 4K–8K tokens and lowers total energy by $G=p^2$04. The paper’s roofline interpretation is that GEMM moves from memory-bound to compute-bound after quantization, while GEMV’s operational intensity rises above the roofline ridge and the redesigned dataflow unlocks the full 32-core potential (Pang et al., 9 Jun 2026).

OliVe reports accuracy retention with post-training quantization and no retraining: BERT-base on GLUE shows $G=p^2$05 average drop versus full precision; BART-base on SQuAD shows $G=p^2$06 F1/EM drop; GPT-2 XL and OPT-6.7B remain within $G=p^2$07 PPL of FP32 at 4 bit, while 8 bit achieves the original PPL. On hardware metrics, the OliVe-based accelerator surpasses GOBO by $G=p^2$08 speedup and $G=p^2$09 energy reduction, and the GPU die-area overhead of OVP decoders is $G=p^2$10, about $G=p^2$11 of a $G=p^2$12 die (Guo et al., 2023).

Winograd tap-wise quantization evaluates both quantization error and system-level hardware impact. Relative quantization error drops from mean $G=p^2$13 for uniform per-layer quantization to $G=p^2$14 for tap-wise quantization, while power-of-two constraints cost $G=p^2$15 Top-1 on ImageNet when combined with distillation. For ResNet-34, FP32 Top-1 of $G=p^2$16 becomes $G=p^2$17 with $G=p^2$18 tap-wise quantization plus Po2 scale and distillation, whereas $G=p^2$19 without tap-wise quantization loses $G=p^2$20. Across 12 CNNs, the Winograd-enhanced DSA achieves $G=p^2$21 end-to-end speed-up and up to $G=p^2$22 energy-efficiency gain versus im2col INT8 (Andri et al., 2022).

TileQ emphasizes model compression and MoE inference regularization. It reports up to $G=p^2$23 reduction in extra low-rank storage, MoE-MLP latency reduced to $G=p^2$24, and, on Mixtral-8×7B, 2-bit TileQ achieving perplexity near $G=p^2$25 versus $G=p^2$26 for GPTQ and MMLU $G=p^2$27 versus $G=p^2$28. At 3 bit, it reports within $G=p^2$29 of FP16 on all studied downstream tasks. JAQ, in turn, reports ImageNet Top-1 of $G=p^2$30, $G=p^2$31, and $G=p^2$32 at three $G=p^2$33 settings where Auto-nba yields $G=p^2$34, $G=p^2$35, and $G=p^2$36, a peak-memory reduction of about $G=p^2$37 from CSQ during search, and accelerator search time reduced from $G=p^2$38 seconds per iteration to $G=p^2$39 seconds (Gu et al., 10 May 2026, Wang et al., 9 Jan 2025).

6. Conceptual boundaries, misconceptions, and open distinctions

A common misconception is to equate any tile-centric mixed-precision computation with a hardware-aware tile quantization scheme. The available evidence does not support that equivalence. The PaRSEC GEMM work shows tile-wise FP64/FP32 execution and receiver-side conversion, but the details explicitly state that the user specifies a global D:S percentage, tiles are randomly assigned to precisions, and there are no runtime criteria, quantization formulas, accuracy reports based on the disabled Frobenius-norm paragraph, or hardware-probing mechanisms. It is therefore better understood as a tile-centric mixed-precision execution framework rather than a fully worked-out adaptive hardware-aware quantizer (Zhang et al., 20 Aug 2025).

Another misconception is that “hardware-aware” must imply the same mechanism across domains. In the training setting of TAH-Quant, hardware awareness is expressed through tuning tile size, bit ratios, and transform frequency to network bandwidth, scratchpad limits, and SIMD cost. In TileFuse, it is carried by layout, metadata placement, and fused dequantization matched to XDNA2’s single weight stream, DMA restrictions, and 4×8 AIE topology. In OliVe, it appears as memory-aligned 1-byte tiles and decode logic that fits tensor cores or systolic arrays. In Winograd $G=p^2$40, it means per-tap power-of-two scaling and shift-add transform engines that preserve an integer-only pipeline. In TileQ, it means reorganizing expert computation so that two large GEMMs and regular gather/scatter dominate instead of many tiny irregular kernels. In JAQ, it appears as joint search over bit-widths, channel sparsity, tiling modes, and accelerator parameters (He et al., 2 Jun 2025, Pang et al., 9 Jun 2026, Guo et al., 2023, Andri et al., 2022, Gu et al., 10 May 2026, Wang et al., 9 Jan 2025).

The broader literature therefore suggests a plural rather than singular definition. A hardware-aware tile quantization scheme is not one fixed algorithm, but a class of co-designed numerical representations and execution strategies in which tile-level quantization choices are constrained by, and often fused directly into, the hardware’s memory system, compute primitives, or compiler mapping. What varies from work to work is the tile definition, the quantizer family, the degree of adaptivity, and whether the design target is communication efficiency, integer-only inference, NPU deployability, MoE compression, or accelerator-search tractability.