TileQ: 2D Tiled Quantization for MoE

• TileQ is a post-training quantization method for MoE models that employs a 2D tiling scheme to share low-rank factors across experts.
• It uses activation-aware biclustering and greedy spatial allocation to structure expert parameters and minimize extra memory overhead.
• The approach fuses numerous small GEMMs into dense operations, achieving significant memory savings and reduced inference latency.

Searching arXiv for the TileQ paper and closely related MoE/PTQ references. TileQ is a fine-tuning-free post-training quantization method for Mixture-of-Experts (MoE) models that uses a 2D-tiling structured low-rank quantization scheme to share low-rank factors across both input and output dimensions of experts, together with a fused sparse low-rank inference path called LoTileMoE (Gu et al., 10 May 2026). It is designed for the deployment regime in which MoE models obtain sparse compute through routing while still incurring a severe memory bottleneck because all expert parameters must remain resident in memory (Gu et al., 10 May 2026). The method targets three limitations identified for prior low-rank PTQ in MoE settings: large accuracy degradation at extremely low bit-width, nonnegligible “extra-bit” overhead from per-expert low-rank factors, and inference latency caused by many small GEMMs and lack of fusion (Gu et al., 10 May 2026).

1. Conceptual setting and problem formulation

In TileQ, the underlying model class is an MoE MLP layer with experts $W=\{W_k\}_{k=1}^K$, where each expert weight satisfies $W_k \in \mathbb{R}^{o \times i}$, and a router $G$ sparsely activates only a small subset per token (Gu et al., 10 May 2026). For an input $X \in \mathbb{R}^{B \times i}$, the MoE layer is written as

$$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$

where $e_{k,X}$ is the set of selected experts and $G_k(X)$ are routing scores (Gu et al., 10 May 2026).

The deployment difficulty addressed by TileQ is not the dense compute cost of using every expert at inference, but the requirement that all expert parameters reside in memory even under sparse activation (Gu et al., 10 May 2026). The paper frames PTQ as attractive because it avoids retraining, while noting three concrete obstacles for existing low-rank PTQ methods in MoE models. First, extremely low-bit scalar quantization, such as 2-bit or 3-bit, can induce large accuracy drops. Second, experts are small and numerous, so storing low-rank factors separately for each expert produces substantial low-rank overhead. Third, expert-wise low-rank decompositions generate many small matrix multiplications and inhibit fusion, thereby creating a latency bottleneck (Gu et al., 10 May 2026).

TileQ’s central idea is to replace isolated per-expert low-rank structure with a tiled, shared structure. Experts are arranged on an $M \times N$ grid and share left factors along rows and right factors along columns. This suggests a representation in which the number of shared bases grows with $M+N$ rather than with $K$, while preserving expert-specific residual information through residual quantization (Gu et al., 10 May 2026).

2. Structured low-rank model and 2D tiling

The baseline low-rank approximation discussed in the paper is

$W_k \in \mathbb{R}^{o \times i}$0

with $W_k \in \mathbb{R}^{o \times i}$1 and $W_k \in \mathbb{R}^{o \times i}$2 (Gu et al., 10 May 2026). TileQ generalizes this by imposing a two-dimensional sharing structure across experts. All $W_k \in \mathbb{R}^{o \times i}$3 experts are placed into an $W_k \in \mathbb{R}^{o \times i}$4 grid, and for expert $W_k \in \mathbb{R}^{o \times i}$5 with tile coordinate $W_k \in \mathbb{R}^{o \times i}$6, the factorization is

$W_k \in \mathbb{R}^{o \times i}$7

where $W_k \in \mathbb{R}^{o \times i}$8, $W_k \in \mathbb{R}^{o \times i}$9, $G$0 is diagonal, and $G$1 is a quantized residual (Gu et al., 10 May 2026).

The construction is activation-aware. TileQ first incorporates diagonal calibration scaling into the weights,

$G$2

where $G$3 is a diagonal scaling obtained from calibration activations (Gu et al., 10 May 2026). According to the paper, this aligns left singular vectors with input features for clustering $G$4 and right vectors with output features for clustering $G$5 (Gu et al., 10 May 2026).

Feature extraction then proceeds by computing a rank-$G$6 SVD on the scaled weights,

$G$7

followed by vectorization and $G$8-normalization of the singular subspace embeddings,

$G$9

These embeddings are clustered by KMeans into $X \in \mathbb{R}^{B \times i}$0 row clusters and $X \in \mathbb{R}^{B \times i}$1 column clusters,

$X \in \mathbb{R}^{B \times i}$2

thereby producing ideal tile coordinates $X \in \mathbb{R}^{B \times i}$3 for each expert (Gu et al., 10 May 2026).

Because the independent row and column cluster assignments can collide, TileQ resolves placement by a greedy spatial allocation minimizing $X \in \mathbb{R}^{B \times i}$4 deviation from the ideal positions,

$X \in \mathbb{R}^{B \times i}$5

The tiled matrix is then assembled as

$X \in \mathbb{R}^{B \times i}$6

where $X \in \mathbb{R}^{B \times i}$7 is an $X \in \mathbb{R}^{B \times i}$8 basis matrix with a $X \in \mathbb{R}^{B \times i}$9 at $$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$0 and $$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$1 denotes block Kronecker placement (Gu et al., 10 May 2026).

A rank-$$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$2 decomposition of $$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$3 yields shared factors $$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$4, from which the per-tile factors are sliced:

$$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$5

The low-rank reconstruction for expert $$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$6 is then unscaled as

$$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$7

This produces a shared low-rank component together with an expert-specific residual (Gu et al., 10 May 2026).

3. Quantization pipeline and calibration procedure

TileQ applies residual quantization after constructing the shared low-rank approximation. For each expert,

$$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$8

where the quantizer is instantiated using mainstream PTQ backends with Hessian-aware optimization (Gu et al., 10 May 2026). The paper explicitly lists scalar GPTQ and vector GPTVQ, and also describes optional low-rank rotation via LOPRO to improve quantization robustness of the residual (Gu et al., 10 May 2026).

The quantization mapping is presented in standard uniform affine form:

$$\mathrm{MoE}(X)=\sum_{k\in e_{k,X}} G_k(X)\cdot W_k(X),$$9

with symmetric or asymmetric formats, while the implementation adopts GPTQ/GPTVQ learned scales and rounding or grouped vector-wise codebooks as appropriate (Gu et al., 10 May 2026). Group size $e_{k,X}$0, for example $e_{k,X}$1, controls per-group scales (Gu et al., 10 May 2026).

TileQ uses a mixed-precision storage strategy. The shared factors $e_{k,X}$2 and $e_{k,X}$3 are stored in fp8, with $e_{k,X}$4 bits, the diagonal $e_{k,X}$5 is stored in fp16, and residual weights are quantized to 2-bit or 3-bit depending on the target setting (Gu et al., 10 May 2026). The paper states that this halves low-rank storage relative to fp16 factors (Gu et al., 10 May 2026).

Calibration is performed without fine-tuning. The reported configuration uses 128 C4 sequences of length 2048 for calibration (Gu et al., 10 May 2026). The layer-wise flow per MoE MLP is: collect activations, compute $e_{k,X}$6, extract sketch-SVD features, run biclustering, perform greedy placement, decompose $e_{k,X}$7 by sketch-SVD, slice $e_{k,X}$8 and $e_{k,X}$9, reconstruct $G_k(X)$0, and quantize the residual with GPTQ or GPTVQ using layer-wise Hessian proxies and optional rotation (Gu et al., 10 May 2026). The paper emphasizes that no fine-tuning is performed (Gu et al., 10 May 2026).

To avoid full SVD cost, TileQ uses a Gaussian sketch for rank extraction. With Gaussian vector $G_k(X)$1,

$G_k(X)$2

from which rank-1 components $G_k(X)$3 are formed, accumulated, and the matrix is deflated iteratively for $G_k(X)$4 steps (Gu et al., 10 May 2026). This sketch-based low-rank extraction is part of the implementation path described as lightweight relative to the quantization backend (Gu et al., 10 May 2026).

4. Fused inference and LoTileMoE

TileQ’s inference path splits the MoE computation into a residual quantized path and a fused low-rank path:

$G_k(X)$5

The purpose of LoTileMoE is to eliminate the many tiny GEMMs that arise in expert-wise low-rank execution and replace them with a small number of dense operations aligned with tensor-core execution (Gu et al., 10 May 2026).

The LoTileMoE workflow has four reported stages. First, for each selected token–expert pair $G_k(X)$6, the selected expert $G_k(X)$7 is mapped to a tile coordinate $G_k(X)$8 (Gu et al., 10 May 2026). Second, inputs are projected once using the shared left factor:

$G_k(X)$9

Third, routing-weighted contributions are accumulated into a compact tile-column buffer $M \times N$0:

$M \times N$1

Fourth, $M \times N$2 is reshaped to $M \times N$3 and a single right projection is computed,

$M \times N$4

This sequence is explicitly described as projecting inputs once, aggregating at tile granularity, and finishing with one right-factor GEMM (Gu et al., 10 May 2026).

The reported complexity is

$M \times N$5

The paper states that this replaces many tiny GEMMs with two well-shaped dense GEMMs plus vectorized gathers and scatters, reduces irregular memory traffic in per-expert dispatch, and enables single-pass fused execution for both prefill and decode (Gu et al., 10 May 2026).

This fused design is presented as a hardware-oriented complement to the 2D factor sharing. A plausible implication is that TileQ’s compression and latency properties are inseparable: the same shared tiling that reduces low-rank storage also makes a fused computational schedule possible, whereas disjoint expert-specific low-rank blocks obstruct such fusion (Gu et al., 10 May 2026).

5. Memory overhead, latency, and empirical performance

A central analytical component of TileQ is the “extra-bit” overhead calculation for low-rank factors beyond the residual bit-width $M \times N$6 (Gu et al., 10 May 2026). For per-expert low-rank storage,

$M \times N$7

For 1D sharing,

$M \times N$8

For TileQ’s 2D sharing,

$M \times N$9

with the $M+N$0 term described as negligible at large $M+N$1 (Gu et al., 10 May 2026).

Under $M+N$2 and $M+N$3, the three overheads simplify to

$M+N$4

The paper interprets this as a $M+N$5 reduction relative to 1D sharing and a factor of $M+N$6 relative to per-expert low-rank storage, and summarizes the empirical effect as “up to 90% low-rank extra memory” saved (Gu et al., 10 May 2026).

A concrete numerical example is given for Qwen3-30B-A3B with expert shape $M+N$7, $M+N$8, $M+N$9, $K$0, residual bit-width $K$1, and low-rank bit-width $K$2 (Gu et al., 10 May 2026). The base quantization cost is

$K$3

bits per weight (Gu et al., 10 May 2026). The reported overheads are:

Scheme	Extra bits per weight
Per-expert low-rank	$K$4
1D shared	$K$5
TileQ with $K$6	$K$7

The paper states that this yields about $K$8 lower low-rank extra bits than per-expert storage and about $K$9 lower than 1D sharing in that example (Gu et al., 10 May 2026).

Latency results are reported on A800, H800, and RTX 5090 hardware. TileQ’s fused MLP-block latency overhead during prefill is stated to be less than 5%, and the paper attributes this to replacing irregular expert dispatch with a single structured GEMM (Gu et al., 10 May 2026). It also states that the overhead remains low in decode due to contiguous layouts and tile-aggregated memory access (Gu et al., 10 May 2026).

The empirical evaluation covers Qwen1.5-MoE-A2.7B, Qwen3-30B-A3B, Qwen3-Next-80B-A3B, Mixtral-8×7B, and Deepseek-MoE-16B, using WikiText-2 perplexity and downstream task accuracy on ARC-Challenge, ARC-Easy, PIQA, Winogrande, MMLU, and HellaSwag via lm-eval (Gu et al., 10 May 2026). The paper highlights the following results.

Model / setting	Reported result
Mixtral-8×7B, 2-bit	FP16 PPL 3.87; GPTQ 15.3; GPTVQ 5.28; LOPRO 5.01; TileQ_V 4.78; TileQ_S 4.98
Mixtral-8×7B, 3-bit	TileQ_S reaches 4.10 PPL and MMLU 69.4, near FP16 71.2
Qwen3-30B-A3B, 2-bit	TileQ_V PPL 7.35 vs GPTQ 12.5 and LOPRO 7.52
Qwen3-30B-A3B, 3-bit	TileQ_S PPL 6.94, matching LOPRO
Deepseek-MoE-16B, 2-bit	TileQ_S matches LOPRO on PPL and downstream metrics
Deepseek-MoE-16B, 3-bit	TileQ_S slightly improves several tasks

The reported ablations further state that removing 2D tiling keeps accuracy but increases extra bits by $W_k \in \mathbb{R}^{o \times i}$00, while removing low-rank causes performance collapse, with 2-bit PPL 12.5 and accuracy 38.5 (Gu et al., 10 May 2026). Rank and tile-size ablations indicate that rank 32 and near-square tilings such as $W_k \in \mathbb{R}^{o \times i}$01 or $W_k \in \mathbb{R}^{o \times i}$02 balance accuracy and latency (Gu et al., 10 May 2026).

6. Theoretical interpretation, implementation profile, and limitations

The paper summarizes TileQ’s reconstruction error by the decomposition

$W_k \in \mathbb{R}^{o \times i}$03

where $W_k \in \mathbb{R}^{o \times i}$04 is the per-expert low-rank-plus-quantization error and $W_k \in \mathbb{R}^{o \times i}$05 is the additional error induced by 2D sharing (Gu et al., 10 May 2026). Appendix-level analysis is described as controlling $W_k \in \mathbb{R}^{o \times i}$06 through KMeans optimality and Davis–Kahan-type subspace stability bounds, with the conclusion that when experts’ activation-aware subspaces cluster well, TileQ approaches per-expert reconstruction quality while using $W_k \in \mathbb{R}^{o \times i}$07 shared bases instead of $W_k \in \mathbb{R}^{o \times i}$08 (Gu et al., 10 May 2026).

From an implementation perspective, the procedure is explicitly modular. Preprocessing consists of calibration on 128 C4 sequences of length 2048, computation of per-expert scaling $W_k \in \mathbb{R}^{o \times i}$09, sketch-SVD feature extraction, KMeans biclustering, greedy tile placement, construction of $W_k \in \mathbb{R}^{o \times i}$10, sketch-based rank-$W_k \in \mathbb{R}^{o \times i}$11 decomposition, residual formation, and residual PTQ via GPTQ or GPTVQ, optionally with LOPRO (Gu et al., 10 May 2026). Deployment then replaces the MoE MLP experts with a composite of the quantized residual path and the LoTileMoE path, using reshaped $W_k \in \mathbb{R}^{o \times i}$12, compact accumulation buffers, and single-pass kernels (Gu et al., 10 May 2026).

The reported hardware and software profile emphasizes dense GEMMs and vectorized scatter_add, contiguous tile layouts, and experiments on NVIDIA A800-80GB as the main platform, with latency results also on H800 and RTX 5090 (Gu et al., 10 May 2026). Mixtral downstream evaluation required 2$W_k \in \mathbb{R}^{o \times i}$13A800 (Gu et al., 10 May 2026).

The limitations identified in the paper are specific. The dominant bottleneck in end-to-end runtime is residual quantization with GPTQ or GPTVQ, especially for MoE models with many small experts (Gu et al., 10 May 2026). Performance is sensitive to the tiling dimensions $W_k \in \mathbb{R}^{o \times i}$14 and the rank $W_k \in \mathbb{R}^{o \times i}$15; larger $W_k \in \mathbb{R}^{o \times i}$16 increases gather/scatter overhead and decode latency, while rank above 32 provides diminishing accuracy returns but increases memory and latency (Gu et al., 10 May 2026). The method also depends on representative calibration data and on expert subspaces being sufficiently clusterable; if activation-aware expert subspaces do not cluster well, the clustering error term can grow (Gu et al., 10 May 2026). Finally, very sparse activation and highly skewed tilings can degrade memory coalescing, though the compact tile buffers are presented as a mitigation (Gu et al., 10 May 2026).

Taken together, TileQ occupies a specific point in the MoE compression design space. It is a PTQ method rather than a retraining method, a structured low-rank method rather than a purely scalar quantizer, and a deployment-oriented system whose inference design is co-optimized with its compression structure (Gu et al., 10 May 2026). This suggests that its main contribution is not only lower storage for low-rank factors, but a representation that makes low-rank MoE execution compatible with fused, high-utilization kernels.

7. Position within MoE quantization research

TileQ is evaluated against GPTQ, GPTVQ, LOPRO, and MOEQuant, with FP16 used as an upper bound (Gu et al., 10 May 2026). Within that comparison, the paper’s stated contribution is threefold: a fine-tuning-free PTQ approach for MoE based on a 2D-tiling layout and singular subspace clustering, an optimized fusion algorithm that combines multiple low-rank expert GPU kernels across all tokens into a single-pass operation for both prefill and decode, and experimental evidence that TileQ saves up to 90% low-rank extra memory and reduces inference latency to about 5% while preserving state-of-the-art accuracy (Gu et al., 10 May 2026).

The method’s relation to earlier low-rank PTQ is defined in the paper through its sharing pattern. Per-expert low-rank methods preserve expert independence but incur maximal low-rank overhead. Shared-1D schemes reduce storage by sharing only one side of the factorization, but still keep disjoint per-expert blocks on the other side, which blocks fusion and was reported in the paper as adding more than 50% latency on newer sparse MoEs (Gu et al., 10 May 2026). TileQ’s 2D sharing replaces this with joint row-wise and column-wise sharing, enabling both lower extra-bit overhead and the LoTileMoE execution path (Gu et al., 10 May 2026).

The paper’s summary characterizes the overall result as follows: TileQ shares low-rank factors in both input and output dimensions via 2D tiling guided by activation-aware biclustering, quantizes only residuals with GPTQ or GPTVQ while storing $W_k \in \mathbb{R}^{o \times i}$17 in fp8 and $W_k \in \mathbb{R}^{o \times i}$18 in fp16, and fuses all experts’ low-rank computations into one-pass dense operations (Gu et al., 10 May 2026). It reports up to $W_k \in \mathbb{R}^{o \times i}$19 additional memory reduction for low-rank overhead and MLP-block prefill latency of approximately 5%, while maintaining strong accuracy at 2-bit and 3-bit precision across diverse MoE models (Gu et al., 10 May 2026).