Test-Time Adaptation Quantization (TTAQ)

• Test-Time Adaptation Quantization (TTAQ) is a technique that dynamically recalibrates quantized neural network models to overcome domain shift and maintain inference performance.
• TTAQ methods adjust parameters like quantization scales and normalization statistics in real time using approaches such as BN-statistics adaptation and finite-difference estimators, eliminating the need for backpropagation.
• Empirical results demonstrate that TTAQ improves accuracy and reduces error rates across various applications—including edge vision, large-scale LLM inference, and diffusion models—while keeping memory and latency overhead minimal.

Test-Time Adaptation Quantization (TTAQ) refers to a family of techniques designed to adapt quantized neural network models on the fly, addressing domain shift and maintaining performance when inference-time data distributions diverge from those seen during training or calibration. Unlike conventional post-training quantization (PTQ), which fixes quantization parameters after a one-off calibration, TTAQ frameworks aim to dynamically recalibrate, regularize, or otherwise adapt quantization behavior or associated normalization/statistics in response to continuously evolving test-time inputs. TTAQ has become central in enabling robust, memory- and energy-efficient deployment of deep models on edge devices, large-scale LLM inference, and in scenarios characterized by distributional non-stationarity.

1. Problem Statement and Motivation

Quantized models are widely used in resource-constrained environments to reduce computational and memory overhead. PTQ methods typically employ a small, static calibration dataset to determine quantization scales and zero-points, assuming test data distributions are similar. In realistic scenarios—e.g., sensor drift, environmental changes, dataset shifts—test distributions may differ substantially, leading to compounding quantization errors across layers. These issues are especially pronounced for low-precision (e.g., 2–8 bit) models, which are more sensitive to input and activation shifts. As a result, performance degradation is often observed in quantized models under distribution shift, exceeding that of corresponding full-precision models. Standard continuous test-time adaptation (TTA) approaches, often designed for full-precision models and relying on gradient backpropagation, are generally impractical for quantized models due to vanishing gradients, non-differentiable quantizers, and stringent memory/latency budgets (Xiao et al., 2024, Deng et al., 4 Aug 2025, Dong et al., 20 Mar 2025, Koike-Akino et al., 11 Mar 2026, So et al., 2023, Patarlapalli et al., 7 May 2026).

2. Key Approaches and Mathematical Foundations

Several TTAQ variants have emerged, each targeting different network architectures and operational constraints:

A. LeanTTA

LeanTTA operates on quantized models with batch normalization, using a backpropagation-free, stateless approach. For each incoming sample, LeanTTA updates only normalization statistics (mean and variance), operating in forward mode only. Notably, it:

• Computes per-feature means and variances over the sample,
• Combines them with train-time statistics via a momentum parameter $\tau$,
• Measures distribution shift using the Mahalanobis distance,
• Recombines source and adapted statistics using a scaling parameter $\lambda$,
• Applies the updated normalization,
• Resets the statistics after each sample, ensuring statelessness,
• Uses partial fusion: only shallow (input-adjacent) layers keep separate BN for adaptation, while deeper layers are fused for memory/performance (Dong et al., 20 Mar 2025).

Mathematically, at each adapted BN layer: $$\begin{aligned} \mu_t &= \frac{1}{H\cdot W}\sum x_t\ \sigma_t^2 &= \operatorname{Var}(x_t)\ \mu_b &= \tau\mu_s + (1-\tau)\mu_t,\qquad \sigma_b^2 = \tau\sigma_s^2 + (1-\tau)\sigma_t^2\ d &= 1 - \exp\left(-(\mu_b-\mu_s)^\top\Sigma_s^{-1}(\mu_b-\mu_s)\right),\quad \Sigma_s=\operatorname{diag}(\sigma_s^2)\ \mu_\text{new} &= (d\lambda)\mu_s + (1-d\lambda)\mu_b\ \sigma_\text{new}^2 &= (d\lambda)\sigma_s^2 + (1-d\lambda)\sigma_b^2\ x_t^{l+1} &= \gamma\cdot\frac{x_t^l-\mu_\text{new}}{\sqrt{\sigma_\text{new}^2+\epsilon}} + \beta \end{aligned}$$ (Dong et al., 20 Mar 2025).

B. Zeroth-Order Adaptation (ZOA)

ZOA enables quantized model adaptation with only two forward passes per batch (no backpropagation). Adaptation parameters are updated using a finite-difference (one-sided SPSA) gradient estimator and aggregated domain knowledge. It maintains a bank of adaptation vectors for continual learning and efficiently updates only a few low-dimensional or aggregation parameters: $$\theta = \theta^0 + \sum_j \alpha_j\Delta_j + \tilde{\theta}$$ where $\{\Delta_j\}$ are domain adaptation deltas, $\alpha$ is a softmax aggregation vector, and $\tilde{\theta}$ are new adaptation parameters. Adaptation is driven by aligning current predictions and normalization statistics to the original domain (Deng et al., 4 Aug 2025).

C. Post-Training Quantization Adaptation with Stability (TTAQ: PEM, PCR, ABL)

The PEM (Perturbation Error Mitigation) regularizer recenters and rescales quantized weights to match the first two moments of the original weights, reducing error propagation under perturbed activations: $$\hat{W}_{i,j} = \alpha\frac{W_{i,j}-\mu_{W_i}}{\sigma_{W_i}}$$ PCR (Perturbation Consistency Reconstruction) enforces consistency of outputs when activations are perturbed (by small noise) and uses logit/KL matching for stability. ABL (Adaptive Balanced Loss) corrects classifier logits using adaptive, momentum-updated class priors that capture class-frequency and gradient magnitude, counteracting class imbalance and catastrophic forgetting (Xiao et al., 2024).

D. Activation- and Context-Aware Quantization for LLMs

TTQ for LLMs performs online, prompt-wise groupwise quantization using diagonal approximations of layer input autocorrelation, scaling each weight matrix column by per-prompt activation statistics. This results in optimal quantization parameters per context, reducing loss under domain shift and enabling integer computation at scale (Koike-Akino et al., 11 Mar 2026). BitCal-TTS applies runtime proxies for uncertainty and stability with bit-conditioned scaling in token generation, making adaptive halting more robust to quantizer-induced noise (Patarlapalli et al., 7 May 2026).

E. Temporal Dynamic Quantization (TDQ) in Diffusion Models

TDQ modules adapt quantization scale as a function of the denoising step $t$ via a small step-conditioned MLP generator, achieving consistent quantization error across the entire trajectory. This is crucial since diffusion activations are highly time-dependent and static quantization scales are suboptimal (So et al., 2023).

3. Algorithmic Implementations

TTAQ pipelines are highly varied but share the principle of real-time or near-real-time adaptation of quantizer or normalization parameters. Algorithmic sketches include:

• BN-statistics adaptation per sample (LeanTTA): collects statistics, updates parameters statelessly, and resets after each sample (Dong et al., 20 Mar 2025).
• Zeroth-order online adaptation: two forward passes per batch, finite-difference gradient estimate, adaptation of a small number of parameters, and domain bank management (Deng et al., 4 Aug 2025).
• Quantizer calibration with consistency and balance (TTAQ/PEM/PCR/ABL): block-wise scale updates via stochastic perturbation, class-balancing via logit adjustment, and limited updated parameter scope (Xiao et al., 2024).
• Prompt-/step-aware scale adaptation: real-time diagonal activation profiling or per-step context encoding for time-dependent tasks, folded into scaling for quantization (Koike-Akino et al., 11 Mar 2026, So et al., 2023).

4. Empirical Results and Comparative Analysis

Empirical studies across edge vision, LLMs, and structured prediction domains demonstrate that TTAQ methods consistently outperform static PTQ and non-adaptive TTA under distribution shift. Selected results include:

• LeanTTA on Raspberry Pi Zero 2W (ResNet18, CIFAR10-C, batch=1): INT8 LeanTTA achieves 78.8% accuracy vs. 76.3% baseline and best SOTA (RealisticTTA) at 63.4%, for a 15.7% relative error reduction. Adaptation overhead is negligible in memory and latency (Dong et al., 20 Mar 2025).
• ZOA on ImageNet-C (W6A6 ViT-B): ZOA yields 56.3% average accuracy (10th adaptation round) vs 51.3% for FOA and 47.7% non-adaptive, with similar memory/latency cost as inference (Deng et al., 4 Aug 2025).
• TTAQ (with PEM/PCR/ABL) on ImageNet-C (ResNet-50 W2A4): Reduces mean error from 66.2% (QDrop) and 67.5% (Brecq) to 62.05%. For W2A2, QDrop fails but TTAQ obtains 80.59%. For COCO-C detection, TTAQ mAP improves by +1.8 points on W2A4 (Xiao et al., 2024).
• TTQ for LLMs: 3–5 bit quantization with TTQ matches or outperforms strong AWQ baselines on OPT, Qwen3, and Gemma LLM families (up to 32B), and achieves stable perplexity under domain shift (Koike-Akino et al., 11 Mar 2026).
• TDQ for diffusion models: On CIFAR-10 and LSUN benchmarks, TDQ consistently matches or outperforms LSQ, NIPQ, and PTQ4DM at 4–8 bits in both IS and FID across PTQ and QAT settings, with negligible runtime cost (So et al., 2023).
• BitCal-TTS on GSM8K with Qwen2.5-7B/14B: In partial-shard tests with capped token budget, BitCal-TTS improves accuracy by up to +3.7 pp and reduces premature stops (11.1–11.4% vs. 14.8–17.1%) relative to adaptive non-bit-aware baselines, with maintained sample efficiency (Patarlapalli et al., 7 May 2026).

5. Design, Trade-offs, and Practical Considerations

TTAQ methods differ significantly in scope, parameterization choices, and resource/accuracy trade-offs:

• Stateful vs. Stateless: LeanTTA adopts strict statelessness, resetting all adaptation state per-sample, preventing drift; ZOA and TTAQ/PEM approaches maintain light state (parameter deltas, domain banks) for continual adaptation (Deng et al., 4 Aug 2025, Xiao et al., 2024, Dong et al., 20 Mar 2025).
• Backpropagation: All surveyed TTAQ approaches avoid or minimize backpropagation, leveraging forward-only statistics, finite-difference gradients, or sidecar calibration without needing full gradient computation (Deng et al., 4 Aug 2025, Dong et al., 20 Mar 2025).
• Batch Size and Latency: Methods like LeanTTA, TTQ, and BitCal-TTS are robust at single-sample (batch=1) operation—critical for streaming and edge inference. In contrast, traditional TTA and some continual adaptation baselines degrade at small batch sizes or incur substantial memory and latency costs (Dong et al., 20 Mar 2025, Xiao et al., 2024).
• Parameter Scope: TTAQ pipelines typically restrict updates to a small subset (e.g., scale/shift in BN, quantization interval scales, aggregation logits, or classifier head), rather than full-network adaptation (Xiao et al., 2024, So et al., 2023).
• Domain and Modalities: Approaches generalize across vision, language (LLMs), and even temporal generative models, but some methods require specific architectural modules (e.g., BN layers for LeanTTA, step-awareness for TDQ in diffusion models) (Dong et al., 20 Mar 2025, So et al., 2023).
• Limitations: Variance in zeroth-order updates (ZOA), difficulty at extreme quantization (W2A2), dependence on BN or similar normalizers, and limited efficacy on static or narrow-shift scenarios are all acknowledged constraints—or targets of future work (Xiao et al., 2024, Dong et al., 20 Mar 2025, Deng et al., 4 Aug 2025).

6. Extensions and Open Challenges

Several research directions have been highlighted:

• Normalization Beyond BN: Extending LeanTTA and related forward-adaptive methods to models without BN (i.e., with GroupNorm or LayerNorm) remains unresolved (Dong et al., 20 Mar 2025).
• Hyperparameter Adaptivity: Selection or meta-learning of adaptive parameters (e.g., momentum $\tau$, scaling $\lambda$0) potentially per-layer and per-stream, is a pertinent question for robust TTAQ (Dong et al., 20 Mar 2025, Koike-Akino et al., 11 Mar 2026).
• Hybrid Adaptation: Combining TTAQ approaches with lightweight backpropagation on real-valued adapters or with test-time pruning/compression is suggested for further gains (Deng et al., 4 Aug 2025, Koike-Akino et al., 11 Mar 2026).
• Task and Modality Expansion: Adapting TTAQ frameworks for detection, segmentation, structured prediction, audio, and multi-modal tasks is an evolving frontier (Xiao et al., 2024, Deng et al., 4 Aug 2025).
• Robustness under Extreme Stream Nonstationarity: Handling cases of abrupt or highly non-i.i.d. distribution shift, out-of-memory conditions, and designing smarter domain shift detection is an ongoing challenge (Deng et al., 4 Aug 2025, Patarlapalli et al., 7 May 2026, Dong et al., 20 Mar 2025).

7. Impact and Comparative Summary

TTAQ has enabled quantized neural networks to remain robust and efficient under dynamic operational conditions across diverse environments. Table 1 summarizes key representative TTAQ frameworks:

Method	Adaptation Principle	Quantized/Model Types
LeanTTA (Dong et al., 20 Mar 2025)	BN stat update, stateless, no BP	Edge vision, BN-based CNNs
ZOA (Deng et al., 4 Aug 2025)	Zeroth-order, domain-bank	Quantized CNN/Transformer
TTAQ (PEM/PCR/ABL) (Xiao et al., 2024)	Block/scale reg + consistency	Classification, detection
TTQ (Koike-Akino et al., 11 Mar 2026)	Prompt-wise activation-aware Q	LLMs, Transformer
TDQ (So et al., 2023)	Step-wise scale MLP (diffusion)	Diffusion, U-Net
BitCal-TTS (Patarlapalli et al., 7 May 2026)	Bit-aware halting/confidence ctrl.	Reasoning LLMs, math

Performance improvements are substantial versus static PTQ (e.g., 8–15% error reduction, multi-point accuracy increases), with little to no runtime or memory penalty. TTAQ research continues to expand its reach across architectures, application domains, and operational scenarios.