Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers

Abstract: Data-Free Quantization (DFQ) addresses data security concerns by synthesizing samples, without accessing real data. It has garnered increasing attention in the context of Vision Transformers (ViTs), owing to the superiority of the self-attention mechanism compared to classical convolutional operation. However, previous DFQ arts for ViTs often suffer from a distribution mismatch between synthetic samples and input distribution expected by quantized models Q, resulting in the suboptimal performance. In this paper, we propose a novel Masked Attention Alignment approach for Data-Free Quantization of ViTs, named MaskAQ, revealing that: 1) the semantics in the self-attention mechanism is predominantly localized to a sparse subset of patches, called informative regions; 2) the informative regions dominate the mutual information between synthetic samples and Q's outputs. To these ends, we incorporate differential entropy maximum over patch similarity of synthetic samples, to decouple informative regions from noisy background. To couple with varied Q, the informative regions are selected to align full-precision models with Q via a masked attention alignment objective, thus yielding high-quality synthetic samples. Furthermore, a periodic sample refreshing strategy comes up to endow MaskAQ with the capacity to continually adapt to the evolving state of Q throughout the training process, to preserve desirable mutual information with synthetic samples. Extensive experiments verify the merits of MaskAQ over state-of-the-art approaches across multiple backbones and downstream tasks. Our code is available at https://github.com/hfutqian/MaskAQ.

• The paper introduces MaskAQ, a data-free quantization method that decouples informative image regions, aligns full- and low-precision attention, and refreshes synthetic samples during calibration.
• MaskAQ consistently outperforms prior methods on ImageNet, including a 3.10-point Top-1 gain at 3-bit DeiT-T quantization and improvements on COCO detection and ADE20K segmentation.
• The method shows that focusing calibration on discriminative patches is more effective than matching the full data distribution, although iterative synthesis adds overhead and results below 3 bits remain untested.

Motivation and problem setting

Data-Free Quantization (DFQ) synthesizes calibration samples from a pre-trained full-precision model $P$ without accessing original training data, addressing privacy constraints in domains such as healthcare. While DFQ for CNNs benefits from Batch Normalization statistics as a distributional prior, Vision Transformers (ViTs) rely on Layer Normalization and self-attention, so this prior is unavailable and naive migration of CNN-era DFQ methods degrades sharply, particularly at ultra-low bit widths (2606.04373).

The paper identifies two failure modes of existing ViT-DFQ sample synthesis methods such as PSAQ-ViT, CLAMP-ViT, and MimiQ:

• Semantic dispersion: synthetic semantics are spread across the entire image rather than concentrated in a coherent object structure, diluting the supervision signal for calibrating the quantized model $Q$.
• Attentional disparity: synthetic samples lack discriminative regions that low-precision $Q$ can reliably recognize, so $Q$ cannot align its attention with $P$, which is especially damaging under aggressive quantization.

The central reframing is an information bottleneck (IB) perspective: rather than approximating the real data distribution, synthesis should preserve the mutual information between synthetic samples and $Q$'s outputs subject to a precision-dependent information budget $C$, i.e., $\max I(z_q; y)$ subject to $I(x; z_q) \le C$. Under this budget, representational capacity should be allocated to the most informative patches.

Method

MaskAQ comprises three components: informative region decoupling, masked attention coupling, and periodic sample refreshing.

Informative region decoupling. An Informative Region (IR) is defined as the set of patches whose attention weights in $P$ exceed the $Q$0-th largest attention value. To decouple IRs from noisy background, the method maximizes the differential entropy of pairwise cosine similarities between attention vectors at each layer. Since histogram-based Shannon entropy estimates are binning-sensitive, the similarity distribution is approximated as Gaussian, giving the tractable surrogate

$Q$1

where $Q$2 is the empirical variance of off-diagonal attention similarities. Maximizing this entropy discourages collapsed, degenerate attention patterns and promotes structural coherence in synthesized images.

Masked attention coupling. A stricter binary mask selects a subset of informative patches (with the threshold $Q$3 annealed from 50% to 10% of patches over training), followed by stochastic dropout ($Q$4) to avoid overfitting to a fixed patch set. The alignment objective penalizes the $Q$5 discrepancy between attention maps of $Q$6 and $Q$7 restricted to masked regions, normalized by mask cardinality:

$Q$8

Restricting alignment to informative regions avoids over-regularizing background patches whose quantized attention is inherently distorted. The full synthesis loss combines one-hot classification, total variation, inter-head attention similarity priors, plus $Q$9 and $Q$0 with weights $Q$1 and $Q$2.

Masked calibration and periodic refreshing. Calibration of $Q$3 uses a weighted head-output matching loss that upweights informative-region tokens (weight factor $Q$4). Because $Q$5 evolves during calibration, synthetic samples are periodically re-synthesized via the full objective at fixed intervals, maintaining mutual information with the current state of $Q$6.

Theoretical analysis

Two theorems formalize the IB argument using entropy continuity bounds on total variation distances. Theorem 1 shows that if the joint distributions of $Q$7 and conditional distributions $Q$8 are within TV distance $Q$9 of their $Q$0 counterparts, then the gap in predictive mutual information $Q$1 is bounded by a quantity vanishing as $Q$2. Theorem 2 extends this to synthetic samples: if the synthetic informative region $Q$3 matches the real informative region's label informativeness within $Q$4, and analogous TV conditions hold, then $Q$5.

The two losses map directly onto these conditions: $Q$6 targets the region-label informativeness gap $Q$7, while $Q$8 targets the TV alignment conditions. It should be noted that these are sufficient-condition results under discrete-alphabet assumptions; they do not quantify how tightly the practical objectives satisfy the stated TV bounds during optimization.

Empirical results

On ImageNet classification across ViT-T/B, DeiT-T/S/B, and Swin-T, MaskAQ consistently outperforms CNN-oriented baselines (GDFQ, Qimera, AdaDFQ), ViT-specific methods (PSAQ-ViT V2, CLAMP-ViT), generative-data GenQ, and MimiQ. Representative results:

Setting	Backbone	Best prior art	MaskAQ
4w4a	ViT-B	67.50 (GenQ)	70.53
4w4a	DeiT-B	74.10 (MimiQ)	74.41
5w5a	DeiT-S	72.59 (MimiQ)	73.18
3w3a	DeiT-T	19.55 (MimiQ)	22.65
3w3a	Swin-T	42.90 (MimiQ)	44.98

The largest reported gain is up to 3.10% Top-1 accuracy at 3-bit on DeiT-T, and up to 7.62% over MimiQ overall; gains over PSAQ-ViT variants reach at least 18.91% at 4w4a. Notably, most competing methods collapse to near-random accuracy at 3w3a, so only reproduced MimiQ numbers are reported there — meaning the ultra-low-bit comparison rests on a single baseline. On downstream tasks with Swin-T, MaskAQ improves over MimiQ by up to 0.92 AP on COCO detection and 1.24 mIoU on ADE20K segmentation at 4w4a.

Ablations confirm each component contributes: removing $Q$9 causes the largest degradation (1.57–2.05%), followed by $P$0 (DeiT-T 3w3a drops from 22.65% to 20.97%), periodic refreshing (0.79–1.22%), and unmasked calibration (smallest effect). Hyperparameter sweeps identify optima at $P$1 and $P$2. Visualizations show MaskAQ's synthetic samples exhibit more coherent structure and larger attended object regions, with improved $P$3–$P$4 attention alignment.

Limitations and open questions

The authors acknowledge that MaskAQ depends on iterative image synthesis, incurring nontrivial generation overhead relative to single-pass alternatives, and that applicability to more aggressive quantization settings (below 3 bits) remains unexplored. Additional caveats bear on the interpretation of results: the theoretical guarantees assume discrete alphabets and TV closeness conditions that are not verified empirically; the 3-bit comparison lacks most baselines due to instability; and the Gaussian approximation to attention-similarity distributions, while empirically stabilizing, is an approximation whose validity across architectures is not characterized. Whether the informative-region selection transfers reliably to domain-shifted or fine-grained settings where attention may be diffuse is left open.

Conclusion

MaskAQ reformulates data-free sample synthesis for ViTs around preserving calibration-relevant mutual information rather than mimicking real data. By decoupling sparse informative regions via differential entropy maximization, coupling them to the evolving quantized model through masked attention alignment, and periodically refreshing samples, it achieves consistent state-of-the-art results across backbones, tasks, and bit widths, with its clearest advantage at ultra-low precision where distribution mismatch between synthetic samples and $P$5's expected inputs is most severe.