Quantization-Induced Masked Bias Flipping

• The paper demonstrates that post-training quantization can flip response-level bias labels, even though overall bias measures remain unchanged.
• It employs paired analysis and uncertainty metrics to compare weight-only and weight-activation quantization regimes across multiple benchmarks.
• The study suggests mitigation strategies, including Fair-GPTQ and favoring 8-bit precisions, to balance model compression with reduction of subgroup bias disparities.

Searching arXiv for the cited papers to ground the article in current literature. Quantization-induced masked bias flipping is the phenomenon whereby post-training quantization changes the bias state of individual outputs or pairwise preferences in a LLM while leaving aggregate bias summaries apparently stable. In recent studies of weight-only and weight-activation quantization, the effect appears when compression increases uncertainty, compresses probability gaps, and thereby makes conventional aggregate metrics look neutral even though response-level or subgroup-level behavior has changed materially, sometimes in opposite directions across demographic groups (Marcuzzi et al., 25 Aug 2025, Hua et al., 5 Feb 2026).

1. Definition and conceptual scope

In the response-level formulation, let $y_i\in\{0,1\}$ denote the pre-quantization bias label for example $i$, with $0=\text{unbiased}$ and $1=\text{biased}$, and let $\tilde y_i$ be the post-quantization label. The bias-flip indicator is

$F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$

and the overall flip rate is

$$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$

Under this definition, masked bias flipping refers not merely to a shift in mean bias, but to hidden churn in the mapping from inputs to biased or unbiased outputs (Hua et al., 5 Feb 2026).

A paired-input formulation appears in fairness-aware quantization work. If $X_0$ and $X_1$ are stereotypical and anti-stereotypical inputs, $W$ are full-precision weights, and $i$0 are quantized weights, quantization-induced flipping occurs when a model’s relative preference reverses after quantization. One concrete condition is that

$i$1

Equivalently, for $i$2, the scalar $i$3 can change sign even though $i$4 had the opposite sign. This formulation isolates flipping as a change in ordering, not only a change in score magnitude (Proskurina et al., 18 Sep 2025).

The qualifier “masked” is crucial. In these studies, masking does not mean that the bias is removed. It means that the change is obscured by aggregate metrics that remain near their original values even as individual responses reverse. This suggests that quantization can preserve dataset-level averages while altering the local decision geometry of the model.

2. Quantization regimes in which the phenomenon is studied

The underlying compression setting is standard post-training quantization (PTQ), in which real-valued tensors $i$5 are replaced or simulated by discretized tensors

$i$6

with $i$7 as a per-tensor or per-channel scale factor. The evaluated methods include AWQ and GPTQ as weight-only PTQ methods, and SQ as a weight-activation method that also quantizes activations to $i$8 after optionally rescaling outliers. The reported bit-widths are AWQ at W3, W4, and W8; GPTQ at W4 and W8; and SQ at W4A8 and W8A8 (Marcuzzi et al., 25 Aug 2025).

A larger-scale study broadens the PTQ landscape to Round-to-Nearest (RTN) at W4A16 and W8A16, GPTQ at W4A16, AWQ at W4A16, and SmoothQuant + RTN at W4A16, all applied in place on pre-trained weights without further fine-tuning. The models span ten instruction-tuned systems across LLaMA 3.1/3.2, Mistral 8B, and Qwen 2/2.5 from 0.5B to 14B parameters (Hua et al., 5 Feb 2026).

The compression ratios are substantial. For LLaMA-8B, the original size is 15 GB; AWQ W3 reduces it to $i$9 GB, described as $0=\text{unbiased}$0 shrink, while W4 and W8 reduce it to 5.3 GB and 8.6 GB. For Qwen-14B, the original size is 27.5 GB; corresponding compressed sizes include 5.3 GB, 8.6 GB, 5.2 GB, and 8.5 GB depending on the method and precision setting. These gains motivate quantization in practice, but they also define the compression regime in which masked bias flipping has been documented (Marcuzzi et al., 25 Aug 2025).

3. Measurement frameworks and why aggregate scores can be misleading

The phenomenon is visible only when quantized outputs are compared directly against their float32 or higher-precision baselines using paired analyses. One line of work evaluates probability-based stereotypes, decision-level fairness, and generated text using nine benchmarks: StereoSet, RedditBias, WinoBias, DiscrimEval, DiscrimEvalGen, DT-Fairness, BOLD, DT-Toxicity, and MMLU as a capability reference. Probability-based stereotype change is measured through quantities such as $0=\text{unbiased}$1, while fairness is measured with $0=\text{unbiased}$2, $0=\text{unbiased}$3, Demographic Parity Difference, and Equalized Odds Difference. Toxicity and sentiment are evaluated through generated continuations using Toxic-BERT and VADER, with inter-subgroup disparity measured by average absolute pairwise subgroup differences (Marcuzzi et al., 25 Aug 2025).

A second line of work introduces PostTrainingBiasBench, a unified framework spanning thirteen datasets and three capability groupings: bias identification, equal outcomes under informative context, and safe or uncertain response to biased prompts. Its datasets include CEB-Recognition, Jigsaw, Adult, Credit, BiasLens-Choices, SocialStigmaQA, BBQ, IAT, StereoSet, BiasLens-GenWhy, CEB-Continuation, CEB-Conversation, and FairMT10K. Each dataset yields an aggregate bias score in $0=\text{unbiased}$4, with higher values indicating more bias, but closed-ended tasks additionally supply per-response bias labels, making it possible to observe flips that aggregate scores obscure (Hua et al., 5 Feb 2026).

This measurement distinction explains why the same quantized model can appear approximately unchanged under a dataset-level score and yet exhibit substantial response-level instability. A plausible implication is that benchmark design determines whether quantization-induced bias changes are observable: probability-based averages can conceal local reversals, whereas paired response labels or generated-text decisions reveal them directly.

4. Empirical signatures across stereotypes, fairness, toxicity, and subgroup effects

The most direct quantitative signature is the response-level flip rate. Across 50 quantized variants and 13 datasets, up to $0=\text{unbiased}$5 of individual responses flip bias label even when the dataset’s aggregate bias score is not statistically significant. Quantization strength strongly modulates this effect: 4-bit quantizations produce $0=\text{unbiased}$6 more overall behavior-flips than 8-bit quantizations. The reported averages are approximately $0=\text{unbiased}$7 for RTN W8A16, $0=\text{unbiased}$8 for GPTQ W4A16, $0=\text{unbiased}$9 for AWQ W4A16, $1=\text{biased}$0 for RTN W4A16, and $1=\text{biased}$1 for SmoothQuant + RTN W4A16 (Hua et al., 5 Feb 2026).

On stereotype benchmarks, the reported picture is mixed across metric types. For probability-based StereotypeScore, shifts are mild and tend toward $1=\text{biased}$2. On RedditBias with LLaMA, the reported examples are W3 AWQ with $1=\text{biased}$3, W4 GPTQ with $1=\text{biased}$4, and W8 GPTQ with $1=\text{biased}$5. Yet the effect-size formulation based on Cohen’s $1=\text{biased}$6 can flip sign at low bits, meaning that models begin to prefer anti-stereotype sentences. In generated text, however, stereotypes can increase under heavy quantization: with W3 AWQ on LLaMA, historical bias in WinoBias rises by $1=\text{biased}$7 percentage points, from $1=\text{biased}$8 percentage points to $1=\text{biased}$9 percentage points (Marcuzzi et al., 25 Aug 2025).

On fairness benchmarks, probability-based metrics can again look small while generative metrics worsen. DiscrimEval changes are reported as small, under $\tilde y_i$0 percentage points except W4A8. By contrast, in DiscrimEvalGen the unbiased-answer rate drops from $\tilde y_i$1 to $\tilde y_i$2 under AWQ W8, while subgroup disparity rises from $\tilde y_i$3 percentage points to $\tilde y_i$4 percentage points. For DT-Fairness on DS-LLaMA under AWQ W3, Demographic Parity Difference increases from $\tilde y_i$5 percentage points to $\tilde y_i$6 percentage points and Equalized Odds Difference rises from $\tilde y_i$7 percentage points to $\tilde y_i$8 percentage points, both marked with $\tilde y_i$9 (Marcuzzi et al., 25 Aug 2025).

Toxicity and sentiment exhibit a different profile. Raw toxicity can decrease markedly with aggressive quantization: on DS-LLaMA using BOLD, AWQ W3 yields $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$0, AWQ W4 yields $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$1, and GPTQ W4 yields $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$2. Inter-subgroup toxicity gaps remain within $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$3 percentage points. Sentiment shifts are described as minor neutralization under heavy quantization, with at most a $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$4 percentage point drop, while inter-subgroup sentiment differences remain below $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$5 percentage points in most cases (Marcuzzi et al., 25 Aug 2025).

Subgroup asymmetry is a central empirical point. For the same quantized model, bias can worsen by up to $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$6 for some groups while improving by $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$7 for others, producing aggregate outcomes that appear neutral. One reported example is Qwen 2.5 0.5B with RTN W4A16, where “male” prompts worsen by $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$8 and “short” prompts improve by $F_i=\mathbf{1}\{y_i\neq \tilde y_i\},$9. Aggregated over all models, “male” prompts see a net $$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$0 change and “short” prompts a net $$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$1 change (Hua et al., 5 Feb 2026).

5. Mechanistic explanations: uncertainty, compression, and metric-dependent masking

The central mechanism identified in recent work is quantization-induced uncertainty. In closed-ended tasks, uncertainty is measured by normalized Shannon entropy,

$$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$2

Responses are stratified into Low $$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$3, Medium $$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$4, and High $$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$5 uncertainty. High-uncertainty responses flip $$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$6 of the time, while low-uncertainty responses flip less than $$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$7. High-uncertainty examples are $$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$8 more likely to flip than confident ones. Preference tuning with SimPO and explicit entropy maximization further establish a dose–response relation in which increasing pre-quantization entropy causally raises post-quantization flip rate (Hua et al., 5 Feb 2026).

A complementary mechanistic account explains why probability-based stereotype metrics may suggest bias reduction. In the quantified ablations, quantization lowers the average log-likelihood of both pro-stereotype and anti-stereotype sentences symmetrically. Because probability-based metrics rely on relative differences between two small numbers, those differences shrink toward zero as both likelihoods decline. The result is an apparent reduction in measured bias even when the model has not become more equitable in behavior. Generated metrics such as WinoBias accuracy gaps or DiscrimEvalGen counts are less susceptible to this masking because they require actual resolution of fine-grained reasoning or adherence to choose-one instructions, capacities that aggressive quantization degrades unevenly across subgroups (Marcuzzi et al., 25 Aug 2025).

Standard GPTQ helps clarify why this can occur. Its layer-wise objective is

$$\mathrm{FlipRate}=\frac{1}{N}\sum_{i=1}^N F_i \times 100\%.$$9

or, in the Optimal Brain Surgeon formalism, minimization of $X_0$0 under per-weight quantization constraints. Because this objective minimizes reconstruction error on calibration activations but does not constrain how stereotype pairs $X_0$1 are mapped, it can amplify existing biases. This suggests that masked bias flipping is not an incidental artifact of one benchmark family, but can arise from the mismatch between generic reconstruction objectives and fairness-sensitive pair structure (Proskurina et al., 18 Sep 2025).

6. Mitigation strategies and fairness-aware quantization

One practical response is to moderate the quantization regime. Reported recommendations are to use 8-bit weights W8 or 4-bit weights W4 via GPTQ or AWQ, which minimize bias amplification while still providing a $X_0$2 compression, and to avoid ultra-aggressive settings such as W3 or W4A8 in SQ, which degrade reasoning and can spike fairness or stereotype gaps by up to $X_0$3 percentage points. The same work recommends evaluating both probability-based and generative metrics across multiple bias dimensions, preferring subgroup-level analyses, and noting that reasoning-oriented DeepSeek models exhibit lower baseline bias and smaller post-quantization increases (Marcuzzi et al., 25 Aug 2025).

A more explicit intervention is Fair-GPTQ, described as the first quantization method explicitly designed to reduce unfairness in LLMs. It augments the GPTQ objective with a group-fairness penalty,

$X_0$4

where $X_0$5 and $X_0$6 are calibration activations for stereotypical and anti-stereotypical inputs and $X_0$7 weights the bias penalty. The added term explicitly minimizes differences in the quantized representations of paired inputs and thereby guides rounding toward less-biased text generation for protected groups defined by gender, race, and religion (Proskurina et al., 18 Sep 2025).

Empirically, Fair-GPTQ preserves at least $X_0$8 of baseline zero-shot accuracy, reduces unfairness relative to a half-precision model, retains the memory and speed benefits of 4-bit quantization, and incurs runtime increase of approximately $X_0$9 because the only extra cost over GPTQ is computing $X_1$0 and one additional matrix–matrix multiply $X_1$1. On Mistral-7B and OPT-6.7B, CrowS-Pairs and StereoSet scores drop by $X_1$2 points, with especially strong reductions when debiasing lower layers only. Weight-level analysis identifies the attention output projection and FC2 in OPT-6.7B, and MLP down-projection layers in Mistral-7B, as the matrix types receiving the largest first-order debiasing updates (Proskurina et al., 18 Sep 2025).

The broader post-quantization protocol proposed in the uncertainty study aligns with these interventions: always perform post-quantization bias evaluation at the response level and within subgroups; favor W8A16 over W4A16 in sensitive settings because 8-bit quantization exhibits $X_1$3 fewer flips; and use uncertainty as a pre-screening signal for high-risk questions or groups (Hua et al., 5 Feb 2026).

7. Terminological distinctions, misconceptions, and open questions

A recurrent misconception is to equate masked bias flipping with the “mask-flipping” error described for quantized masked softmax. In masked softmax, quantization of logits can make positions that should receive zero probability acquire tiny nonzero probability because a large negative masking constant becomes finite after quantization and dequantization. That work defines a separate correction problem: quantization of softmax outputs introduces a total bias

$X_1$4

and the proposed remedy is to absorb a constant offset into the asymmetric quantizer’s zero-point so that normalization is restored offline without extra runtime cost. This is a distinct phenomenon from social-bias masking, even though both involve quantization-induced bias and both can use the language of “flipping” (Pandey et al., 2023).

Another misconception is that larger models are inherently more robust to quantization-induced social bias changes. The available evidence does not support a monotonic robustness advantage: within the Qwen 2.5 family from 0.5B to 14B, flip rates vary erratically by model size and dataset, sometimes decreasing and sometimes increasing, with no clear monotonic trend (Hua et al., 5 Feb 2026).

Several open questions remain explicitly identified. Fair-GPTQ experiments are limited to OPT and Mistral and to English calibration data, so multilingual fairness and transfer to LLaMA-3, Qwen, or multimodal transformers remain to be explored. StereoSet calibration pairs are short, and longer narrative contexts may require extended minimal-pair datasets. Residual individual-example flips persist even when average bias is reduced, suggesting that instance-level pairing or contextual debiasing in higher-precision outlier channels may be necessary. More broadly, the observed fairness–accuracy trade-off indicates that small accuracy drops remain even under fairness-aware rounding, especially for rare or intersectional stereotypes (Proskurina et al., 18 Sep 2025).

Taken together, the literature characterizes quantization-induced masked bias flipping as a metric-sensitive and uncertainty-driven alteration of social bias under PTQ. The phenomenon is not reducible to a single benchmark or protected attribute, and it is not adequately described by aggregate bias scores alone. Its defining feature is that compression changes which examples are treated as biased or unbiased, often asymmetrically across subgroups, while conventional summaries may imply stability.