Class-Feature Bias: Mechanisms & Mitigation

• Class-feature bias is the class-dependent asymmetry in feature contributions, where a feature’s discriminative power is measured pair-wise rather than globally.
• It is induced by factors like class imbalance, spurious correlations, and spectral variations that skew feature representations towards majority classes.
• Mitigation strategies include regularization methods, geometric adjustments, and sample selection techniques to enhance fairness and generalization.

Class-feature bias denotes class-dependent asymmetry in how features contribute to prediction, explanation, or representation. In the multi-class explainability setting, it refers to the fact that a feature is not equally good at making distinctions between different classes, so its utility is better characterized by pair-wise class-distinguishing capabilities than by a single global importance score (Sadhukhan et al., 2022). Across adjacent literatures, the same expression is used for several closely related phenomena: systematic dependence of predictions on proxy features for unavailable sensitive attributes, dominance of majority-class directions in imbalanced embedding spaces, class-specific performance gaps induced by spectral geometry, and over-reliance on features that are informative for only a subset of classes (Zhao et al., 2021, Kishanthan et al., 23 Dec 2025, Kaushik et al., 2024, Zuo et al., 9 Aug 2025). This suggests a unifying interpretation: class-feature bias is a mismatch between the feature structure learned or exploited by a model and the class structure that should govern robust, fair, and generalizable decision-making.

1. Conceptual scope and principal meanings

The expression is not attached to a single canonical definition. Instead, different subfields operationalize it according to the object being analyzed: raw input variables, latent embeddings, proxy attributes, spectral statistics, or classifier-head parameters. In "Reconnoitering the class distinguishing abilities of the features, to know them better" the emphasis is explicitly on explaining features by their class or category-distinguishing capabilities, especially for pair-wise class combinations in multi-class data, and on reusing those scores in a latent feature context together with a “refuse to render decision” option (Sadhukhan et al., 2022).

A broader fairness-oriented usage defines class-feature bias as indirect dependence of the predicted class on non-sensitive features that are highly correlated with sensitive attributes when the sensitive attributes themselves are unavailable during training (Zhao et al., 2021). In imbalance-aware representation learning, the term is used for systematic skew in learned feature representations where majority classes dominate the embedding space during deep network training (Kishanthan et al., 23 Dec 2025). In medical diagnosis, class-feature bias is defined as reliance on features that are strongly correlated with only a subset of classes, thereby producing biased performance and poor generalization on the remaining classes (Zuo et al., 9 Aug 2025).

Setting	Operational meaning	Representative source
Multi-class explainability	Pair-wise class-distinguishing capability of features	(Sadhukhan et al., 2022)
Fairness without sensitive attributes	Dependence of predictions on proxy features correlated with sensitive attributes	(Zhao et al., 2021)
Class imbalance in deep learning	Majority-class domination of embedding directions and separability	(Kishanthan et al., 23 Dec 2025)
Medical diagnosis	Reliance on features informative for only a subset of classes	(Zuo et al., 9 Aug 2025)
Dataset bias analysis	Feature importance disparity across subgroups	(Chang et al., 2023)

These formulations differ in emphasis, but they converge on a common structural problem: feature relevance is class-conditional rather than globally uniform. Some works study this as an interpretability issue, others as unfairness, others as a source of catastrophic forgetting or poor calibration. The shared concern is that aggregate accuracy or global feature importance can conceal class-specific distortions.

2. Mechanisms that generate class-feature bias

One major mechanism is class imbalance. In deep representation learning, the optimizer sees more gradients from majority-class samples, and these gradients shape the representation subspace to favor majority clusters. The reported manifestations include majority classes occupying most “energy” directions, minority examples being mapped into directions aligned with majority features, higher within-class scatter for minority classes, and lower between-class separation relative to majority classes (Kishanthan et al., 23 Dec 2025). In binary facial-attribute classification, an even more specific pathology appears: majority-class predictions can be driven by global, non-informative signals or even by the final-layer bias term rather than by localized semantic evidence, whereas minority-class activations are often more reasonable and localized (Zhang et al., 2024).

A second mechanism is dataset-level spurious correlation. In image benchmarks, common-sense co-occurrence patterns such as target objects appearing with specific contexts can distort the decision boundary, so that performance varies across groups defined by feature presence or absence. The relevant object is not only a protected attribute but any human-interpretable feature whose empirical co-occurrence with the label becomes a shortcut (Zhang et al., 2024). Related work on debiasing via bias-contrastive pairs makes the same point spatially: deep networks rely on bias attributes that are spuriously correlated with a target class, and the key failure is a shift from intrinsic features to peripheral, non-intrinsic cues (Park et al., 2024).

A third mechanism is representational geometry. Spectral imbalance theory treats class disparities as a property of class-conditional covariance spectra rather than of sample counts. Different classes can have different eigenspectra even in balanced datasets, and these spectral differences directly control per-class margins and errors for downstream linear classifiers (Kaushik et al., 2024). In supervised and unsupervised contrastive learning, a distinct but related mechanism arises through simplicity bias in gradient methods: supervised contrastive learning can collapse subclasses within a class, while unsupervised contrastive learning can suppress harder class-relevant features by focusing on easy class-irrelevant ones (Xue et al., 2023).

Regularization and normalization create yet another route. For lasso, ridge, and elastic net on binary features, class balances $p_j$ directly influence the regression coefficients, and the effect depends on the normalization scheme. In this setting, the feature’s proportion of ones alters shrinkage and selection, so coefficient magnitudes can reflect feature prevalence as much as predictive structure (Larsson et al., 7 Jan 2025). Earlier work on feature-wise bias amplification connects this to gradient-descent inductive bias: with insufficient training data, moderately predictive weak features can be overestimated, and if more such features are oriented toward one class than the other, the classifier amplifies the class disparity beyond the ground truth (Leino et al., 2018).

3. Formalizations and diagnostic measures

The most direct formalization treats class-feature bias as pair-wise separability. The 2022 multi-class explainability formulation estimates class-distinguishing capabilities of variables for pair-wise class combinations, then uses the resulting scores both for explanation and for latent decision making (Sadhukhan et al., 2022). This formulation is diagnostic rather than explicitly fairness-based: it asks which features separate which classes, and with what asymmetry.

A fairness-oriented formalization uses dependence penalties. Let $\hat{y}=f_\theta(x)$ and let $z_i$ denote non-sensitive related features that are highly correlated with the unavailable sensitive attribute. The core objective combines binary cross-entropy with a covariance-based regularizer,

$$\min_{\theta,\boldsymbol{\lambda}} \ \mathcal{L}_{\text{cls}(\theta)} +\eta \sum_{i=1}^{m}\lambda_i\,R(\hat{\mathbf{y}},z_i) +\beta\|\boldsymbol{\lambda}\|_2^2,$$

subject to $\lambda_i\ge 0$ and $\sum_i \lambda_i=1$, where $R$ is the absolute covariance between predictions and each related feature (Zhao et al., 2021). The accompanying correlation-propagation theorem states that if a proxy feature is strongly correlated with the sensitive attribute and training drives $\operatorname{corr}(z_i,\hat{y})$ toward $0$, then $\operatorname{corr}(s,\hat{y})$ is also driven toward $\hat{y}=f_\theta(x)$0 under the stated conditions (Zhao et al., 2021).

A complementary diagnostic tradition searches over feature-defined subgroups. In predictive-bias scanning, the null assumes that the odds of the outcome are correctly given by the predicted probabilities, while the alternative multiplies the subgroup odds by a factor $\hat{y}=f_\theta(x)$1. The resulting generalized likelihood-ratio statistic is

$\hat{y}=f_\theta(x)$2

which allows statistically significant over- or under-prediction to be detected over exponentially many subgroups via subset scan (Zhang et al., 2016). A related subgroup-centric diagnostic is Feature Importance Disparity, defined by the gap between global and subgroup feature importance, $\hat{y}=f_\theta(x)$3, with efficient search over rich subgroup classes (Chang et al., 2023).

Representation-level diagnostics are more geometric. In the high-dimensional spectral framework, the per-class probability of error is

$\hat{y}=f_\theta(x)$4

making class-conditional covariance structure an explicit determinant of class disparity (Kaushik et al., 2024). The same work proposes the Spectral Quantile Score to summarize cross-class spectral imbalance. In medical diagnosis, the diagnostic quantity is class-wise loss inequality: if $\hat{y}=f_\theta(x)$5, then either class-feature bias or class imbalance, or both, are present. This motivates the inequality penalty

$\hat{y}=f_\theta(x)$6

which treats equal class-wise loss as a proxy for equal informativeness across classes (Zuo et al., 9 Aug 2025).

4. Mitigation strategies

One family of methods directly regularizes prediction–feature dependence. FairRF uses related features both for prediction and for fairness, with dynamic regularization weights that adapt to the current covariance between predictions and proxies (Zhao et al., 2021). In dataset-level debiasing, CSBD extracts noun phrases from captions, clusters them into semantic features, measures pairwise $\hat{y}=f_\theta(x)$7 correlations, and then uses weighted sampling so that $\hat{y}=f_\theta(x)$8, thereby removing dataset-level dependence of a sensitive feature on the target (Zhang et al., 2024).

A second family modifies representation geometry. OGAB introduces an activation layer that combines an exactly orthogonal Cayley-transform map,

$\hat{y}=f_\theta(x)$9

with an implicit group-aware bias

$z_i$0

followed by $z_i$1. The reported rationale is that orthogonality preserves feature independence and norms, while the learned group-aware bias shifts embeddings to enhance separability without explicit supervision (Kishanthan et al., 23 Dec 2025). A related image-fairness approach identifies a dominant bias direction from differences between protected class prototypes, then removes its projection from the feature vector,

$z_i$2

while also replacing one-hot labels with protected-value-specific label embeddings (Thong et al., 2021).

A third family operates through explicit feature selection or balancing. Feature-wise bias amplification can be mitigated by “Feature parity” or “Experts,” both of which remove low-influence features whose aggregate orientation amplifies class disparity while preserving or improving accuracy in the reported experiments (Leino et al., 2018). In regularized regression, the class-balance effect of binary predictors can be mitigated by scaling binary features with their variance for lasso and with their standard deviation for ridge; for elastic net, equivalent mitigation can be achieved by scaling the penalty weights instead of the features (Larsson et al., 7 Jan 2025).

A fourth family uses sample selection, pair construction, or self-training. DCAST performs class-aware pseudo-labeling with diversity constraints by selecting $z_i$3 confident candidates per class, clustering them, and then choosing the highest-probability sample from each cluster; this is designed to counter confirmation bias while leveraging unlabeled data under class-aware bias (Tepeli et al., 2024). In spatial debiasing, bias-contrastive pairs identify common class-discerning features between a bias-aligned sample and a bias-conflicting sample, and the model amplifies those under-exploited intrinsic regions using an intrinsic-feature mask derived from similarity and Grad-CAM-based relative exploitation (Park et al., 2024).

Finally, explicit class-wise balancing can target the loss itself. Class-Unbias combines the inequality penalty with a class-wise group DRO objective,

$z_i$4

where the group DRO weights are a stop-gradient softmax over class-wise losses. The reported effect is simultaneous mitigation of class imbalance and class-feature bias in binary medical diagnosis (Zuo et al., 9 Aug 2025).

5. Specialized manifestations in incremental learning and unlearning

In class-incremental learning, class-feature bias is typically decomposed into representation bias and classifier bias. Generative Feature Replay splits the model into a feature extractor and a classifier, uses feature distillation to keep $z_i$5 close to $z_i$6, and trains the classifier on current-task real features together with generated old-task features. The explicit aim is to counter classifier skew toward new classes and feature drift for old classes without storing exemplars (Liu et al., 2020).

PASS++ adopts the same two-bias language but with a different mechanism. Self-Supervised Transformation in input space learns generic and diverse representations, while prototype augmentation in feature space explicitly or implicitly augments old-class prototypes to preserve old decision boundaries. The total objective is

$z_i$7

and the method further adds hardness-aware prototype augmentation and multi-view ensemble (Zhu et al., 2024). In few-shot class-incremental learning, the same structural issue is abstracted as a unified model-bias problem: freezing the feature extractor yields a Base/Inc imbalance, while fine-tuning it can create a current-versus-past incremental imbalance, motivating mapping ability stimulation, separately dual-feature classification, and self-optimizing classifiers (Zhao et al., 2024).

Class-level unlearning reveals a closely related parameter-level version. Under retain-set-only softmax cross-entropy, the bias gradient for an absent class $z_i$8 is

$z_i$9

so gradient descent monotonically decreases $$\min_{\theta,\boldsymbol{\lambda}} \ \mathcal{L}_{\text{cls}(\theta)} +\eta \sum_{i=1}^{m}\lambda_i\,R(\hat{\mathbf{y}},z_i) +\beta\|\boldsymbol{\lambda}\|_2^2,$$0. This creates a bias-dominated shortcut: forgotten classes can be suppressed largely by shifting their classification-head biases downward, even if feature-level traces remain (Zheng et al., 9 May 2026). The resulting diagnostic baseline, BiasShift, can satisfy conventional unlearning metrics while leaving abnormal bias patterns that reveal the forgotten labels. To reduce this dependence, the work introduces TS-BGRM and LB-HR, together with bias-oriented metrics BSC, MBG, and MBS (Zheng et al., 9 May 2026).

These continual and unlearning settings show that class-feature bias is not restricted to static fairness audits. It also appears whenever class exposure is temporally uneven, supervision is partial, or optimization pressure can be absorbed by a low-cost shortcut in the head rather than by a genuine redistribution of representation-level evidence.

6. Limitations, controversies, and research directions

A central limitation is definitional heterogeneity. Some works reserve class-feature bias for pair-wise class-distinguishing ability of features, others use it for proxy discrimination, majority-class domination, subgroup-specific importance disparity, or parameter-level head suppression. This plurality has been productive, but it also makes cross-paper comparison difficult. A plausible implication is that future work will need a clearer hierarchy separating feature-level, subgroup-level, representation-level, and head-level notions.

Several methods also inherit strong assumptions from their measurement strategies. Proxy-based fairness regularization assumes the existence of non-sensitive related features that are highly correlated with the unavailable sensitive attribute, and it relies primarily on covariance or Pearson correlation, which may miss nonlinear dependence (Zhao et al., 2021). CSBD requires per-image text descriptions, depends on clustering quality and a human-in-the-loop review, and analyzes pairwise rather than higher-order correlations (Zhang et al., 2024). CAM-based analyses of majority-class “bias activation” remain post-hoc and, as explicitly noted, indicate correlation rather than causation (Zhang et al., 2024).

On the modeling side, not every mitigation comes with a formal proxy for the targeted bias. OGAB demonstrates improved balanced metrics and t-SNE separation, but the paper does not provide a formal measurable proxy such as scatter matrices $$\min_{\theta,\boldsymbol{\lambda}} \ \mathcal{L}_{\text{cls}(\theta)} +\eta \sum_{i=1}^{m}\lambda_i\,R(\hat{\mathbf{y}},z_i) +\beta\|\boldsymbol{\lambda}\|_2^2,$$1 or cluster-overlap metrics (Kishanthan et al., 23 Dec 2025). Spectral imbalance theory provides exact expressions in a high-dimensional mixture setting, but its analysis is built around linear downstream classifiers, Gaussian structure, and asymptotics (Kaushik et al., 2024). Incremental-learning methods such as PASS++ reduce forgetting without exemplars, yet the same work reports vulnerability under distribution shift, indicating that bias reduction in one sense does not automatically deliver robustness in another (Zhu et al., 2024).

Recurring future directions include better proxy discovery, nonlinear dependence penalties such as HSIC, causal or counterfactual validation of feature use, multivariate bias induction beyond pairwise correlations, and more explicit representation-level leakage metrics in settings such as machine unlearning (Zhao et al., 2021, Zhang et al., 2024, Zheng et al., 9 May 2026). The multi-class explainability line initiated by pair-wise class-distinguishing scores also points toward richer decision protocols, including latent-space scoring and abstention when the evidence structure itself is diagnostically unstable (Sadhukhan et al., 2022).