Generative Classifiers Avoid Shortcut Solutions (2512.25034v1)

Abstract: Discriminative approaches to classification often learn shortcuts that hold in-distribution but fail even under minor distribution shift. This failure mode stems from an overreliance on features that are spuriously correlated with the label. We show that generative classifiers, which use class-conditional generative models, can avoid this issue by modeling all features, both core and spurious, instead of mainly spurious ones. These generative classifiers are simple to train, avoiding the need for specialized augmentations, strong regularization, extra hyperparameters, or knowledge of the specific spurious correlations to avoid. We find that diffusion-based and autoregressive generative classifiers achieve state-of-the-art performance on five standard image and text distribution shift benchmarks and reduce the impact of spurious correlations in realistic applications, such as medical or satellite datasets. Finally, we carefully analyze a Gaussian toy setting to understand the inductive biases of generative classifiers, as well as the data properties that determine when generative classifiers outperform discriminative ones.

• The paper demonstrates that generative classifiers, by modeling the full input distribution p(x|y), robustly avoid shortcut solutions compared to discriminative models.
• The paper employs state-of-the-art diffusion models for images and autoregressive transformers for text, achieving higher in-distribution and out-of-distribution accuracy on multiple benchmarks.
• The paper’s analysis reveals that sustained gradient signals in generative models underpin their effective robustness against spurious correlations and distribution shifts.

Generative Classifiers and Shortcut Avoidance

Introduction

The paper "Generative Classifiers Avoid Shortcut Solutions" (2512.25034) critically examines the robustness of generative classifiers—specifically, those utilizing modern deep class-conditional generative models—to distribution shift in classification tasks. It addresses a core deficiency of discriminative classifiers: a pronounced susceptibility to shortcut learning, where spurious correlations in the training distribution are exploited, undermining generalization under even minor distributional shifts. The authors advance the thesis that generative classifiers, due to their inherent requirement to model the full input distribution $p(x|y)$, are significantly less prone to overfitting to such spurious features compared to discriminative models, which optimize $p(y|x)$ and often disregard core features once high-confidence predictions can be achieved via easy shortcuts.

Methodology

The generative classifier paradigm models $p(x|y)$ for each class independently and leverages Bayes' rule at inference, computing $p(y|x) \propto p(x|y)p(y)$. The study repurposes state-of-the-art diffusion models for image classification and autoregressive transformers for text, deploying them as generative classifiers without architectural or objective modifications. Notably, no special augmentations, explicit regularization, or annotation of spurious correlations are necessary, facilitating ease of deployment relative to fairness-centric ERM variants or explicit debiasing.

For images, diffusion classifiers are constructed by estimating the class-conditional likelihood via the variational bound used in standard diffusion training, applying the method of Li et al. (2023). For text, the authors slightly adapt the sequence prefix to correspond to the class token, enabling straightforward estimation of $p(x|y)$ by autoregressive log-likelihood aggregation.

Experimental Evaluation

Benchmarks and Baselines

Robustness is systematically assessed on five canonical distribution shift benchmarks—Waterbirds, CelebA, Camelyon17, FMoW, and CivilComments—spanning both domain and subpopulation shifts. Discriminative baselines include ERM and advanced group-robust variants such as LfF, JTT, and DFR/RWY. All models are trained from scratch for a fair comparison of inductive biases.

Main Results

Generative classifiers exhibit consistently superior out-of-distribution (OOD) and worst-group accuracy across all five datasets. Strikingly, they also yield higher in-distribution accuracy on a majority of datasets, contradicting the common compromise between ID and OOD performance seen with prior debiasing techniques. A salient property, "effective robustness" (OOD performance exceeding discriminative models at matched ID accuracy), is observed robustly in settings such as CelebA and CivilComments. These empirical trends persist under large-scale ablation and are not attributable to increased model parameter counts; increasing discriminative model size does not close the gap, nor does adding unconditional generative objectives to discriminative training.

Analysis of Inductive Bias

Gradient Signal and Shortcut Suppression

Empirical gradient norm tracking reveals that generative classifiers maintain substantive learning signal across both majority (shortcut-consistent) and minority (shortcut-inconsistent) groups throughout training. Conversely, discriminative training rapidly decays the gradient for majority group instances, effectively "starving" the model of signal to learn core features once the shortcut suffices. This aligns with the theoretical expectation: the objective of generative models compels comprehensive input modeling, making reliance on only a subset of spurious features infeasible.

Gaussian Toy Setting

A suite of controlled experiments using Gaussian mixtures, varying the strength of core, spurious, and noisy features, demonstrates the different regimes in which generative classifiers' inductive biases outperform those of discriminative approaches. With low core feature variance and strong spurious correlations, generative classifiers (LDA) downweight spurious features more efficiently and generalize better on rare/minority groups. As core feature variance increases, the generative approach's advantage is attenuated, establishing that no method is universally superior—performance depends on the reliability of the core feature versus the strength of spurious and noisy dimensions.

These findings are encapsulated in "generalization phase diagrams," delineating regions where each approach is superior for both ID and OOD performance, as functions of spurious correlation strength and noise.

Implications and Future Directions

Theoretical

The study challenges the entrenched assumption that discriminative paradigms are universally preferable in high-dimensional machine learning. It empirically and analytically substantiates that generative classifiers, benefited by modern deep generative modeling, are inductively biased towards robust, core features. This property is especially salient in scenarios where the true data-generating process contains reliable, low-variance features amidst strong spurious correlations or high-dimensional noise.

Practical

Generative classifiers promise a compelling, hyperparameter-free approach for robust real-world deployment in domains like medical imaging or satellite data, where annotation of potential spurious correlations is infeasible. Their robustness to distribution shift is not achieved at the cost of ID accuracy, and they synergize with existing generative modeling infrastructure.

However, inference latency and computational cost—especially with diffusion models—remain a bottleneck. Additionally, integrating data augmentations or more complex compositional splits within generative classification frameworks is an open challenge.

Future Directions

• Scalability and Efficiency: Reduction of generative classifier inference cost—through amortized likelihood estimation or distillation—is necessary for deployment.
• General Applications: Extension of generative classification to complex structured output scenarios (e.g., language modeling tasks beyond classification).
• Understanding Data Properties: Deeper characterization of when generative biases are beneficial, allowing practitioners to select the correct paradigm based on empirical data properties.

Conclusion

This work rigorously demonstrates that modern generative classifiers, leveraging advances in diffusion and autoregressive models, avoid shortcut solutions endemic to discriminative classifiers under distribution shift. Their robust generalization emerges from modeling the full data distribution rather than optimizing only predictive margin; this is substantiated both on real-world benchmarks and theoretical analysis. These results strongly motivate further investigation and practical adoption of generative classification paradigms in high-stakes and variable environments.