Manifold-Constrained Adversarial Training for Long-Tailed Robustness via Geometric Alignment

Abstract: Adversarial training is effective on balanced datasets, but its robustness degrades under longtailed class distributions, where tail classes suffer high robust error and unstable decision boundaries. We propose Manifold-Constrained Adversarial Training (MCAT), a unified framework that enforces the semantic validity of adversarial examples by penalizing deviations from class-conditional manifolds in feature space, while promoting balanced geometric separation across classes via an ETF-inspired regularization. We provide theoretical results that link geometric separation to lower bounds on adversarially robust margins, and show that manifold-constrained adversarial risk upperbounds robust risk on high-density semantic regions. Extensive experiments on standard longtailed benchmarks demonstrate consistent improvements in overall, balanced, and tail-class adversarial robustness.

• The paper introduces MCAT, which enforces semantic manifold constraints and ETF-inspired weight regularization to mitigate tail-class vulnerabilities in imbalanced data.
• It employs Manifold-Supported PGD and simplex ETF penalties to improve robust accuracy on CIFAR-10-LT, CIFAR-100-LT, and Tiny-ImageNet-LT benchmarks.
• Empirical and theoretical analyses demonstrate that MCAT enhances robust margins by maximizing inter-class angles and controlling off-manifold adversarial drift.

Manifold-Constrained Adversarial Training for Long-Tailed Robustness via Geometric Alignment

Problem Formulation and Motivations

The reliability of adversarial robustness under long-tailed data regimes remains a critical challenge for deep neural networks, with existing adversarial training (AT) protocols exhibiting substantial degradation of robust performance, especially for tail classes. Extensive empirical and theoretical analyses demonstrate that head-class-dominated optimization results in geometric misalignment (i.e., inter-class margin compression) and pronounced off-manifold adversarial drift for tail classes. These geometric failures manifest as fragile, spurious, and unstable decision boundaries under adversarial attack, severely limiting model trustworthiness in imbalanced domains.

Figure 1: Standard adversarial training leads to geometric imbalance and off-manifold adversarial drift for tail classes, while MCAT constrains class geometry and adversarial perturbations to semantically meaningful manifolds.

The MCAT Framework

Architectural Overview

MCAT (Manifold-Constrained Adversarial Training) introduces a dual approach: (1) direct regulation of adversarial perturbations in feature space via explicit class-conditional manifold constraints and (2) regularization of classifier weight geometry through a simplex Equiangular Tight Frame (ETF) penalty. The adversarial inner maximization employs Manifold-Supported PGD (MS-PGD) to bias the search within high-density semantic support, while the classifier head is aligned to exhibit maximized and uniform angular separation across all classes.

Figure 2: MCAT enforces a feature-space manifold distance penalty and ETF-inspired classifier weight regularization, leading to robust, margin-balanced boundaries.

Semantic Manifold Constraints

Class-conditional semantic manifolds are modeled via lightweight MLP generators trained on the feature embeddings of each class. For input $x$ and class $y$, the generator $G_y$ synthesizes an embedding approximating the class support. The off-manifold distance $d_{\mathcal{M}_y}(u) = \min_z \| u - G_y(z) \|_2^2$ is efficiently approximated and penalized in the training objective. Critically, these constraints are computationally practical even for scarce tail classes due to the lowered intrinsic dimensionality in representation space compared to pixel space.

ETF-Inspired Geometric Alignment

To counteract imbalance-induced geometric distortion, MCAT regularizes the classifier weight matrix $W$ to approach a simplex ETF structure—maximizing the minimum inter-class angle and enforcing norm equality. The ETF penalty on the classifier Gram matrix provably increases certifiable robust margins due to the direct relationship between angular separation and adversarial robustness radius, as formalized in Theorem 1.

Unified Min–Max Objective

The overall MCAT objective is

$$R_{MCAT}(\Theta) = \mathbb{E}_{(x, y) \sim D}\left[\max_{\|\delta\|_\infty \leq \epsilon}\left(\ell(f_\Theta(x + \delta), y) - \lambda d_{\mathcal{M}_y}(\phi_\Theta(x + \delta))\right)\right] + \beta \mathcal{R}_{geom}(\Theta),$$

where $\lambda$ and $\beta$ are tunable weights for semantic consistency and geometric alignment, respectively.

Theoretical Guarantees

The authors establish two main analytical results:

1. Geometric Lower Bound on Robust Radii: The minimum inter-class angle $\theta_{min}$ lower-bounds the certifiable adversarial robustness radius, with ETF geometry achieving the theoretically optimal separation under fixed dimension. When head-class optimization compresses tail-class angles, robust margins for tails collapse rapidly.
2. Manifold Constraint on Robust Risk: Penalizing the off-manifold drift in adversarial optimization ensures the robust risk within the true semantic class support is upper bounded (modulo an $O(\lambda^{-1})$ term due to imperfect constraint enforcement).

Empirical Results

Core Performance Metrics

MCAT is evaluated on CIFAR-10-LT, CIFAR-100-LT, and Tiny-ImageNet-LT with imbalance ratios up to IR=100. Under AutoAttack and strong PGD attacks, MCAT achieves new state-of-the-art robustness, with robust accuracy improvements that are especially pronounced for tail and balanced accuracy metrics.

Figure 3: Overall AutoAttack robustness as a function of increasing class imbalance severity shows MCAT outperforming all baselines, especially as imbalance intensifies.

Sensitivity and Ablations

Hyperparameter sweeps demonstrate the monotonic improvement of robustness and suppression of off-manifold drift with increasing $y$0 (manifold penalty) and that $y$1 governs the trade-off between enlarged angular margins and over-regularization. Component ablations reveal that each constituent—manifold constraint and ETF alignment—contributes distinctly and that their combination yields maximal effect.

Figure 4: Robustness versus $y$2 shows the effect of semantic manifold constraint weight.

Figure 5: Minimum inter-class angle increases with stronger geometric regularization, directly enhancing tail-class robustness.

Figure 6: MCAT substantially suppresses off-manifold adversarial drift for all class-frequency groups.

Feature Geometry Diagnostics

Direct visualization of feature embeddings confirms that MCAT preserves tighter, well-separated tail clusters while maintaining head-class compactness, as seen in 2D projections.

Figure 7: Embedding projections show MCAT’s effect in tightening and separating tail clusters relative to standard AT.

Per-Class Robustness

Robustness versus class frequency rank demonstrates that MCAT raises the worst-case adversarial performance floor, mitigating head-dominated skew and enhancing practical fairness in robust recognition.

Figure 8: Per-class robust accuracy under AutoAttack, sorted by class frequency rank, demonstrating consistent MCAT advantage.

Practical and Theoretical Implications

MCAT provides a framework for integrating geometric inductive biases and semantically-aware adversarial constraints with negligible computational overhead. This approach offers significant improvements for reliability and fairness in robust learning scenarios with label imbalance—a setting typical in real-world deployed systems. The analytic results unify recent ETF and neural collapse insights with robust training, reinforcing the criticality of feature geometry for certified and empirical robustness under distributional skew.

The method is broadly compatible with state-of-the-art adversarial protocols and can be extended to more complex vision architectures and other forms of data imbalance. Open questions include scaling MCAT's manifold estimation to high-dimensional, fine-grained, or non-image domains, and exploring interactions with distributionally robust optimization and self-supervised pretraining.

Conclusion

The manifold-constrained and geometry-regularized adversarial training framework of MCAT demonstrates significant improvements in adversarial robustness for long-tailed distributions, particularly benefitting tail classes and enhancing balanced accuracy. Its integration of class-conditional manifold penalties and ETF-inspired classifier alignment advances both theoretical understanding and practical robustness in realistic, imbalanced data settings (2605.02183).