Alignment Collapse in AI Systems

• Alignment Collapse is a phenomenon where optimization or adaptation procedures concentrate outputs, causing degenerate, low-diversity behaviors and distorted feature geometry.
• It manifests in generative RL, safety-aligned language models, and representation learning, often transforming high-reward regions into zero-entropy or brittle regimes.
• Mitigation strategies include entropy regularization, prompt augmentation, reward-direction correction, and geometric realignment to preserve diversity and robust safety.

Across recent literature, the expression alignment collapse is used for several related phenomena in which procedures intended to improve alignment instead induce degenerate concentration, brittle safety behavior, or unstable geometric structure. In reinforcement fine-tuning for generative models, the alignment objective can collapse a conditional policy toward a zero-entropy solution concentrated on a narrow set of high-reward outputs (Liu et al., 18 Jan 2026). In safety-aligned LLMs, benign downstream fine-tuning, reasoning augmentation, or even inference-time systems choices such as KV cache quantization can degrade refusal behavior and increase harmful compliance (Springer et al., 17 Feb 2026). In representation learning and adaptation, the term also denotes failures of feature–classifier self-duality, semantic or positional crowding in shared embedding spaces, or exact constant collapse of latent means (Chen et al., 11 Dec 2025).

1. Scope, terminology, and recurrent structure

The term is not monosemous. In one cluster of work, alignment collapse names an optimization pathology: reward maximization or preference optimization concentrates mass on a small subset of outputs, often approaching a Dirac delta or a low-diversity mode family. In another, it denotes degradation of already aligned behavior under fine-tuning, test-time adaptation, multimodal reasoning, or systems compression. In a third, it refers to geometric distortions in latent or feature spaces, where desired alignments between modalities, prototypes, class means, or safety-relevant directions fail, drift, or become degenerate (Liu et al., 18 Jan 2026, Springer et al., 17 Feb 2026, Wang et al., 25 Nov 2025).

A recurrent structure nevertheless appears across these usages. First, there is a privileged objective or subspace: a scalar reward, a safety-sensitive Fisher eigenspace, a classifier-weight direction, or a shared multimodal embedding. Second, optimization, adaptation, or compression amplifies certain directions while suppressing alternatives. Third, the resulting system becomes brittle: diversity shrinks, support vanishes, feature geometry loses separability, or safety behavior flips under perturbation. This suggests that “collapse” is less a single mechanism than a family of concentration phenomena induced by asymmetric pressures in objective space, representation space, or systems space.

The literature also draws careful distinctions from adjacent concepts. In diffusion reinforcement fine-tuning, diversity collapse is distinguished from classical GAN-style mode collapse and from reward hacking: collapse can arise even with sensible rewards because exponential tilting and on-policy exploitation concentrate probability mass on high-reward patterns for a prompt (Liu et al., 18 Jan 2026). In safety work, orthogonality-based reassurance is criticized because initial near-orthogonality between benign fine-tuning updates and safety-critical directions is not preserved under gradient flow on curved landscapes (Springer et al., 17 Feb 2026).

Context	Manifestation	Representative papers
Generative RLHF and diffusion alignment	Diversity collapse, preference mode collapse, lineage collapse	(Liu et al., 18 Jan 2026, Chen et al., 30 Dec 2025, Shekhar et al., 8 Apr 2026)
Safety-aligned language and multimodal models	Guardrail degradation, refusal loss, jailbreak susceptibility	(Springer et al., 17 Feb 2026, Lou et al., 10 May 2025, Xu et al., 1 Jun 2026)
Representation geometry	Sample-wise misalignment, NC3 failure, semantic or positional collapse	(Chen et al., 11 Dec 2025, Wang et al., 25 Nov 2025, Moon et al., 31 Oct 2025, Jeong et al., 13 Mar 2026)

2. Reward maximization and collapse in generative alignment

In reinforcement fine-tuning for conditional generation, the basic phenomenon is mathematically explicit. With model $p_\theta(x\mid c)$ and scalar reward $R(x,c)$, the naive objective

$$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$

encourages concentration on the highest-reward sample. Without entropy regularization or KL control, the optimum is

$p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$

a zero-entropy policy. For KL-constrained reward maximization with reference policy $\pi_{\mathrm{ref}}$, the optimal policy is

$$\pi^*(x_0|c)=\frac{1}{Z(c)}\,\pi_{\mathrm{ref}}(x_0|c)\,\exp\!\left(\frac{1}{\beta}r(x_0,c)\right),$$

so that decreasing $\beta$ exponentially amplifies small reward differences; in the limit $\beta\to 0^+$, the solution collapses to a Dirac delta. The same work attributes this to on-policy gradient dynamics: once $\pi_\theta$ shifts toward a high-reward region, it samples that region more often, while gradients from unvisited regions vanish (Liu et al., 18 Jan 2026).

This formulation is the basis for the “curse of diversity collapse” in image-generation RL fine-tuning. The DRIFT framework addresses the problem from three directions: reward-concentrated subset selection that filters out reward outliers, noise-conditioned prompt augmentation that broadens the conditioning manifold, and potential-based intra-group diversity shaping using DreamSim embeddings. On Stable Diffusion v1.5 with LoRA, DRIFT reports a $9.08\%\!\sim\!43.46\%$ increase in diversity at equivalent alignment levels and a $R(x,c)$0 increase in alignment at equivalent diversity levels, measured with DreamSim Diversity, CLIP Diversity, Generalized Recall, Vendi, and normalized PickScore or HPSv2 reward (Liu et al., 18 Jan 2026).

A closely related formulation appears in text-to-image RLHF as Preference Mode Collapse (PMC). There, the failure is attributed to over-optimization along reward-model bias directions, producing homogeneous outputs such as glossy or overexposed images that score well yet degrade identity, style, layout, and tonal diversity. The work introduces DivGenBench with four diversity dimensions—Identity Divergence Score, Artistic Style Coverage, Spatial Dispersion Index, and Photographic Variance Score—and proposes Directional Decoupling Alignment, which learns a directional correction in the frozen reward embedding space and uses a guided reward during optimization. On FLUX.1.Dev, this method reports best DivGenBench scores under both HPS-v2.1 and HPS-v2.1+CLIP reward settings (Chen et al., 30 Dec 2025).

Inference-time diffusion alignment exhibits an analogous but genealogical version of collapse. Sequential Monte Carlo samplers under strong selection pressure suffer lineage collapse because multinomial resampling coalesces trajectories early. Fleming–Viot Diffusion replaces multinomial resampling with independent reward-based survival decisions and stochastic rebirth noise. On CIFAR-10 with $R(x,c)$1 and $R(x,c)$2, FK-Diffusion collapses to $R(x,c)$3 lineages while FVD preserves $R(x,c)$4; the same work reports that FVD is up to $R(x,c)$5 times faster than value-based approaches and improves FID by roughly $R(x,c)$6–$R(x,c)$7 on class-conditional tasks (Shekhar et al., 8 Apr 2026).

3. Safety-alignment collapse in language and multimodal models

One major line of work treats alignment collapse as degradation of safety guardrails after benign downstream fine-tuning. In an information-geometric formulation, each safety skill induces a Fisher Information Matrix $R(x,c)$8 whose top eigenspace $R(x,c)$9 is the alignment sensitivity subspace. The Alignment Instability Condition consists of three jointly sufficient properties: low-rank sensitivity, initial near-orthogonality of the fine-tuning gradient to $$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$0, and nontrivial curvature coupling

$$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$1

Under these conditions, gradient flow develops a second-order drift into the safety-sensitive subspace, and alignment loss obeys a quartic onset law,

$$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$2

even when the first update appears harmless (Springer et al., 17 Feb 2026).

A complementary explanation emphasizes data similarity. In this account, high representation similarity between upstream safety-alignment datasets and downstream fine-tuning tasks weakens guardrail durability. Safety-aligned subsets selected to be low-similarity relative to the downstream task consistently produce lower Harmfulness Score than high-similarity subsets, with the most salient matched-size result being a $$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$3 HS reduction on SAMSum for Llama-2-13B with $$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$4K subsets. The operational hypothesis is that narrow, homogeneous safety alignment forms fragile guardrails that downstream updates overwrite more easily (Hsiung et al., 5 Jun 2025).

Multimodal large reasoning models introduce a further variant. Across $$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$5 models and five benchmarks, reasoning variants exhibit an average ASR increase of approximately $$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$6 on jailbreak robustness tasks relative to their base models, while safety-awareness benchmarks often show smaller degradation or even improvement. The same study reports that the thought process itself can be unsafe and can rationalize harmful outputs. A multimodal dataset with safety-oriented thought processes, used for supervised fine-tuning, sharply reduces ASR and improves safety-awareness scores on FigStep, MMSafetyBench, JailBreaKV, SIUO, and MSSBench (Lou et al., 10 May 2025).

Systems optimizations can trigger the same failure. Under KV cache quantization, low-bit inference can silently destroy safety alignment while leaving perplexity nearly unchanged. Across eleven instruction-tuned models and five benchmarks, Mistral-7B loses $$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$7 of its refusals at only $$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$8 perplexity, and there is no universal safe bit-width because collapse thresholds are model-specific. The proposed diagnostic, Per-Channel Reduction, classifies failures into “outlier-crushes-safety,” “outlier-as-safety,” and “multi-layer dilution,” and the training-free mitigation protocol recovers up to $$\max_{\theta}\; \mathbb{E}_{x\sim p_{\theta}(x\mid c)}[R(x, c)]$$9 of lost alignment, including in production KIVI and vLLM settings (Xu et al., 1 Jun 2026).

4. Geometric and representational formulations

In test-time adaptation, alignment collapse appears as a failure of sample-wise feature–classifier alignment. Extending Neural Collapse to the sample level, NC3+ states that for a labeled training sample $p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$0 with normalized feature $p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$1 and normalized classifier weight $p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$2, the ground-truth FCA distance

$p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$3

decreases monotonically and converges to zero during the terminal phase of training. Under domain shift, this relation breaks: features drift toward incorrect classifier weights, pseudo-labels become unreliable, and entropy minimization can reinforce errors. NCTTA addresses this by combining geometric proximity and predictive confidence into hybrid targets and applying an NC-guided alignment loss (Chen et al., 11 Dec 2025).

In long-tailed learning, the central issue is persistent failure of NC3 self-duality between feature and classifier spaces. Even when one space is made ETF-like, the other can remain misaligned, reducing margins and degrading tail performance. The paper formalizes this with a uniform misalignment angle $p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$4 and proves that the optimal error exponent under misalignment obeys

$p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$5

so nonzero angles shrink effective margins by $p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$6. Three plug-and-play strategies—similarity regularization, spherical linear interpolation, and gradient projection—explicitly target the NC3 gap (Wang et al., 25 Nov 2025).

Generalized Category Discovery and few-shot class-incremental learning provide two further geometric uses. NC-GCD argues that prior GCD methods suffer objective inconsistency and category confusion because supervised and unsupervised branches optimize against drifting targets; its response is a shared, fixed simplex ETF basis plus a Semantic Consistency Matcher that stabilizes pseudo-label identities across clustering iterations (Han et al., 7 Jul 2025). By contrast, NC-inspired FSCIL uses “alignment collapse” in a constructive sense: features are deliberately collapsed onto a pre-assigned simplex ETF of non-learnable classifier prototypes so that feature–classifier alignment is preserved session after session, mitigating catastrophic forgetting (Yang et al., 2023).

Outside classification, direct alignment objectives can distort embedding spaces in multimodal retrieval and recommendation. In Partially Relevant Video Retrieval, pairwise video-level supervision causes semantic collapse: distinct events within the same video are pulled together, while semantically similar events across videos are pushed apart. The proposed remedy combines Text Correlation Preservation Learning with Cross-Branch Video Alignment and order-preserving token merging (Moon et al., 31 Oct 2025). In multimodal recommender systems, direct unified-space alignment can blur modality-specific structure and amplify ID dominance; AnchorRec describes the resulting positional collapse and avoids it by indirect, anchor-based alignment in a lightweight projection domain (Jeong et al., 13 Mar 2026).

5. Diagnostics, indices, and certificates

Because collapse can be invisible to surface metrics, the literature places heavy emphasis on intrinsic diagnostics. In generative RL fine-tuning, alignment–diversity tradeoffs are tracked with DreamSim Diversity, CLIPScore Diversity, Generalized Recall, Vendi, and Pareto-front measures such as Diversity Gain and Reward Gain (Liu et al., 18 Jan 2026). Preference mode collapse in text-to-image models is diagnosed by DivGenBench’s Identity Divergence Score, Artistic Style Coverage, Spatial Dispersion Index, and Photographic Variance Score (Chen et al., 30 Dec 2025). Safety degradation is tracked with Harmfulness Score, Attack Success Rate, and ConditionalFlip, the last measuring the fraction of FP16 refusals that become compliance after perturbation or quantization (Hsiung et al., 5 Jun 2025, Xu et al., 1 Jun 2026).

Domain	Diagnostic family	Reported use
Generative RL fine-tuning	DreamSim Diversity, CLIP Diversity, Generalized Recall, Vendi, DG/RG	Reward–diversity Pareto analysis (Liu et al., 18 Jan 2026)
Preference mode collapse	IDS, ASC, SDI, PVS	Identity, style, layout, tonal breadth (Chen et al., 30 Dec 2025)
Safety degradation	Harmfulness Score, ASR, ConditionalFlip	Refusal loss and jailbreak vulnerability (Hsiung et al., 5 Jun 2025, Xu et al., 1 Jun 2026)

Intrinsic geometry is also used directly. The Geometric Overlap Score approximates Fisher-weighted projection into alignment-sensitive subspaces and predicts risk before fine-tuning (Springer et al., 17 Feb 2026). AQI proposes a prompt-invariant latent diagnostic based on clustering quality between safe and unsafe activations, combining the Davies-Bouldin Score, Dunn Index, Xie-Beni Index, and Calinski-Harabasz Index across layer-wise pooled representations (Borah et al., 16 Jun 2025). In feedback alignment, effective-rank diagnostics reveal a different collapse: FA error signals and hidden activities become low-rank, confining updates to a small subspace and stalling alignment in deep networks (Boeshertz et al., 9 Jun 2026).

Some works go further and turn collapse into a certifiable property. For variational autoencoders, a fixed simplex witness head defines a teacher–student alignment loss

$p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$7

and the constant-predictor baseline is the teacher information

$p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$8

If $p_{\theta}(x\mid c) = \delta(x - x^\*), \quad x^\* = \arg\max_x R(x,c),$9, then $\pi_{\mathrm{ref}}$0 cannot be input-independent constant collapsed. This converts exact constant collapse from an after-the-fact pathology into a design-and-certificate problem (Zhang et al., 18 May 2026).

6. Mitigation strategies and research directions

Mitigation strategies largely follow the mechanism identified in each domain. For reward-induced diversity collapse, DRIFT broadens support by filtering reward outliers, expanding the conditioning space with prompt noise, and adding potential-based diversity shaping that preserves optimal-policy invariance (Liu et al., 18 Jan 2026). For preference mode collapse, Directional Decoupling Alignment modifies the reward direction rather than only the optimization magnitude, learning a correction vector in reward-embedding space while keeping the reward model frozen (Chen et al., 30 Dec 2025). For iterative RLHF, Foresighted Policy Optimization restores the parameter-steering term missing from standard myopic updates, regularizing the policy’s effect on future reward-model parameters (Gauthier et al., 5 May 2026).

Where the problem is geometric, the remedies explicitly preserve or rebuild geometry. NCTTA realigns features to classifier weights with hybrid targets under unreliable pseudo-labels (Chen et al., 11 Dec 2025). Long-tailed learning uses SpA-Reg, SpA-SLERP, and SpA-Proj to align class means and classifier weights (Wang et al., 25 Nov 2025). NC-GCD fixes ETF prototypes and stabilizes label identities with SCM (Han et al., 7 Jul 2025). Anchor-based or cross-branch methods in recommendation and retrieval decouple alignment from representation learning so that modality-specific topology is not erased (Jeong et al., 13 Mar 2026, Moon et al., 31 Oct 2025).

Safety-oriented interventions likewise match the diagnosed mechanism. Low-similarity upstream safety-alignment data can make guardrails more durable under downstream fine-tuning (Hsiung et al., 5 Jun 2025). Safety-oriented thought processes can reduce jailbreak robustness failures in multimodal reasoning models (Lou et al., 10 May 2025). PCR-guided mixed-precision protection can recover most of the safety lost under KV cache quantization (Xu et al., 1 Jun 2026). At inference time for diffusion models, Fleming–Viot birth–death dynamics preserve broader trajectory support than multinomial resampling under strong reward selection (Shekhar et al., 8 Apr 2026).

Open problems are correspondingly diverse. In diffusion RL fine-tuning, outstanding directions include model-internal diversity potentials, extensions to video and 3D generation, adaptive shaping ratios, and cross-modal generalization (Liu et al., 18 Jan 2026). In safety collapse, curvature-aware methods, dynamic subspace tracking, and practical second-order controls remain unresolved (Springer et al., 17 Feb 2026). In iterative RLHF, extending Stackelberg-style steering control beyond strongly convex follower analyses and beyond small reward heads is still open (Gauthier et al., 5 May 2026). Taken together, these works indicate that alignment collapse is best understood not as a single pathology, but as a recurring concentration failure at the intersection of objective design, geometry, and systems constraints.