Manifold Misalignment Overview

• Manifold misalignment is a mismatch between the learned geometric structure and the intended semantic, correspondence, or intrinsic properties of data.
• It manifests in various settings, from MoE LLM routing and 3D point cloud distortions to conflicting objectives in visual tokenization, challenging traditional alignment metrics.
• Recent methods integrate supervised cues, filtering, or topological separation to reconcile geometry with task-specific invariants and enhance downstream performance.

Manifold misalignment denotes a mismatch between the geometry of a learned, aligned, or operational manifold and the structure that the manifold is supposed to preserve. Across recent work, this mismatch appears in several distinct but technically related forms: routing weights that fail to reflect task semantics in Mixture-of-Experts LLMs, shared embeddings that are geometrically aligned yet not predictive for classification, latent geometries that distort the intrinsic geometry of 3D surfaces, machine perceptual manifolds whose dimensionality is far larger than that of human concepts, and visual tokenizers in which reconstruction and semantics induce opposing geometric pressures (Li et al., 10 Nov 2025, Rhodes et al., 2024, Alonso et al., 8 May 2026, Salvatore et al., 3 Mar 2026, Yang et al., 7 May 2026). In all of these settings, the central issue is not merely poor optimization, but a structural discrepancy between the manifold actually produced by the model and the manifold implied by semantics, correspondence, downstream prediction, or intrinsic geometry.

1. Conceptual scope and defining characteristics

In semi-supervised manifold alignment, the misalignment problem is often not that two domains fail to enter a common coordinate system, but that the resulting joint embedding is geometrically aligned while remaining class-incoherent, not label-separating, or too weakly connected across domains to transfer discriminative structure. One formulation describes the failure explicitly as a mismatch between alignment quality in the geometric or correspondence sense and usefulness of the embedding for supervised tasks, noting that many methods “often fail to generate embeddings that capture sufficient information for downstream classification” (Rhodes et al., 2024).

A related but more classical use arises in alignment of nonlinear dynamical systems. For two slightly different double pendulums, misalignment means that corresponding configurations are not mapped close together after manifold learning and alignment. That work quantifies misalignment by the average normalized distance between corresponding embedded points, $\Delta$, and its standard deviation, $\sigma$, thereby treating alignment as a correspondence-preservation problem under noise rather than solely a visualization problem (Aziz et al., 2018).

Recent foundation-model work uses the term more geometrically. In sparse MoE LLMs, pretrained routers may produce routing-weight manifolds that do not preserve the cluster structure visible in a task embedding manifold. In 3D point cloud classification, adversarial fragility is attributed to a manifold misalignment between the latent geometry learned by the model and the intrinsic geometry of the underlying surface. In unified visual tokenization, the conflict between pixel reconstruction and semantic abstraction is cast as manifold misalignment because one objective “unfolds” the latent space while the other “collapses” nuisance variation (Li et al., 10 Nov 2025, Alonso et al., 8 May 2026, Yang et al., 7 May 2026).

Taken together, these usages suggest that manifold misalignment is best understood as a family of failures in which manifold structure is optimized for the wrong invariants. A plausible implication is that “alignment” cannot be assessed from geometry alone; it must be assessed relative to the task-specific structure the geometry is intended to encode.

2. Cross-domain alignment, filtering, and predictive usefulness

A central line of work studies manifold misalignment in domain adaptation and multimodal data fusion. Semi-supervised alignment is formulated between two domains, $\mathcal{X}$ and $\mathcal{Y}$, using partial correspondence from anchor points or shared class labels, with the goal of constructing a shared low-dimensional manifold that preserves local geometry within each domain and meaningful cross-domain relationships. Within this setting, several graph-based methods are positioned as predecessors or comparators, including SSMA, MAPA, JLMA, KEMA, MALI, MAGAN, DTA, MASH, and SPUD (Rhodes et al., 2024).

The main criticism of earlier graph-based methods is that improvements in correspondence metrics do not necessarily translate into downstream classification gains. In response, “Random Forest-Supervised Manifold Alignment” initializes within-domain graphs using RF-GAP proximities, a supervised similarity derived from random forests. These proximities form a row-stochastic diffusion operator within each domain. RF-MASH then powers the joint similarity matrix $\mathcal{P}$ to obtain a potential distance before applying MDS, whereas RF-SPUD builds cross-domain structure through shortest paths and geodesic connectivity before the same embedding step. The stated motivation is that random forest proximities encode supervised geometry, so local neighborhoods are shaped by class structure rather than only unsupervised distance (Rhodes et al., 2024).

A complementary strategy is “Filtered Manifold Alignment,” which explicitly separates denoising from cross-domain joining. FMA first projects and filters each domain independently, then aligns the reduced representations via cross-domain correspondences. The joint Laplacian is decomposed as

$\mathbf{L}=\mathbf{L}^*+\mathbf{A}\mathbf{A}^T,$

with $\mathbf{L}^*$ block diagonal and $\mathbf{A}\mathbf{A}^T$ a low-rank update induced by correspondences. This “filter then align” design is intended to ensure that the filter of one domain does not affect the second domain or the cross-domain correspondences, while also reducing complexity relative to direct joint alignment (Dernbach et al., 2020).

Visual alignment under geometric corruption introduces another variant of the same theme. “Manifold Constrained Low-Rank Decomposition” augments low-rank decomposition with a manifold prior so that recovered samples must remain on or near a nonlinear manifold $\mathcal{M}$. The method uses ADMM and a projection step

$$V_{m,i}^{'[k+1]} = \Phi_{\alpha,\epsilon}(V_{m,i}^{[k+1]},V_m^{[k+1]}),$$

thereby enforcing local neighbor-preserving structure during alignment and reconstruction. The rationale is that low rank alone is too weak when image sets contain occlusion, noise, illumination variation, rotation, viewpoint changes, and geometric misalignment (Chen et al., 2017).

The double-pendulum study supplies an early comparative picture of what these choices optimize. Semi-supervised feature-level manifold alignment using global distance produced the most convincing visualisations, whereas semi-supervised feature-level local alignment methods produced smaller alignment errors, were more robust to noise, and were faster than the alternatives. This result is significant because it separates visual plausibility from measured correspondence quality: the manifold that looks most coherent need not minimize $\sigma$0 and $\sigma$1 (Aziz et al., 2018).

3. Routing manifolds and task semantics in MoE LLMs

In sparse Mixture-of-Experts LLMs, manifold misalignment is formulated as a mismatch between the geometry of routing weights and the geometry of task embeddings. The motivating observation is that semantically similar samples in a task embedding space can receive very different routing weights from pretrained routers, even though similar tasks should ideally induce similar expert selections. The paper characterizes this as a failure of the routing-weight manifold to preserve the cluster structure visible in the task embedding manifold, and reports a severe performance gap, “e.g., 10–20% in accuracy,” relative to oracle routing (Li et al., 10 Nov 2025).

“Routing Manifold Alignment” (RoMA) addresses this problem through post-training router fine-tuning with all non-router parameters frozen. Successful training samples are first defined as

$\sigma$2

For each sample, neighbors are drawn from $\sigma$3 in task embedding space using either $\sigma$4-NN or an $\sigma$5-ball, with Gaussian similarity

$\sigma$6

Normalized adjacency weights $\sigma$7 then define a manifold regularizer

$\sigma$8

and the full objective is

$\sigma$9

The key design choice is that RoMA imitates only successful neighbors rather than arbitrary neighbors, so routing patterns from failed examples are not propagated (Li et al., 10 Nov 2025).

Operationally, RoMA is intentionally lightweight. Only the router parameters are updated; all expert parameters are frozen; and the paper states that this modifies about 0.0095% of the base model in the reported setup. It also reports that fine-tuning only the routers in the last five layers gives the best accuracy/efficiency tradeoff (Li et al., 10 Nov 2025).

The empirical effect is framed as stable expert binding across layers. On MMLU, the reported gains are 46.2% $\mathcal{X}$0 56.8% for DeepSeekMoE, 57.8% $\mathcal{X}$1 69.0% for OLMoE, and 74.2% $\mathcal{X}$2 78.8% for Qwen3-MoE. Across eight benchmarks, average performance rises from 66.6 to 74.7 for DeepSeekMoE, from 67.6 to 76.2 for OLMoE, and from 74.0 to 79.5 for Qwen3-MoE. RoMA is also compared with C3PO, with the paper noting that C3PO requires about 6–7× more FLOPs due to test-time optimization, whereas RoMA keeps inference cost nearly unchanged (Li et al., 10 Nov 2025).

Conceptually, this formulation shifts manifold misalignment from a static embedding problem to a computational allocation problem. The “manifold” is no longer a latent representation alone, but the layerwise geometry of expert choices induced by the router.

4. Intrinsic geometry, latent distortion, and adversarial robustness

In 3D point cloud recognition, manifold misalignment is defined as a mismatch between intrinsic surface geometry and the latent geometry induced by the classifier. A point cloud is treated as a finite sampling of a low-dimensional surface manifold $\mathcal{X}$3, while the network induces a latent manifold $\mathcal{X}$4. The local latent metric is

$\mathcal{X}$5

and robustness is associated with the alignment condition

$\mathcal{X}$6

Deviation from this condition is the paper’s formal notion of manifold misalignment. The corresponding distortion measure is

$\mathcal{X}$7

so large $\mathcal{X}$8 indicates strong anisotropy and severe local distortion (Alonso et al., 8 May 2026).

MAPR addresses this problem with intrinsic feature augmentation and intrinsic consistency regularization. The intrinsic map $\mathcal{X}$9 includes curvature estimated by the random-walk Laplacian,

$\mathcal{Y}$0

and multi-scale diffusion descriptors $\mathcal{Y}$1 and $\mathcal{Y}$2 for $\mathcal{Y}$3. The consistency term uses symmetric KL divergence normalized by intrinsic feature change:

$\mathcal{Y}$4

The reported effect is an average robustness gain of +20.02% on ModelNet40 and +8.58% on ScanObjectNN, without adversarial training or additional data (Alonso et al., 8 May 2026).

A more global formulation appears in work on adversarial examples via perceptual manifolds. For a class $\mathcal{Y}$5, the network’s perceptual manifold is

$\mathcal{Y}$6

with $\mathcal{Y}$7 in the CIFAR-10 experiments. Misalignment is called exponential misalignment because machine PMs are orders of magnitude higher-dimensional than natural human concepts, so the volume of machine-recognized-but-human-unrecognized inputs grows exponentially with dimension. On CIFAR-10, the natural image manifold is reported as around $\mathcal{Y}$8 by participation ratio and $\mathcal{Y}$9 by 2NN, whereas model PMs are around $\mathcal{P}$0 by participation ratio and at least $\mathcal{P}$1 by 2NN. Even the most robust models remain around $\mathcal{P}$2 by participation ratio and $\mathcal{P}$3 by 2NN. Across 18 networks, the paper reports a negative correlation between robust accuracy and PM dimensionality, and a dual increase in distance from random points to the PM as PM dimension decreases (Salvatore et al., 3 Mar 2026).

The two formulations differ in scale. MAPR studies local metric distortion on an intrinsic manifold, whereas perceptual-manifold analysis studies the global dimensionality of class concepts in input space. Their conjunction suggests that adversarial vulnerability can be viewed both as local anisotropic stretching and as global overexpansion of class manifolds.

5. Visual tokenization, multimodal locality, and topological orthogonality

Unified visual tokenization introduces a distinct form of manifold misalignment. “MUSE” argues that joint optimization for high-fidelity pixel reconstruction and semantic abstraction creates a zero-sum game because reconstruction prefers spatial equivariance and high-frequency detail while semantics prefers invariance and abstraction. The paper describes this as destructive interference from conflicting gradients: pixel gradients “unfold” the manifold, semantic gradients “collapse” it, and naïve shared optimization entangles both tendencies in the same parameter subspace (Yang et al., 7 May 2026).

The proposed solution is Topological Orthogonality, implemented through a Synergistic Block that separates attention routing from semantic content. The topology stream computes

$\mathcal{P}$4

while the semantic stream computes

$\mathcal{P}$5

Structural gradients are intended to update $\mathcal{P}$6 and $\mathcal{P}$7, semantic gradients to update $\mathcal{P}$8, and a stop-gradient operator is used so that reconstruction does not overwrite semantic alignment. The information-theoretic decomposition

$\mathcal{P}$9

motivates the training curriculum of topology warmup, semantic injection, and end-to-end synergistic tuning (Yang et al., 7 May 2026).

The gradient analysis makes the alignment claim explicit. Under naïve shared optimization, the reported gradient cosine is about $\mathbf{L}=\mathbf{L}^*+\mathbf{A}\mathbf{A}^T,$0; soft regularization improves it to $\mathbf{L}=\mathbf{L}^*+\mathbf{A}\mathbf{A}^T,$1; MUSE reports $\mathbf{L}=\mathbf{L}^*+\mathbf{A}\mathbf{A}^T,$2. The main tokenizer metrics are gFID 3.08, zero-shot accuracy 76.1, linear probing 85.2%, mIoU 46.5, rFID 0.62, PSNR 24.9, and SSIM 0.78. The paper highlights that linear probing at 85.2% exceeds the teacher InternViT-300M at 82.5% (Yang et al., 7 May 2026).

A complementary issue arises in vision-language matching, where global alignment can conceal local inconsistencies. “Extract Free Dense Misalignment from CLIP” argues that scalar CLIPScore captures only global alignment and cannot identify which words are inconsistent with an image. CLIP4DM therefore revises gradient-based attribution so that negative gradients of individual text tokens indicate misalignment:

$\mathbf{L}=\mathbf{L}^*+\mathbf{A}\mathbf{A}^T,$3

Aggregating these signals yields F-CLIPScore,

$\mathbf{L}=\mathbf{L}^*+\mathbf{A}\mathbf{A}^T,$4

which combines global mismatch with token-level negative attribution mass (Nam et al., 2024).

The method is evaluated on FOIL, nocaps-FOIL, HAT, SeeTRUE-Feedback, and Rich-HF. The reported results include LA 0.716 and AP 0.794 on nocaps-FOIL with ViT-H/14, 0.348 localization accuracy on HAT, 0.660 on SeeTRUE-Feedback with ViT-H/14, and on Rich-HF F1 0.427, precision 0.365, recall 0.516, Pearson 0.368, and Spearman 0.433. On nocaps-FOIL, the paper reports roughly 44× lower inference time than ALOHa (Nam et al., 2024).

This line of work shows that manifold misalignment in multimodal systems is not exhausted by whole-example similarity. A pair may remain globally near in representation space while still containing token-level inconsistencies, just as a tokenizer may reconstruct well while degrading semantic structure.

6. Recurring evaluation patterns and methodological implications

Several recurring patterns emerge from these literatures. First, geometric alignment metrics and task utility are repeatedly shown to diverge. In RF-supervised alignment, improvements in FOSCTTM and CE do not guarantee better downstream classification. In noisy double-pendulum alignment, global-distance feature alignment gives the most convincing visual structure, whereas local methods yield the smallest measured $\mathbf{L}=\mathbf{L}^*+\mathbf{A}\mathbf{A}^T,$5 and $\mathbf{L}=\mathbf{L}^*+\mathbf{A}\mathbf{A}^T,$6 and are faster (Rhodes et al., 2024, Aziz et al., 2018).

Second, successful methods usually insert additional structure before or during alignment rather than relying on unconstrained joint optimization. RF-MASH and RF-SPUD use RF-GAP proximities as supervised geometry; FMA filters each domain separately before joining; MeADMM projects onto a learned manifold during low-rank recovery; RoMA regularizes routers toward successful neighbors in task-embedding space; MAPR normalizes predictive consistency by intrinsic geometric change; and MUSE physically separates topology and semantics in attention (Dernbach et al., 2020, Chen et al., 2017, Li et al., 10 Nov 2025, Alonso et al., 8 May 2026, Yang et al., 7 May 2026).

Third, local and global notions of misalignment often coexist. CLIP4DM distinguishes global CLIP similarity from token-level negative attribution. MAPR measures local metric distortion, whereas perceptual-manifold analysis measures global dimensional mismatch. RoMA operates locally through successful-neighbor smoothing, yet its goal is global improvement in expert binding across layers and downstream generalization (Nam et al., 2024, Salvatore et al., 3 Mar 2026, Li et al., 10 Nov 2025).

A common misconception is that alignment is a single scalar property. The literature instead operationalizes it through multiple incompatible criteria: correspondence distance, classification accuracy, potential distance, oracle-routing gap, Jacobian anisotropy, participation ratio, 2NN intrinsic dimension, gradient cosine, dense token attribution, and benchmark robustness. This suggests that manifold misalignment is best treated as a structural diagnosis whose meaning depends on what the manifold is intended to preserve. A plausible implication is that future work will continue to move away from purely geometric matching toward alignment criteria that are explicitly tied to semantics, prediction, and intrinsic structure.