Manifold-Geometric Transformer (MGT)

• Manifold-Geometric Transformers are architectures that integrate explicit geometric priors to constrain feature updates along data manifolds.
• They employ techniques like tangent-bundle constraints, proximal projections, and mixture-of-experts routing to ensure stability and interpretability.
• MGT models excel in handling non-Euclidean data across vision, graphs, and sequences, achieving measurable gains such as 2–3% accuracy improvements and up to 21% error reduction.

The Manifold-Geometric Transformer (MGT) refers to a family of Transformer architectures that explicitly encode, constrain, or exploit the geometry of data manifolds in their representations and feature update laws. These models, introduced independently across multiple domains—ultra-deep sequence modeling, vision, graph learning, and material property prediction—share a unifying principle: the integration of manifold geometry into the core transformer attention, update, and fusion mechanisms. Key variants include deep-sequence MGTs with tangent-bundle constraints and delta dynamics (Su et al., 3 Jan 2026), proximal manifold projection for visual representations (Yun et al., 23 Aug 2025), multi-curvature expert mixtures for graphs (Jyothish et al., 9 Jul 2025), and multi-view fusions for crystalline materials (Zhang et al., 21 Jul 2025).

1. Geometric Collapse and Tangent-Bundle Constraints in Deep Transformers

Standard deep Transformers deploy residual connections of the form $x_{l+1} = x_{l} + F(x_l)$, where $F$ is MHSA or MLP output. Empirical analysis reveals that as depth $L$ increases ($L\gg 50$), hidden states $x_l$ increasingly lose their effective rank, suffer semantic drift off the underlying data manifold $\mathcal{M}\subset\mathbb{R}^D$, and accumulate irrecoverable noise. The phenomenon persists despite state-of-the-art normalization and initialization. This is posited as a fundamentally geometric failure: valid feature updates should be constrained to $T_{x_l}\mathcal{M}$, the tangent space at $x_l$.

MGT addresses this by introducing manifold-constrained hyper-connections (mHC), which project or rectify $F(x_l)$ into $T_{x_l}\mathcal{M}$ using a learnable gating function $$g_l = \sigma(\mathrm{LN}(W_{\rm m}x_l + b_{\rm m})) \in [0,1]^D$$, so $V_{\rm mHC} = F(x_l)\odot g_l$. Deep delta learning (DDL) further augments this with a non-monotonic update law: $$x_{l+1} = x_l + \beta_l \odot (V_{\rm mHC} - \gamma x_l/\|x_l\|)$$, where $\beta_l$ is a signed, data-dependent scalar, and $\gamma$ is a learned scale. This decouples update direction (constrained to $T_{x_l}\mathcal{M}$) from sign/magnitude, enabling both accumulation and explicit erasure, and yields stable evolution of representations even for $L \geq 100$ (Su et al., 3 Jan 2026).

2. Multi-View and Equivariance Modules in Material and Graph Data

MGT variants for structured non-Euclidean data use explicit geometric encodings that capture intrinsic symmetries and topologies.

In crystalline material property prediction, MGT fuses two parallel graph-encoder branches (Zhang et al., 21 Jul 2025):

• An SE(3)-invariant branch leveraging distances and bond angles for rotation- and translation-invariant features, with transformers updating scalar node and edge embeddings.
• An SO(3)-equivariant branch using vector displacements and spherical harmonics, with tensor-product layers preserving equivariance under 3D rotations.

A lightweight mixture-of-experts (MoE) self-attention router then learns per-task weights for these two geometric representations, producing a fused embedding that adapts to the downstream property (e.g., formation energy, bandgap). Self-supervised pretraining combines denoising (reconstruction of perturbed angles/edges) and contrastive alignment between the two views.

On graphs, MGT prepends a Riemannian mixture-of-experts (GraphMoRE) projection to SGFormer (Jyothish et al., 9 Jul 2025). Each node is routed via a learned softmax to mixture experts with chosen curvatures $c_k$ ($>0$: spherical, $=0$: Euclidean, $<0$: hyperbolic), each admitting closed-form exponential/logarithmic maps for manifold transition. Embeddings are aggregated and fused with the original features via cross-attention, providing latent spaces whose curvature matches the local graph topology.

3. Proximal Geometry and Global Feature Alignment in Computer Vision

In vision, the Proximal Vision Transformer interprets classical ViT attention as constructing local tangent-bundle approximations of the data manifold $\mathcal{M}$ (Yun et al., 23 Aug 2025). Each head and layer builds a linear local chart for $T_p\mathcal{M}$, but original ViT is confined to intra-image attention, limiting global consistency.

MGT supplements ViT with a second stage that collects class-token outputs from a batch (as columns of $Z\in\mathbb{R}^{d\times m}$) and maps them from tangent space back onto the base manifold via a learned proximal operator: $$W_{k+1} = \mathrm{prox}_{\gamma_k\,g + \iota_C}\Big(W_k - \gamma_k Z^\top (ZW_k - Z) R_k \Big)$$ Here $g$ is a regularization (typically $\ell_1$), $C$ enforces constraints (e.g., nonnegativity), and $R_k$ is an optional preconditioner. This section map $s_W:Z\mapsto \hat Z = ZW$ aligns global features across samples, promoting intra-class clustering and inter-class separation on $\mathcal{M}$.

Experimentally, such geometric projection yields consistent accuracy improvements ($\sim$2–2.4% absolute for 15-Scene/Mini-ImageNet) and induces tighter class clusters and larger inter-class distances in embedding space (Yun et al., 23 Aug 2025).

4. Mixture-of-Experts Routing and Adaptive Geometry

MGT exploits MoE routers in two principal ways:

• For crystalline graphs, an attention-based MoE learns whether SE(3) or SO(3) encoders are more task-informative, with weights $g=(g_1,g_2)$ adapting per domain and task (Zhang et al., 21 Jul 2025).
• For graph data, per-node routing uses local topological descriptors to set softmax weights $\alpha_{i,k}$ over manifold experts with different curvatures, allowing individual nodes to adaptively choose their embedding geometry (Jyothish et al., 9 Jul 2025).

These routers not only improve performance but afford interpretability: learned mixture weights can be examined to reveal structure—e.g., nodes with high hyperbolic weights correspond to hierarchical regions; shifts in the SE(3)/SO(3) MoE weights reflect task structure or symmetry.

5. Theoretical Insights and Empirical Results

A central claim from deep-sequence MGTs is that geometric validity of updates, plus dynamic erasure, is necessary and sufficient to prevent representational collapse: mHC alone averts drift but not feature accumulation, DDL alone enables forgetting but not tangent-constrained updates. Only the full decoupling stably preserves effective rank even at $L=200$ layers, with gate statistics confirming non-monotonic update dynamics: early layers “write,” deep layers perform “erase” (Su et al., 3 Jan 2026).

On graph benchmarks, inclusion of manifold mixtures (especially hyperbolic and spherical) in MGT consistently yields up to 3-point absolute gains in macro-F1 and weighted-F1 (Jyothish et al., 9 Jul 2025), with ablations revealing that removal of individual curvature experts degrades performance on graphs whose topologies naturally fit those geometries.

For crystalline materials, multi-task self-supervised pretraining, MoE fusion, and the two-branch geometric encoder architecture reduce mean absolute errors by up to 21% on property prediction, and transfer learning improvements of 25–58% are reported for catalyst adsorption and perovskite bandgap tasks (Zhang et al., 21 Jul 2025).

In vision, the global manifold-consistency stage in MGT leads to more discriminative embeddings in feature space, as measured both by t-SNE plots and by increased inter-class Wasserstein distances (Yun et al., 23 Aug 2025).

6. Interpretability and Domain-Generalization

MGT provides intrinsic interpretability through its explicit geometric parameters:

• In graph MGTs, mixture weights $\alpha_{i,k}$ furnish local geometric explanations; the relative magnitude of weights can signal tree-likeness (hyperbolic), cluster structure (spherical), or flat regularity (Euclidean) per node (Jyothish et al., 9 Jul 2025).
• In crystal MGTs, router weights and t-SNE analysis reveal the complementarity of invariant and equivariant representations and the adaptivity of the fusion to the target property (Zhang et al., 21 Jul 2025).
• In ultra-deep sequence settings, the nature/distribution of DDL gates $\beta_l$ reflects where the network is writing new features versus erasing redundancy (Su et al., 3 Jan 2026).

This capability is coupled with demonstrated transferability across heterogeneous domains and tasks: domain-agnostic MGT architectures retain or improve performance in vision, materials science, and graph learning.

7. Summary Table: MGT Instantiations Across Domains

Domain	Geometric Mechanism	Router/Fusion
Deep Sequence	Tangent-bundle mHC, DDL	Layerwise gating
Vision	Tangent-bundle ViT + Proximal Map	None (batch section)
Graph	MoE mixture (sph/hyp/Eucl emb.)	Nodewise softmax
Materials	SE(3)-inv. + SO(3)-eq. branches	MoE self-attention

All Manifold-Geometric Transformers build on the insight that explicitly encoding geometric priors, constraints, or mixtures into the feature-update or fusion logic of the Transformer yields more stable, interpretable, and adaptable representations across data modalities with complex non-Euclidean structure (Su et al., 3 Jan 2026, Yun et al., 23 Aug 2025, Jyothish et al., 9 Jul 2025, Zhang et al., 21 Jul 2025).