Manifold Anchor Regularization

• MAR is a geometry-aware regularization framework that enforces dispersive constraints to prevent intra-modal representation collapse and maintains bounded cross-modal alignment.
• It integrates seamlessly into existing training pipelines for multimodal and continual learning without requiring architectural changes.
• Empirical studies demonstrate that MAR improves accuracy, boosts resilience against modality corruption, and preserves legacy task performance.

Manifold Anchor Regularization (MAR) is a geometry-aware regularization framework for neural networks, targeting the explicit control of representation geometry during learning. MAR is designed to prevent intra-modal representation collapse and constrain cross-modal inconsistency, particularly in multimodal and continual learning settings. By enforcing dispersion within modalities and anchoring across modalities or tasks, MAR mitigates both loss of unimodal expressiveness and degradation of joint or legacy representations. MAR has been instantiated in both multimodal fusion architectures and continual learning as detailed in recent work (Xia et al., 29 Jan 2026, Kobs, 20 Mar 2026).

1. Conceptual Foundations

MAR introduces two complementary constraints on intermediate embeddings:

• Intra-modal dispersive regularization: This penalizes the collapse of representations for different samples within a modality, encouraging diversity by maximizing spread on the embedding manifold.
• Inter-modal (or inter-task) anchoring regularization: This softly restricts the divergence of embeddings for the same underlying semantic entity across modalities (or tasks), bounding the cross-modal (or cross-temporal) feature drift within a prescribed tolerance.

MAR explicitly augments the primary task loss without requiring architectural modifications and remains compatible with standard supervised or self-supervised learning algorithms. Its mechanisms are plug-and-play and can be injected into existing training pipelines as an additional regularizer.

2. Mathematical Formulation in Multimodal Learning

Let $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ be a multimodal dataset, with sample $x_i = (x_i^1, \ldots, x_i^M)$ spanning $M$ modalities and label $y_i$, and $f_m$ denote the encoder for modality $m$, producing embeddings $z_i^m \in \mathbb{R}^d$. For each modality and batch:

• Normalization: Each embedding is projected to the unit hypersphere,

$\tilde{z}_i^m = \frac{z_i^m}{\|z_i^m\|_2}$

• Dispersive loss: For modality $m$, MAR penalizes clustering of embeddings using a potential function $\phi$ (often RBF/log-uniformity), averaged over all sample pairs:

$x_i = (x_i^1, \ldots, x_i^M)$0

Global dispersive loss aggregates across modalities.

• Anchoring loss: For each sample $x_i = (x_i^1, \ldots, x_i^M)$1 and pair $x_i = (x_i^1, \ldots, x_i^M)$2,

$x_i = (x_i^1, \ldots, x_i^M)$3

where $x_i = (x_i^1, \ldots, x_i^M)$4 and $x_i = (x_i^1, \ldots, x_i^M)$5 sets the tolerance radius.

The total loss is:

$x_i = (x_i^1, \ldots, x_i^M)$6

with $x_i = (x_i^1, \ldots, x_i^M)$7, $x_i = (x_i^1, \ldots, x_i^M)$8 controlling the strength of dispersion and anchoring, respectively.

3. Instantiation in Continual Learning

In continual learning, MAR is used as an anchor geometry-preserving regularizer within frameworks like Support-Preserving Manifold Assimilation (SPMA-OG) (Kobs, 20 Mar 2026):

• Anchor selection: A fixed, small set of anchor samples $x_i = (x_i^1, \ldots, x_i^M)$9 from old tasks are stored with their teacher (pre-update) embeddings $M$0.
• Global distance preservation: The student (current) model aims to match pairwise distances between anchor embeddings to the teacher, using

$M$1

and penalizing deviations:

$M$2

• Local smoothing: Weighting these differences by local kernel affinity to emphasize local neighborhood preservation.
• Chart-assignment preservation: Clusters (charts) are fitted on teacher anchors; soft assignments of new student embeddings to these charts are matched by KL divergence.

The full MAR loss in this context is a weighted sum of global, local, and chart-preserving terms, typically added to the standard cross-entropy, output distillation, and parameter drift penalties.

4. Optimization and Implementation

MAR augments the training loop as follows:

• Normalize batch embeddings.
• Compute intra-modal dispersive loss using chosen $M$3 and RBF temperature $M$4.
• Compute inter-modal anchoring loss with threshold $M$5.
• (Optionally) Use Pareto-balanced weighting, where regularization strengths $M$6 and $M$7 are adaptively computed each step by minimizing the squared $M$8 norm of the weighted regularizer gradients.

In continual learning, additional steps comprise anchor batch sampling, anchor memory updates, and explicit preservation of anchor geometry and chart soft-assignments, following the SPMA-OG framework pseudocode.

5. Geometric Interpretation

• Dispersion: MAR's intra-modal regularizer creates repulsive forces between embeddings of different samples within each modality over the sphere, thus preserving embedding diversity and guarding against low-rank collapse. Theoretically, this maximizes Rényi-2 entropy and increases the effective rank of the batch embedding covariance (Xia et al., 29 Jan 2026).
• Anchoring: The cross-modal (or temporal) anchoring regularizer draws paired representations (e.g., audio and video for the same utterance, or the same sample across time) together only if their $M$9 distance exceeds a soft threshold $y_i$0. This enforces bounded alignment, allowing semantic identity while preserving modality- or task-specific structure within the $y_i$1-radius.

In combination, these constraints shape the learned embedding manifold into well-dispersed, unimodal manifolds with bounded, adaptive multimodal (or multi-task) clustering.

6. Empirical Findings and Benchmarks

MAR has demonstrated effectiveness across diverse settings:

Benchmark	Setting	Improvement	Notes
CREMA-D	Audio-Visual	+0.5–1.2 pp	Unimodal and multimodal accuracy, see below
Kinetics-Sounds	Video-Audio	+0.5–1.2 pp	Both fusion and unimodal boost
CUBICC	Image-Text Clust.	+3.5 pp ACC	Also NMI, ARI improved
XRF55	RF-Vision	+2.7 pp	All settings improved

On CREMA-D, ablation shows that dispersive and anchoring regularization each improve both unimodal and fusion accuracy, but their combination (MAR) produces the highest overall gains (e.g., Multi: $y_i$2 vs. baseline $y_i$3).

Robustness experiments under audio/visual corruption, frame/feature dropping, and Gaussian channel noise reveal that MAR yields smoother degradation and higher average accuracy, confirming enhanced resilience to unreliable modalities.

In continual learning (e.g., CIFAR-10 compatible shift), MAR improves legacy task retention and representation metrics (CKA, anchor correlation) compared to replay-only or distillation-only approaches. On synthetic manifold benchmarks, MAR achieves near-perfect anchor geometry preservation (CKA $y_i$4) (Kobs, 20 Mar 2026).

7. Practical Recommendations and Hyperparameters

Empirical sensitivity studies recommend moderate regularization strengths:

• $y_i$5–$y_i$6,
• $y_i$7–$y_i$8,
• RBF temperature $y_i$9–$f_m$0,
• Tolerance $f_m$1–$f_m$2.

Pareto-balanced weighting (base scale $f_m$3) is suggested to automate trade-offs between dispersion and anchoring. In continual learning, appropriate anchor sampling and memory capacity are vital for effective geometry preservation.

---

MAR introduces geometry-aware inductive biases that are architecture-agnostic, computationally lightweight, and empirically validated across multimodal fusion and continual learning scenarios. Its explicit control of intra- and inter-manifold geometry constitutes a principled addition to the set of tools for mitigating representation collapse and catastrophic forgetting while enhancing robustness and fusion (Xia et al., 29 Jan 2026, Kobs, 20 Mar 2026).