Cross-Modal Relational Topology Regularization

• The paper introduces MIRROR, a method that uses a semi-inverse Gromov–Wasserstein formulation to regularize visual token spaces by preserving pairwise relational structures from language.
• It employs attention-derived geometries with techniques like layer decoupling, head selection, and token suppression to effectively bridge language and vision modalities.
• Empirical results on benchmarks such as GQA, BLINK, and VLM tests show notable improvements in relational reasoning and overall multimodal performance.

Searching arXiv for the specified paper to ground the article and citation. I’ll check available tools and then look up the arXiv record for (Wang et al., 28 Jun 2026). Cross-modal relational topology regularization denotes a class of training constraints that aim to preserve the pairwise relational structure of concepts across modalities rather than aligning only token-level semantics. In the formulation introduced by MIRROR—“Mapping Inter-concept Relations from language to visual Representation via Optimal-transport-based Regularization”—the objective is to regularize the visual token space so that its distance structure mirrors relational priors encoded in language representations, using the model’s own cross-attention as a soft correspondence between text and image tokens. The method is developed for decoder-only multimodal LLMs (MLLMs), where standard projection-based alignment is argued to supervise visual tokens primarily at the identity or semantic level while leaving cross-modal relational geometry underconstrained (Wang et al., 28 Jun 2026).

1. Problem Setting and Failure Mode

The setting is the standard decoder-only MLLM pipeline in which an image is encoded into visual tokens, projected into the LLM embedding space by an adapter, concatenated with text tokens, and trained autoregressively with

$$\mathcal{L}_{\text{LM}} = -\sum_t \log p\!\left(w_t \mid f(v_1),\ldots,f(v_{n_v}),\, w_{<t}\right). \tag{1}$$

Within this setup, the supervision induced by next-token prediction mainly encourages each visual token to carry the correct semantics for textual prediction. It does not, however, require the relations among concepts to remain geometrically consistent when transferred from language to vision (Wang et al., 28 Jun 2026).

The central diagnosis is therefore structural rather than merely representational. Projection adapters align what tokens mean, but not how concepts relate to one another. If two concepts are close or far in language geometry, the corresponding visual geometry is not explicitly required to preserve that pattern. MIRROR identifies this missing supervision as relational topology preservation. A common misconception, in this framing, is that semantic alignment alone suffices for multimodal reasoning transfer; the paper argues instead that a model may retain the right conceptual prior in language while failing to apply it in vision because the cross-modal relational structure is not preserved (Wang et al., 28 Jun 2026).

2. MIRROR as a Semi-Inverse Gromov–Wasserstein Formulation

MIRROR casts cross-modal relational topology regularization as an inverse geometric problem derived from the Gromov–Wasserstein (GW) distance. For metric measure spaces with pairwise distance matrices $D^X$ and $D^Y$, and coupling $C \in \Pi(\mu,\nu)$, the GW objective is

$$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$

with

$$\Pi(\mu,\nu)= \{C \in \mathbb{R}_+^{n_t \times n_v} \mid C \mathbf{1}_{n_v} = \mu,\; C^\top \mathbf{1}_{n_t} = \nu\}. \tag{3}$$

Rather than solving for the coupling in the usual direction, MIRROR fixes the language geometry $D^t$ and the cross-modal coupling $C$, then asks what the visual geometry $D^v$ should be. This yields the Semi-Inverse Gromov–Wasserstein (SI-GW) problem:

$$\widehat{D}^v = \arg\min_{D^v \in \mathbb{R}^{n_v\times n_v}} \sum_{i,j=1}^{n_t}\sum_{k,\ell=1}^{n_v} \bigl|d_t(i,j)-d_v(k,\ell)\bigr|^2\, C_{ik}\,C_{j\ell}. \tag{4}$$

Here $D^X$0 is the text relational matrix, $D^X$1 is the visual relational matrix, $D^X$2 is the cross-attention coupling, and $D^X$3 gives the column marginals. The term “semi-inverse” reflects the asymmetry of the setup: language geometry and the cross-modal coupling serve as fixed teachers, while visual geometry is the unknown to be recovered (Wang et al., 28 Jun 2026).

This formulation makes cross-modal regularization explicitly topological in the sense used by the paper: the constraint is not imposed on isolated token embeddings, but on the modality-specific geometry encoded by pairwise distances.

3. Closed-Form Solution and Surrogate Loss

A defining property of the SI-GW formulation is that its objective admits a unique closed-form minimizer:

$D^X$4

where $D^X$5 denotes elementwise division and $D^X$6. The minimizer is interpreted as a conditional expectation. Defining

$D^X$7

one obtains

$D^X$8

Each visual pair $D^X$9 is thus assigned a language-derived target distance equal to the expected text distance between the text tokens that attend to those visual tokens (Wang et al., 28 Jun 2026).

Training uses the Frobenius discrepancy between the current visual relational matrix and this target:

$D^Y$0

The full objective is

$D^Y$1

The significance of this surrogate is computational as well as conceptual. Direct optimization of the pairwise-pairwise SI-GW objective in Eq. 4 is quartic in the token counts. The surrogate instead prescribes an explicit relational target $D^Y$2, turning topology transfer into a tractable regularization term. This suggests that the method treats language not merely as supervisory text, but as a source of geometric priors over inter-concept structure.

4. Extraction of Relational Priors from Attention

MIRROR extracts both intra-modal geometry and cross-modal coupling from attention maps. Using self-attention matrices $D^Y$3 for text and $D^Y$4 for vision, it symmetrizes the matrices and converts attention into distance-like dissimilarities:

$D^Y$5

Under this transformation, high attention corresponds to small distance and low attention to large distance. The text-to-vision cross-attention matrix $D^Y$6 is used as the soft correspondence that pushes language geometry $D^Y$7 into the implied visual target $D^Y$8 (Wang et al., 28 Jun 2026).

The transfer pipeline is therefore structured as follows in the paper’s own organization: relational structure is extracted from text self-attention; token correspondences are obtained from cross-attention; the implied visual relational target $D^Y$9 is computed; and visual attention geometry is regularized toward that target. This is the operational meaning of cross-modal relational topology regularization in MIRROR.

Quantity	Extracted from	Function
$C \in \Pi(\mu,\nu)$0	early LLM layer	text relational geometry
$C \in \Pi(\mu,\nu)$1	intermediate LLM layer	cross-modal coupling
$C \in \Pi(\mu,\nu)$2	final ViT layer	visual relational geometry

Applying this design inside decoder-only Transformers requires explicit stabilization strategies. The paper states that naïve extraction from a single layer is unstable because causal self-attention mixes text-text and text-image attention under one softmax, producing noisy gradients. MIRROR therefore uses layer decoupling, extracting $C \in \Pi(\mu,\nu)$3 from an early LLM layer, $C \in \Pi(\mu,\nu)$4 from an intermediate LLM layer, and $C \in \Pi(\mu,\nu)$5 from the final ViT layer; it also splits causal attention blocks with padding masks to isolate text-text and text-vision submatrices (Wang et al., 28 Jun 2026).

A second stabilization component is head selection. For each attention head $C \in \Pi(\mu,\nu)$6, entropy is computed as

$C \in \Pi(\mu,\nu)$7

Low-entropy heads are treated as more concentrated and hence more informative. MIRROR retains the top-$C \in \Pi(\mu,\nu)$8 lowest-entropy heads:

$C \in \Pi(\mu,\nu)$9

A third component is token suppression. System prompts, image placeholders such as <image>, and formatting markers are treated as non-semantic or noisy for geometry extraction and are downweighted by a soft position-aware factor $$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$0:

$$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$1

Row normalization is then applied before computing $$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$2. Ablations establish a clear hierarchy among these mechanisms: layer decoupling is described as indispensable, averaging all heads dilutes signal, and token filtering adds smaller but consistent gains (Wang et al., 28 Jun 2026).

5. Computational Characteristics

The paper’s efficiency claim is tied directly to the closed-form SI-GW solution. Direct evaluation of the original objective has complexity

$$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$3

whereas the surrogate target

$$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$4

can be computed via matrix multiplications with cost

$$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$5

In the reported breakdown, computing $$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$6 costs $$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$7, the dominant term $$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$8 costs $$\mathrm{GW}^2(\mu,\nu) = \min_{C \in \Pi(\mu,\nu)} \sum_{i,j=1}^{n_t} \sum_{k,\ell=1}^{n_v} \bigl| d_\mathcal{X}(i,j) - d_\mathcal{Y}(k,\ell) \bigr|^2 \, C_{ik} C_{j\ell}, \tag{2}$$9, and the elementwise division and Frobenius norm are lower-order (Wang et al., 28 Jun 2026).

This complexity reduction is important because the target domain is long token sequences typical in MLLMs. The regularizer is used only during training and adds no inference cost. A plausible implication is that MIRROR is designed to be inserted into existing decoder-only MLLM pipelines without altering deployment-time architecture or latency, provided the necessary attention tensors are already available during training.

6. Empirical Behavior and Reported Effects

Evaluation is reported on GQA for structured relational reasoning, BLINK for multi-level reasoning, and VQAv2, POPE, and RealWorldQA for general vision-language performance (Wang et al., 28 Jun 2026). On GQA, MIRROR improves overall performance consistently across four model settings: +0.77 for LLaVA-1.5-7B, +0.34 for LLaVA-1.5-13B, +0.82 for LLaVA-NeXT-7B, and +0.62 for LLaVA-NeXT-13B. The largest improvements are reported in Global and Category reasoning, which the paper associates with relational composition.

On BLINK, the average gains are likewise consistent: +2.5, +1.4, +2.4, and +2.5 depending on configuration. The largest gains occur in mid-level spatial reasoning and several high-level semantic tasks, including localization, counting, semantic correspondence, and functional reasoning. By contrast, low-level tasks such as reflection and depth change little. The paper explicitly treats this pattern as aligned with the method’s purpose: the regularizer targets relational geometry rather than local visual features.

On more general VLM benchmarks, the trade-off is reported as favorable. VQAv2 is mostly stable or better, POPE improves in all settings, which the paper interprets as suggesting reduced hallucination, and RealWorldQA remains stable. In this sense, the regularizer is presented not as a replacement for the language-model objective but as an auxiliary constraint that improves relational consistency while preserving general vision-language ability (Wang et al., 28 Jun 2026).

7. Ablations, Hyperparameter Sensitivity, and Conceptual Significance

The ablation results reinforce the implementation claims. Performance improves cumulatively as the three design choices are added; removing layer decoupling causes divergence, removing head selection severely degrades performance, and token filtering produces smaller but consistent gains. The weight of the SI-GW term is reported to be robust in the range

$$\Pi(\mu,\nu)= \{C \in \mathbb{R}_+^{n_t \times n_v} \mid C \mathbf{1}_{n_v} = \mu,\; C^\top \mathbf{1}_{n_t} = \nu\}. \tag{3}$$0

with best performance at

$$\Pi(\mu,\nu)= \{C \in \mathbb{R}_+^{n_t \times n_v} \mid C \mathbf{1}_{n_v} = \mu,\; C^\top \mathbf{1}_{n_t} = \nu\}. \tag{3}$$1

Too large a $$\Pi(\mu,\nu)= \{C \in \mathbb{R}_+^{n_t \times n_v} \mid C \mathbf{1}_{n_v} = \mu,\; C^\top \mathbf{1}_{n_t} = \nu\}. \tag{3}$$2 is reported to hurt because the geometric regularizer overwhelms the language-model objective (Wang et al., 28 Jun 2026).

Attention visualizations are used to support the architectural choices: intermediate layers provide the best balance between diversity and concentration, low-entropy heads yield clearer cross-modal correspondences, and MIRROR reshapes visual geometry toward the language-derived target $$\Pi(\mu,\nu)= \{C \in \mathbb{R}_+^{n_t \times n_v} \mid C \mathbf{1}_{n_v} = \mu,\; C^\top \mathbf{1}_{n_t} = \nu\}. \tag{3}$$3. The qualitative motivating example shows a model that is biased by vision despite possessing the correct language prior, illustrating the distinction between identity-level alignment and relational transfer.

The conceptual takeaway is tightly delimited. MIRROR does not claim that semantic projection is unnecessary; rather, it identifies a missing regularization signal in standard decoder-only MLLMs. Cross-modal relational topology regularization, in this formulation, means turning language attention geometry into a teacher for visual geometry by extracting intra-modal geometry from self-attention, extracting cross-modal coupling from cross-attention, solving a semi-inverse GW problem in closed form, and minimizing the discrepancy between the current visual geometry and the coupling-induced target geometry. This suggests a more geometric view of multimodal alignment in which successful transfer depends not only on semantic compatibility across modalities, but also on preservation of relational structure itself (Wang et al., 28 Jun 2026).