Cross-Modal Decompositionality

• Cross-modal decompositionality is a framework that factorizes joint embedding spaces to separate shared and modality-specific features.
• It employs methods like group-sparse modeling, low-rank plus sparse factorization, and codebook quantization to enhance semantic control and model interpretability.
• Empirical analyses demonstrate improved zero-shot performance, reduced inactive neurons, and effective bias mitigation for explainable AI applications.

Cross-modal decompositionality is the property and methodology whereby joint embedding spaces or inference processes in multimodal systems are factorized or partitioned into components that are modality-specific, modality-invariant, or otherwise aligned/disentangled across participating modalities. This principle enables mechanistic interpretability, actionable control, and systematic benchmarking of models handling data sources such as vision, language, audio, and beyond. Core approaches include group-sparse modeling, structured factorization, discrete codebooks, sheaf-theoretic analysis, and information-theoretic partitioning, with applications extending from representation analysis to explainable AI and multimodal evaluation.

1. Theoretical Foundations and Definitions

Cross-modal decompositionality posits that representations in multimodal models can be analyzed, factorized, or linearly decomposed into distinct semantic bases capturing modality-specific or shared information. In the context of embedding spaces, given embeddings $X \in \mathbb{R}^{n \times d}$ from modalities such as vision ($X^{(1)}$) and language ($X^{(2)}$), the goal is to find a dictionary $W \in \mathbb{R}^{d \times p}$ and sparse codes $Z \in \mathbb{R}^{p \times n}$ such that $X \approx W Z$, with specific constraints to align codes across modalities and extract human-interpretable “concept” vectors (Kaushik et al., 27 Jan 2026).

This decomposition framework generalizes to a variety of domains:

• Group-sparse autoencoders induce shared support on sparse codes for paired modality embeddings, ensuring that aligned semantic features activate across modalities rather than only in a unimodal fashion (Kaushik et al., 27 Jan 2026).
• Tripartition via Modality Dominance Score (MDS): A principled metric $R(k)$ quantifies the degree to which feature $k$ is image-dominant, text-dominant, or cross-modal, enabling explicit separation of modality roles at the neuron or atom level (Yan et al., 16 Feb 2025).
• Low-rank plus sparse factorization decomposes input features into a shared, low-rank modality-invariant component $L$ and modality-specific sparse residuals $S_i$, $X^{(1)}$0 (Tian et al., 8 Jun 2025).
• Iso-Energy principle enforces that bimodal dimensions exhibit matched statistics across modalities, so that “energy” (squared norm) on any concept axis is invariant—defining truly shared directions in embedding space (Dhimoïla et al., 5 Feb 2026).

A plausible implication is that these decompositions provide interpretability and controllability, allowing both researchers and downstream applications to isolate semantic signals relevant to specific modalities or forms of multimodal integration.

2. Methodologies and Model Structures

A diversity of architectural and algorithmic techniques has been devised for achieving and quantifying cross-modal decompositionality.

Sparse Autoencoder-Based Decomposition

A canonical approach uses overcomplete sparse autoencoders, with loss objectives:

• Standard SAE: Minimizes reconstruction error on $X^{(1)}$1, with strong sparsity on $X^{(1)}$2.
• Group-Sparse SAE (GSAE/MGSAE): Adds a group-sparsity penalty

$X^{(1)}$3

to couple paired image-text activations. Cross-modal random masking further enforces that both modalities select active units from the same random pool, increasing multimodal neuron prevalence and reducing dead or modality-exclusive atoms (Kaushik et al., 27 Jan 2026).

Modality Dominance Scoring (MDS)

Given paired embeddings, the Modality Dominance Score for each feature (dimension) $X^{(1)}$4 is

$X^{(1)}$5

with MDS used to categorize features as image-dominant, text-dominant, or cross-modal, after thresholding by empirically computed mean and standard deviation $X^{(1)}$6 (Yan et al., 16 Feb 2025).

Discrete Codebooks and Vector Quantization

Other schemes build a shared vocabulary of discrete codes via vector quantization. Each modality projects fine-grained features into a common codebook $X^{(1)}$7, then uses a cross-modal code-matching loss to enforce that paired sequence-level embeddings use similar code distributions:

$X^{(1)}$8

Thus, semantic clusters in the codebook align across modalities, supporting both retrieval and fine-grained localization (Liu et al., 2021).

Low-Rank + Sparse Factorization

In affective computing, feature matrices are decomposed as $X^{(1)}$9, $X^{(2)}$0—with $X^{(2)}$1 shared and $X^{(2)}$2, $X^{(2)}$3 sparse and modality-specific. An augmented Lagrangian framework directly optimizes over this structure, and a dynamic soft prompt fuses these components for prediction (Tian et al., 8 Jun 2025).

Sheaf-Theoretic and Obstruction-Based Approaches

At a more abstract, topological level, cross-modal decompositionality is formalized via cellular sheaves over a modality-independent sample graph. Here, global compatibility between modalities reduces to the existence and structure of global sections (shared alignment maps). Projection hardness quantifies the minimal model complexity for a global map, while sheaf Laplacian obstruction measures the minimal spatial variation (parameter inconsistency) needed when only locally consistent alignment maps exist (Sloboda, 8 Apr 2026).

3. Empirical Evidence and Evaluation Metrics

Quantitative and qualitative analyses demonstrate the efficacy of cross-modal decompositionality.

• Neuron Activity Partitioning: MGSAE achieves $X^{(2)}$4 neurons that co-activate for both image and text, compared to $X^{(2)}$5 for vanilla SAE; “dead” neurons fall from $X^{(2)}$6 (SAE) to $X^{(2)}$7 (MGSAE) (Kaushik et al., 27 Jan 2026).
• Multimodal Monosemanticity Score (MMS): MGSAE dramatically increases the count of high-MMS (well-aligned) neurons versus groupless SAEs.
• Zero-shot Cross-modal Task Performance:
  • On CLIP (ViT-B/16, CC3M): MGSAE achieves image/text accuracy $X^{(2)}$8 on CIFAR-10 and $X^{(2)}$9 on ImageNet, substantially outperforming standard SAEs (Kaushik et al., 27 Jan 2026).
  • Audio/text zero-shot (CLAP): MGSAE recovers $W \in \mathbb{R}^{d \times p}$0 (GTZAN genre), $W \in \mathbb{R}^{d \times p}$1 (NSynth instrument), and retrieval MRR $W \in \mathbb{R}^{d \times p}$2, again outperforming non-group methods.
• Downstream manipulation: MDS-based decompositions enable direct bias analysis, cross-modal adversarial defenses (TextD-feature alignment disables text-only adversarial attacks with success rate dropping from $W \in \mathbb{R}^{d \times p}$3 to $W \in \mathbb{R}^{d \times p}$4), and style/semantic disentanglement in generative models (Yan et al., 16 Feb 2025).
• Statistical Decomposition: Bimodal (cross-modal) features comprise $W \in \mathbb{R}^{d \times p}$5 of all neurons in CLIP’s last layer, with clear human-interpretable semantics (Yan et al., 16 Feb 2025).
• Low-rank plus sparse ablations: Removing the shared component $W \in \mathbb{R}^{d \times p}$6 degrades F1 by $W \in \mathbb{R}^{d \times p}$7–$W \in \mathbb{R}^{d \times p}$8pp, removing the sparse components $W \in \mathbb{R}^{d \times p}$9 has smaller but consistent negative impact (Tian et al., 8 Jun 2025).

4. Application Domains and Use Cases

Cross-modal decompositionality has concrete operational value in multiple areas:

• Interpretable retrieval and control: Sparse aligned concept bases allow direct editing of semantic or stylistic aspects of cross-modal content (e.g., linear interventions on “violin” neuron re-weight music features in CLAP; blending text- and image-dominant subsets in text-to-image generation controls content and visual style) (Kaushik et al., 27 Jan 2026, Yan et al., 16 Feb 2025).
• Counterfactual and explainable AI: Decompose and Explain (DeX) employs cross-modal perturbations along text-driven directions in embedding space, generating per-instance semantic counterfactuals without retraining. By collecting dominant concept removals across samples, DeX quantifies model decision factors and exposes dataset bias (e.g., privacy classifiers on VISPR vs PrivacyAlert) (Baia et al., 21 Dec 2025).
• Benchmark construction and analysis: Multimodal Item Response Theory (M3IRT) explicitly decomposes model ability into image-only, text-only, and cross-modal reasoning axes, identifying “shortcut” items and prioritizing those requiring genuine cross-modal fusion (Uebayashi et al., 3 Mar 2026).
• Affective computing: Explicit decomposition into shared/contrast components enhances performance on tasks such as multimodal sentiment/emotion analysis and hateful meme detection (e.g., Twitter-15 accuracy $Z \in \mathbb{R}^{p \times n}$0, Hateful Meme Challenge AUROC $Z \in \mathbb{R}^{p \times n}$1) (Tian et al., 8 Jun 2025).
• Neuroscience/brain signal analysis: Cross-modal decoding with representational similarity analysis identifies time-varying, modality-independent codes that align with semantic and visual—rather than lexical—feature spaces in MEG data (Dirani et al., 2023).

5. Geometric and Algebraic Perspectives

Geometry and linear algebra provide a deeper layer of structure:

• Iso-Energy/Modality Score Partitioning: Each code axis’ average energy is measured per domain; “bimodal” (shared) atoms are those for which $Z \in \mathbb{R}^{p \times n}$2, enabling separation into three linear spaces: shared/bimodal, image-only, text-only (Dhimoïla et al., 5 Feb 2026).
• Discrete shared codebooks: Discretized embedding vector quantization across modalities creates a finite dictionary of cross-modal semantic “atoms.” Distribution matching at the codebook level enforces modality-invariant concept usage (Liu et al., 2021).
• Sheaf Laplacian theory: The existence of a global alignment map (a global section) versus local, spatially varying parameter fields corresponds to two distinct failure modes: projection hardness and sheaf-Laplacian obstruction. Combinatorial Laplacian eigenstructure (spectral gap) bounds the “excess error” incurred by globalizing local fits (Sloboda, 8 Apr 2026).

A plausible implication is that cross-modal decompositionality operationalized in latent geometry provides actionable means to eliminate the modality gap, project queries into in-distribution space, or diagnose limits of fusion in learned representations.

6. Limitations, Extensions, and Open Problems

While cross-modal decompositionality has proven utility, each approach has inherent limitations:

• Supervision or pairing: Many methods require some degree of paired (aligned) multimodal data for group sparsity, iso-energy alignment, or codebook matching to be effective (Kaushik et al., 27 Jan 2026, Liu et al., 2021).
• Tuning/stability and scalability: The efficacy and interpretability of decompositions (e.g., group-sparsity hyperparameters $Z \in \mathbb{R}^{p \times n}$3, masking rate $Z \in \mathbb{R}^{p \times n}$4) depend on careful tuning to balance reconstruction fidelity and feature alignment (Kaushik et al., 27 Jan 2026).
• Obstruction/non-transitivity: Sheaf-based analysis proves that compatibility and alignment may fail for two distinct reasons—no low-complexity aligning map exists (hardness), or spatial variation prevents a single global alignment (obstruction). Furthermore, compatibility is generally non-transitive: bridging via intermediate modalities (composition) can reduce the needed complexity, but not always (Sloboda, 8 Apr 2026).
• Robustness for arbitrary modalities: While most frameworks generalize in principle to $Z \in \mathbb{R}^{p \times n}$5 modalities, many have only been validated for $Z \in \mathbb{R}^{p \times n}$6 (e.g., vision-language, audio-language). Extending group-sparsity, codebook alignment, or Laplacian obstruction to more complex multimodal graphs is an open direction.
• Actionability and human-labeled concepts: Decomposed features are not always perfectly aligned with human semantics; in some settings, extra supervision or interpretability tools may be required for labeling and controlling decomposed components (Yan et al., 16 Feb 2025).

7. Comparative Overview of Techniques

Approach	Factorization Type	Key Modality Alignment Mechanism
Group-Sparse SAE/MGSAE	Sparse linear, shared	Group $Z \in \mathbb{R}^{p \times n}$7 penalty, cross-modal masking
Modality Dominance (MDS)	Feature scoring/tripart.	Mean abs. activation ratios, thresholded partition
Iso-Energy SAE	Sparse linear iso-energy	Energy (sec. moment) balancing per atom
Vector Quantization (CQDRL)	Discrete codebook	Code matching loss ($Z \in \mathbb{R}^{p \times n}$8)
Low-Rank + Sparse LMR	Matrix factorization	Nuclear + $Z \in \mathbb{R}^{p \times n}$9 norm, attention fusion
Sheaf Laplacian/Obstruction	Category-theoretic/geometric	Laplacian energy, projection hardness
Concept Counterfactuals (DeX)	Per-image, linear	Text-driven embedding subtraction

Each method is optimized for different axes: interpretability, granularity, scalability, or theoretical generality.

---

In summary, cross-modal decompositionality organizes, analyzes, and manipulates shared and modality-specific structure in multimodal models. It encompasses sparse, low-rank, discrete, and geometric formulations, enabling interpretable mechanistic understanding, performance gains, and robust benchmarking. The ongoing challenge is to develop efficient, generalizable decompositional schemes that scale to many modalities, respect human semantics, and provide new theoretical guarantees for integration and alignment (Kaushik et al., 27 Jan 2026, Yan et al., 16 Feb 2025, Liu et al., 2021, Dhimoïla et al., 5 Feb 2026, Sloboda, 8 Apr 2026, Uebayashi et al., 3 Mar 2026, Tian et al., 8 Jun 2025, Baia et al., 21 Dec 2025, Dirani et al., 2023).