CM-SSA: Cross-Modal Scene Semantic Alignment

• CM-SSA is a multimodal alignment strategy that explicitly maps scene-level semantics such as entities, attributes, and relations between visual and textual modalities.
• It leverages structured intermediates like scene graphs, semantic distributions, and pooled embeddings to bridge modality-specific differences and enforce visual grounding.
• Empirical results indicate that CM-SSA improves reasoning fidelity, captioning quality, and retrieval performance by reducing hallucinations and misalignments.

Cross-Modal Scene Semantic Alignment (CM-SSA) denotes a family of multimodal learning objectives in which semantic content at the scene level is made consistent across modalities, rather than merely forcing heterogeneous features to occupy a nearby location in a shared embedding space. In the most explicit recent formulation, SceneAlign presents CM-SSA as the problem of making a multimodal LLM’s chain-of-thought faithful to the actual visual structure in complex scenes, so that text reasoning and visual scene representation correspond at the level of entities, relations, and inference structure (Wang et al., 9 Jan 2026). Across the broader literature, closely related formulations pursue a shared, modality-agnostic scene representation, semantic consistency after cross-modal translation, scene-graph alignment, or semantic-only matching after separating modality-specific nuisance information (Aytar et al., 2016).

1. Core definition and problem setting

In SceneAlign, the CM-SSA problem is defined over an image-question pair $(I,q)$ for which an MLLM should produce a reasoning chain $\tau$ and answer $a$. The paper’s central claim is that in visually crowded scenes, the main failure is not just wrong answers, but reasoning that drifts away from the image—hallucinating objects, swapping relations, skipping evidence, or adding irrelevant steps. It therefore treats reasoning faithfulness, visual grounding, and cross-modal semantic alignment as three linked properties: reasoning faithfulness means the chain-of-thought faithfully follows the evidence; visual grounding means each reasoning step refers to actual objects, attributes, and relations in the image; and cross-modal semantic alignment is the broader objective of making the text reasoning and visual scene representation correspond at the level of entities, relations, and inference structure (Wang et al., 9 Jan 2026).

This scene-level view differs from formulations that emphasize only instance matching or class preservation. Earlier cross-modal scene work already treated the goal as learning a shared representation that is agnostic of the modality, using natural images, line drawings, descriptions, clip art, and spatial text images labeled with the same scene categories but not paired across modalities. In that setting, semantic alignment meant that higher layers should encode scene concepts compatibly across modalities even when low-level statistics differ substantially (Aytar et al., 2016).

A recurring conceptual distinction in this literature is that semantic consistency is not identical to embedding coincidence. DSTC makes this explicit for cross-modal retrieval by arguing that two paired samples from different modalities need not become the same point in a shared space; for semantic retrieval, it is enough that they remain semantically equivalent, meaning that class identity is preserved after translation across modalities and after translation back (Parida et al., 2021).

2. Historical development and major variants

An early ancestor of CM-SSA is the weakly supervised scene-representation program exemplified by “Cross-Modal Scene Networks,” which introduced CMPlaces and learned a representation with modality-specific lower layers and shared higher layers, notably pool5, fc6, and fc7. Its two principal alignment mechanisms were modality tuning, which freezes higher semantic layers while adapting lower layers for a new modality, and statistical regularization, which constrains hidden activations to match a prior learned from Places-CNN (Aytar et al., 2016).

A second line of work replaced direct point-matching by semantic preservation under translation. DSTC uses modality-specific encoders, classifiers, and translators, and defines alignment by whether a translated sample is still classified correctly by the target-modality classifier. The resulting formulation distinguishes geometry-driven pointwise consistency from semantics-driven transitive consistency, and explicitly argues that distance minimization alone may be too strict for semantic retrieval (Parida et al., 2021).

A third line made scene structure explicit. Cross2StrA uses visual scene graphs and language scene graphs as pivots in unpaired cross-lingual image captioning, aligning image semantics with pivot-language caption semantics through a contrastive objective over scene-graph nodes. SceneAlign then sharpened this idea for multimodal reasoning by using scene graphs as a structured bridge between image and text and by perturbing reasoning-critical graph components to produce hard negative rationales (Wu et al., 2023).

More recent methods also broadened the notion of “scene semantics.” CDDS argues that standard image-text alignment methods optimize embedding consistency even though embeddings mix semantic and modality-specific information. It therefore decouples each modality into semantic and modality components and aligns only the semantic component through distribution sampling, so that the model aligns the true semantics rather than the whole mixed embedding (Ma et al., 5 Mar 2026). CrossOver extends scene-level alignment to 3D settings, shifting from rigid object-instance alignment to flexible, scene-level modality alignment across RGB images, point clouds, CAD models, floorplans, and text descriptions (Sarkar et al., 20 Feb 2025).

3. Structural intermediates and alignment units

CM-SSA methods differ primarily in the structural object they treat as the carrier of scene meaning.

Paper	Structural intermediate	Alignment unit
SceneAlign (Wang et al., 9 Jan 2026)	Scene graph $SG=(E,A,R)$ and CoT-grounded subgraph $SG_q^c$	Reasoning trajectories grounded in entities, attributes, and relations
Cross2StrA (Wu et al., 2023)	Visual SG $SG^V$ and language SG $SG^L$	Object, attribute, and relation nodes
CDDS (Ma et al., 5 Mar 2026)	Semantic components and x-semantic components	Patch/word semantic distributions
CrossOver (Sarkar et al., 20 Feb 2025)	Scene embeddings from pooled modality features	Whole-scene modality-agnostic embeddings

SceneAlign formalizes the scene graph as

$SG=(E,A,R),$

where $E$ is the set of entities, $A$ the set of attribute pairs, and $\tau$0 the set of relational triples $\tau$1. For each training instance, GPT-4o is used to extract a positive scene graph $\tau$2 and a grounded positive CoT $\tau$3. The model then extracts the CoT-grounded subgraph $\tau$4, defines the residual pool

$\tau$5

and perturbs only the reasoning-critical subgraph. The four perturbation operators are

$\tau$6

$\tau$7

$\tau$8

and

$\tau$9

These simulate relation mis-grounding or role confusion, hallucination-like substitution, incomplete grounding, and over-specification, respectively. Negative scene graphs are then formed as

$a$0

filtered by a Jaccard-overlap mid-range interval, and selected by a diversity-based max–min rule; the final set contains diverse, medium-difficulty negatives, with three negatives per instance by default (Wang et al., 9 Jan 2026).

Cross2StrA also centers scene graphs, but in a different regime. There, both modalities are represented as

$a$1

with nodes representing objects, attributes, and relations and edges encoding graph structure. A GCN encodes the visual SG, a Transformer decoder generates the pivot caption from SG features, and a contrastive loss aligns semantically corresponding visual and language nodes. The paper distinguishes this cross-modal semantic structure alignment from constituency-tree alignment, which handles fluency and grammaticality in the pivot-to-target translation stage (Wu et al., 2023).

Other CM-SSA variants do not require scene graphs. CDDS uses semantic distributions over image patches and text words, sparsifies cross-modal distribution correlations with an adaptive soft-threshold, and samples an x-semantic distribution in the other modality’s descriptive form. CrossOver instead aggregates instance features into scene embeddings and later replaces explicit instance semantics at inference by unified 1D, 2D, and 3D encoders. This suggests that the structural intermediate can range from symbolic scene graphs to pooled scene embeddings to semantic distributions, provided that it localizes the scene content that should remain invariant across modalities (Ma et al., 5 Mar 2026).

4. Objective functions and optimization regimes

The optimization strategy in CM-SSA depends on what is being aligned. SceneAlign uses Direct Preference Optimization (DPO) over scene-graph-grounded reasoning trajectories. Its context is

$a$2

and the preference dataset is

$a$3

The model is trained to assign higher likelihood to the positive scene-faithful rationale than to the structured negative under the standard DPO form, with the novelty residing in the fact that the preference pairs are not answer-level or purely text-level; they are scene-graph-grounded reasoning trajectories (Wang et al., 9 Jan 2026).

Contrastive learning remains the dominant alternative. Cross2StrA defines a node-level contrastive loss $a$4 over visual and language SG embeddings using cosine similarity and a confidence threshold for positive-pair selection (Wu et al., 2023). In audio-visual localization, semantic alignment is decomposed into spatial localization similarity $a$5 and global semantic similarity $a$6, and both are optimized jointly with a multi-positive contrastive loss so that localization and scene semantic understanding are trained together (Senocak et al., 2023). S-CMRL defines a semantic alignment optimization loss

$a$7

which aligns residual audio and visual features from the same sample and time step in a shared semantic space (He et al., 18 Feb 2025). MCFA for zero-shot remote sensing combines VAE reconstruction, cross-modal feature reconstruction, cross-modal distribution alignment, and supervised contrastive losses $a$8, $a$9, and $SG=(E,A,R)$0 to align image scenes with semantic class descriptors at both single-instance and cross-instance levels (Liu et al., 2023).

A distinct family of methods explicitly opposes semantic alignment to naïve embedding matching. DSTC uses

$SG=(E,A,R)$1

where the pointwise terms focus on geometry and DSTC focuses on semantics (Parida et al., 2021). CDDS similarly uses

$SG=(E,A,R)$2

where $SG=(E,A,R)$3 is semantic consistency via distribution sampling, $SG=(E,A,R)$4 preserves modality-specific consistency, and $SG=(E,A,R)$5 and $SG=(E,A,R)$6 enforce information integrity (Ma et al., 5 Mar 2026).

In some applications, semantic alignment is auxiliary rather than primary. The ICA method named CM-SSA uses a complexity regression branch and a scene semantic alignment branch on top of CLIP. Its total objective is

$SG=(E,A,R)$7

with empirical weights $SG=(E,A,R)$8 and $SG=(E,A,R)$9. The alignment branch aligns images with scene descriptions generated by InstructBLIP, but the paper states that this branch is a feature-guidance mechanism rather than the final predictor (Luo et al., 21 Oct 2025).

5. Empirical behavior across tasks

In multimodal reasoning, SceneAlign is evaluated on Qwen2.5-VL-3B, Qwen3-VL-4B, Qwen2.5-VL-7B, InternVL3-8B, and LLaVA-Next-8B, using training data constructed from A-OKVQA and benchmarks MME-RealWorld, EMMA-mini, ScienceQA, MMMU-Reasoning, HallusionBench, GQA, and SeedBench. The paper reports consistent gains across all models and benchmarks, with average gains of roughly 3%–5% on reasoning-heavy benchmarks. The largest gains are on HallusionBench and EMMA. On LLaVA-Next-8B, SceneAlign also outperforms AoT and LLaVA-Reasoner across nearly all metrics, especially on HallusionBench and EMMA. Its ablations show that removing any of the four perturbation operators hurts performance, that relation-level grounding is the most important, that the overlap-threshold sensitivity follows an inverted-U pattern with default $SG_q^c$0 and $SG_q^c$1 working best, and that three negatives per instance are best overall (Wang et al., 9 Jan 2026).

In captioning, Cross2StrA reports a full-model average score of 45.0, while removing $SG_q^c$2 yields 42.3, a drop of 2.7 points. The associated metric drops are Zh→En BLEU: 57.7 → 54.4, Zh→En ROUGE: 33.5 → 29.7, En→Zh BLEU: 52.8 → 50.1, and En→Zh ROUGE: 36.1 → 34.9. Human evaluation on the MSCOCO (En) → AIC-ICC (Zh) transfer gives the full model Relevancy: 9.70, Diversification: 9.53, and Fluency: 9.22; without SG, relevancy drops to 8.35 and diversification to 7.75 (Wu et al., 2023).

In perceptual assessment, the ICA method named CM-SSA reports IC9600: SRCC 0.958, PLCC 0.961, RMSE 0.009, VISC-C: SRCC 0.823, PLCC 0.805, RMSE 0.018, and Savoias: SRCC 0.883, PLCC 0.875, RMSE 0.026. Its branch ablation on IC9600 gives Branch_C only: SRCC 0.947, PLCC 0.951, Branch_A only: SRCC 0.915, PLCC 0.922, and both branches: SRCC 0.958, PLCC 0.961, supporting the paper’s claim that the semantic alignment branch acts as guidance rather than a standalone complexity estimator (Luo et al., 21 Oct 2025).

In retrieval and scene understanding, CDDS reports improvements of 6.6\% to 14.2\% overall and representative retrieval outcomes of rSum = 552.5 on Flickr30K with Swin-384 and rSum = 472.1 on MS-COCO 5K with Swin-384 (Ma et al., 5 Mar 2026). CrossOver reports $SG_q^c$3 top-1 scene category recall of 64.74 versus 42.95 for the instance baseline, $SG_q^c$4 top-1 recall of 55.77 versus 13.14 for ULIP-2, and $SG_q^c$5 top-1 recall of 54.64, while also demonstrating emergent modality behavior for pairs not directly trained together (Sarkar et al., 20 Feb 2025). In audio-visual localization, joint localization and semantic alignment yield VGG-SS cIoU 42.64 and AUC 41.48, and the retrieval results on VGG-SS reach A → I R@1 16.47, R@5 36.99, R@10 49.00 and I → A R@1 20.09, R@5 42.38, R@10 53.66, supporting the paper’s claim that higher sound localization performance does not guarantee better audio-visual semantic understanding (Senocak et al., 2023).

6. Distinctions, limitations, and acronym ambiguity

A common misconception is that CM-SSA is equivalent to collapsing modalities into a single shared embedding by direct similarity maximization. Several papers argue against that view. DSTC states that exact point-to-point matching is unnecessary for semantic retrieval and that the correct condition is class preservation after translation across modalities and after translation back (Parida et al., 2021). CDDS similarly argues that embeddings contain more than semantics and that aligning such mixed representations can induce semantic deviation and information loss / distortion (Ma et al., 5 Mar 2026). SceneAlign adds a further distinction: a rationale can be linguistically plausible yet visually false, so fluent explanation is not evidence of cross-modal semantic alignment (Wang et al., 9 Jan 2026).

Another misconception is that CM-SSA always requires explicit scene graphs or dense semantic supervision. The weakly supervised scene-network work used only scene labels and no paired examples (Aytar et al., 2016). CrossOver is explicitly designed so that inference does not require semantic instance information, even though scene-level training can use instance features (Sarkar et al., 20 Feb 2025). By contrast, some methods do depend on stronger supervision or auxiliary generation: SceneAlign uses GPT-4o to extract positive scene graphs and positive CoTs from A-OKVQA (Wang et al., 9 Jan 2026); DSTC requires paired data and class labels (Parida et al., 2021); and S-CMRL assumes paired audio-visual samples and synchronized time steps, with semantic alignment applied to residual features inside a Transformer-based spiking architecture (He et al., 18 Feb 2025).

The acronym itself is not stable across the literature. In the RGB-thermal semantic segmentation paper CM-SSM, CM-SSA denotes Cross-Modal State Space Association, not Cross-Modal Scene Semantic Alignment. There, CM-SS2D constructs global cross-modal associations through state-space recurrence, and CM-SSA fuses those global associations with local spatial features extracted through convolutional operations. The module improves CART ablation performance from 72.6 mIoU for simple addition to 74.6 for CM-SSA, but the term refers to a fusion block in a state-space segmentation architecture rather than to the scene-semantic alignment problem as formulated in SceneAlign or related scene-level alignment papers (Guo et al., 22 Jun 2025).

This suggests that CM-SSA is best understood not as a single canonical algorithm, but as a design principle: preserve scene meaning across modalities while ensuring that the alignment signal operates at the level most relevant to the task—scene categories, semantic class identity, scene-graph nodes, reasoning trajectories, semantic distributions, or modality-agnostic scene embeddings.