Cross-Modal Options Synthesis (CmOS)

• Cross-modal Options Synthesis (CmOS) is a framework for generating target modal options by leveraging complementary evidence across modalities.
• It employs staged processes such as content evaluation, latent fusion, and cross-modal alignment via shared geometry and attention mechanisms.
• Evaluations focus on both the fidelity of synthesized options and their downstream utility, as seen in improved diagnostic and content discrimination metrics.

Searching arXiv for the named framework and adjacent cross-modal synthesis literature to ground the article in cited papers. {} Searching "Cross-modal Options Synthesis" Cross-modal Options Synthesis (CmOS) denotes a family of multimodal methods that synthesize, infer, or assemble target options in one modality from evidence available in another. In the narrowest and explicitly named sense, CmOS is the framework introduced for educational multiple-choice question generation with visual answer options, where multimodal instructional content $\mathcal{C}=(T,I,A)$ is transformed into a question, a correct visual option, and visual distractors through staged reasoning, retrieval, and image generation (Wang et al., 26 Aug 2025). Taken more broadly, the papers surveyed here suggest a wider methodological pattern: a target modality or option is not treated as an isolated output, but as a structure that must be aligned to source modalities through shared geometry, shared semantics, cross-attention, transport, or probabilistic latent fusion, and then validated by downstream usefulness rather than appearance alone (Sikka et al., 2018).

1. Scope and definitional boundaries

The explicit label “Cross-modal Options Synthesis” appears in the educational MCQ framework of "Beyond the Textual: Generating Coherent Visual Options for MCQs" (Wang et al., 26 Aug 2025). That framework has four stages: evaluate content convertibility, generate alternative questions and reasons, select the optimal question-reason pair, and generate option descriptions and visual options. Its input is $\mathcal{C}=(T,I,A)$, where $T$ is textual context, $I$ is an associated image, and $A$ is the target answer; its outputs are a question $Q$, a correct visual option $A'$, and distractors $D_s$ (Wang et al., 26 Aug 2025).

Several adjacent works are directly relevant to CmOS without using the term as a formal system name. These include one-to-one medical image synthesis from MRI to FDG-PET (Sikka et al., 2018), arbitrary-subset missing-modality synthesis with hierarchical variational models (Dorent et al., 2024), cross-modal MRI reconstruction using Optimal Transport and alternating synthesis-reconstruction updates (Wang et al., 2023), joint RGB-and-geometry novel-view generation with attention transfer (Kwak et al., 13 Jun 2025), bridge-modality ultrasound synthesis for speech enhancement (Yu et al., 2024), and controllable anomaly synthesis from non-matching text-image references (He et al., 2024). This suggests that CmOS functions less as a single canonical architecture than as an umbrella designation for systems that convert partial multimodal evidence into a target option space.

A recurrent boundary condition in this literature is that not every multimodal method is a full CmOS system. "Modality Selection and Skill Segmentation via Cross-Modality Attention" explicitly does not present a complete option-learning framework with initiation sets, termination functions, or a trained high-level option selector; it is better understood as an enabling mechanism for modality relevance estimation and primitive discovery (Jiang et al., 20 Apr 2025). Likewise, "What to align in multimodal contrastive learning?" is a representation-learning substrate rather than an option-synthesis model, despite being highly relevant to how multimodal option latents should be formed (Dufumier et al., 2024).

2. Problem formulations and recurrent computational pattern

A common formal pattern is partial observation over modalities followed by synthesis of a missing or target modality. In MRI-to-PET synthesis, the problem is supervised paired cross-modal regression from MRI-derived gray matter volumes to normalized whole-brain FDG-PET volumes, using a deterministic mapping where “every voxel in the given input scan is used to predict every voxel in the output scan” (Sikka et al., 2018). In MMHVAE, the complete multimodal variable is $\mathbf{X}=(X_1,\dots,X_M)$, with a missingness indicator $\mathbf{R}\in\{0,1\}^M$, and the model receives any non-empty observed subset $\mathcal{C}=(T,I,A)$0 and synthesizes missing modalities through a shared hierarchical latent representation (Dorent et al., 2024). In the educational CmOS framework, convertibility is itself part of the problem definition: the system first decides whether a piece of content is suitable for visual-option synthesis before generating any options (Wang et al., 26 Aug 2025).

A second recurring pattern is the use of an intermediate alignment space before synthesis. In USpeech, large-scale video-audio data are used to align video and audio in a shared semantic space, and only then is ultrasound synthesized from audio via an encoder-decoder trained on a small paired audio-ultrasound dataset (Yu et al., 2024). In CrossModalityDiffusion, modality-specific encoders produce geometry-aware feature volumes that are placed in a common 3D coordinate frame and rendered into target-view feature images that condition a modality-specific diffusion decoder (Berian et al., 16 Jan 2025). In COMET, new modalities are incrementally mapped into a unified discrete codebook through a mediator modality, with pseudo-modality replay used to preserve prior semantic structure under continual modality expansion (Xia et al., 1 Apr 2025).

A third pattern is that the target “option” is frequently richer than a single image. In the MCQ setting, the synthesized option set is a structured bundle comprising a question, a correct visual answer, and semantically plausible visual distractors (Wang et al., 26 Aug 2025). In novel-view synthesis, the option may be the pair $\mathcal{C}=(T,I,A)$1, because the requested output is jointly specified by modality and pose (Berian et al., 16 Jan 2025). In robotics, the option-like unit is closer to a primitive skill segment associated with a time-varying modality relevance profile, although the paper stops short of formal options (Jiang et al., 20 Apr 2025). This suggests that “option” in CmOS is best interpreted as the target unit of multimodal decision or generation, not necessarily as a reinforcement-learning option.

3. Alignment and synthesis mechanisms

The mechanisms used to realize CmOS-like systems differ sharply across domains, but most fall into a small number of families.

Mechanism family	Representative use	Paper
Whole-volume supervised regression	MRI gray matter $\mathcal{C}=(T,I,A)$2 FDG-PET with a 3D U-Net	(Sikka et al., 2018)
Hierarchical latent fusion	Arbitrary-subset missing-modality synthesis with hierarchical PoE and mixture ELBO	(Dorent et al., 2024)
Geometry-aware shared space	Multi-modal novel-view synthesis via overlapped feature volumes and volumetric rendering	(Berian et al., 16 Jan 2025)
Cross-modal attention transfer	Image-attention maps injected into geometry denoising for aligned RGB/pointmap synthesis	(Kwak et al., 13 Jun 2025)
Structural phrase-region alignment	Attribute-Phrase to garment-part matching and bundled attention in text-guided diffusion	(Zhang et al., 2023)
Cross-modal semantic priors	Text-image anomaly reference encoded into a controllable diffusion prior	(He et al., 2024)

In the medical-imaging lineage, the decisive design choice is often how much global structure is exposed to the synthesis model. The MRI-to-PET system adopts a modified 3D U-Net rather than a patch-based CNN, motivated by the claim that MRI–PET correspondence is distributed and nonlinear rather than purely local; empirically, the proposed method improves over a replicated patch baseline on all reported image-fidelity metrics, with SSIM $\mathcal{C}=(T,I,A)$3, MAE $\mathcal{C}=(T,I,A)$4, and PSNR $\mathcal{C}=(T,I,A)$5 (Sikka et al., 2018). MMHVAE pursues a different route: a Multimodal Hierarchical Variational Autoencoder with a hierarchical product-of-experts posterior, a mixture-over-subsets ELBO, and dataset-level adversarial marginal regularization, so that any non-empty subset of modalities can be fused into a posterior that estimates missing information needed for synthesis (Dorent et al., 2024).

Geometry-grounded systems make the alignment space explicit. CrossModalityDiffusion places modality-specific feature volumes into a common 3D latent space, fuses them by summation at sampled points, and renders target-view feature images with volumetric rendering before decoding with a target-modality denoiser (Berian et al., 16 Jan 2025). "Aligned Novel View Image and Geometry Synthesis via Cross-modal Attention Instillation" instead formulates novel-view RGB image and pointmap generation as paired inpainting conditioned on warped partial geometry, with a one-way attention transfer in which attention maps from the image diffusion branch are injected into the geometry branch during both training and inference (Kwak et al., 13 Jun 2025). The paper’s proximity-based mesh conditioning further augments the shared conditioning interface with depth and normal cues (Kwak et al., 13 Jun 2025).

Text-conditioned diffusion systems often replace flat prompt conditioning with more structured cross-modal controls. DiffCloth parses prompts into Attribute-Phrases such as adjective-plus-part-noun bundles, aligns them to segmented garment parts through bipartite matching, and regularizes bundled attention maps so that adjectives and their associated nouns attend to spatially consistent regions (Zhang et al., 2023). AnomalyControl uses a non-matching prompt pair—a text-image anomaly reference and a targeted text prompt—to learn cross-modal semantic features through a Cross-modal Semantic Modeling module, sharpen them with Anomaly-Semantic Enhanced Attention, and inject them into Stable Diffusion through a Semantic Guided Adapter based on decoupled cross-attention (He et al., 2024). A plausible implication is that CmOS systems become more controllable when the conditioning interface is a structured semantic prior rather than an undifferentiated text embedding.

4. Evaluation: fidelity, utility, and option quality

A defining methodological feature of this literature is that evaluation often extends beyond direct reconstruction or perceptual quality. In MRI-to-PET synthesis, the main claim is not only that PET-like images can be generated, but that the synthesized PET contributes complementary diagnostic information: MRI-only classification is reported as $\mathcal{C}=(T,I,A)$6, whereas MRI plus synthesized PET reaches $\mathcal{C}=(T,I,A)$7 with a paired-sample $\mathcal{C}=(T,I,A)$8-test giving $\mathcal{C}=(T,I,A)$9 (Sikka et al., 2018). The same paper explicitly cautions that synthesized PET does not match real PET performance and should be treated as a machine-learning aid rather than as a surrogate biomarker (Sikka et al., 2018).

This downstream-first evaluation philosophy is even more explicit in clinically oriented augmentation systems. In "Efficient endometrial carcinoma screening via cross-modal synthesis and gradient distillation," unpaired MRI-to-ultrasound synthesis is used to enlarge the pathological ultrasound pool, and the downstream lightweight screening network reports accuracy $T$0, sensitivity $T$1, specificity $T$2, and ROC AUC $T$3 at $T$4 GFLOPs (Shan et al., 23 Feb 2026). The paper’s ablations separate the value of synthetic pre-training from gradient distillation, showing that the cross-modal augmentation strategy contributes materially to minority-pathology learning (Shan et al., 23 Feb 2026).

In USpeech, the downstream criterion is speech enhancement rather than image similarity. The system trained with synthesized ultrasound reports PESQ $T$5, STOI $T$6, LSD $T$7, and MOS $T$8, compared with PESQ $T$9, STOI $I$0, LSD $I$1, and MOS $I$2 when trained with physical ultrasound (Yu et al., 2024). The paper interprets this as evidence that the synthetic ultrasound is comparable to physical data for downstream learning, which suggests a broader CmOS lesson: a synthesized modality can be valuable even when the primary objective is not the fidelity of that modality itself.

The explicitly named educational CmOS framework evaluates option quality in a different way. It reports $I$3 average accuracy for content discrimination, average BLEU-4/ROUGE-L of $I$4 for question generation, and average SSIM/CLIP-T of $I$5 for visual option generation, outperforming reported baselines across all three tasks (Wang et al., 26 Aug 2025). Because its output is an option set rather than a single image, the evaluation combines discrimination accuracy, textual question quality, and visual-option coherence, which is distinctive within the broader cross-modal synthesis literature (Wang et al., 26 Aug 2025).

5. Representation learning, attention, and option-like structure

The theoretical question of what should be aligned in a CmOS system is addressed most directly by CoMM. That paper decomposes task-relevant information for two modalities $I$6 as

$I$7

where $I$8 is redundant information, $I$9 are modality-unique terms, and $A$0 is synergy (Dufumier et al., 2024). Its central claim is that standard CLIP-style cross-modal contrastive learning mostly targets redundancy, whereas aligning multimodal representations rather than modality-specific pairs can preserve shared, unique, and synergistic information (Dufumier et al., 2024). This suggests that a CmOS system built only on pairwise alignment may discard precisely the complementary structure needed for robust option synthesis.

Several papers operationalize this broader view through attention. In medical evidence synthesis for Parkinson’s Disease, Multi-Head Co-Attention computes a joint similarity distribution between 3D brain SPECT features and selected DNA methylation features, achieving $A$1 accuracy and AUROC $A$2, while also exposing attention coefficients that the authors use for pathway analysis (Taylor et al., 2019). In the robotics paper on cross-modality attention, modality embeddings from two consecutive timesteps are stacked into $A$3, processed by an 8-head, 2-layer transformer, and analyzed through attention weights that vary across primitive actions such as Reach Base, Insert, and Screw (Jiang et al., 20 Apr 2025). The paper explicitly states that this is not yet a full options framework, but it provides evidence that cross-modal attention can support unsupervised primitive clustering and modality-sensitive state abstraction (Jiang et al., 20 Apr 2025).

Other works make the “option-like” structure more explicit by turning semantic bundles or discrete codes into reusable control units. DiffCloth treats Attribute-Phrases and segmented garment parts as aligned phrase-region control units that can be swapped during editing (Zhang et al., 2023). COMET maintains a shared discrete codebook across continually added modalities through a Continual Mixture of Experts Adapter and Pseudo-Modality Replay, thereby preserving a modality-agnostic semantic inventory that can support unseen modality-pair transfer (Xia et al., 1 Apr 2025). This suggests a plausible route toward explicit CmOS libraries: option units could be represented as shared discrete codes or structured phrase-region bundles rather than as opaque continuous embeddings.

6. Limitations, misconceptions, and open problems

A first limitation is terminological. The data suggest that CmOS is not yet a uniformly standardized field label. One paper names a specific educational framework “CmOS” (Wang et al., 26 Aug 2025), whereas many others are described as relevant “from a CmOS perspective” without claiming that label as their primary formulation (Sikka et al., 2018, Dorent et al., 2024). This suggests that the term currently functions more as a unifying lens than as a settled taxonomy.

A second limitation is dependence on pairing, registration, or structured correspondence. MRI-to-PET synthesis is fully supervised on paired ADNI scans and depends on careful preprocessing and registration to template space (Sikka et al., 2018). MMHVAE handles incomplete multimodal datasets but still benefits from at least some paired supervision among modalities (Dorent et al., 2024). CrossModalityDiffusion assumes known camera poses and matched scene geometry across sensor modalities (Berian et al., 16 Jan 2025). Unpaired training is possible, as shown by the structure-guided MRI-to-ultrasound translator in endometrial carcinoma screening, but doing so requires additional constraints such as cycle consistency, modality-agnostic feature consistency, and domain-adversarial structure preservation (Shan et al., 23 Feb 2026).

A third misconception concerns fidelity. Several papers explicitly warn that good reconstruction metrics do not imply full semantic or biological equivalence. The MRI-to-PET paper states that nothing in its evaluation proves synthesized PET is a faithful surrogate biomarker; it demonstrates utility for classification, not biomarker equivalence (Sikka et al., 2018). The educational CmOS paper likewise reports that human evaluation scores for visual options remain fairly low in absolute terms, indicating that high automatic scores do not eliminate issues such as limited detail, hallucinated content, stylistic inconsistency, or weak distractibility (Wang et al., 26 Aug 2025).

A fourth open problem is explicit option formalization. The robotics attention paper identifies primitive structure and modality relevance but does not learn $A$4 or a high-level option selector (Jiang et al., 20 Apr 2025). CoMM supplies a representation-theoretic argument for preserving synergistic information, but it does not provide temporal abstractions, action conditioning, or planning over options (Dufumier et al., 2024). This suggests that future CmOS work would need to combine cross-modal synthesis with initiation models, termination models, controllable composition, and uncertainty-aware decision layers if it is to become a full option-synthesis framework rather than a collection of synthesis and fusion techniques.

A final practical limitation is reproducibility and specification. Several papers leave key details underspecified, including exact diffusion losses, checkpoint-selection criteria, or codebook-expansion policies (Kwak et al., 13 Jun 2025, Xia et al., 1 Apr 2025, He et al., 2024). A plausible implication is that CmOS, as it currently appears across arXiv, is methodologically rich but still unevenly standardized. Its most stable common principle is not a single architecture, but an evaluation doctrine: cross-modal synthesis should be judged by whether the synthesized option preserves the complementary structure needed for the downstream task that motivated its creation.