Cross-Modal Generative Injection (CMGI)

• CMGI is a design pattern where generative processes dynamically shape one modality’s representation to improve cross-modal alignment and control.
• Key technical strategies include training-time generative injection, latent refinement via diffusion, and unified frameworks for both generation and discrimination.
• CMGI is applied in multimodal retrieval, long-context image generation, and even exposes security vulnerabilities through adversarial cross-modal prompt injection.

Searching arXiv for the supplied papers and adjacent CMGI-related work. Cross-Modal Generative Injection (CMGI) is an Editor’s term for a family of multimodal methods in which a model does not merely align or compare modalities, but uses a generative process in one modality to shape representation learning, conditioning, control, or behavior in another. Across the literature, this pattern appears in at least three technically distinct forms: generative enhancement of cross-modal retrieval, LLM-to-diffusion conditioning for long-context multimodal generation, and unified diffusion-based training of generation and discrimination; it also appears in a security setting, where generative models are used to embed adversarial instructions across modalities (Gu et al., 2017). A common denominator is that cross-modal information is injected through trainable generative pathways, latent refinement, or diffusion dynamics rather than through static feature concatenation alone. This suggests that CMGI is best understood as a design pattern rather than a single architecture.

1. Definition and scope

CMGI denotes architectures in which generative mechanisms are used to inject cross-modal structure into another modality’s representation or output space. In the retrieval setting, the generative component is used during training so that embeddings capture not only “global abstract features” but also “local grounded features,” rather than operating as purely discriminative joint embeddings (Gu et al., 2017). In multimodal generation, the injection pathway can operate through an LLM embedding space that is subsequently mapped into the conditioning interface of a diffusion model, while additional structural controls such as object layouts are injected into the image generator (Kumar et al., 2024). In unified diffusion training, the same diffusion backbone can be used for image synthesis and for image-text discrimination by formulating discrimination as a diffusion process over text embeddings conditioned on images (Huang et al., 2023).

The term also has a security interpretation. In multimodal agents, “cross-modal prompt injection attacks” exploit the fact that malicious content can be embedded across images, text, and external documents so that the agent’s behavior is redirected toward unauthorized tasks (Wang et al., 19 Apr 2025). In that setting, generation is not used to improve alignment or synthesis, but to create latent cross-modal cues that hijack downstream reasoning.

A plausible implication is that CMGI spans both constructive and adversarial uses of the same underlying principle: one modality can be made to carry generative information that reconfigures another modality’s semantics, conditioning path, or action policy.

2. Historical emergence from retrieval and joint embedding models

An early and conceptually important instance appears in “Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models,” which explicitly argues against image-text retrieval systems that embed image-text pairs as single feature vectors in a common representational space and instead incorporates “generative processes into the cross-modal feature embedding” (Gu et al., 2017). The paper states that this enables learning not only global abstract features but also local grounded features, and reports state-of-the-art cross-modal retrieval results on MSCOCO.

Within this line of work, the crucial shift is from a purely discriminative retrieval objective to a hybrid regime in which the learned representation must also support generation or reconstruction across modalities. The paper’s title itself captures the pattern: “Look” for encoding, “Imagine” for cross-modal generation, and “Match” for retrieval. Even without access to the exact PDF contents in the supplied material, the stated contrast is clear: retrieval quality is improved by requiring representations to support cross-modal generative structure rather than only pairwise similarity.

This historical step matters because it reframes cross-modal representation learning. Instead of asking only whether an image and a sentence are close in a shared space, the model is pressured to encode semantically grounded content that could plausibly generate the counterpart modality. This suggests a general CMGI principle: cross-modal generation can regularize or enrich retrieval embeddings without necessarily being used at inference time.

3. Core technical patterns

Across the cited works, CMGI appears through several recurring mechanisms.

Shared representations with generative supervision: In cross-modal retrieval, generative processes are attached to the same feature embeddings used for matching, so the embedding space is optimized simultaneously for retrieval and for cross-modal generation or reconstruction (Gu et al., 2017). The supplied material characterizes this as training-time generative injection that can be discarded at test time.

Cross-modal refinement in an intermediate latent space: MGCC introduces a “Cross-Modal Refinement module” that explicitly learns cross-modal dependencies between text and image in the LLM embedding space, refining special image output tokens before they are mapped to the diffusion model’s text-conditioning interface (Kumar et al., 2024). This is a direct injection mechanism: multimodal context is first aggregated in the LLM, then distilled into refined tokens, then translated into diffusion-compatible conditioning.

Structural injection alongside semantic injection: MGCC supplements semantic refinement with a “contextual object grounding module” that produces object bounding boxes for scenes with multiple objects (Kumar et al., 2024). The diffusion model is therefore conditioned not only by a CLIP-like embedding sequence derived from refined multimodal context, but also by layout information. The supplied material identifies these two pathways as complementary forms of injection.

Diffusion over embeddings for discriminative alignment: DiffDis unifies generation and discrimination by casting image-text discrimination as “a generative diffusion process of the text embedding from the text encoder conditioned on the image” (Huang et al., 2023). The denoised text embedding is then used in a contrastive image-text alignment loss. Here CMGI takes a strong form: the discriminative function is realized through a generative denoising trajectory in another modality’s embedding space.

Cross-guided diffusion with adaptive parameters: In MedM2G, cross-modal injection occurs within a multi-flow latent diffusion framework. The target modality’s denoising trajectory is guided by the source modality’s latent state at the same diffusion step, together with trainable adaptive parameters and cross-attention conditioning (Zhan et al., 2024). The framework further uses central alignment in a unified space and preserves a “medical visual invariant” for each imaging modality.

These patterns differ architecturally, but all instantiate the same operational motif: one modality is not merely encoded as context; it is transformed, refined, denoised, or structurally converted so that it becomes an active generative controller of another modality.

4. Representative systems

The following systems exemplify distinct CMGI regimes.

System	CMGI role	Main setting
“Look, Imagine and Match” (Gu et al., 2017)	Generative processes in cross-modal feature embedding	Textual-visual retrieval
MGCC (Kumar et al., 2024)	LLM-space refinement plus layout grounding into diffusion	Multimodal prompt-to-image generation
DiffDis (Huang et al., 2023)	Diffusion-based unification of generation and discrimination	Image synthesis and image-text alignment
MedM2G (Zhan et al., 2024)	Cross-guided multi-flow diffusion with visual invariant	Medical text-image and image-image generation
CrossInject (Wang et al., 19 Apr 2025)	Adversarial cross-modal generative prompt injection	Multimodal agents and security

In MGCC, the model comprises a frozen OPT-6.7B LLM, CLIP ViT-L image features mapped into the LLM token space, special image tokens, a Cross-Modal Refinement module, a Transformer mapper, GLIGEN as a frozen diffusion backbone, and a Contextual Object Grounding Module (Kumar et al., 2024). The reported effect is improved generation from complex multimodal prompt sequences, including long stories and dialogues.

In DiffDis, the image branch and discriminative branch share the diffusion U-Net, and the model “outperforms single-task models on both the image generation and the image-text discriminative tasks,” including a 1.65% improvement on average zero-shot classification accuracy over 12 datasets and a 2.42 improvement on FID in zero-shot image synthesis (Huang et al., 2023). The paper therefore shows that CMGI can be used not only for richer conditioning but also for joint pretraining objectives.

In MedM2G, CMGI is specialized for clinical settings. The model is described as the first medical generative model that unifies text-to-image, image-to-text, and unified generation of medical modalities including CT, MRI, and X-ray, performing 5 medical generation tasks across 10 datasets (Zhan et al., 2024). The supplied material emphasizes that cross-modal injection is mediated by central alignment, visual invariant preservation, and adaptive cross-guided parameters inside the diffusion process.

5. Applications and empirical effects

The application profile of CMGI is broad.

In retrieval, the benefit is improved matching between images and sentences with complex content, attributed to the use of generative processes in the feature embedding (Gu et al., 2017). The key empirical claim available in the supplied material is state-of-the-art cross-modal retrieval on MSCOCO.

In multimodal image generation, MGCC targets prompt sequences rather than single prompts. On VIST with multimodal inputs, it reports a CLIP Similarity score of $0.652$ compared to GILL’s $0.641$, and on VisDial it reports a CLIP score of $0.660$ compared to an existing state-of-the-art score of $0.645$ (Kumar et al., 2024). The ablations in the supplied material further report that adding CMRM improves FID from 15.31 to 14.67 and CLIP similarity from 0.641 to 0.646, while the full model with CMRM and COGM reaches FID 14.23 and CLIP 0.652. These results indicate that semantic refinement and structural grounding contribute complementary gains.

In unified generative-discriminative pretraining, DiffDis reports better generation ability and cross-modal semantic alignment in one architecture (Huang et al., 2023). The supplied material states that zero-shot image synthesis on MSCOCO improves from FID 10.8 for Stable Diffusion to 9.8 for DiffDis, while average zero-shot classification accuracy over 12 datasets improves from 22.7% for CLIP ViT-L/14 to 27.4%, and mean retrieval R@1 over Flickr30K and MSCOCO improves from 19.1 to 33.6.

In medicine, MedM2G is reported to outperform various state-of-the-art methods across report generation, text-to-image generation, MRI synthesis, MRI–CT translation, and unified multimodal generation (Zhan et al., 2024). The supplied material includes examples such as MIMIC-CXR FID 1.7, BraTS MRI synthesis PSNR 29.89 versus 28.26 for CoLa-Diff, and T2-to-CT PSNR 27.45 versus 26.86 for SynDiff.

A plausible implication is that CMGI is especially effective when the target task depends on long-context semantics, multi-object structure, or missing-modality completion, since static single-vector alignment is weakest in precisely those regimes.

6. Adversarial and security interpretation

The most explicit adversarial formulation appears in “Manipulating Multimodal Agents via Cross-Modal Prompt Injection,” which identifies a “critical yet previously overlooked security vulnerability in multimodal agents” and proposes CrossInject (Wang et al., 19 Apr 2025). Here CMGI is no longer a training strategy for better representations, but an attack surface.

CrossInject has two components. “Visual Latent Alignment” uses a text-to-image generative model to create a target image corresponding to malicious instructions, then optimizes adversarial features so that a benign image aligns with that malicious target in visual embedding space. “Textual Guidance Enhancement” uses a LLM to infer the defensive system prompt through adversarial meta prompting and generate a malicious textual command that improves compliance with the attacker’s request (Wang et al., 19 Apr 2025). The paper reports at least a +26.4% increase in attack success rates across diverse tasks and validates the attack in real-world multimodal autonomous agents.

This setting clarifies a common misconception. CMGI is not inherently beneficial or benign. The same principle that allows one modality to enrich another through generative conditioning also allows malicious semantics to be encoded in one modality and operationalized in another. In the supplied material, the attack affects language outputs, tool use, and even physical-world behavior in an autonomous driving setting.

Another misconception is that cross-modal injection must involve visible prompts or explicit text rendered into images. CrossInject instead relies on latent alignment under an imperceptibility constraint, which means the injected semantics need not be human-readable in the attacked modality (Wang et al., 19 Apr 2025). This suggests that robust defenses must reason over multimodal latent consistency rather than only surface-form filtering.

7. Conceptual synthesis and open questions

Taken together, the cited works support a general encyclopedia-level definition of CMGI as the use of generative processes to make one modality actively shape another modality’s latent geometry, conditioning pathway, or decision behavior. In “Look, Imagine and Match,” the injected signal improves retrieval embeddings (Gu et al., 2017). In MGCC, it enables long-context multimodal generation through LLM-space refinement and grounding (Kumar et al., 2024). In DiffDis, it turns discrimination itself into a generative denoising problem in text-embedding space (Huang et al., 2023). In MedM2G, it becomes a multi-flow clinical generation framework with modality-preserving invariants (Zhan et al., 2024). In CrossInject, it becomes a security vulnerability in multimodal agents (Wang et al., 19 Apr 2025).

Several research questions remain open within the supplied material. One concerns the balance between shared alignment and modality-specific preservation. MedM2G argues that naïve unified alignment can mix clinical properties and therefore introduces visual invariants to preserve modality-specific structure (Zhan et al., 2024). Another concerns inference cost: DiffDis gains discriminative power through diffusion-based denoising, but this is more expensive than encoder-only retrieval or classification (Huang et al., 2023). A further question concerns training-time versus inference-time injection. Retrieval-oriented CMGI may use generative modules only during training, whereas MGCC and CrossInject both rely on active inference-time injection pathways (Gu et al., 2017).

A plausible implication is that future CMGI systems will be judged not only by whether they improve alignment or generation, but by how explicitly they control the interface between modalities: whether through refined latent tokens, diffusion trajectories, cross-attention over time-dependent states, or structural controls such as layouts. The same evidence also implies that safety analysis must become part of CMGI research, because cross-modal generative control and cross-modal prompt injection are technically adjacent phenomena rather than unrelated domains.