CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval

Abstract: Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG), improving code discovery, reuse, and the reliability of LLM-based coding. Yet existing code IR models remain largely text-centric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data visualizations, SVGs, schematic diagrams, and UML. To bridge this gap, we introduce MMCoIR, the first comprehensive benchmark for evaluating multimodal code IR across five visual domains, eight programming languages, eleven libraries, and show the challenge of the task through extensive evaluation. Therefore, we then propose CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images into a shared semantic space through instruction-based multimodal alignment. CodeMMR achieves strong generalization across modalities and languages, outperforming competitive baselines (e.g., UniIR, GME, VLM2Vec) by an average of 10 points on nDCG@10. Moreover, integrating CodeMMR into RAG enhances code generation fidelity and visual grounding on unseen code generation tasks, underscoring the potential of multimodal retrieval as a core enabler for next-generation intelligent programming systems. Datasets are available at HuggingFace.

• The paper introduces CodeMMR, a unified retriever model that embeds natural language, code, and image inputs into a shared semantic space using instruction-driven contrastive learning.
• The work presents MMCoIR, a comprehensive benchmark that standardizes retrieval tasks across diverse visual code domains with explicit natural language instructions.
• Experimental results show that CodeMMR outperforms prior methods by up to 10 absolute nDCG@10 points and significantly enhances retrieval-augmented code generation.

Unified Multimodal Retrieval for Code, Text, and Images: An Expert Analysis of "CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval" (2604.15663)

Introduction and Motivation

Information retrieval (IR) on source code is a critical component for intelligent programming systems, enabling not only more productive code search but also retrieval-augmented generation (RAG) to improve code synthesis reliability. Traditional code IR models are overwhelmingly text-centric, typically modeling queries and targets as natural language descriptions or raw code tokens. However, the practical landscape of software engineering is inherently multimodal: user interfaces, data visualizations, SVG graphics, and software diagrams are all instantiated via source code, but consumed visually. There is a pressing need for models and benchmarks that unify retrieval across natural language, code, and image modalities.

"CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval" introduces two core contributions to address this underexplored problem: (1) MMCoIR, a comprehensive multimodal, multilingual benchmark spanning five diverse visual code domains, and (2) CodeMMR, a unified retriever model embedding natural language, code, and image inputs into a shared semantic space using instruction-driven contrastive alignment.

Figure 1: CodeMMR enables unified embedding of text, image, and code, evaluated on MMCoIR across multiple cross-modal retrieval scenarios.

MMCoIR: A Comprehensive Multimodal Code Retrieval Benchmark

MMCoIR is presented as the first testbed that systematically covers the highly heterogeneous retrieval landscape encountered in real-world code search and generation contexts. The benchmark unifies datasets spanning:

• WebUI: Realistic and hand-drawn web page screenshots and sketches paired with HTML/CSS/JS.
• Data Visualization: Chart images paired with Python code for chart generation using multiple plotting libraries.
• SVG: Scalable vector graphics images mapped to their XML-based code.
• Schematic Diagrams: Programmatic visuals (LaTeX/TikZ, Graphviz DOT) for scientific or engineering diagrams.
• UML: PlantUML scripts for software engineering diagrams.

Each subdomain is multilingual (eight languages, eleven libraries), supporting retrieval in both directions—e.g., image-to-code, code-to-image, text-to-code, and compositional queries (e.g., text+image-to-code).

MMCoIR establishes a unified schema with explicit natural-language instructions accompanying every query, standardizing the retrieval intent and ensuring consistent evaluation across domains. This design also includes specialized settings for instruction-driven code editing and repair, testing generalization on entirely novel cross-modal retrieval tasks.

CodeMMR Model: Unified Retrieval via Multimodal Instruction-based Contrastive Learning

CodeMMR is a unified embedding model that projects text, code, and image inputs (as well as mixtures thereof) into a shared space through an instruction-conditioned contrastive learning objective. It is built on Qwen2-VL-2B-Instruct, extended to structured code representations, with the following distinctive properties:

• Instruction-based Alignment: Every retrieval query is conditioned on an explicit instruction specifying modality and retrieval goal, increasing task controllability and cross-domain compatibility.
• Joint Multimodal Embedding: Code, images, and natural language are processed with symmetric multimodal encoders, enabling direct similarity computation in the embedding space for retrieval.
• Contrastive Training Objective: Using the InfoNCE loss on batches of positive and negative cross-modal query-target pairs, with hard negative mining to improve discrimination.
• Input Adaptation: Model supports variable-length inputs up to at least 512 tokens, mitigating information loss for long-form structured modalities (e.g., SVG XML).

Experimental Results

Benchmarking on MMCoIR

CodeMMR demonstrates strong generalization and a substantial performance gap over prior work (UniIR, GME, VLM2Vec, LamRA):

• Average nDCG@10 improvement: CodeMMR (2B variant) outperforms best prior art by an average of ~10 absolute points on nDCG@10 across the full MMCoIR suite.
• Domain breakdown: Highest retrieval accuracy is seen in structured domains like UML and PlantUML (Hit@1 of 100%), with pronounced gains in data visualization and web UI domains as well.
• Hard domains: SVG-based retrieval (mapping code to images and vice versa) is uniquely challenging due to geometric/structural complexity and long code sequences, and remains the toughest case in all models.

Figure 2: Impact of input sequence length during training—longer contexts (up to 512 tokens) benefit structure-heavy domains, especially SVG and schematic diagrams.

Generalization to Unseen Tasks

On evaluation with out-of-distribution domains and tasks, including pure code editing and schematic diagram synthesis, CodeMMR maintains top performance, though some compositional and sketch-based inputs continue to present difficulties for all models. Fine-tuning on MMCoIR is critical for achieving these gains; conventional multimodal retriever pretraining offers negligible benefit when transferred directly, underscoring the value of the domain-specific supervision MMCoIR provides.

Retrieval-Augmented Generation (RAG)

Integrating CodeMMR into code RAG pipelines yields notable empirical benefits for image-to-code generation:

• On ChartMimic Direct: +10.0% Execution Rate, +7.6% High-Level Score over no-retrieval baseline, and 4–5 point improvements versus the best non-CodeMMR retriever.
• On WebCode2M-Mid: +9.4% Visual Accuracy, +10.8% CLIP Similarity, with consistent gains across model sizes and backbones (LLaVA-7B, 13B; Qwen2VL-7B).

Figure 3: CodeMMR-based RAG consistently improves both execution and structural scores on code generation benchmarks ChartMimic and WebCode2M.

Figure 4: Cross-VLM comparison of retrieval-augmented code generation, showing CodeMMR's contribution across MLLMs and datasets.

Low-level visual analysis reveals improvements in chart/text localization, layout, object type, and color fidelity on ChartMimic Figure 5, with manual inspection indicating error mitigation in text/labeling and layout, lessening hallucinations and improving structural faithfulness.

Figure 5: Evaluation on visual metrics—CodeMMR-augmented retrieval raises text, layout, type, and color scores for VLM-based code generation.

Qualitative Synthesis

Qualitative inspection on the WebCode2M-Mid dataset further demonstrates that CodeMMR-based RAG facilitates code generation with higher visual fidelity and layout consistency when compared to strong baselines and specialized VLMs.

Figure 6: LLaVA-7B output: baseline generative model captures only rudimentary page structure.

Figure 7: LLaVA-13B output: improvement in header and containers, but still omits style/visual details.

Figure 8: LLaVA-13B + CodeMMR: RAG induces visually and structurally accurate code consistent with the ground truth.

Figure 9: WebCoder: HTML structure and content mostly recovered, but with misaligned layout.

Figure 10: Ground Truth web page as rendered from reference code.

Ablations and Analysis

• Input Length Sensitivity: Increasing the maximum input tokens during training correlates with higher task performance, especially in code structures (SVG, TikZ, UML) that are highly verbose or compositional Figure 2.
• Pretraining Data Mixing: Incorporating large conventional image-text retrieval data into CodeMMR training (CodeMMR-Mix) does not improve, and at times degrades, model accuracy—even on outside domains—confirming the task/structure specificity required for realistic cross-modal code retrieval.
• Error Analysis: The hardest settings involve abstract or compositional tasks (e.g., text+code → image), and cases with significant domain shift (e.g., hand-drawn sketches in Sketch2Code) where all retrieval models plateau at low performance, motivating future improvements in compositional visual-language reasoning.

Implications and Future Prospects

This work establishes that unified multimodal embedding models—when built upon appropriate instruction-driven, cross-modal datasets—can achieve strong retrieval performance and significantly empower RAG-style code generation pipelines. The evidence presented for MMCoIR and CodeMMR directly supports future research in several key directions:

• Extension to Additional Modalities: The architecture and benchmark are extensible to video, audio, or 3D code-generation domains, providing the basis for further generalist retrievers.
• Reasoning-Intensive Retrieval: The pronounced challenges in compositional and high-abstraction settings spotlight the importance of incorporating explicit structured reasoning and long-context dependencies.
• Enhanced Cross-Modal Alignment: Continued improvements on visually complex, code-structured domains like SVG will require research that bridges the gap between semantic/geometric representation and raw language tokens.

Conclusion

The introduction of MMCoIR provides a rigorous, standardized testbed for multimodal, cross-lingual code retrieval, capturing the realities of software as a visual and structural artifact. CodeMMR, by unifying code, text, and image modalities in a shared, instruction-driven semantic space, sets a new SOTA in both retrieval and code RAG tasks, manifesting robust empirical gains across all metrics and domains. The evidence highlights the necessity for domain-specialized multimodal supervision and methodology, charting a clear research trajectory toward more general, semantically grounded, and context-sensitive program synthesis architectures.