MIRA: Multimodal Intelligent Retrieval & Augmentation

• Multimodal Intelligent Retrieval and Augmentation (MIRA) is a framework that retrieves and fuses multimodal evidence from text, images, audio, and video to ground generation, reasoning, and predictions.
• MIRA architectures utilize shared embedding spaces, dual encoders, query rewriting, and modality-aware routing to optimize evidence retrieval and relevance.
• Recent studies highlight challenges such as retrieval noise, modality imbalance, and evaluation limitations, paving the way for advanced layered multimodal systems.

Searching arXiv for the core MIRA papers and adjacent multimodal retrieval-augmentation work to ground the article in current literature. Multimodal Intelligent Retrieval and Augmentation (MIRA) denotes a class of retrieval-augmented multimodal systems in which an agent retrieves, filters, and reuses evidence from multiple modalities to ground generation, reasoning, or prediction. In the medical RAG literature, MIRA is introduced as a framework for optimizing factual accuracy in multimodal LLMs by combining dynamic retrieval control with image-text fusion (Wang et al., 10 Jul 2025). In adaptive video understanding, the same term is used as a design paradigm in which an agent perceives, indexes, and selectively reuses information from video, audio, and text to ground generation, adapt to context, and reduce cognitive burden in dynamic environments (Mao et al., 29 May 2025). Across benchmark, retrieval, and deployment papers, the term is associated with shared embedding spaces, modality-aware routing, query rewriting, retrieval-augmented prompting, and evaluation regimes that test both answer quality and evidence use (Liu et al., 24 Feb 2025, Martin et al., 28 Oct 2025).

1. Conceptual scope and historical development

MIRA emerged from earlier retrieval-augmented multimodal systems that treated external multimodal corpora as non-parametric memory. A foundational instance is the decoupled DXR/XTRA design for visual question answering, in which a cross-modal retriever is trained once to align images and captions in a common space and a separate reader concatenates retrieved captions to the text stream, enabling hot-swapping of indices and domain adaptation without retraining the reader (Gur et al., 2021). This architecture already exhibits core MIRA properties: a retriever trained outside the task head, a searchable external corpus, and inference-time augmentation of a multimodal model.

Subsequent work generalized the pattern beyond question answering. In radiology, FactMM-RAG uses RadGraph-mined factual pairs to train a retriever that selects clinically faithful reports for report generation, explicitly targeting hallucination reduction and factual completeness (Sun et al., 2024). In multimodal web generation, M$^2$RAG formalizes a task in which a system retrieves mixed text-image web content and generates a multimodal response that interleaves relevant images with explanatory text (Ma et al., 2024). In open-domain multimodal RAG benchmarking, a separate M$^2$RAG benchmark evaluates how multimodal LLMs exploit retrieved text, images, and image-text pairs across image captioning, multimodal question answering, multimodal fact verification, and image reranking (Liu et al., 24 Feb 2025).

By 2025 and 2026, the term spans both named systems and a broader systems perspective. It is instantiated in adaptive video understanding, medical VQA and report generation, visually rich document retrieval, geospatial cyberinfrastructure, multimodal information retrieval, federated learning with missing modalities, and residue-level protein active site identification (Mao et al., 29 May 2025, Wang et al., 10 Jul 2025, Xu et al., 1 May 2025, Kang et al., 14 Jun 2026, Su et al., 25 Feb 2026, Poudel et al., 2024, Wu et al., 2 Mar 2026). This breadth suggests that MIRA is less a single algorithm than a recurring architectural program: retrieve multimodal evidence, convert it into model-usable context, and control how that context is trusted and fused.

2. Retrieval substrates, indexing strategies, and representation design

A central MIRA problem is how to encode heterogeneous artifacts into retrievable representations. Many systems adopt a bi-encoder or dual-encoder structure in which queries and candidates are mapped to vectors and scored by cosine or dot-product similarity. In the multimodal RAG benchmark, retrieval is defined by $h_q = Enc(q)$, $h_d = Enc(d)$, and $S(q,d)=\mathrm{cosine}(h_q,h_d)$, with top-$k$ search over a unified VISTA embedding space that supports text-text, image-image, text-image, and image-text matching (Liu et al., 24 Feb 2025). In Omni-Embed-Nemotron, queries and corpus items across text, image, audio, and video are independently encoded into a shared embedding space by a bi-encoder built on Qwen-Omni / Qwen2.5-Omni-3B, and the reported results show that separate-stream late fusion outperforms early fusion for audio-video retrieval on both LPM and FineVideo (Xu et al., 3 Oct 2025).

Other work emphasizes universal embeddings learned inside a multimodal LLM. Unite uses Qwen2-VL as backbone, extracts the final-token hidden state under an Explicit One-word Limitation prompt, and trains with bidirectional contrastive learning plus Modal-Aware Masked Contrastive Learning, which restricts negatives to the same target modality combination to reduce inter-modal competition (Kong et al., 26 May 2025). On MMEB, Unite 7B reports Overall $70.3$, IND $73.6$, and OOD $66.3$, while on WebVid-CoVR-Test it reports $R@1=72.5$ and $^2$0 (Kong et al., 26 May 2025).

A contrasting design externalizes reasoning before embedding. Reasoning-Augmented Representations uses a strong vision-LLM to densely caption corpus images, resolve ambiguous multimodal references in queries, and rewrite modification instructions into concise retrieval constraints; it reports that inference-only enhancement degrades performance because of distribution shift, whereas training on enhanced data improves average $^2$1 from $^2$2 to $^2$3 on M-BEIR (Zhang et al., 6 Feb 2026). RetLLM takes a different route: it keeps coarse retrieval lightweight with CLIP-ViT-L/14@336px, forms a top-$^2$4 candidate pool with default $^2$5, and then asks an MLLM to directly predict similarity scores in a fine stage, with entropy-based tie-breaking and a visual enhancement module applied inside selected FFN layers with injection ratio $^2$6 (Su et al., 25 Feb 2026).

MIRA systems also vary in index structure and routing. Multi-RAG converts video frames, audio transcripts, timestamps, and summaries into Markdown, embeds chunks with LangChain OpenAIEmbeddings, and stores them in Chroma, so that multimodal signals are unified into a single text-centric retrieval space (Mao et al., 29 May 2025). I-GUIDE Smart Search instead operates with multiple production-maintained retrieval substrates at once: OpenSearch BM25, OpenSearch vector search, OpenSearch geo_shape spatial search, and a Neo4j provenance graph, with retrieval-method routing combining hand-curated triggers and a few-shot LLM router (Kang et al., 14 Jun 2026). This multi-index view makes routing, not only encoding, part of the retrieval substrate.

3. Augmentation mechanisms and context construction

Retrieval alone does not define MIRA; the distinctive step is augmentation, namely the packaging of retrieved multimodal evidence into a form that changes downstream inference. In the multimodal RAG benchmark, retrieved documents are concatenated with the query and a task-specific instruction template according to

$^2$7

where $^2$8 and $^2$9 may be text tokens, visual tokens, or image-plus-caption tokens; fine-tuning uses LoRA on MiniCPM-V 2.6 and Qwen2-VL for $h_q = Enc(q)$0 epochs, batch size $h_q = Enc(q)$1, and a maximum of $h_q = Enc(q)$2 tokens (Liu et al., 24 Feb 2025). The associated MM-RAIT method improves average performance by $h_q = Enc(q)$3 for MiniCPM-V and $h_q = Enc(q)$4 for Qwen2-VL over vanilla RAG baselines (Liu et al., 24 Feb 2025).

In adaptive video understanding, augmentation is performed by transforming video and audio into timestamped textual context. Multi-RAG samples frames, transcribes audio with Whisper, adds temporal summaries and preliminary analyses, chunks the resulting Markdown with overlap, retrieves top-$h_q = Enc(q)$5 chunks by cosine similarity, and inserts them into one of two prompt templates: a creativity-forward prompt or a fact-grounded prompt that requires the prefix “Speculative —” or the answer “Unknown” when evidence is unclear (Mao et al., 29 May 2025). The best setting is top-$h_q = Enc(q)$6, which yields Overall $h_q = Enc(q)$7, Perception $h_q = Enc(q)$8, and Reasoning $h_q = Enc(q)$9 under Prompt 2 (Mao et al., 29 May 2025).

Medical MIRA augments multimodal reasoning with retrieval control. Its pipeline rewrites the query with Qwen-2.5-8B, retrieves from offline and online medical sources via Faiss-CUDA and DuckDuckGo, rearranges contexts with an inner-product reranker based on image and text embeddings, and fuses modalities with an attention-based gate

$h_d = Enc(d)$0

before a rethink-and-refine stage validates the answer against retrieved evidence (Wang et al., 10 Jul 2025). The system is explicitly designed to avoid two failure modes: insufficient retrieval and excessive retrieval, the latter being a source of factual degradation when noisy evidence overrides an initially correct answer (Wang et al., 10 Jul 2025).

Augmentation can also be generative rather than purely contextual. MIRAGE for medical education performs a two-stage retrieval over ROCO: it first retrieves top-$h_d = Enc(d)$1 captions and images for a prompt, uses Dolly-v2-3b to generate an enriched description, re-embeds that description for second-pass retrieval, and in parallel synthesizes an image with Prompt2MedImage (Benito et al., 6 May 2026). In visually rich document retrieval, augmentation includes OCR text, page or region images, generated captions, and layout metadata such as bounding boxes and reading order, which are then passed to a generator with explicit grounding instructions (Xu et al., 1 May 2025). These designs indicate that MIRA augmentation can mean retrieved context concatenation, retrieved-to-generated reformulation, or retrieved evidence used as a scaffold for subsequent synthesis.

4. Benchmarks, tasks, and evaluation regimes

Benchmarking in MIRA has moved from answer-only scoring toward task suites that explicitly test multimodal evidence use. The open-domain M$h_d = Enc(d)$2RAG benchmark contains four tasks—image captioning, multimodal question answering, multimodal fact verification, and image reranking—with corpora reaching hundreds of thousands of images and texts, and it evaluates captioning and QA with BLEU-4, ROUGE-L, and CIDEr, fact verification with ACC and F1, and image reranking with FID$h_d = Enc(d)$3 (Liu et al., 24 Feb 2025). Representative results show, for example, that Qwen2-VL under MM-RAIT reaches Image Captioning BLEU-4 $h_d = Enc(d)$4, ROUGE-L $h_d = Enc(d)$5, and CIDEr $h_d = Enc(d)$6, compared with vanilla top-$h_d = Enc(d)$7 BLEU-4 values around $h_d = Enc(d)$8–$h_d = Enc(d)$9 (Liu et al., 24 Feb 2025).

A different M$S(q,d)=\mathrm{cosine}(h_q,h_d)$0RAG benchmark studies multimodal output generation rather than multimodal input conditioning. It is built from $S(q,d)=\mathrm{cosine}(h_q,h_d)$1 long-form open-ended queries drawn from ELI5, spans $S(q,d)=\mathrm{cosine}(h_q,h_d)$2 topic categories, and evaluates outputs with four text-modal metrics—Fluency, Relevance, Context Precision, Faithfulness—and four multimodal metrics—Image Coherence, Image Helpfulness, Image Reference, Image Recall—combined as

$S(q,d)=\mathrm{cosine}(h_q,h_d)$3

Its central empirical finding is that multi-stage joint modeling substantially outperforms single-stage modeling: average overall score rises from $S(q,d)=\mathrm{cosine}(h_q,h_d)$4 to $S(q,d)=\mathrm{cosine}(h_q,h_d)$5 for LLMs and from $S(q,d)=\mathrm{cosine}(h_q,h_d)$6 to $S(q,d)=\mathrm{cosine}(h_q,h_d)$7 for MLLMs (Ma et al., 2024).

Evaluation itself has become a specialized MIRA problem. MiRAGE introduces a claim-centric framework for multimodal RAG evaluation with two metrics: InfoF1 for factuality and information coverage, and CiteF1 for citation support and completeness (Martin et al., 28 Oct 2025). Its definitions decompose predictions and references into subclaims and score support either against a source collection or against a reference proxy. Human-applied MiRAGE is reported to align strongly with extrinsic quality judgments, and the paper argues that text-centric metrics such as ACLE, ARGUE, and RAGAS are limited in multimodal, reasoning-intensive settings because they do not verify information directly against multimodal sources (Martin et al., 28 Oct 2025).

These benchmarking efforts make a methodological shift visible. Earlier multimodal retrieval work frequently treated retrieval quality or downstream accuracy as sufficient end points. Current MIRA evaluation adds tests for attribution, citation correctness, multimodal coverage, and the ability to remain grounded when evidence is distributed across text, images, audio, or video.

5. System instantiations across domains

MIRA has been instantiated in markedly different operational settings, often with domain-specific retrieval substrates and domain-specific notions of correctness. In adaptive video understanding, Multi-RAG benchmarks on MMBench-Video and reports that at $S(q,d)=\mathrm{cosine}(h_q,h_d)$8 fps it reaches Overall $S(q,d)=\mathrm{cosine}(h_q,h_d)$9, matching GPT-4o’s $k$0, while surpassing GPT-4o in Reasoning mean, $k$1 versus $k$2; under Prompt 2 it also reports Hallucination score $k$3 versus GPT-4o’s $k$4 (Mao et al., 29 May 2025). In medical RAG, MIRA achieves the best reported MIMIC-CXR scores among listed systems, including BLEU-1 $k$5, BLEU-4 $k$6, ROUGE-L $k$7, CIDEr $k$8, and SPICE $k$9, while on PMC-VQA it attains Conv $70.3$0 and Details $70.3$1 without PMC-VQA pretraining (Wang et al., 10 Jul 2025). In radiology report generation, FactMM-RAG improves MIMIC-CXR retrieval-augmented generation to F1CheXbert $70.3$2 and F1RadGraph $70.3$3, compared with Med-MARVEL’s $70.3$4 and $70.3$5 (Sun et al., 2024).

Visually rich documents and geospatial knowledge introduce different retrieval problems. The multi-granularity framework for MMDocIR and M2KR combines ColQwen2-7B at page level, GME-7B over OCR and region images, layout-aware search, Reciprocal Rank Fusion, and Qwen2.5-VL reranking, reaching a top aggregate score of $70.3$6 (Xu et al., 1 May 2025). I-GUIDE Smart Search integrates keyword, vector, spatial, and graph retrieval over $70.3$7 public knowledge elements, $70.3$8 nodes, and $70.3$9 graph relationships; on a single A100 deployment it supports interactive use up to about $73.6$0 concurrent simulated users, reaches $73.6$1 requests per second, and shows p50 latency near $73.6$2 seconds despite $73.6$3–$73.6$4 LLM calls per query (Kang et al., 14 Jun 2026).

The same architectural logic has been specialized to protein science, multimodal retrieval, and federated healthcare learning. MERA performs chain-, sequence-, and active-site-level retrieval with residue-level mixture-of-experts gating and reliability-aware fusion via Dempster–Shafer evidence theory, reporting about $73.6$5 AUPRC on ProTAD-Gen and strong gains on TS125 peptide-binding site identification (Wu et al., 2 Mar 2026). RetLLM reformulates multimodal information retrieval as direct similarity score generation and reports Recall@1 values of $73.6$6 and $73.6$7 on Flickr30K, $73.6$8 and $73.6$9 on COCO, and Overall Precision@1 $66.3$0 on MMEB, outperforming the strongest zero-shot baseline UniME by $66.3$1 (Su et al., 25 Feb 2026). CAR-MFL treats retrieval as cross-modal augmentation under federated missing-modality constraints and reports macro AUC gains across multiple client compositions, such as $66.3$2 versus $66.3$3 for mFedAvgP in the $66.3$4 homogeneous split and $66.3$5 versus $66.3$6 in the heterogeneous $66.3$7 setting (Poudel et al., 2024).

Domain	System	Reported outcome
Adaptive video understanding	Multi-RAG	Overall $66.3$8 at $66.3$9 fps; Reasoning mean $R@1=72.5$0 (Mao et al., 29 May 2025)
Medical multimodal RAG	MIRA	MIMIC-CXR BLEU-4 $R@1=72.5$1, ROUGE-L $R@1=72.5$2, CIDEr $R@1=72.5$3 (Wang et al., 10 Jul 2025)
Radiology report generation	FactMM-RAG	F1CheXbert $R@1=72.5$4, F1RadGraph $R@1=72.5$5 on MIMIC-CXR (Sun et al., 2024)
Geospatial knowledge discovery	I-GUIDE Smart Search	$R@1=72.5$6 requests/sec; p50 latency near $R@1=72.5$7 seconds (Kang et al., 14 Jun 2026)
Visually rich documents	Multi-granularity retrieval	Top score $R@1=72.5$8 on MMDocIR/M2KR configuration (Xu et al., 1 May 2025)
Protein active site identification	MERA	About $R@1=72.5$9 AUPRC on ProTAD-Gen (Wu et al., 2 Mar 2026)

These examples show that MIRA is not tied to any single modality mix or single downstream task. It has been realized in settings where the retrieved object is a caption, a web page element, a video chunk, a medical report, a geospatial artifact, or a protein-context representation.

6. Limitations, open problems, and research directions

The current literature repeatedly identifies retrieval quality as a hard bottleneck. In the multimodal RAG benchmark, performance is bounded by retrieval accuracy, models still rely heavily on textual data in some tasks, and noisy retrieval harms generation; vanilla RAG often improves from top-$^2$00 to top-$^2$01 but can degrade at top-$^2$02, indicating sensitivity to noisy or less-relevant contexts (Liu et al., 24 Feb 2025). In adaptive video understanding, hallucination risk remains in open-ended reasoning, robustness depends strongly on transcript quality, and very long videos challenge context continuity because no specialized memory module is used (Mao et al., 29 May 2025). In medical MIRA, online retrieval can introduce irrelevant content, such as the reported encephalomalacia snippet in a chest CT VQA case study, making stronger filters and curated medical knowledge bases a continuing need (Wang et al., 10 Jul 2025).

Another limitation concerns modality balance and reliability. Omni-Embed-Nemotron reports that lecture-style content favors text from transcripts and OCR, audio carries more signal than video frames in open-domain videos, and video-only inputs can reach sequence lengths of $^2$03 tokens in the provided example, making processor-argument tuning critical (Xu et al., 3 Oct 2025). Unite shows that mixing modality tasks in instruction tuning can degrade performance on specific domains unless negatives are restricted by modality type through MAMCL (Kong et al., 26 May 2025). MERA makes the same point from a different angle: existing multimodal fusion methods can let unreliable modalities dominate, so it introduces reliability-aware fusion with belief mass functions, discounting coefficients, and Dempster’s rule (Wu et al., 2 Mar 2026).

Evaluation remains incomplete. MiRAGE argues that text-centric metrics are insufficient for multimodal, reasoning-intensive settings because they do not verify claims directly against sources, while automatic multimodal verification still depends on the calibration of LLM and VLM judges (Martin et al., 28 Oct 2025). The geospatial deployment study adds that retrieval-needed queries demand not just stronger models but better metadata, richer graph provenance, reliable routing, explicit refusal policies, and interface contracts between components (Kang et al., 14 Jun 2026). This suggests that future MIRA progress will depend not only on larger encoders, but also on adaptive retrieval, better cross-modal alignment, learnable gating, long-context handling, stronger citation-aware evaluation, and domain-specific data engineering.

A plausible implication is that MIRA is converging toward a layered systems discipline. At one layer lie universal or modality-aware retrieval models; at another, query rewriting, routing, and reranking; at another, context fusion and augmentation; and at a final layer, claim-centric evaluation of factuality and citation support. The recent literature indicates that improvements at any one layer can be neutralized by weaknesses at another, which is why MIRA increasingly appears not as a standalone model family but as an end-to-end multimodal retrieval and grounding stack.