---
title: 'MMORE: Unified Multimodal Retrieval'
url: https://www.emergentmind.com/topics/mmore
type: topic
---

# MMORE: Unified Multimodal Retrieval

MMORE, short for **Massive Multimodal Open Retrieval-Augmented Generation and Extraction**, is an open-source end-to-end pipeline for converting large collections of heterogeneous files into a unified, searchable knowledge base for retrieval-augmented generation and downstream extraction. It is designed for settings in which useful information is distributed across PDFs, scanned pages, slide decks, spreadsheets, emails, images, audio, video, HTML, markdown, and related formats, and where ingestion, normalization, indexing, and retrieval must be handled within a single modular system rather than a fragmented toolchain. The system supports more than 15 file types, exposes both interactive APIs and batch RAG endpoints, and emphasizes scalable parallelization across CPUs and GPUs through distributed execution [2509.11937].

## 1. Scope and motivating problem

MMORE addresses a practical bottleneck in multimodal AI systems: real-world corpora rarely arrive as clean text. Instead, knowledge is embedded in heterogeneous document and media formats, and existing workflows often require separate components for OCR, parsing, embedding, retrieval, and orchestration. MMORE is designed to combine these stages into a single open-source pipeline for **document ingestion at scale**, **multimodal extraction**, **indexing for RAG**, **efficient distributed processing**, and **open and extensible deployment**.

The system is explicitly intended for research, enterprise, and institutional settings in which heterogeneous files must be transformed into a common representation that can support downstream LLM applications. A central premise is that ingestion and retrieval should be treated as one continuous pipeline rather than as loosely coupled preprocessing and serving stages. This design makes MMORE a task-agnostic foundation for multimodal RAG over diverse, real-world data [2509.11937].

## 2. Layered architecture and canonical representation

MMORE is organized into two main layers. The **processing / ingestion layer** extracts content from files, normalizes the output into a common multimodal representation, and supports both single-node and distributed execution. The **RAG layer** post-processes extracted content, builds hybrid retrieval indices, and serves query-time retrieval and batch workflows. Each stage can be run independently, which makes the architecture modular rather than monolithic.

The unifying representation is a JSON-based format called **MultimodalSample**. A MultimodalSample contains a `"text"` field holding document text, textual placeholders where non-text content appears, and a `"modalities"` array storing the embedded objects with their modality type and file path or location. The appendix gives the following structure:

```json
{
  "text": "A report containing a cool image <attachment> and a chart <attachment>...",
  "modalities": [
    {"type": "image", "value": "chart_url_2.png"},
    {"type": "image", "value": "chart_url_1.png"}
  ]
}
```

This representation preserves alignment between text and visual or media content. In MMORE’s formulation, that alignment is essential not only for multimodal RAG but also for later multimodal training or extraction tasks, because non-text assets remain explicitly linked to their textual context rather than being flattened into disconnected metadata [2509.11937].

## 3. Ingestion, extraction, and modality-specific transformation

MMORE supports a broad multimodal range. The formats named in the system description include **PDF**, **DOCX**, **PPTX**, **XLSX / spreadsheets**, **TXT**, **HTML**, **Markdown**, **CSV**, image formats such as **PNG, JPEG, SVG, TIFF, BMP**, as well as **audio**, **video**, and **EML** emails. The appendix’s evaluation set specifically lists **PDF, DOCX, EML, MD, MP4, MP3, PPTX, TXT,** and **XLSX**.

A notable design choice is that MMORE does not reimplement every extractor. Instead, it reuses mature open-source tools and focuses on orchestration and unification. For PDF parsing, OCR, and layout analysis it uses **Surya**; for audio transcription it uses **Whisper**; and for office formats and email it relies on standard Python libraries and tools such as `moviepy`, `python-docx`, `python-pptx`, `openpyxl`, and `BeautifulSoup`. MMORE also exposes two processing modes: **default mode**, which prioritizes accuracy, and **fast mode**, which prioritizes speed and may disable costly steps such as OCR.

| Format class | Default / accurate mode | Fast mode |
|---|---|---|
| PDF | `marker-pdf` | `PyMuPDF` |
| Audio / Video | `moviepy` + `whisper-large-v3-turbo` | `whisper-tiny` |
| DOCX / PPTX / XLSX / HTML / MD / TXT / EML | Standard format-specific libraries | No fast alternative for some formats |

The default extractor mapping includes `python-docx` for DOCX, `markdown` and `markdownify` for MD, `python-pptx` for PPTX, `openpyxl` for XLSX, Python `open()` for TXT, the Python email module for EML, and `BeautifulSoup` for HTML. This transformation strategy makes MMORE a coordination layer over heterogeneous parsers while still producing a standardized multimodal output format [2509.11937].

## 4. Distributed execution, post-processing, and retrieval

A major contribution of MMORE is its support for **parallel and distributed execution**. It supports **intra-node parallelism**, **inter-node parallelism**, multi-CPU and multi-GPU orchestration, and cluster deployment including **Kubernetes**. The execution layer is built on **Dask**, which provides automatic workload balancing, fault tolerance, and scalable scheduling across nodes. As a result, the same pipeline can run on a laptop or workstation, on a single GPU server, or on a multi-node cluster.

After extraction, MMORE performs post-processing, indexing, and retrieval. The post-processing stage leverages **datatrove** and includes support for **Named Entity Recognition**, **Chunking**, and **Tagging**. For retrieval, MMORE uses a **hybrid dense-sparse index**, combining sparse representations for lexical matching and interpretability with dense embeddings for semantic search. Query-time retrieval returns the **top-\(k\)** most relevant documents by similarity and prepends them to the user question as context for the LLM.

The service model supports both **API-based querying** and **batch mode** over JSONL input/output files. Users can customize the model, prompt template, index source, and related settings through configuration files or API parameters. This arrangement makes the RAG layer simultaneously interactive and suitable for large offline workloads [2509.11937].

## 5. Empirical evaluation

MMORE is evaluated in two dimensions: **processing** and **RAG**. The processing evaluation benchmarks MMORE against **Docling** for speed, efficiency, and extraction quality. The setup includes a single **A100 80GB GPU**, synthetic document lengths from **36 to 720 pages**, a **4-node Kubernetes cluster** with one A100 per node, and a heterogeneous throughput benchmark over **19 files across 9 file types**.

The reported scaling behavior is explicitly nonuniform across document lengths. On short documents, Docling can be slightly faster; around **90 pages**, MMORE catches up; beyond **180 pages**, MMORE becomes clearly faster and scales more nearly linearly, whereas Docling slows down super-linearly on longer documents. MMORE’s **fast mode** yields about **2–3× additional speedup**, and running default MMORE on **4 nodes** gives a **3.8× latency reduction** relative to single-node MMORE. Over the 19-file heterogeneous benchmark, default MMORE reduces total processing time by **45.48%** versus Docling, while fast mode reduces total processing time by **155.38%** versus Docling.

Extraction quality is reported on two Project Gutenberg books using **BLEU**, **ROUGE-L**, and **CER**. For **The Blue Castle**, all systems perform well and the differences are minor: MMORE reports **BLEU 0.8608, ROUGE-L 0.9940, CER 0.0241**; MMORE fast reports **BLEU 0.8639, ROUGE-L 0.9963, CER 0.0206**; and Docling reports **BLEU 0.8643, ROUGE-L 0.9959, CER 0.0199**. For **The Great Gatsby**, which is scanned and OCR-heavy, MMORE default reports **BLEU 0.7973, ROUGE-L 0.9813, CER 0.0295**, Docling reports **BLEU 0.5451, ROUGE-L 0.6582, CER 0.5518**, and MMORE fast reports **BLEU 0.0000, ROUGE-L 0.0000, CER 1.0000** because OCR is disabled. The system summary characterizes this as about **40% higher layout accuracy on scanned PDFs** than Docling.

The RAG evaluation indexes PubMed abstracts and conclusions and measures downstream performance on **PubMedQA** using **Meditron3-8B** and **Meditron3-70B**. Accuracy improves with retrieval depth. For **Meditron3-8B**, the reported accuracies are **75.4%** at \(k=0\), **79.0%** at \(k=1\), and **80.0%** at \(k=3\). For **Meditron3-70B**, the corresponding values are **80.2%**, **81.0%**, and **82.0%**. These results indicate that MMORE’s retrieval layer contributes measurable domain-specific gains when additional context is prepended at inference time [2509.11937].

## 6. Comparative position, practical uses, and limitations

MMORE is positioned against systems such as **Docling**, **NV-Ingest**, **LangChain**, **LlamaIndex**, **Unstructured**, and **Haystack**. Relative to Docling, it is described as offering **broader modality coverage**, **built-in RAG**, **distributed multi-node execution**, **better performance on scanned/OCR-heavy documents**, and **more end-to-end integration**. Relative to more general multimodal RAG stacks assembled from separate loaders, OCR tools, embedding tools, retrievers, and orchestration frameworks, MMORE is characterized as a **single unified ingestion-to-retrieval system** with explicit support for audiovisual assets as well as conventional documents. The appendix further distinguishes MMORE from NV-Ingest through support for audio, video, and emails, together with native indexing, embedding, and RAG functionality.

The practical implications named for MMORE include **enterprise document intelligence**, **multimodal knowledge bases**, **scientific literature assistants**, **biomedical RAG systems**, **document QA over scanned archives**, **email/document/media search**, **batch processing pipelines for large corpora**, and **cluster-based ingestion in production**. Because its supported inputs include office documents, scanned PDFs, emails, audio, and video, MMORE is particularly suited to environments where knowledge is distributed across heterogeneous sources rather than concentrated in text-only repositories.

The limitations are also explicit. **Fast mode can be inaccurate for OCR-heavy documents**, because it may omit OCR entirely. The published evaluation is **limited in scope**: extraction accuracy is shown on only two books, and RAG is tested mainly on PubMedQA. The future directions named for the system include **multilingual retrieval**, **audiovisual alignment**, and **federated processing for privacy-sensitive settings**. A plausible implication is that MMORE should be understood less as a closed benchmark winner than as an extensible systems substrate whose principal contribution lies in unifying extraction, normalization, indexing, and retrieval for large-scale multimodal corpora [2509.11937].

Source: https://www.emergentmind.com/topics/mmore