MMORE: Unified Multimodal Retrieval

• MMORE is a unified multimodal system that transforms heterogeneous files—including PDFs, images, audio, and video—into a common JSON-based representation for seamless retrieval-augmented generation.
• It integrates ingestion, normalization, indexing, and extraction into a continuous, scalable pipeline using distributed processing frameworks like Dask for efficient execution.
• Leveraging mature open-source tools such as Surya, Whisper, and standard Python libraries, MMORE streamlines document intelligence for enterprise, research, and multimodal RAG applications.

MMORE, short for Massive Multimodal Open Retrieval-Augmented Generation and Extraction, is an open-source end-to-end pipeline for converting large collections of heterogeneous files into a unified, searchable knowledge base for retrieval-augmented generation and downstream extraction. It is designed for settings in which useful information is distributed across PDFs, scanned pages, slide decks, spreadsheets, emails, images, audio, video, HTML, markdown, and related formats, and where ingestion, normalization, indexing, and retrieval must be handled within a single modular system rather than a fragmented toolchain. The system supports more than 15 file types, exposes both interactive APIs and batch RAG endpoints, and emphasizes scalable parallelization across CPUs and GPUs through distributed execution (Sallinen et al., 15 Sep 2025).

1. Scope and motivating problem

MMORE addresses a practical bottleneck in multimodal AI systems: real-world corpora rarely arrive as clean text. Instead, knowledge is embedded in heterogeneous document and media formats, and existing workflows often require separate components for OCR, parsing, embedding, retrieval, and orchestration. MMORE is designed to combine these stages into a single open-source pipeline for document ingestion at scale, multimodal extraction, indexing for RAG, efficient distributed processing, and open and extensible deployment.

The system is explicitly intended for research, enterprise, and institutional settings in which heterogeneous files must be transformed into a common representation that can support downstream LLM applications. A central premise is that ingestion and retrieval should be treated as one continuous pipeline rather than as loosely coupled preprocessing and serving stages. This design makes MMORE a task-agnostic foundation for multimodal RAG over diverse, real-world data (Sallinen et al., 15 Sep 2025).

2. Layered architecture and canonical representation

MMORE is organized into two main layers. The processing / ingestion layer extracts content from files, normalizes the output into a common multimodal representation, and supports both single-node and distributed execution. The RAG layer post-processes extracted content, builds hybrid retrieval indices, and serves query-time retrieval and batch workflows. Each stage can be run independently, which makes the architecture modular rather than monolithic.

The unifying representation is a JSON-based format called MultimodalSample. A MultimodalSample contains a "text" field holding document text, textual placeholders where non-text content appears, and a "modalities" array storing the embedded objects with their modality type and file path or location. The appendix gives the following structure:

{ "text": "A report containing a cool image <attachment> and a chart <attachment>...", "modalities": [ {"type": "image", "value": "chart_url_2.png"}, {"type": "image", "value": "chart_url_1.png"} ] }

This representation preserves alignment between text and visual or media content. In MMORE’s formulation, that alignment is essential not only for multimodal RAG but also for later multimodal training or extraction tasks, because non-text assets remain explicitly linked to their textual context rather than being flattened into disconnected metadata (Sallinen et al., 15 Sep 2025).

3. Ingestion, extraction, and modality-specific transformation

MMORE supports a broad multimodal range. The formats named in the system description include PDF, DOCX, PPTX, XLSX / spreadsheets, TXT, HTML, Markdown, CSV, image formats such as PNG, JPEG, SVG, TIFF, BMP, as well as audio, video, and EML emails. The appendix’s evaluation set specifically lists PDF, DOCX, EML, MD, MP4, MP3, PPTX, TXT, and XLSX.

A notable design choice is that MMORE does not reimplement every extractor. Instead, it reuses mature open-source tools and focuses on orchestration and unification. For PDF parsing, OCR, and layout analysis it uses Surya; for audio transcription it uses Whisper; and for office formats and email it relies on standard Python libraries and tools such as moviepy, python-docx, python-pptx, openpyxl, and BeautifulSoup. MMORE also exposes two processing modes: default mode, which prioritizes accuracy, and fast mode, which prioritizes speed and may disable costly steps such as OCR.

Format class	Default / accurate mode	Fast mode
PDF	marker-pdf	PyMuPDF
Audio / Video	moviepy + whisper-large-v3-turbo	whisper-tiny
DOCX / PPTX / XLSX / HTML / MD / TXT / EML	Standard format-specific libraries	No fast alternative for some formats

The default extractor mapping includes python-docx for DOCX, markdown and markdownify for MD, python-pptx for PPTX, openpyxl for XLSX, Python open() for TXT, the Python email module for EML, and BeautifulSoup for HTML. This transformation strategy makes MMORE a coordination layer over heterogeneous parsers while still producing a standardized multimodal output format (Sallinen et al., 15 Sep 2025).

4. Distributed execution, post-processing, and retrieval

A major contribution of MMORE is its support for parallel and distributed execution. It supports intra-node parallelism, inter-node parallelism, multi-CPU and multi-GPU orchestration, and cluster deployment including Kubernetes. The execution layer is built on Dask, which provides automatic workload balancing, fault tolerance, and scalable scheduling across nodes. As a result, the same pipeline can run on a laptop or workstation, on a single GPU server, or on a multi-node cluster.

After extraction, MMORE performs post-processing, indexing, and retrieval. The post-processing stage leverages datatrove and includes support for Named Entity Recognition, Chunking, and Tagging. For retrieval, MMORE uses a hybrid dense-sparse index, combining sparse representations for lexical matching and interpretability with dense embeddings for semantic search. Query-time retrieval returns the top-$k$ most relevant documents by similarity and prepends them to the user question as context for the LLM.

The service model supports both API-based querying and batch mode over JSONL input/output files. Users can customize the model, prompt template, index source, and related settings through configuration files or API parameters. This arrangement makes the RAG layer simultaneously interactive and suitable for large offline workloads (Sallinen et al., 15 Sep 2025).

5. Empirical evaluation

MMORE is evaluated in two dimensions: processing and RAG. The processing evaluation benchmarks MMORE against Docling for speed, efficiency, and extraction quality. The setup includes a single A100 80GB GPU, synthetic document lengths from 36 to 720 pages, a 4-node Kubernetes cluster with one A100 per node, and a heterogeneous throughput benchmark over 19 files across 9 file types.

The reported scaling behavior is explicitly nonuniform across document lengths. On short documents, Docling can be slightly faster; around 90 pages, MMORE catches up; beyond 180 pages, MMORE becomes clearly faster and scales more nearly linearly, whereas Docling slows down super-linearly on longer documents. MMORE’s fast mode yields about 2–3× additional speedup, and running default MMORE on 4 nodes gives a 3.8× latency reduction relative to single-node MMORE. Over the 19-file heterogeneous benchmark, default MMORE reduces total processing time by 45.48% versus Docling, while fast mode reduces total processing time by 155.38% versus Docling.

Extraction quality is reported on two Project Gutenberg books using BLEU, ROUGE-L, and CER. For The Blue Castle, all systems perform well and the differences are minor: MMORE reports BLEU 0.8608, ROUGE-L 0.9940, CER 0.0241; MMORE fast reports BLEU 0.8639, ROUGE-L 0.9963, CER 0.0206; and Docling reports BLEU 0.8643, ROUGE-L 0.9959, CER 0.0199. For The Great Gatsby, which is scanned and OCR-heavy, MMORE default reports BLEU 0.7973, ROUGE-L 0.9813, CER 0.0295, Docling reports BLEU 0.5451, ROUGE-L 0.6582, CER 0.5518, and MMORE fast reports BLEU 0.0000, ROUGE-L 0.0000, CER 1.0000 because OCR is disabled. The system summary characterizes this as about 40% higher layout accuracy on scanned PDFs than Docling.

The RAG evaluation indexes PubMed abstracts and conclusions and measures downstream performance on PubMedQA using Meditron3-8B and Meditron3-70B. Accuracy improves with retrieval depth. For Meditron3-8B, the reported accuracies are 75.4% at $k=0$, 79.0% at $k=1$, and 80.0% at $k=3$. For Meditron3-70B, the corresponding values are 80.2%, 81.0%, and 82.0%. These results indicate that MMORE’s retrieval layer contributes measurable domain-specific gains when additional context is prepended at inference time (Sallinen et al., 15 Sep 2025).

6. Comparative position, practical uses, and limitations

MMORE is positioned against systems such as Docling, NV-Ingest, LangChain, LlamaIndex, Unstructured, and Haystack. Relative to Docling, it is described as offering broader modality coverage, built-in RAG, distributed multi-node execution, better performance on scanned/OCR-heavy documents, and more end-to-end integration. Relative to more general multimodal RAG stacks assembled from separate loaders, OCR tools, embedding tools, retrievers, and orchestration frameworks, MMORE is characterized as a single unified ingestion-to-retrieval system with explicit support for audiovisual assets as well as conventional documents. The appendix further distinguishes MMORE from NV-Ingest through support for audio, video, and emails, together with native indexing, embedding, and RAG functionality.

The practical implications named for MMORE include enterprise document intelligence, multimodal knowledge bases, scientific literature assistants, biomedical RAG systems, document QA over scanned archives, email/document/media search, batch processing pipelines for large corpora, and cluster-based ingestion in production. Because its supported inputs include office documents, scanned PDFs, emails, audio, and video, MMORE is particularly suited to environments where knowledge is distributed across heterogeneous sources rather than concentrated in text-only repositories.

The limitations are also explicit. Fast mode can be inaccurate for OCR-heavy documents, because it may omit OCR entirely. The published evaluation is limited in scope: extraction accuracy is shown on only two books, and RAG is tested mainly on PubMedQA. The future directions named for the system include multilingual retrieval, audiovisual alignment, and federated processing for privacy-sensitive settings. A plausible implication is that MMORE should be understood less as a closed benchmark winner than as an extensible systems substrate whose principal contribution lies in unifying extraction, normalization, indexing, and retrieval for large-scale multimodal corpora (Sallinen et al., 15 Sep 2025).