NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

Abstract: Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal LLMs. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.

• The paper presents NeoMME, a model which processes raw image patches and multilingual text using a single bidirectional Transformer, designed for efficient fine-tuning and inference, especially in visual document retrieval tasks.
• NeoMME-260m, with 260M parameters, reaches an impressive 0.523 nDCG@10 on ViDoRe V3.
• NeoMME outperforms competitors with lower parameters and faster throughput for encoding $2,048 imes 2,048$ pages, demonstrating efficient scaling and retrieval with smart compression techniques and without sacrificing retrieval quality

Research objective and central contribution

NeoMME presents a multimodal-native alternative to the dominant strategy of adapting generative VLMs or combining independently pretrained vision and text encoders for representation learning. The paper’s central claim is architectural: a compact, bidirectional Transformer can process multilingual text and raw image patches through a single shared computational pathway, while supporting both dense and late-interaction retrieval. The resulting models, NeoMME-260M and NeoMME-800M, contain 262.9M and 793.7M parameters, respectively, and support a 16,384-token context.

The work targets visual document retrieval in particular, where systems such as ColPali retain patch-level representations but inherit much of the computational and parameter overhead of a VLM. NeoMME instead trains its multimodal encoder from random initialization. Text tokens and $32 \times 32$ RGB patches enter through modality-specific input projections, after which all representations are processed by the same bidirectional Transformer. The paper evaluates this design through pretraining diagnostics, visual and text retrieval, transfer learning, indexing throughput, representation compression, and query latency (2609.01657).

The empirical thesis is deliberately multi-dimensional. NeoMME does not consistently dominate larger or specialized models on every benchmark. Rather, it combines competitive visual retrieval quality with substantially lower parameter count, high-resolution page encoding, and a deployment-oriented representation architecture. Its strongest result is obtained by the late-interaction head: NeoMME-260M reaches 0.523 nDCG@10 on ViDoRe v3, and NeoMME-800M reaches 0.556. The smaller model is reported as the strongest evaluated system below 300M parameters and remains competitive with models several times larger.

Multimodal-native encoder design

NeoMME uses a single bidirectional Transformer rather than a vision tower connected to a LLM or a dual-tower alignment architecture. The image pathway converts an RGB image into non-overlapping $32 \times 32$ patches and maps each patch through layer normalization and a two-layer MLP. There is no pretrained vision encoder, patch-merging module, image tokenizer, pixel-reconstruction target, or causal language-model decoder.

Dynamic resolution is implemented through a sampled longest-side cap between 1,024 and 2,048 pixels. Aspect ratios are preserved, and images are downsampled only when they exceed the sampled cap. This design yields fewer image tokens than a $16 \times 16$ patch scheme while avoiding the information loss associated with forcing every document into a fixed square resolution.

Figure 1: Dynamic-resolution image processing with a variable side-length cap.

The text input path uses a factorized ALBERT-style embedding. A 131,072-entry vocabulary is represented through a 256-dimensional lookup table followed by projection to the model width, with the factors tied to the masked-token output layer. This substantially reduces embedding parameters while enabling a tokenizer trained for multilingual text, code, mathematics, and machine-generated image transcripts.

The tokenizer emits 44.4% fewer tokens than ModernBERT on a 14-language FLORES-200 evaluation, 39.4% fewer than LFM2.5-Encoder-230M, 6.3% fewer than mmBERT-base, and 16.9% fewer than EuroBERT-210M. However, this advantage is geographically and linguistically uneven. Across all 204 FLORES-200 language-script pairs, NeoMME emits 65.29% more tokens than mmBERT and 15.28% more than EuroBERT. The tokenizer is therefore efficient on the selected target languages but does not establish uniformly strong multilingual coverage, particularly for Tibetan, South Asian, and Southeast Asian scripts.

The encoder combines bidirectional attention with interleaved local and global attention. Most layers use symmetric sliding-window attention, with alternating half-windows of 256 and 1,024 tokens; every sixth layer and the final layer use global attention. Both configurations use grouped-query attention, query-key normalization, partial RoPE in global layers, and parameter-free RMS normalization.

Figure 2: Alternating sliding-window and global-attention layers in the NeoMME encoder stack.

Several architectural modifications are composed within each block: squared-ReLU MLPs, query-dependent sigmoid gating of attention outputs, $(2L)^{-1/2}$ residual scaling, token-indexed value embeddings at selected global layers, residual mixing with the initial multimodal input, and a learnable form of exclusive self-attention. Attention-output and MLP down-projections are zero-initialized, so each residual block begins as an identity mapping. These choices are intended to stabilize optimization at long context lengths, although the paper does not provide ablations isolating their individual contributions.

Pretraining objective and cross-modal behavior

The pretraining corpus is configured for approximately 524B packed input tokens: 55% from text-only sources and 45% from multimodal sources. The text stream includes web and PDF text, multilingual data, synthetic educational material, Wikipedia, mathematics, question-answering datasets, and code. The multimodal stream contains natural images, rendered documents, OCR data, PDF pages, charts, diagrams, and synthetic multilingual document images.

NeoMME uses a masked discrete-diffusion objective rather than conventional BERT masking. Each text segment samples a corruption rate from $\mathcal{U}(0,1)$ for text-only examples and $\mathcal{U}(0.30,1)$ for multimodal examples. Selected tokens are always replaced by the mask token, and the loss is evaluated only at masked positions. Image patches remain visible and condition the reconstruction of masked text.

The higher minimum corruption rate in multimodal examples is important to the paper’s argument. At low masking rates, masked text can often be recovered from neighboring text without visual evidence. By removing at least 30% of the text and frequently much more, the objective is intended to force the model to use the image. The loss additionally applies inverse corruption-rate weighting, capped at 20, to compensate for the varying number of masked targets across segments.

The cross-modal ablation probe supports the claim that image patches affect text prediction. At 90% masking, image conditioning produces a final mean accuracy gain of 38.4 percentage points for NeoMME-260M and 40.5 percentage points for NeoMME-800M relative to an identical input in which image patches are replaced by zeros.

Figure 3: Pretraining loss and image gain at 90% masking for both model sizes, with raw and smoothed measurements.

This result establishes image dependence in masked-token recovery, but it does not demonstrate general visual understanding. The probe measures exact token recovery on pretraining-style examples, and the paper does not report an average captioning or OCR score. The qualitative generation experiments show that a pretrained NeoMME-260M can produce captions and document-like text from fully masked canvases, but the authors explicitly treat these outputs as demonstrations rather than a generation evaluation.

The absence of an explicit visual prediction objective has measurable consequences. Frozen natural-image classification is weak, averaging 13.2 accuracy across ten tasks. Fine-tuning improves Food101 to $77.1 \pm 0.3$, Oxford Pets to $63.9 \pm 0.2$, and Stanford Cars to $46.8 \pm 6.3$. Document-image transfer is considerably stronger: on RVL-CDIP, a frozen first-token probe reaches $51.6 \pm 0.3$ accuracy, while fine-tuning on only 6,000 examples reaches $32 \times 32$0. The contrast indicates that the pretraining objective and data mixture produce representations more adapted to document structure and text-bearing images than to generic visual semantics.

Retrieval formulation and dual-head training

NeoMME-Retriever fine-tunes the backbone with two jointly optimized heads. The late-interaction head projects each final hidden state to a normalized 128-dimensional vector. Text produces one vector per token, while images produce one vector per patch and structural token. Query-document similarity is computed with MeanMaxSim: each query vector selects its maximum similarity to any document vector, and the maxima are averaged over query positions.

Figure 4: Dense, cross-encoder, and late-interaction retrieval scoring patterns.

The dense head mean-pools hidden states and produces a normalized single-vector embedding. It is trained with Matryoshka objectives at widths of 128, 256, 512, and 1,024 dimensions for NeoMME-260M, with an additional 1,792-dimensional width for NeoMME-800M. This allows the same backbone to support ANN retrieval, dense-first retrieval pipelines, and late-interaction reranking.

The retrieval loss is the sum of dense and late-interaction InfoNCE losses, both using temperature $32 \times 32$1. Training uses hard negatives, GradCache, and the authors’ Late-Interaction Kernels. The visual retrieval mixture contains 952,741 queries over 760,826 documents after adding 191,915 generated queries to pages sampled from the original collection. The generated queries cover multi-hop, comparison, open-ended, boolean, enumerative, and numerical forms across six languages.

The dual-head design produces a clear asymmetry. On ViDoRe v3, joint training raises the 260M late-interaction score from 0.5088 to 0.5226, a gain of 1.38 nDCG points. A paired analysis over 14,514 judged queries estimates a 1.39-point gain with a 95% confidence interval of 1.11 to 1.67 and $32 \times 32$2. The dense score changes by only 0.01 point, with a confidence interval spanning negative and positive effects.

The result is not a general benefit to both representations. On BEIR-15, joint training improves late-interaction by 1.07 points but reduces dense retrieval by 1.85 points. Thus, the paper’s evidence supports one-directional transfer from dense supervision to late-interaction on visual retrieval, not a universally synergistic dual-objective effect. Moreover, each ablation uses one training seed, so the reported confidence intervals do not capture run-to-run variability.

Visual document retrieval performance

NeoMME’s strongest evaluation is ViDoRe v3. The late-interaction NeoMME-260M obtains 0.5226 nDCG@10, compared with 0.2612 for ColModernVBERT and 0.2073 for ColSmol-256M. Relative to the best evaluated model below 300M parameters, the paper reports gains of 26.1 nDCG points on ViDoRe v3, 11.5 points on v2, and 5.4 points on v1.

NeoMME-800M reaches 0.5560 on ViDoRe v3, 0.5814 on v2, and 0.8801 on v1. Scaling from 260M to 800M improves the aggregate visual scores by 3.3 points on v3, 3.7 points on v2, and 1.5 points on v1. The 800M system is within 0.9 nDCG points of the approximately 850M Vultron Flash model on ViDoRe v3 and exceeds the reported 3.75B ColQwen2.5-v0.2 score on that benchmark.

Figure 5: ViDoRe v3 nDCG@10 as a function of model size, showing NeoMME-260M and NeoMME-800M on the Pareto frontier.

The results vary substantially by domain. NeoMME-260M performs best on computer science, with nDCG@10 of 0.6743, followed by pharmaceuticals at 0.5963 and energy at 0.5922. It is weaker on industrial documents, physics, and French finance, with scores of 0.3989, 0.4246, and 0.3782, respectively. NeoMME-800M improves most domains but remains below Vultron Flash on industrial, pharmaceutical, computer-science, and physics subsets in the reported comparison. Consequently, the Pareto-frontier claim is accurate at the aggregate parameter-performance level but should not be interpreted as uniform task-level superiority.

Late-interaction is consistently more effective than dense pooling. On ViDoRe v3, NeoMME-260M obtains 0.5226 with late interaction versus 0.3907 with dense retrieval; NeoMME-800M obtains 0.5560 versus 0.4391. On BEIR-15, the corresponding late-interaction scores are 0.4881 and 0.5126, while dense scores are 0.3055 and 0.3686. For NeoMME-260M, the late-interaction advantage on BEIR is 18.3 points; for NeoMME-800M, it is 14.4 points.

Figure 6: ViDoRe v3 quality as a function of hierarchical token-pooling factor for both NeoMME model sizes.

The magnitude of this gap supports the paper’s interpretation that patch-level representations preserve document evidence that is lost through mean pooling. It also exposes a limitation of the dense head: despite Matryoshka training and high-dimensional output, single-vector retrieval remains substantially weaker for both visual documents and text.

Resolution, storage, and inference efficiency

Resolution determines both retrieval quality and the number of late-interaction vectors. At a 2,048-pixel longest-side cap, a square page produces 4,162 vectors and 2.13 MB of uncompressed float32 representation. Reducing the cap to 1,536 pixels reduces the vector count to 2,354 and the raw storage to 1.21 MB. Across ViDoRe v1, v2, and v3, the quality loss from 2,048 to 1,536 pixels is at most 1.8% for either model. At 1,024 pixels, losses range from 3.0% to 12.7%; at 768 pixels, they range from 11.7% to 36.1%.

This provides a practical operating point: 1,536 pixels is a substantially cheaper setting with limited measured degradation, whereas 768 pixels can cause severe losses on resolution-sensitive tasks. Upsampling smaller pages to 2,048 pixels does not improve either retrieval head.

Hierarchical token pooling clusters similar document vectors and replaces each cluster with its mean. At pool factor 7, both model sizes retain more than 99% of factor-one ViDoRe v3 nDCG@10 while reducing document-vector storage by approximately $32 \times 32$3. At factor 10, NeoMME-260M retains 99.2% of its baseline quality and NeoMME-800M retains 98.9%; factor 20 retains 97.8% and 98.0%, respectively.

The paper’s most substantial deployment result combines pooling with asymmetric quantization. With factor-8 pooling, int8 queries, and binary document vectors, the NeoMME-260M representation falls from approximately 1.54 MB to 6.0 kB per document, a 255.5-fold reduction, while retaining 95.19% of baseline nDCG@10. A less aggressive factor-10/int8 configuration reduces storage to 39.0 kB, a 39.4-fold reduction, while retaining 99.16% of baseline quality.

Quantizing both queries and documents to int8 has almost no quality effect: the 260M score changes from 0.5226 to 0.5224, and the 800M score from 0.5560 to 0.5559. Binary document representations reduce storage by $32 \times 32$4 relative to float32, but the quality penalty is larger when only documents are binarized: 1.58 and 1.91 nDCG points for the 260M and 800M models, respectively. The paper does not report optimized end-to-end retrieval latency for binary scoring, because its experiments do not include kernels operating directly on packed bits.

On an NVIDIA L40S, NeoMME-260M encodes $32 \times 32$5 pages at 51.3 pages per second, compared with 26.0 for ColModernVBERT under the matched benchmark configuration. This is a $32 \times 32$6 throughput advantage at the same nominal resolution. NeoMME-800M reaches 21.2 pages per second at $32 \times 32$7, and 39.2 pages per second at $32 \times 32$8. The comparison with Vultron Flash at 1,344 pixels is favorable to NeoMME-800M in pages per second but not resolution-matched: NeoMME processes 31% more pixels.

Figure 7: NVIDIA H100 80GB HBM3 used in the indexing and kernel efficiency measurements.

Query encoding is also comparatively efficient. NeoMME-260M has mean batch-one latency of 21.0 ms on an L40S, 78.3 ms on a 128-core CPU host, and 15.9 ms on an Apple M5 Pro. It is second only to ColModernVBERT across all tested devices. These measurements cover tokenization, query processing, the encoder forward pass, and output transfer, but exclude candidate generation, ANN search, MaxSim scoring, and end-to-end response latency.

Figure 8: Batch-one text query-encoding latency across CPU, NVIDIA L40S, and Apple M5 Pro execution environments.

The Late-Interaction Kernels address memory rather than principally improving throughput in the NeoMME training configuration. At 4,096 document tokens, the exact kernel reduces MaxSim peak memory from 672 MB to 193 MB and backward time from 1.82 ms to 0.54 ms. At 8,192 tokens, the naive implementation runs out of memory while the tiled kernel uses 321 MB. In matched NeoMME-260M training runs, throughput differs only within run-to-run variation, so the principal contribution is the ability to train and score longer documents without materializing the complete similarity tensor.

Transfer beyond retrieval

The 260M backbone is evaluated on language classification, token labeling, document classification, and natural-image tasks. On a 17-task GLUE, SuperGLUE, and multilingual suite, it achieves a mean score of 75.3, compared with 79.3 reported for the similarly sized LFM2.5-Encoder-230M. The result is competitive given that NeoMME’s pretraining budget and capacity are shared with image patches and document data, but the comparison is not a same-harness reproduction of the competing model’s recipe.

NeoMME reaches $32 \times 32$9 accuracy on the 8K-context LEDGAR task and 89.3 entity F1 on a one-seed CoNLL-2003 experiment. These results indicate that the long-context bidirectional encoder transfers to document classification and sequence labeling. They do not establish superiority over specialized text encoders.

The visual-transfer results reinforce the asymmetry between document and natural-image competence. Fine-tuned performance on Food101, Oxford Pets, and Stanford Cars is moderate, while RVL-CDIP reaches $16 \times 16$0 with only 6,000 training examples. Since RVL-CDIP contains known annotation noise and train-test overlap, the paper appropriately interprets this result as rapid adaptation rather than clean out-of-distribution generalization.

Limitations and open questions

The principal limitation is experimental confounding. NeoMME is trained on approximately 524B packed tokens, including roughly 290B text-only tokens, whereas ModernBERT reportedly uses about 2T text tokens. Similarly, NeoMME-Retriever processes approximately 430K pure-text query examples and 850K image-query examples, while mLateOn uses approximately 660M contrastive examples followed by about 16M multilingual hard-negative examples. The observed performance differences therefore cannot be attributed solely to architecture or objective.

The paper also does not isolate the effects of its numerous architectural choices. There are no matched ablations for patch size, local-global attention scheduling, QK normalization, residual mixing, zero initialization, value embeddings, exclusive self-attention, or the masked-diffusion corruption distribution. The retrieval processor’s ten learned query-expansion mask tokens are likewise not ablated.

The visual objective leaves a specific unresolved question. NeoMME demonstrates image-conditioned masked-text prediction but does not directly predict image content or optimize an image-level representation. The weak frozen natural-image results may therefore reflect the objective rather than a limitation of the shared encoder architecture. A controlled comparison among image-token prediction, pixel or latent reconstruction, image-text contrastive learning, and the present text-denoising objective is needed.

The multilingual claims also require qualification. The tokenizer is highly efficient on the selected 14-language target set but substantially less competitive across all FLORES-200 languages. Retrieval training covers six languages in the visual stream, with narrower page-language coverage than query-language coverage. The paper does not evaluate mixed-modality retrieval over a joint corpus of text chunks and page images, despite training a shared embedding space for that purpose.

Finally, compression results exclude ANN routing overhead, filesystem overhead, index construction, serving latency, and specialized binary scoring kernels. The reported 255-fold reduction is consequently a representation-storage result, not an end-to-end system-compression guarantee. The retrieval ablation uses one seed, and the visual benchmarks remain dependent on the coverage and judgment quality of their task collections.

Conclusion

NeoMME establishes a coherent design for compact multimodal representation learning: raw image patches and multilingual text are processed by one bidirectional long-context Transformer trained from scratch with an image-conditioned masked discrete-diffusion objective. Its strongest empirical case is visual document retrieval. NeoMME-260M achieves 0.5226 nDCG@10 on ViDoRe v3 and nearly doubles ColModernVBERT’s measured page-indexing throughput at matched $16 \times 16$1 resolution, while NeoMME-800M reaches 0.5560 nDCG@10.

The work’s most practically important contribution is the combination of late-interaction retrieval with aggressive representation compression. Hierarchical pooling and asymmetric quantization reduce approximately 1.5 MB document embeddings to 6 kB while retaining 95.19% of baseline retrieval quality. At the same time, the results show clear boundaries: dense retrieval is substantially weaker, natural-image transfer is poor without fine-tuning, broad multilingual tokenization is inconsistent, and architectural and data-scale effects remain insufficiently disentangled.

NeoMME therefore provides evidence that a single-tower multimodal-native encoder can achieve favorable quality-efficiency trade-offs for visual document retrieval, while leaving open whether the same design remains competitive under matched pretraining and retrieval supervision, broader multilingual coverage, explicit visual objectives, and end-to-end indexed retrieval evaluation.