NeoMME: Multimodal Encoders for Retrieval

• NeoMME is a family of single-tower, multimodal-native, multilingual bidirectional Transformers designed for efficient fine-tuning, inference, and visual-document retrieval, comprising NeoMME-260M and NeoMME-800M with 262,937,906 and 793,715,032 parameters, respectively.
• NeoMME-Retriever, the principal downstream system, jointly supports dense and late-interaction representations, achieving competitive performance on ViDoRe and BEIR metrics with significant compression and memory efficiency.
• NeoMME-260M achieved an nDCG@10 of 0.5226 on ViDoRe v3, while NeoMME-800M reached 0.5560, demonstrating strong performance with late-interaction retrieval.

NeoMME is a family of single-tower, multimodal-native, multilingual bidirectional Transformer encoders designed for efficient fine-tuning, inference, and visual-document retrieval. Unlike generative vision-LLMs that combine a separately pretrained vision encoder with a causal LLM, NeoMME processes multilingual text tokens and raw image patches jointly through one bidirectional Transformer trained from scratch. The family comprises NeoMME-260M and NeoMME-800M, with 262,937,906 and 793,715,032 parameters, respectively. Its principal downstream system, NeoMME-Retriever, jointly supports dense and late-interaction representations (Lac et al., 31 Aug 2026).

1. Design rationale and system characteristics

NeoMME addresses the computational mismatch between generative vision-language architectures and representation-learning tasks. Generative VLMs typically contain a vision tower, a projection or cross-attention connector, and a causal LLM. Although this design supports captioning and visual question answering, it retains substantial parameter and inference overhead when the target task is retrieval. Visual-document retrievers such as ColPali repurpose generative VLMs as encoders while retaining their architectural costs.

NeoMME is encoder-native rather than generative-model-derived. It contains no pretrained vision tower, causal decoder, separate text and image towers, OCR requirement, pixel-reconstruction objective, or discrete image tokenizer. Raw image patches and text tokens are projected into a common hidden space and processed jointly by every Transformer layer. A single forward pass produces both conventional dense embeddings and token-level late-interaction embeddings.

The two released configurations are:

Configuration	NeoMME-260M	NeoMME-800M
Parameters	262,937,906	793,715,032
Hidden width	1,024	1,792
Encoder layers	17	20
Global-attention layers	3	4
Query heads	16	28
Key/value heads	4	7
Context length	16,384	16,384
Patch size	$32\times32$	$32\times32$

Both models use grouped-query attention, query-key RMS normalization, partial rotary embeddings in global layers, squared-ReLU MLPs, gated attention outputs, and residual-branch scaling by $(2L)^{-1/2}$, where $L$ is the number of Transformer layers. The 800M model scales primarily through width while retaining a compact 20-layer backbone.

The architecture differs from CLIP and SigLIP, which use separate image and text towers; from generative VLMs, which connect vision encoders to causal LLMs; and from ColModernVBERT, which retains a pretrained SigLIP2 visual tower. NeoMME instead exposes both modalities to the same contextual computation from the first Transformer layer.

2. Input representation and long-context processing

Text tokenization

NeoMME uses a tokenizer trained from scratch with a 131,072-entry vocabulary. It is a byte-level, whitespace-unconstrained BPE tokenizer with byte fallback, separate digit splitting, 64 reserved identifiers for fixed special tokens, and a maximum merge length of 48 bytes. The tokenizer mixture includes multilingual web text, PDF text, code, mathematics, and machine-produced image transcripts.

On the paper’s 14-language FLORES evaluation, NeoMME emitted 44.4% fewer tokens than ModernBERT, 39.4% fewer than LFM2.5-Encoder-230M, 6.3% fewer than mmBERT-base, and 16.9% fewer than EuroBERT-210M. This reduction does not extend uniformly to all 204 FLORES languages: across all language-script pairs, NeoMME emitted 65.29% more tokens than mmBERT and 15.28% more than EuroBERT, with particularly weak results for several Tibetan, South Asian, and Southeast Asian scripts.

Text embeddings use a factorized ALBERT-style input representation:

$\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$

where $E\in\mathbb{R}^{V\times d_e}$ is the vocabulary embedding table, $V=131{,}072$, $d_e=256$, and $P\in\mathbb{R}^{d\times d_e}$ projects the embedding into the Transformer hidden width. The output vocabulary logits reuse the same factors:

$\boldsymbol{\ell}_i = E P^\top \mathbf{h}_i.$

This factorization avoids a separate full-width vocabulary projection.

Raw image patches

Images are converted to RGB and divided into non-overlapping $32\times32$0 patches. Each patch therefore contains $32\times32$1 pixel values. Layer normalization and a two-layer MLP project each raw patch into the model input dimension.

Images preserve their aspect ratio. During pretraining and fine-tuning, the processor samples a longest-side cap between 1,024 and 2,048 pixels and downsizes only when necessary. A $32\times32$2 square produces $32\times32$3 raw image patches. Including structural and row-boundary tokens, the late-interaction representation contains 4,162 vectors.

NeoMME does not use a pretrained ViT or SigLIP tower, patch merging, a discrete image tokenizer, or a separate pixel-prediction head. Image patches remain visible during masked-text prediction and condition the model’s textual representations.

Context and attention

Both models support 16,384 tokens. The long-context design combines mostly symmetric sliding-window attention with global-attention layers every sixth layer and at the end. Sliding windows alternate between half-windows of 256 and 1,024 tokens.

Packed documents are isolated with variable-length attention boundaries and segment offsets, allowing multiple examples to occupy a sequence without attending to one another. Image patches receive two-dimensional rotary coordinates. A patch at image-grid row $32\times32$4, column $32\times32$5, with image coordinate base $32\times32$6, receives approximately

$32\times32$7

whereas text uses diagonal coordinates $32\times32$8. The 16,384-token context is intended to accommodate long text and up to two standard $32\times32$9 4K UHD images after $(2L)^{-1/2}$0-pixel patching.

3. Pretraining objective and data

NeoMME is pretrained from scratch with a masked discrete-diffusion text objective rather than ordinary BERT masking. The objective uses an absorbing-mask process in which selected text tokens are replaced by a mask token and the model predicts the original tokens. There is no image reconstruction target.

The planned pretraining run processes

$(2L)^{-1/2}$1

packed input tokens. The mixture contains 55% text-only data, approximately 288.358 billion tokens, and 45% multimodal data, approximately 235.930 billion tokens.

Text sources include FineWeb-Edu, FineWeb2-HQ, FinePDFs, Nemotron-CC, Cosmopedia, Wikipedia, FineMath, question-answering datasets, and StarCoderData. The multimodal mixture includes FineVision, PDFA and LightOnOCR, DocAtlas, synthetic multilingual OCR, and PixelProse. The visual stream contains natural images, document pages, charts, diagrams, OCR text, captions, Markdown transcriptions, and synthetic text.

Pretraining used 16 H100 GPUs for NeoMME-260M and 32 H100 GPUs for NeoMME-800M, with bfloat16 computation, packed sequences of 1,048,576 tokens per global step, 300-step warmup, warmup–stable–decay learning-rate scheduling, and decay during the final 10% of training to 1% of the peak learning rate. Matrix parameters use NorMuon, while embeddings and scalars use AdamW variants.

For each document segment $(2L)^{-1/2}$2, the corruption rate is sampled as

$(2L)^{-1/2}$3

for text-only examples and

$(2L)^{-1/2}$4

for multimodal examples. The higher minimum corruption rate in multimodal segments reduces visible textual context and encourages image-conditioned prediction. Each eligible text position is independently replaced with the mask token with probability $(2L)^{-1/2}$5. Padding, image patches, and structural markers are excluded from masking.

Unlike the BERT 80/10/10 rule, selected positions are always replaced by the mask token, and loss is computed only at selected positions. The original token is the target. With inverse-corruption weighting,

$(2L)^{-1/2}$6

the rank-local loss is

$(2L)^{-1/2}$7

The reciprocal weight is capped at 20 in implementation. The optimized loss averages rank-local losses over data-parallel ranks.

The objective can be interpreted as learning denoising transitions across corruption levels. At inference, a fully masked text canvas can be progressively denoised by predicting and revealing tokens, optionally conditioned on image patches. In an image-sensitivity probe at 90% masking, the final 50 probe events showed image-conditioned masked-token accuracy gains of 38.4 percentage points for NeoMME-260M and 40.5 percentage points for NeoMME-800M. These measurements support the use of visible image content rather than merely residual textual context.

Qualitative image-conditioned captioning and document-text generation were also demonstrated with the pretrained 260M model, although these examples were not presented as a quantitative generation benchmark.

4. Retrieval representations and optimization

NeoMME-Retriever is fine-tuned with jointly trained dense and late-interaction heads. Both representations are generated by a single backbone forward pass.

Late interaction

The late-interaction head applies a learned linear projection from final hidden states to 128 dimensions, followed by normalization. Text yields one vector per text token, while images yield one vector per image patch and structural position.

For query $(2L)^{-1/2}$8 and document $(2L)^{-1/2}$9,

$L$0

with unit-normalized vectors in $L$1. NeoMME uses MeanMaxSim:

$L$2

Each query token is matched to its most similar document vector, and the maxima are averaged over query tokens. This retains localized evidence that can be lost in a single pooled representation.

Queries contain ten learned expansion mask tokens:

$L$3

Image documents use a structure containing document and image markers followed by row-major patches with row markers.

Dense representation

The dense head mean-pools final hidden states and applies L2 normalization. It is trained with Matryoshka representation learning, so prefixes of the dense vector remain usable. The trained widths are $L$4 for the 260M model and $L$5 for the 800M model.

Dense similarity is the normalized-vector dot product:

$L$6

Dense embeddings support conventional approximate-nearest-neighbor indexing, whereas late interaction retains token- or patch-level matching at higher storage and scoring cost.

Contrastive fine-tuning

For a positive document $L$7 and in-batch candidate corpus $L$8, the late-interaction InfoNCE loss uses temperature $L$9:

$\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$0

The dense loss averages analogous contrastive losses over the Matryoshka dimensions. The total retrieval objective is

$\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$1

with equal weights and no teacher-logit distillation.

Retrieval fine-tuning used one AWS p5.48xlarge node with eight H100 accelerators and 20,000 steps. The text stream contained approximately 1.58 million query examples across English, multilingual, code, and organic retrieval data. The visual stream contained 952,741 queries over 760,826 pages after query augmentation. Visual sources included ColPali training data, a multilingual document-image dataset, VisRAG in-domain data, and VisRAG synthetic data.

Visual hard-negative mining was performed in two stages: initial training without explicit hard negatives, followed by retrieval of a 32-candidate window and retraining with seven negatives sampled from that window. Negatives with mining scores at least $\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$2 times the positive score, together with nonfinite candidates, were removed.

5. Retrieval performance and efficiency

ViDoRe and BEIR

On ViDoRe v3, NeoMME-Retriever achieved the following aggregate results:

Model/head	Parameters	ViDoRe v3 nDCG@10
NeoMME-260M dense	260M	0.3907
NeoMME-260M late interaction	260M	0.5226
NeoMME-800M dense	800M	0.4391
NeoMME-800M late interaction	800M	0.5560

The 260M late-interaction model was the strongest evaluated model below 300M parameters and, according to the paper, the strongest model strictly below 800M. Its 0.5226 nDCG@10 was close to ColQwen2.5’s 0.5244 despite the substantially smaller parameter count. NeoMME-800M reached 0.5560, within 0.9 nDCG points of the approximately 850M Vultron Flash model.

For the 260M and 800M late-interaction models, respectively, the complete ViDoRe v3 metrics were:

Model/head	nDCG@5	nDCG@10	Recall@5	Recall@10	Recall@100	MAP
260M late	0.4991	0.5226	0.4637	0.5664	0.8326	0.4555
800M late	0.5329	0.5560	0.4906	0.5970	0.8607	0.4878

The corresponding dense results were lower. On BEIR-15, late interaction reached 0.4881 for NeoMME-260M and 0.5126 for NeoMME-800M, compared with 0.3055 and 0.3686 for their dense heads.

The late-interaction advantage was 0.1319 nDCG points for the 260M model on ViDoRe v3 and 0.1169 points for the 800M model. On BEIR-15, the paper reports late-interaction improvements over dense retrieval of 18.3 points for 260M and 14.4 points for 800M.

Increasing model size from 260M to 800M improved late-interaction performance by 3.3 nDCG points on ViDoRe v3, 3.7 points on ViDoRe v2, and 1.5 points on ViDoRe v1. Joint training of dense and late-interaction heads improved late interaction by 1.38 points on ViDoRe v3 and 1.07 points on BEIR-15 relative to late-only training. The paired ViDoRe analysis estimated a 1.39-point gain with a 95% confidence interval of 1.11–1.67 points and $\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$3. Dense retrieval did not improve universally: on BEIR, joint training reduced dense performance by 1.85 points.

Resolution dependence

The default ViDoRe evaluation uses a downscale-only longest-side cap of 2,048 pixels. For square pages:

Cap	Vectors	Raw float32 storage	260M nDCG@10	800M nDCG@10
768	602	0.31 MB	0.3340	0.4049
1,024	1,058	0.54 MB	0.4562	0.5029
1,536	2,354	1.21 MB	0.5134	0.5496
2,048	4,162	2.13 MB	0.5226	0.5560

A 1,536-pixel cap produces no more than approximately a 1.8% reduction in nDCG across the reported ViDoRe evaluations. At 1,024 pixels, quality losses range from 3.0% to 12.7%; at 768 pixels, from 11.7% to 36.1%.

Encoding throughput

At a matched $\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$4 image size on an NVIDIA L40S, model-side page-encoding throughput was:

Model	Parameters	Vectors/page	Pages/s
NeoMME-260M	260M	4,162	51.3
ColModernVBERT	250M	1,149	26.0
ColSmol-256M	256M	1,139	24.8
NeoMME-800M	800M	4,162	21.2

NeoMME-260M therefore encoded pages at approximately $\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$5 times the throughput of ColModernVBERT under the reported setup. The measurement included device transfer, model forward execution, output unpadding, conversion to contiguous float32 embeddings, and in-memory serialization, but excluded image decoding, image processing, compression, index construction, and storage I/O.

Mean batch-one query latency for NeoMME-260M was 21.0 ms on an L40S, 78.3 ms on a 128-core CPU host, and 15.9 ms on an Apple M5 Pro. The model was second only to ColModernVBERT on the tested devices.

Late-Interaction Kernels compute tiled similarities while retaining only the running maximum for each query token. For 4,096 document tokens, they reduced forward time from 0.65 ms to 0.12 ms, backward time from 1.82 ms to 0.54 ms, and peak memory from 672 MB to 193 MB. For 8,192 document tokens, naive scoring ran out of memory while Late-Interaction Kernels used 321 MB. Their main benefit for NeoMME training was memory headroom rather than a substantial end-to-end throughput increase.

6. Representation compression, deployment, and limitations

Hierarchical pooling and quantization

Late-interaction storage scales with the number of document vectors and their dimensional precision. Hierarchical token pooling clusters similar document vectors and replaces each cluster with its mean. A pool factor $\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$6 approximately reduces the number of stored vectors from $\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$7 to $\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$8.

A pool factor of 7 provided approximately $\mathbf{h}^{(0)}_i = P E_{x_i,:}^{\top},$9 lower storage with more than 99% of ViDoRe v3 nDCG retained. A pool factor of 10 retained 99.2% of baseline quality for 260M and 98.9% for 800M; a pool factor of 20 retained 97.8% and 98.0%, respectively.

Asymmetric quantization uses higher precision for query vectors than for stored document vectors. The tested formats include float32 queries with float32 documents, int8 queries with int8 documents, int8 queries with binary documents, and binary queries with binary documents.

Model	Query	Document	nDCG@10	Storage
260M	float32	float32	0.5226	1536.7 kB
260M	int8	int8	0.5224	390.2 kB
260M	int8	binary	0.5068	48.0 kB
260M	binary	binary	0.4960	48.0 kB
800M	float32	float32	0.5560	1536.7 kB
800M	int8	int8	0.5559	390.2 kB
800M	int8	binary	0.5369	48.0 kB
800M	binary	binary	0.5253	48.0 kB

Int8 document representations reduce storage by 3.9 times, while binary document representations reduce it by 32 times. Int8 queries cost at most 0.0002 nDCG@10 in the reported experiments. Packed scores matched the float reference within $E\in\mathbb{R}^{V\times d_e}$0 for every query, with an observed maximum difference of $E\in\mathbb{R}^{V\times d_e}$1.

The combined headline operating point applies pool factor 8, int8 query vectors, and binary document vectors. For NeoMME-260M, average page representation storage decreases from 1536.7 kB to 6.0 kB:

$E\in\mathbb{R}^{V\times d_e}$2

This retains 95.19% of baseline ViDoRe v3 nDCG@10. The result is a storage-quality trade-off rather than lossless compression. A less aggressive operating point using pool factor 10 with int8 queries and int8 documents requires 39.0 kB, provides 39.4-times compression, and retains 99.16% of baseline quality.

Software and released resources

NeoMME is contributed to Hugging Face Transformers. Released resources include pretrained backbones, pretraining pre-decay checkpoints, retrieval checkpoints, and a Hugging Face collection. The checkpoints and retrieval-compatible models are released under Apache 2.0.

Relevant checkpoints include:

• Hcompany/NeoMME-260M
• Hcompany/NeoMME-800M
• Hcompany/NeoMME-260M-Pretrain-predecay-s450000
• Hcompany/NeoMME-800M-Pretrain-predecay-s450000
• Hcompany/NeoMME-260M-Retriever
• Hcompany/NeoMME-800M-Retriever

Reproduction of reported retrieval results requires the official processor, dynamic-resolution behavior, 2,048-pixel downscale-only evaluation for ViDoRe, ten learned query-expansion mask tokens, exact MeanMaxSim, and the step-20,000 retrieval checkpoints rather than the step-500,000 pretrained backbone. Throughput comparisons additionally depend on bfloat16 execution, compilation, FlashAttention 2, calibrated batch sizes, and the stated timing scope.

Limitations

NeoMME’s pretraining scale is smaller than that of some comparison systems. It uses approximately 524 billion packed tokens, including about 290 billion text-stream tokens, whereas ModernBERT used approximately 2 trillion text tokens. The accounting methods differ, but ModernBERT saw roughly seven times more text-only data.

The lack of a direct visual pretraining target limits frozen natural-image representations. Frozen 16-shot natural-image classification averaged 13.2 accuracy, whereas fine-tuning produced $E\in\mathbb{R}^{V\times d_e}$3 on Food101, $E\in\mathbb{R}^{V\times d_e}$4 on Oxford Pets, and $E\in\mathbb{R}^{V\times d_e}$5 on Stanford Cars. Document-image transfer was stronger: frozen RVL-CDIP performance was $E\in\mathbb{R}^{V\times d_e}$6, and fine-tuning on 6,000 examples reached $E\in\mathbb{R}^{V\times d_e}$7.

Retrieval supervision is also narrower than in systems such as mLateOn. NeoMME used approximately 430,000 sampled text queries and 850,000 image-query examples, compared with approximately 660 million contrastive examples and 16 million multilingual hard-negative examples reported for mLateOn. Consequently, BEIR differences cannot be attributed purely to architecture.

Language coverage is uneven. Visual pages are primarily English, French, German, Italian, and Spanish; generated queries add Portuguese, while text retrieval includes Arabic and other multilingual pools. Performance for lower-resource scripts and underrepresented languages remains uncertain.

NeoMME-Retriever was trained on both text and image retrieval data, but the paper does not evaluate a unified mixed-modality corpus. Compression experiments exclude index and filesystem overhead, throughput excludes image decoding and processor time, and optimized binary retrieval kernels were not implemented or benchmarked. The dual-head ablation uses one training seed, and the paper does not systematically isolate every architectural, data, or objective choice.

NeoMME is therefore best characterized as a multimodal-native, multilingual, long-context representation encoder rather than a general-purpose generative VLM. Its principal contribution is the combination of a from-scratch single Transformer for raw image patches and text, dual dense and late-interaction retrieval heads, high-resolution visual encoding, efficient inference, and aggressive late-interaction compression.