Most multimodal models today are Frankenstein assemblies: a pretrained vision encoder bolted onto a pretrained language model, wired together with a projector. It works, but it's expensive and architecturally awkward.
Hugging Face just released NeoMME, and it throws that entire playbook out. No separate vision tower. No causal decoder. Just one bidirectional Transformer processing raw image patches and text tokens together, trained from scratch. The result is a pair of compact multimodal encoders—260M and 800M parameters—that punch well above their weight class on visual document retrieval.
One Transformer, Two Modalities
The defining bet behind NeoMME is architectural simplicity. Text tokens use factorized embeddings. Images get split into 32×32 patches and projected through a small MLP. Both enter the same Transformer stack.
This matters because most visual language models inherit the architectural baggage of their components. A separate vision tower means separate optimization paths, separate serving logic, and parameter overhead for tasks that don't need autoregressive generation. Retrieval, classification, and token labeling don't generate text—they don't need a causal decoder.
NeoMME uses a modern encoder stack: grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. Context length is 16,384 tokens—enough for two 4K UHD images. Most layers use sliding-window attention; every sixth layer and the final layer use global attention.
The model handles dynamic image resolution, preserving aspect ratio and allocating tokens proportionally. A high-resolution document page gets more patches than a small icon. This feels obvious but requires careful engineering.
Learning From Images by Masking Text
Pretraining uses a discrete masked-diffusion objective. For text-only examples, NeoMME samples a corruption rate uniformly between 0 and 1, then masks each eligible token independently. Multimodal examples use corruption rates between 0.3 and 1.
The clever part: image patches stay visible while the model reconstructs masked text. Light masking lets the model lean on surrounding text context—"The [MASK] sat on the mat" is solvable without vision. Heavy masking forces the model to ground its predictions in visual evidence.
This design choice sidesteps a common multimodal training trap: language-only shortcuts. If the model can solve most examples using text alone, it won't learn rich cross-modal representations. High text corruption rates remove the shortcut.
The training mix includes multilingual text, code, mathematics, natural images, and document images. Each model sees about 524 billion packed tokens, including 290 billion text-only tokens. That's small compared to ModernBERT's 2 trillion token budget, so the team used the NorMuon optimizer to improve data efficiency.
NeoMME-Retriever: Dense and Late-Interaction in One Pass
The team fine-tuned NeoMME for visual document retrieval using the page-image methodology from ColPali. Instead of extracting text chunks via OCR, the model ranks document page screenshots directly. This preserves layout, charts, tables, font styling—visual cues that OCR discards.
NeoMME-Retriever adds two jointly trained heads:
- Dense head: Mean-pools the backbone's hidden states into a single normalized vector. Compact, works with approximate nearest-neighbor indexes.
- Late-interaction head: Projects each token or patch to a 128-dimensional normalized vector. Finer granularity, better at capturing local matches between query tokens and image regions.
One forward pass returns both representations. Use dense embeddings for massive corpora and ANN retrieval, then rerank with late-interaction. Or skip straight to late-interaction if your corpus is smaller.
Competitive at Compact Sizes
On ViDoRe v3, NeoMME-Retriever-260M hits 0.523 nDCG@10—the highest score among models below 800M parameters. It's within 0.002 nDCG@10 of ColQwen2.5 while using 14× fewer parameters.
NeoMME-Retriever-800M reaches 0.556 nDCG@10, within 0.009 of the similarly sized Vultron Retriever Flash. Both models lie on the model-size Pareto frontier.
On ViDoRe v2 and v1 (which use nDCG@5), the 260M model outperforms ColModernVBERT and the twice-larger ColSmol-500M. The 800M model beats ColPali v1.3 while using 3.6× fewer parameters.
Making Late-Interaction Storage Practical
Late-interaction embeddings scale linearly with image resolution. A 2048×2048 page produces 4,200 vectors with NeoMME-Retriever—about 2.1 MB in float32. Across ViDoRe v3, the average is 1.5 MB per document. That's painful for large corpora.
The team combines two compression techniques:
- Hierarchical token pooling: Clusters similar vectors in each embedding and replaces clusters with their mean, reducing vector count.
- Asymmetric quantization: Quantizes stored document embeddings to int8 or binary. Query embeddings stay at higher precision since they're only generated on-the-fly.
With pooling factor 10 and int8 for both queries and documents, storage drops from 1.5 MB to 39 kB per page (39× reduction) while retaining more than 99% of baseline nDCG@10.
An aggressive configuration—pooling factor 8, int8 queries, binary documents—uses just 6 kB per page (255× smaller) and keeps more than 95% of retrieval quality. Users can pick a compression point based on their storage budget and quality requirements.
Fast Inference, Lower Indexing Costs
Faster encoding means cheaper corpus indexing. The team measured encoding speeds at 2048×2048 resolution on an NVIDIA L40S GPU. The 260M model encodes about 51 pages per second—roughly twice ColModernVBERT's throughput.
This matters for production systems. Building and updating vector indexes requires GPU time. Doubling throughput halves compute cost.
What This Means for Multimodal Encoders
NeoMME demonstrates that you don't need a pretrained vision tower or a causal language model to build a competitive multimodal retriever. A single bidirectional Transformer, trained from scratch with a carefully designed masking objective, can learn strong cross-modal representations.
The architectural simplicity pays dividends: shared computational paths for images and text, no parameter overhead from unused generation machinery, and cleaner serving logic. The dual-head design—dense and late-interaction embeddings in one pass—gives practitioners flexibility without forcing them to pick a representation upfront.
The compression results are particularly interesting. Going from 1.5 MB to 6 kB per page (255× smaller) while retaining 95%+ quality opens up late-interaction retrieval for use cases where storage costs previously killed the economics.
All checkpoints are Apache 2.0 licensed and available in Hugging Face Transformers. The team also released a visual RAG demo and a technical report. If you're building multimodal retrieval systems, this is worth a look.