Molmo2 Backbone: Vision-Language Architecture

• Molmo2 Backbone is an open-weight multimodal framework that integrates a frozen vision encoder, cross-modal connector, and LLM decoder for superior visual understanding.
• It employs innovative token weighting and DP-based packing to balance training across image, video captioning, and pointing tasks efficiently.
• The architecture uses message-tree encoding and block-masked attention to support multi-branch annotations, achieving state-of-the-art video counting, captioning, and tracking.

The Molmo2 backbone refers to the architectural and algorithmic foundations of the Molmo2 family of open-weight multimodal models, specifically its vision-LLM (VLM) component stack. This backbone underpins state-of-the-art performance for open-source models on visual understanding, video captioning, and point-driven visual grounding in both images and video, as demonstrated by superior benchmarks in counting, captioning, and tracking compared to both other open-weight and proprietary models (Clark et al., 15 Jan 2026). The term "Molmo2 backbone" in current literature encompasses the vision encoder, the cross-modal connector, LLM decoder, and their associated architectural, packing, and training strategies.

1. Vision Encoder Architecture

Molmo2 employs a frozen, pre-trained “SigLIP 2 So400m/14” vision transformer (ViT) without any auxiliary convolutional layers. Each input image or video frame is resized or cropped to 384×384 pixels, and tokenized into non-overlapping 14×14 patches, yielding 784 tokens per image. For images, a multicrop strategy is used: one full crop and $K=8$ overlapping crops during training, increasing to $K=24$ at inference. Video frames are single-cropped and sampled at 2 fps, always including the last frame, with a hard cap at $F=128$ frames for standard fine-tuning or $F=384$ for long-context adaptation (Clark et al., 15 Jan 2026).

Intermediate patch embeddings are extracted at two ViT layers: the 3rd-from-last and 9th-from-last. These are pooled spatially—$2 \times 2$ pooling for images and $3 \times 3$ for video—via a small 16-head multi-head attention module, followed by a two-layer MLP with SwiGLU activation, projecting into the LLM’s hidden size. For the 4B variant, the vision transformer backbone is a 27-layer ViT, with embedding dimension $d=1,152$, 16 heads, and roughly 3.38 million parameters, while the connector adds approximately 57 million parameters.

2. Vision–Language Connector and Message-Tree Encoding

The connector bridges the vision encoder to the LLM. It pools spatial patch embeddings as described above, maintaining output dimension equal to the ViT embedding dimension. This enables integration with multi-modal sequences for both image and video input.

Molmo2’s message-tree encoding is designed for efficient representation of multiple annotation branches per visual input (e.g., several captions or pointing tracks for a video or image). The vision tokens are repeated as a prefix for each branch: $$[\text{BOS}, \text{VISION\_TOKENS}, \text{ANNOTATION}_1]$$, $$[\text{BOS}, \text{VISION\_TOKENS}, \text{ANNOTATION}_2]$$, etc. Each branch’s text tokens attend causally within their branch and to all vision tokens, but not to tokens from other branches. All vision tokens attend bidirectionally to each other but not to text. Packing is performed using a dynamic programming (DP)-style knapsack solution over text and vision token budgets, yielding around 3.8 examples per $16\,384$-token training sequence (Clark et al., 15 Jan 2026).

3. LLM Decoder and Attention Strategy

For language decoding, Molmo2 utilizes Qwen3-VL-4B/8B or OLMo3-7B, all large transformer decoders with 36 layers (4B: $K=24$0, $K=24$1-dim MLP, 32 attention heads, SwiGLU activation, dropout 0.1). Vision and text tokens are concatenated, and the decoder applies a block-diagonal masked self-attention: vision tokens attend bidirectionally to all other vision tokens (as a fixed prefix memory), text tokens attend causally within their own annotation branch and attend to all vision tokens, but not to text tokens from other branches. Explicit cross-attention is not required; the integrated prefix approach yields equivalent cross-modal information flow (Clark et al., 15 Jan 2026). This attention mask structure is critical for achieving high performance in multi-branch annotation and grounding tasks.

4. Token-Weighting and Packing Innovations

Molmo2’s backbone incorporates a token-weighting scheme to prevent learning collapse on tasks (like video captioning) with disproportionately long answer sequences. During training, each example’s cross-entropy loss is down-weighted as a function of answer length:

• For video-caption: $K=24$2
• For pointing: $K=24$3
• Otherwise: $K=24$4 This adjustment ensures balanced learning capacity allocation, promoting robust multi-task performance especially when training with long-form video or image captions (Clark et al., 15 Jan 2026).

Packing is performed via a DP-based approach: an in-memory pool of 48 tokenized examples is scored (text length plus $K=24$5 crop count), with constraints on both text and vision token budgets ($K=24$6 and $K=24$7, respectively). In effect, this maximizes data throughput under token budget constraints.

5. Training Regimen and Long-Context Adaptation

Molmo2’s training pipeline consists of three phases:

1. Pre-training on a mixture (60% image captions, 30% image pointing, 10% NLP) for $K=24$8 steps, batch size 128, sequence length 2,560, with AdamW at tiered learning rates: ViT $K=24$9, connector $F=128$0, LLM $F=128$1.
2. Supervised fine-tuning (SFT): $F=128$2 steps, batch 128, sequence $F=128$3, with a fixed mixture of image, video, pointing, and tracking. The same learning rate strategy is employed, with bi-directional vision self-attention and message-tree packing activated (Clark et al., 15 Jan 2026).
3. Long-context post-training: $F=128$4 steps at sequence length $F=128$5 and up to $F=128$6 frames, using Ulysses attention and context parallelism across 8 GPUs. Vision compute is sharded across the CP group to minimize overhead.

Video-specific cropping (multi-crop on images up to $F=128$7 during SFT, up to $F=128$8 at inference), and the synthesis of multi-turn conversations within message-trees, further enhance backbone flexibility for complex visual-linguistic tasks.

6. Empirical Performance and Impact

The Molmo2 backbone achieves state-of-the-art results among open-weight models in image, multi-image, and especially video understanding and grounding. On video-grounding, Molmo2 outperforms the open-weight Qwen3-VL (35.5 vs 29.6 accuracy on video counting), and surpasses proprietary Gemini 3 Pro in video pointing (38.4 vs 20.0 F1) and tracking (56.2 vs 41.1 J&F). Its three-stage training, bidirectional vision-token attention, packing, and token-weighting innovations are collectively responsible for these improvements (Clark et al., 15 Jan 2026). The architectural specification—including the frozen ViT, pooling connector, block-masked cross-modal decoder, and efficient data packing—serves as an open-source blueprint for competitive VLM backbone construction.

7. Architectural and Algorithmic Summary

Component	Key Properties	Parametric Details (4B model)
Vision encoder (ViT)	27 layers, patch size 14×14, d=1,152, frozen weights	~3.38M params
Vision–language connector	2×2 pooling (img), 3×3 (vid), 16-head attn, SwiGLU, 2-layer MLP	9,728-dim, ~57M params
LLM decoder (Qwen3-VL-4B)	36 layers, d=2560, 32 heads, SwiGLU, block-masked cross-modal attention	~4.0B params
Message-tree packing/masking	Custom attention mask, DP packing, branchwise causal text, bidirectional vision	Up to 3.8 examples/16K sequence
Token weighting	Loss reweighting by answer length/task type	$F=128$9 as defined above
Training stages	Pretrain, SFT, long-context finetune, mixture of image/video/pointing	Seq length up to 36,864; batch 128

All hyperparameters and architectural features above are directly specified in the Molmo2 release (Clark et al., 15 Jan 2026). This configuration, combined with innovative packing and training strategies, defines the state of practice for open-weight video-LLM backbones and serves as a reference point for further research and development.