Zamba2-VL: Hybrid Vision-Language Models
- Zamba2-VL is a hybrid vision-language model family that integrates Mamba2 state-space layers with a few Transformer blocks to balance efficient recurrence with precise attention.
- It employs a Qwen2.5-VL ViT encoder and a two-layer MLP adapter to convert visual tokens into the language space, significantly reducing prefill cost and latency.
- A three-stage training curriculum with a targeted data mixture drives competitive performance on image, document, and counting benchmarks while maintaining lower TTFT and memory requirements.
Searching arXiv for the cited Zamba2-VL report and closely related backbone/VLM papers mentioned in the provided data. arXiv search query: "Zamba2-VL Technical Report" Zamba2-VL is a family of open vision-LLMs that extends Zyphra’s Zamba2 hybrid language-model backbone into the multimodal setting. It is presented as evidence that a dense Transformer LLM is not required to build a strong VLM: instead, a hybrid architecture built from Mamba2 state-space layers plus a small number of shared transformer blocks can preserve most of the quality of a comparable Transformer while dramatically reducing prefill cost, latency, and decoding-state memory. The suite comprises 1.2B, 2.7B, and 7B models and is evaluated on image understanding, reasoning, OCR, grounding, and counting benchmarks, where it is reported as competitive with leading Transformer-based open-weight VLMs of comparable scale while substantially outperforming prior SSM-based and hybrid VLMs such as VL-Mamba, Cobra, and mmMamba (Shapourian et al., 29 May 2026).
1. Conceptual position within multimodal modeling
Zamba2-VL follows a standard LLaVA-style pipeline: an image is encoded by a vision backbone, projected into the language embedding space by a lightweight adapter, and then concatenated with text tokens for autoregressive decoding by the LLM. Its distinguishing feature is that the LLM is Zamba2 rather than a dense Transformer. Zamba2 interleaves Mamba2 layers with a small number of shared Transformer blocks, and the shared blocks are lightly specialized at insertion points through LoRA projectors (Shapourian et al., 29 May 2026).
The report frames this design as a specific tradeoff. The Mamba2 layers perform the bulk of the sequence modeling in linear time and with a fixed-size recurrent state, while the shared attention blocks reintroduce the content-addressable retrieval that pure SSMs tend to lose. In this formulation, pure recurrence is fast but can struggle with precise lookup and in-context retrieval, whereas a few attention blocks restore that capability without recovering the full quadratic cost profile of a standard Transformer. The paper explicitly presents this as the main reason Zamba2 is a better multimodal backbone than a pure Mamba model (Shapourian et al., 29 May 2026).
This positioning also clarifies what Zamba2-VL is not. It is not described merely as an LLaVA-style wrapper around an existing LLM, but as a demonstration that hybrid sub-quadratic backbones can work cleanly for vision-language workloads and are especially attractive when context length, memory, or time-to-first-token (TTFT) matter. A plausible implication is that the report seeks to shift multimodal architecture design away from the assumption that strong VLMs must inherit fully dense Transformer backbones.
2. Architectural composition
At the vision front end, Zamba2-VL uses the Qwen2.5-VL ViT as its vision encoder, chosen because it performed best in ablations. The report attributes its strength to two properties: 2D Rotary Position Embeddings applied directly to image patches and native dynamic-resolution processing, both of which help preserve fine spatial detail, especially in document and text-heavy images. Images are resized so height and width are multiples of 28 while preserving aspect ratio, then fed into the ViT with a patch size (Shapourian et al., 29 May 2026).
A two-layer MLP adapter then pools each window of patch embeddings into one vector and projects it to the LLM embedding dimension. This reduces the vision-token count by and aligns the visual features to the text embedding space. The resulting visual tokens are interleaved with text tokens and processed autoregressively by the hybrid backbone (Shapourian et al., 29 May 2026).
The language-side architecture remains the Zamba2 backbone architecture intact. The report repeatedly describes its functional decomposition in compact terms: Mamba2 layers provide linear-time sequence processing and a fixed recurrent state, while the shared Transformer blocks provide the attention-like retrieval missing from pure SSMs. In one compact sense, the efficiency claims are said to boil down to replacing an attention-dominated prefill/decode path with a largely recurrent one, while preserving enough attention to avoid the retrieval weaknesses that pure SSMs exhibit (Shapourian et al., 29 May 2026).
The architecture therefore couples a conventional multimodal tokenization interface with a nonstandard sequence model. This suggests that the central novelty lies less in the image-text fusion protocol than in the claim that hybrid SSM–Transformer backbones can serve as a general-purpose multimodal substrate.
3. Training curriculum and supervision formats
Training is organized as a three-stage curriculum (Shapourian et al., 29 May 2026):
| Stage | Trainable components / scale | Purpose |
|---|---|---|
| Stage 1 | Only the MLP connector; low-resolution caption data from LLaVA-ReCap-558K | Adapter alignment |
| Stage 2 | Full multimodal pretraining on roughly 30B tokens | Capability acquisition across heterogeneous multimodal tasks |
| Stage 3 | Supervised fine-tuning on roughly 20B tokens | Curated instruction following with emphasis on multi-turn dialogue, grounding, and instruction following |
In Stage 1, only the MLP connector is trained, while the vision encoder and Zamba2 backbone remain frozen. This stage uses low-resolution caption data from LLaVA-ReCap-558K, with loss computed over text tokens only, to initialize the image-to-language interface. Stage 2 performs full multimodal pretraining on roughly 30B tokens from a heterogeneous mixture covering captioning, VQA, OCR/document understanding, chart and figure understanding, grounding, and multimodal reasoning. Stage 3 applies supervised fine-tuning on roughly 20B tokens of curated instruction-following data, with more emphasis on multi-turn dialogue, grounding, and instruction following (Shapourian et al., 29 May 2026).
Across stages, the training uses a lightweight chat template with newly introduced <|vision_start|> and <|vision_end|> tokens around each image, while turn boundaries are signaled using the tokenizer’s normal BOS/EOS markers. From Stage 2 onward, the cross-entropy loss is computed only over answer tokens, not on the input question or context. The report states that this makes longer responses count more heavily and reflects the model’s role as a conditional generator. Because that choice reduces the effective gradient signal per step, the authors compensate with a relatively large nominal batch size (Shapourian et al., 29 May 2026).
The grounding supervision is resolution-independent. Points are emitted in XML-style inline tags with normalized percentage coordinates, while boxes use normalized coordinates. The report emphasizes that these formats match the dynamic-resolution image pipeline. This suggests a deliberate alignment between the supervision schema and the image-processing stack rather than an ad hoc output convention.
4. Data mixture and capability targeting
The data mixture is described as intentionally broad and capability-oriented. It is built from open-source corpora spanning image understanding and captioning, document/OCR, grounding and perception, image-grounded QA, multimodal reasoning, and a small amount of text-only math data. The training corpus includes FineVision, PixMo-Cap, MAmmoTH-VL, M4-Instruct, The Cauldron, PDFA, UCSF Industry Documents Library, DocMatix, PixMo-Docs, UniChart, ECD-10K, PixMo-Point, PixMo-Count, MultiUI, OS-Atlas, UGround, AutoGUI, Aria-UI, OpenImages, Objects365, M³CoT, Geometry3K, Geo170K, ViRL39K, and GSM8K (Shapourian et al., 29 May 2026).
A notable aspect of the mixture is that document and OCR data are upsampled relative to natural frequency. The stated motivation is empirical: under matched training, Zamba2 backbones already compete well on general image understanding but lag on text-heavy document/OCR tasks, and upsampling those sources closes that gap in the final model. This creates a direct connection between ablation findings and corpus design (Shapourian et al., 29 May 2026).
The benchmark profile reported later is consistent with that training choice. The model is said to be consistently strong on document and chart benchmarks, which the report explicitly aligns with the OCR/document upsampling strategy. A plausible implication is that the system’s capability distribution is not an accidental byproduct of scale, but a consequence of targeted data allocation designed to counter known architectural weaknesses.
5. Model scales, positional nuances, and implementation constraints
Zamba2-VL is released in three sizes: 1.2B, 2.7B, and 7B. These correspond to the three Zamba2 backbones and are intended to span the small-to-mid-scale deployment regime. The 1.2B and 2.7B models are identified as the most relevant to edge or on-device settings because the efficiency gap versus dense Transformer baselines is largest there, while the 7B model is presented as the strongest overall and as evidence that the hybrid recipe still scales competitively at a more standard open-model size (Shapourian et al., 29 May 2026).
The 2.7B backbone differs somewhat from the others in its positional encoding setup. The report states that the 1.2B and 7B variants use standard 1D RoPE in shared attention blocks, while the 2.7B has no rotary embedding there due to implementation and timing details during the original Zamba2 architecture search. This variant is not presented as fundamentally different, but it does show a weaker profile in some ablations (Shapourian et al., 29 May 2026).
Several implementation details are directly tied to efficiency. The model retains the 4k context length inherited from the base Zamba2 models, and the report notes that increasing it would require a separate context-extension stage. It also keeps the backbone’s positional setup unchanged rather than adopting a multimodal RoPE variant, because such changes were deemed compute- and data-intensive relative to the available training budget. The two-layer MLP adapter compresses blocks of visual patches before injection into the LLM, reducing both sequence length and adapter complexity (Shapourian et al., 29 May 2026).
These choices indicate that the technical report prioritizes deployable efficiency under finite training budgets rather than exhaustive multimodal architectural redesign. This suggests that Zamba2-VL should be understood as a constrained but practical extension of Zamba2, rather than as a maximalist multimodal pretraining program.
6. Empirical profile, ablations, and deployment implications
Zamba2-VL is evaluated on AI2D, ChartQA, DocVQA, InfoVQA, TextVQA, OCRBench, VQAv2, MathVista, MMMU, SEED, Blink, RealWorldQA, CountBenchQA, and PixMoCount. Across these tasks, the report’s broad claim is that the models are competitive with leading open-weight Transformer VLMs of comparable scale, including the Molmo2, Qwen3-VL, and InternVL3.5 families, while substantially outperforming prior SSM-based and hybrid VLMs such as VL-Mamba, Cobra, and mmMamba (Shapourian et al., 29 May 2026).
The clearest standout area is counting. Zamba2-VL-1.2B is reported to score 62.5 on PixMoCount, compared with 32.8 for InternVL3.5-1B and 17.7 for PerceptionLM-1B. The 2.7B and 7B variants reach 82.5 and 85.3 on PixMoCount, respectively, and are also reported as competitive on CountBenchQA. The report further states that the model is consistently strong on document and chart benchmarks. Its weakest relative area is multi-step reasoning, especially MMMU and MathVista, where it trails the strongest Transformer baselines more noticeably; the paper interprets this as a consequence of emphasizing capability acquisition rather than reasoning-specialized RL post-training, which is left for future work (Shapourian et al., 29 May 2026).
The controlled ablations are central to the paper’s argument. With the language backbone fixed to Zamba2-2.7B, CLIP and SigLIP-2 are compared against the Qwen2.5-VL ViT, and the Qwen2.5-VL ViT clearly wins. With the vision encoder held fixed, Zamba2 is compared against Llama-3.x references at roughly matched scales. The report states that the hybrid backbone trails the Transformer by about 5.5 points on average at the smallest scale, by more at 2.7B, but is nearly indistinguishable from Llama-3.1-8B at the top end. The per-benchmark breakdown suggests that the gap is not uniform: Zamba2 does especially well on AI2D, ChartQA, TextVQA, and RealWorldQA, while the main weakness is on DocVQA and InfoVQA in the ablation setting. The authors argue that this validates the decision to upsample document and OCR data in the main training run. The 2.7B variant is also described as the weakest of the Zamba2 family on the ablation suite, likely because of its lack of RoPE in the shared attention blocks, though that weakness shrinks in the full multimodal training pipeline (Shapourian et al., 29 May 2026).
The most important comparative story in the report is score versus latency rather than raw score alone. The paper plots average benchmark score against TTFT measured on a 32k-token prefill and places Zamba2-VL in the favorable upper-left region at all three scales. It reports roughly an order-of-magnitude lower TTFT than the nearest Transformer baseline at comparable accuracy. This is attributed to the Zamba2 backbone’s linear-time prefill and small recurrent state. During autoregressive decoding, a Transformer must maintain a KV cache that grows with generated length, whereas Zamba2 replaces that with a fixed-size state. The report stresses that long multimodal prefill is exactly where quadratic attention cost becomes especially painful because images can contribute thousands of visual tokens, making the efficiency advantage most pronounced in the multimodal regime and especially at the 1.2B and 2.7B scales (Shapourian et al., 29 May 2026).
This deployment framing also addresses a common misconception: that multimodal competitiveness necessarily requires a dense Transformer backbone. The report argues instead that hybrid SSM–Transformer backbones are a credible alternative to dense Transformers for multimodal AI because they preserve the parts of attention that matter, offload most computation to a sub-quadratic recurrence, and make multimodal systems materially cheaper to serve. At the same time, the weaker relative performance on MMMU and MathVista indicates that the approach does not eliminate the tradeoffs associated with reasoning-specialized post-training.