Pixtral 12B: Open Multimodal LLM

• Pixtral 12B is a cutting-edge, open-source multimodal language model merging a 12-billion-parameter text decoder with a 400M-parameter vision encoder.
• It achieves state-of-the-art performance on document and diagram tasks, processing extended contexts up to 131K tokens with impressive benchmark scores.
• Despite its strong results, evaluations highlight challenges in multi-image reasoning, uncertainty calibration, and safety robustness under adversarial prompts.

Searching arXiv for relevant papers on Pixtral 12B and recent evaluations. arxiv_search(query="Pixtral 12B Mistral AI vision LLM evaluation", max_results=10, sort_by="relevance") arxiv_search(query="(Agrawal et al., 2024) Pixtral 12B", max_results=5, sort_by="relevance") arxiv_search(query="(Jegham et al., 23 Feb 2025) visual reasoning Pixtral 12B", max_results=5, sort_by="relevance") arxiv_search(query="(Ford et al., 4 Feb 2026) alignment drift multimodal LLMs Pixtral 12B", max_results=5, sort_by="relevance") Pixtral 12B is a 12-billion-parameter, Apache 2.0-licensed, decoder-only multimodal LLM (MLLM) open-sourced by Mistral AI. It was released on September 17, 2024, and is described as retaining the 12 B-parameter decoder of Mistral NeMo 12B while augmenting it with a 400 M-parameter vision encoder, Pixtral-ViT, trained from scratch (Agrawal et al., 2024). The model is designed to understand both natural images and documents, to ingest images at their natural resolution and aspect ratio, and to process any number of images in its long context window of 128K tokens; the paper also specifies a decoder context window of 131 072 tokens and an encoder context window of 4096 visual tokens (Agrawal et al., 2024). Subsequent evaluation studies positioned Pixtral 12B as a notable open multimodal model with strong performance on some visual-language benchmarks, while also identifying limitations in multi-image reasoning stability, rejection calibration, and safety under adversarial prompting (Jegham et al., 23 Feb 2025, Ford et al., 4 Feb 2026).

1. Release, positioning, and stated capabilities

The Pixtral 12B paper presents the model as the first open model at its scale that simultaneously matches state-of-the-art text-only performance on MATH, HumanEval, and MT-Bench while providing leading multimodal understanding, surpassing both similar-size open models such as Llama-3.2 11B and Qwen-2-VL 7B, and much larger baselines such as Llama-3.2 90B on VQAv2, DocVQA, MMMU, MathVista, and MM-MT-Bench (Agrawal et al., 2024). It is released under Apache 2.0, with model weights and inference code provided through mistral-inference, and evaluation code and MM-MT-Bench provided through mistral-evals (Agrawal et al., 2024).

The model is described as able to process any number of images in a long context window and as not compromising on natural language performance in order to excel in multimodal tasks (Agrawal et al., 2024). The paper further states that Pixtral 12B substantially outperforms other open models of similar sizes and also outperforms much larger open models like Llama-3.2 90B while being 7x smaller (Agrawal et al., 2024).

A plausible implication is that Pixtral 12B was positioned not merely as a vision-language specialization layer atop a weaker text backbone, but as a unified foundation model whose multimodal functionality is intended to coexist with strong general text performance. That interpretation is consistent with the paper’s framing, but the specific architectural and benchmark claims should be taken directly from the release paper rather than generalized beyond the reported settings.

2. Architecture and multimodal tokenization

Pixtral 12B retains the decoder of Mistral NeMo 12B and adds a 400 M-parameter vision encoder, Pixtral-ViT, trained from scratch (Agrawal et al., 2024). The decoder has model dimension $d = 5120$, number of layers $L = 40$, attention heads $h = 32$, head dimension $d_h = 128$, $n_{kv\_heads} = 8$, feed-forward hidden dimension $d_{ff} = 14\,336$ with GeLU activation, context window $= 131072$ tokens, and vocab size $= 131072$ (Agrawal et al., 2024). The vision encoder has model dimension $d = 1024$, number of layers $L = 24$, attention heads $L = 40$0, head dimension $L = 40$1, $L = 40$2, feed-forward hidden dimension $L = 40$3 with gated FFN, patch size $L = 40$4 pixels, and context window $L = 40$5 visual tokens (Agrawal et al., 2024).

The decoder layers follow the standard causal transformer block. For an input token embedding $L = 40$6, the paper specifies multi-head self-attention with $L = 40$7, $L = 40$8, $L = 40$9,

$h = 32$0

and

$h = 32$1

The feed-forward sublayer is

$h = 32$2

with layer updates

$h = 32$3

These formulas are reported in the model paper as the decoder’s standard causal transformer block (Agrawal et al., 2024).

Pixtral-ViT is a 24-layer ViT pre-trained from scratch with four stated innovations: variable-aspect-ratio patching, RoPE-2D positional encoding, gated FFN, and sequence packing with block-diagonal attention masks (Agrawal et al., 2024). Images are split into non-overlapping $h = 32$4 patches; to preserve aspect ratio, the number of patches per row and column adapts to the image dimensions; between rows of patches, a special token [IMAGE BREAK] is inserted, and [IMAGE END] is appended after the last patch row (Agrawal et al., 2024).

For RoPE-2D, the paper states that relative rotary position encodings generalize to any image size. For a patch embedding vector $h = 32$5 at 2D location $h = 32$6,

$h = 32$7

where $h = 32$8 is a block-diagonal matrix of $h = 32$9 rotation blocks, and the paper notes that Equation (1) demonstrates the “relative” property: inner products depend only on $d_h = 128$0, not absolute $d_h = 128$1 (Agrawal et al., 2024). Multiple images, or mixtures of images and text tokens, can be concatenated in a batch; the self-attention mask is block-diagonal to prevent cross-image leakage (Agrawal et al., 2024).

The final-layer outputs of Pixtral-ViT, which are 1024-dimensional, are passed through a 2-layer MLP with GeLU to project to the decoder embedding dimension of 5120 (Agrawal et al., 2024). These image tokens are interleaved along the long decoder context window, which uses causal RoPE-1D for position encoding in the decoder, and the paper emphasizes that no special architectural changes are needed because images appear as additional tokens in the decoder’s causal self-attention (Agrawal et al., 2024).

3. Training description and benchmarked performance

The Pixtral 12B paper does not disclose precise pre-training datasets, data scales, filtering procedures, or optimizer hyperparameters (Agrawal et al., 2024). It states only that Pixtral is first pretrained on “large-scale, interleaved image and text documents,” followed by multimodal instruction tuning (Agrawal et al., 2024). The paper also states that no explicit loss functions beyond standard causal LM training are detailed, and that contrastive objectives are not mentioned (Agrawal et al., 2024).

On multimodal benchmarks, the paper reports the following scores for Pixtral 12B: MathVista 58.3, MMMU 52.0, ChartQA 81.8, DocVQA 90.7, VQAv2 78.6, MM-MT-Bench 6.05, and LMSys Vision ELO 1076 (Agrawal et al., 2024). In the same table, Qwen-2-VL 7B is reported at 53.7 on MathVista, 48.1 on MMMU, 41.2 on ChartQA, 94.5 on DocVQA, 75.9 on VQAv2, 5.45 on MM-MT-Bench, and 1040 on LMSys Vision ELO; Llama-3.2 11B is reported at 24.3, 23.0, 14.8, 91.1, 67.1, 4.79, and 1032; Llama-3.2 90B at 49.1, 53.7, 33.8, 85.7, 67.0, 5.50, and 1071; Gemini-1.5 Flash 8B at 56.9, 50.7, 78.0, 79.5, 65.5, 5.93, and 1111; Claude-3 Haiku at 44.8, 50.4, 69.6, 74.6, 68.4, 5.46, and 1000; and GPT-4o at 64.6, 68.6, 85.1, 88.9, 77.8, 7.72, and 1208 (Agrawal et al., 2024).

The paper states that Pixtral 12B leads all open models of comparable size by large margins, with examples including +40 points on ChartQA versus Qwen-2-VL 7B, and that it outperforms Llama-3.2 90B on ChartQA by +48 points and on MathVista by +9 points (Agrawal et al., 2024). On text-only benchmarks, the reported scores are MT-Bench 7.68, MMLU (5-shot) 69.2, MATH (Maj@1) 48.1, and HumanEval (Pass@1) 72.0 (Agrawal et al., 2024). The same table lists Llama-3.2 11B at 7.51, 68.5, 48.3, and 62.8; Qwen-2-VL 7B at 6.41, 68.5, 27.9, and 62.2; LLaVA-OneVision 7B at 6.94, 67.9, 38.6, and 65.9; and Molmo-D 7B at 4.53, 61.2, 10.2, and 3.7 (Agrawal et al., 2024).

The paper concludes from these results that Pixtral does not sacrifice text-only capabilities and can serve as a drop-in replacement for pure-text deployments (Agrawal et al., 2024). It also reports a vision-encoder ablation in which Pixtral-ViT outperforms a strong CLIPA baseline on document-understanding tasks, especially at high resolution, while matching CLIPA on VQAv2 (Agrawal et al., 2024). Flexible parsing ablations are reported to show that Pixtral’s performance remains stable under increasingly loose answer-matching metrics (Agrawal et al., 2024).

MM-MT-Bench, introduced alongside the model, consists of 92 multimodal, multi-turn conversations spanning five image categories: charts (21), tables (19), PDF pages (24), diagrams (20), and misc (8) (Agrawal et al., 2024). Its turn distribution is 69 single-turn, 18 two-turn, four three-turn, and one four-turn; each turn is rated 1–10 by an LLM judge on correctness and completeness given full history and reference answers; and the paper reports a Pearson correlation with LMSys Vision ELO of 0.91 (Agrawal et al., 2024).

4. Multi-image reasoning evaluation

A later study on visual reasoning evaluation argues that traditional evaluations of multimodal LLMs have been limited by their focus on single-image reasoning, and introduces a benchmark integrating multi-image reasoning tasks with rejection-based evaluation and positional bias detection (Jegham et al., 23 Feb 2025). Within that evaluation, Pixtral 12B is assessed across eight visual reasoning tasks derived from MUIRBench. The benchmark consists of 120 questions paired with 376 images, with an average of 3.13 images per question, and includes 40 unanswerable questions (Jegham et al., 23 Feb 2025).

The eight tasks are Difference Spotting, Diagram Understanding, Cartoon Understanding, Counting, Geographic Understanding, Image-Text Matching, Ordering, and Visual Retrieval (Jegham et al., 23 Feb 2025). Each question appears in three variants: the original answer order, a re-ordered choice list, and an unanswerable form in which the correct choice is removed (Jegham et al., 23 Feb 2025). To quantify stability across reordered variants, the study computes entropy for each question-group $d_h = 128$2 as

$d_h = 128$3

where $d_h = 128$4 is the fraction of times option $d_h = 128$5 was selected over $d_h = 128$6 choices (Jegham et al., 23 Feb 2025).

On answerable questions, Pixtral 12B answered correctly 51.7% of the time (Jegham et al., 23 Feb 2025). The same comparison reports ChatGPT-o1 at 82.5%, Gemini 2.0 Flash Experimental at 70.8%, QVQ-72B-Preview at 65.8%, Grok 3 at 55.8%, and Janus 7B at 43.3% (Jegham et al., 23 Feb 2025). Of the 40 unanswerable questions, Pixtral 12B correctly abstained with “None of the choices provided” on 30%, as compared to 70% for ChatGPT-o1 and 85.5% for QVQ-72B-Preview (Jegham et al., 23 Feb 2025). Its overall abstention rate was 15%, while the study identifies 33% as the ideal because that is the true proportion of unanswerable questions (Jegham et al., 23 Feb 2025).

Averaging $d_h = 128$7 over all 120 groups yields a mean entropy of 0.557 bits for Pixtral 12B (Jegham et al., 23 Feb 2025). The study reports ChatGPT-o1 at 0.135 bits, ChatGPT-4o at 0.216 bits, Grok 3 at 0.256 bits, and Janus 7B at 0.8392 bits (Jegham et al., 23 Feb 2025). The paper explicitly interprets higher entropy as greater variability or instability, and describes Pixtral’s value as mid-range, revealing moderate sensitivity to answer-order shuffles (Jegham et al., 23 Feb 2025).

The task-level pattern is uneven. The study identifies Diagram Understanding as a domain of relative strength for Pixtral 12B, with 85% accuracy, and Cartoon Understanding as another relative strength at 62.5% (Jegham et al., 23 Feb 2025). Difference Spotting is identified as a domain of relative weakness, at 21.4%, and Image-Text Matching is reported at 57.1%, below Gemini at 82.1% and QVQ-72B-Preview at 85.7% (Jegham et al., 23 Feb 2025). The study concludes that Pixtral under-abstains, often over-committing to an answer even when none is correct, and that the combination of modest overall accuracy and elevated entropy indicates that correct answers are not reliably robust to minor rephrasings or re-orderings of alternatives (Jegham et al., 23 Feb 2025).

This evaluation is significant because it separates raw answer accuracy from rejection accuracy and positional-bias sensitivity. A plausible implication is that Pixtral 12B’s strengths on conventional document and diagram benchmarks do not automatically transfer to stable multi-image comparative reasoning, especially when answer-order perturbations and unanswerable variants are introduced.

5. Safety evaluation and longitudinal alignment drift

The paper “Alignment Drift in Multimodal LLMs” conducts a two-phase longitudinal harmlessness evaluation using a fixed benchmark of 726 adversarial prompts authored by 26 professional red teamers (Ford et al., 4 Feb 2026). In Phase 1, Pixtral 12B is evaluated alongside GPT-4o, Claude Sonnet 3.5, and Qwen VL Plus; in Phase 2, successors including Pixtral Large are evaluated on the identical prompt set (Ford et al., 4 Feb 2026). The prompts are evenly split into 363 text-only and 363 multimodal variants, targeting illegal activity, disinformation, and unethical behavior; multimodal attacks use embedded text in images, benign images with harmful captions, or harmful images with benign captions (Ford et al., 4 Feb 2026). All prompts are executed in a single-turn setting with context reset between calls (Ford et al., 4 Feb 2026).

Pixtral 12B produced 2,904 responses, which 17 native-English annotators rated on a five-point Likert scale from 1 = Not at all harmful through 5 = Extremely harmful, yielding 47,408 total annotations (Ford et al., 4 Feb 2026). The study defines attack success rate as the proportion of prompts on which a model produces any harmful content. For Pixtral 12B in Phase 1, the reported ASR values are 0.61 for multimodal prompts, 0.63 for text-only prompts, and 0.62 overall (Ford et al., 4 Feb 2026). For Pixtral Large in Phase 2, the corresponding ASR values are 0.50, 0.60, and 0.55 (Ford et al., 4 Feb 2026). The paper therefore reports that Pixtral family ASR decreased by 7 percentage points across generations, even though it remained the most vulnerable family in both phases (Ford et al., 4 Feb 2026).

The paper also models five-point harm ratings via a cumulative link mixed model and reports raw annotation distributions that permit direct computation of average harm per prompt (Ford et al., 4 Feb 2026). For Pixtral 12B, the distribution of non-refusal harm ratings is reported as Level 1: 4,460 (37.6%), Level 2: 1,067 (9.0%), Level 3: 1,874 (15.8%), Level 4: 1,786 (15.1%), and Level 5: 2,665 (22.5%) (Ford et al., 4 Feb 2026). The study states that these sum to $d_h = 128$8 ratings with a weighted total of $d_h = 128$9, giving an average human harm rating of approximately $n_{kv\_heads} = 8$0 (Ford et al., 4 Feb 2026). Its CLMM-derived estimated marginal means are reported as $n_{kv\_heads} = 8$1 for multimodal and $n_{kv\_heads} = 8$2 for text-only prompts, while Pixtral Large shifts to $n_{kv\_heads} = 8$3 and $n_{kv\_heads} = 8$4 respectively (Ford et al., 4 Feb 2026).

The modality-specific pattern is explicit: Phase 1 results highlight that Pixtral 12B was particularly sensitive to text-only attacks, with ASR 0.63 versus 0.61 for multimodal prompts, and the paper states that “Pixtral 12B showed the largest modality gap, with text-only prompts elevating harmfulness more sharply than multimodal prompts” (Ford et al., 4 Feb 2026). Phase 2 preserved this general pattern for Pixtral Large, with 0.60 text-only versus 0.50 multimodal (Ford et al., 4 Feb 2026).

The authors conclude that Pixtral 12B, and by extension the Pixtral family, remains the highest-risk MLLM despite incremental safety work (Ford et al., 4 Feb 2026). They recommend embedding longitudinal, fixed adversarial benchmarks in evaluation pipelines; accounting separately for default refusal behavior; expanding beyond U.S. English single-turn static-image prompts; and developing modelling approaches that treat refusal as an independent safety outcome rather than combining it with “Not at all harmful” (Ford et al., 4 Feb 2026). They also note limitations including opaque API interfaces, a fixed set of 726 prompts covering only three harm categories, single-turn evaluation, and conflation of refusal with benign output on the harm scale (Ford et al., 4 Feb 2026).

6. Interpretation, limitations, and improvement directions

Taken together, the cited papers present a mixed but technically coherent profile. The release paper describes Pixtral 12B as an open multimodal model with a from-scratch vision encoder, native handling of arbitrary image resolutions and aspect ratios, a 128K or 131 072-token long context, and benchmark results that compare favorably against same-scale open models and, in some cases, much larger models (Agrawal et al., 2024). The multi-image reasoning study, however, reports only 51.7% overall accuracy on its answerable questions, 30% rejection accuracy on unanswerable questions, a 15% abstention rate against a 33% ground-truth proportion of unanswerable questions, and mean entropy of 0.557 bits, with particularly weak performance on Difference Spotting at 21.4% (Jegham et al., 23 Feb 2025). The safety study further reports an overall ASR of 0.62 and identifies Pixtral as the most vulnerable family in both phases of its longitudinal benchmark, despite a decrease to 0.55 for Pixtral Large (Ford et al., 4 Feb 2026).

These results do not imply a contradiction so much as benchmark dependence. The release paper emphasizes multimodal benchmark suites such as MathVista, MMMU, ChartQA, DocVQA, VQAv2, and MM-MT-Bench (Agrawal et al., 2024). The later evaluation work stresses multi-image contextual reasoning, rejection behavior, positional bias, and adversarial harmlessness, all of which probe properties that are not reducible to conventional task accuracy (Jegham et al., 23 Feb 2025, Ford et al., 4 Feb 2026). This suggests that Pixtral 12B’s performance is strong in some regimes—especially document-oriented and diagram-oriented settings explicitly reflected in the model paper and in the 85% Diagram Understanding score from the visual reasoning study—while remaining limited in uncertainty calibration, answer-order robustness, and safety alignment under attack (Agrawal et al., 2024, Jegham et al., 23 Feb 2025).

The visual reasoning study proposes several concrete directions for improvement: incorporate uncertainty-aware training that explicitly teaches the model to output “None” on unanswerable inputs and tune decision thresholds toward the true 33% abstention rate; randomize option order aggressively during fine-tuning and penalize answer-order sensitivity, possibly via an entropy-based regularizer; enrich difference-spotting examples and image-text matching pairs, especially high-resolution and multi-view scenarios; add a consistency loss over reordered variants to minimize per-group entropy $n_{kv\_heads} = 8$5; and adopt visual “thought chains” to guide stepwise comparisons across images (Jegham et al., 23 Feb 2025). The safety study, by contrast, focuses on evaluation methodology and recommends longitudinal fixed adversarial benchmarks, separate accounting for refusal behavior, broader multilingual and multi-turn assessment, and a tri-categorical safety framework that rewards substantive, harmless compliance rather than blanket refusal (Ford et al., 4 Feb 2026).

A common misconception is that parameter count alone determines performance. The visual reasoning study explicitly states that model size is not the sole determinant of performance, citing Grok 3 underperformance despite its substantial parameter count (Jegham et al., 23 Feb 2025). Within the narrower context of Pixtral 12B, a related caution is that strong scores on standard multimodal benchmarks do not guarantee strong performance on multi-image stability or adversarial safety. The available evidence instead supports a more differentiated characterization: Pixtral 12B is an open, architecturally distinctive MLLM with strong benchmarked capability in several multimodal and text-only tasks, but with documented weaknesses in rejection calibration, positional-bias susceptibility, reasoning stability under reordered alternatives, and vulnerability under adversarial prompting (Agrawal et al., 2024, Jegham et al., 23 Feb 2025, Ford et al., 4 Feb 2026).