Phi-4-Reasoning-Vision-15B

• Phi-4-Reasoning-Vision-15B is an open multimodal autoregressive model with 15B parameters that fuses visual and textual data via a mid-fusion Transformer architecture.
• The model employs dynamic-resolution vision encoding and explicit reasoning tokens (<think> and <nothink>), achieving competitive benchmark results with lower computational costs.
• Rigorous data curation and staged multimodal training enhance its scientific, mathematical, and UI reasoning, making it suitable for real-time research and practical applications.

Phi-4-Reasoning-Vision-15B is an open-weight, 15B-parameter multimodal autoregressive LLM, architected to excel in scientific, mathematical, and user-interface reasoning that requires coherent understanding across both visual and textual modalities. The model advances compact vision-language architectures by integrating a mid-fusion approach, high-resolution visual perception, rigorous data curation, and explicit control over reasoning style, targeting strong performance at much lower computational cost compared to larger proprietary models (Aneja et al., 4 Mar 2026).

1. Architectural Foundations

Phi-4-Reasoning-Vision-15B employs a mid-fusion decoder-only Transformer, interleaving “soft” visual tokens with text tokens through a unified autoregressive stack. The architecture consists of a SigLIP-2 (NaFlex variant) vision encoder operating with dynamic (image-dependent) resolution: input images are tiled into 16×16-pixel patches, producing up to 3,600 visual tokens per image, each embedded into a 1024-dimensional space. These visual descriptors are projected into the model’s embedding space using a two-layer MLP with GELU nonlinearity, given by

$$H = \mathrm{GELU}(V W_1 + b_1), \quad V' = H W_2 + b_2$$

where $V \in \mathbb{R}^{n \times d_v}$ are encoder outputs.

Text is tokenized via a ∼64k subword BPE vocabulary. The model backbone is a 15B-parameter Phi-4-Reasoning Transformer (64 layers, $d_\mathrm{model}=12,288$, 96-way attention, rotary embeddings), processing sequences of both text and vision tokens.

Reasoning modes are selected via explicit special tokens: <think> (chain-of-thought reasoning) and <nothink> (direct answer mode). This mechanism enables the model to deliver verbose reasoning traces or concise outputs on demand, supporting both analytic evaluation and real-time deployments (Aneja et al., 4 Mar 2026).

2. Data Collection, Curation, and Augmentation

Training data are assembled in a three-stage pipeline focused on high quality and multimodal diversity. Initial MLP pretraining (2M high-purity image/caption pairs) is followed by a large-scale multimodal instruction tuning stage (62.8M samples, 188.5B tokens), drawing from open-source datasets spanning image–caption, question–answer, chart and diagram QA, document OCR, object detection, and scientific math OCR problems. Stage 3 (3.2M samples, 12B tokens) further introduces multi-image reasoning and robustness/RAI data.

Data curation involves systematic manual inspection and correction: datasets are classified as excellent, good but erroneous, low-quality (questions/images), or high-quality with formatting problems. Actions include regenerating or verifying answers via GPT-4o + o4-mini majority voting, excluding poor data, and programmatic error correction (reasoning block, answer-format fixes).

Synthetic augmentation pipelines generate detailed captions for scientific/technical images, multi-image tasks (scrambled caption matching, “what’s changed?”), and prompt diversification via paraphrase. This results in a dataset where approximately 20% of training samples feature explicit chain-of-thought reasoning.

A strong emphasis is placed on data quality, with ablation results establishing data curation—not scale—as the primary driver of scientific reasoning accuracy (Aneja et al., 4 Mar 2026).

3. Training Strategy and Efficiency

The training regime follows three distinct epochs:

1. MLP Pretraining: 2M samples targeting vision–language alignment.
2. Instruction Tuning: 188.5B tokens over 62.8M single-image tasks (mixed domains), joint tuning of all model components.
3. Long-Context/Multi-Image: 3.2M samples crafted for robustness, multi-image, and user interface generalization.

AdamW is used for all stages, with learning rates annealing by cosine schedules, bf16 mixed precision, and a global batch size of 1920 in main stages. Sequence lengths are progressively scaled up to 16,384.

A key architectural decision is the use of dynamic-resolution vision encoding: high-resolution, image-adaptive tokenization outperforms multi-crop strategies on fine-grained tasks (e.g., GUI, high-res synthetic benchmarks), as established by ablation on MathVista, ScreenSpot, and related benchmarks (Aneja et al., 4 Mar 2026).

4. Evaluation Benchmarks and Comparative Performance

Phi-4-Reasoning-Vision-15B is benchmarked across open structured reasoning tasks including MathVista, ChartQA, MMMU, AI2D, and ScreenSpot, with particular attention to chain-of-thought vs direct-answer performance. The evaluation regime enforces reasoning modes via special tokens: results are reported both in forced-chain-of-thought and forced-direct modes.

Selected benchmark results (forced modes, Table 4/5):

Benchmark	Direct (<nothink>)	Chain-of-thought (<think>)
MathVista_MINI	75.2	75.2
MMMU_VAL	54.3	54.3
ChartQA_TEST	83.3	–

Phi-4-Reasoning-Vision-15B outperforms or matches other open-weight models (e.g., Kimi-VL-A3B-Instruct, gemma-3-12B-it) and approaches leading performance on high-resolution and reasoning-intensive tasks, despite a much smaller parameter count. Mixed-mode default (no explicit token) is found to slightly increase overall performance, confirming that the mode-token mechanism is effectively learned.

Latency–accuracy and output-token–accuracy trade-offs indicate that Phi-4-Reasoning-Vision-15B achieves a new Pareto frontier for compact vision-LLMs, making it suitable for real-time use (e.g., desktop agents) (Aneja et al., 4 Mar 2026).

5. Techniques for Robust Multimodal Reasoning

Best practices codified during development address persistent limitations in multimodal models:

• Data quality supersedes dataset scale for scientific reasoning; careful filtering and error remediation outweigh the benefits of raw sample count.
• Native dynamic-resolution encoding in the vision frontend is critical for high-res diagram and GUI understanding.
• Mode-token training (∼20% chain-of-thought, ∼80% direct) allows the model to segment reasoning modes transparently, yielding both fast responses and traceable solutions.
• Explicit reasoning supervision: inclusion of human-like chain-of-thought samples encourages disentanglement of visual grounding from symbolic inference, a property supported by results from visual chain-of-thought methodologies on other vision–LLMs (Singh et al., 2023).
• Cross-modality projection with minimal trainable parameters (2-layer MLP), combined with a pre-trained, frozen or modestly adapted vision encoder, streamlines training and improves transferability to new visual domains.

6. Impact, Limitations, and Future Directions

Phi-4-Reasoning-Vision-15B demonstrates that rigorous data curation, architecture selection, and standardized multimodal reasoning protocols enable competitive performance from a moderate-scale (15B) multimodal model—making research-grade scientific reasoning accessible without requiring 100B+ parameter proprietary models.

Limitations center on persistent challenges in diagram–reasoning coupling, especially in physics, as highlighted in SeePhys (Xiang et al., 25 May 2025); models—including Phi-4-Reasoning-Vision-15B—still struggle with robust visual–symbolic binding and can default to text-based reasoning shortcuts when diagrams are optional. Performance lags in tasks demanding deep abstraction, such as ARC, or requiring precise visual feature parsing in cluttered or high-density inputs (Singh et al., 2023).

Recommended future directions include:

• Visual chain-of-thought augmentation: Refined prompting and model-internal modularization separating fact extraction from reasoning (Singh et al., 2023).
• Domain-specific curriculum: Stage-wise training on incrementally complex diagrams, explicit graph-based representations, and symbolic–neural integration, as adopted in leading academic benchmarks (Xiang et al., 25 May 2025).
• Expanded multilingual and multi-domain coverage, with augmented OCR capabilities.
• Hybrid neuro-symbolic extensions, such as Python-based computation modules or explicit constraint-checking, for enhanced mathematical and structural faithfulness (Singh et al., 2023).

Phi-4-Reasoning-Vision-15B offers a state-of-the-art paradigm for compact, open multimodal reasoning, driven by an overview of careful engineering, quality-centric data curation, and explicit control over reasoning modalities (Aneja et al., 4 Mar 2026).