Phi-4 Reasoning Vision 15B Overview

• Phi-4-reasoning-vision-15B is a mid-fusion multimodal transformer that combines high-resolution visual encoding with text for joint scientific and mathematical reasoning.
• It employs a dynamic-resolution vision encoder generating up to 3,600 tokens to achieve fine spatial grounding and improved performance on GUI and chart tasks.
• The model leverages explicit mode tokens for chain-of-thought and direct responses and uses a three-stage training curriculum to balance efficiency and accuracy.

Phi-4-reasoning-vision-15B is a 15-billion-parameter, open-weight, mid-fusion multimodal transformer optimized for efficient joint vision and reasoning tasks. Developed to provide a practical blueprint for compact, high-performance multimodal reasoning agents, the model processes high-resolution visual inputs and text interleaved within a single transformer. It is specifically tuned for scientific and mathematical reasoning as well as complex user interface understanding, combining architectural innovations with a rigorously curated training pipeline. Ablation and benchmarking demonstrate that data quality and dynamic-vision encoding are principal determinants of performance, particularly for high-resolution grounding. Released as open weights alongside code and evaluation logs, the model is intended to guide next-generation multitask multimodal agent development (Aneja et al., 4 Mar 2026).

1. Architectural Overview: Mid-Fusion Transformer Framework

Phi-4-reasoning-vision-15B employs a mid-fusion transformer architecture, wherein visual inputs are first encoded by the SigLIP-2 NaFlex vision encoder. This encoder generates sequences of “soft” visual tokens at dynamic resolutions—up to 3,600 tokens for near-native 720p coverage. These visual representations are mapped via a trainable multi-layer perceptron (MLP) into the model’s embedding space and interleaved with textual tokens for subsequent cross-modal attention in the transformer.

This mid-fusion paradigm preserves pretrained unimodal LLM capabilities while retaining manageable training and inference costs. Unlike early-fusion architectures, which blend modalities at the input stage, mid-fusion entails processing modalities in a way that maintains compartmentalized representations while enabling joint reasoning. The architecture enables accurate perception as a prerequisite for high-fidelity reasoning and minimal latency across a wide range of tasks.

2. Vision Encoding and Dynamic Resolution

Central to the model’s perception capabilities is the SigLIP-2 NaFlex vision encoder’s adaptation to emit a variable number of patches, with capacity for up to 3,600 visual tokens to approximate native 720p input resolution. An extensive ablation on a 5B-parameter proxy model demonstrates that dynamic-resolution encoding yields consistently superior results on fine-grained graphical user interface (GUI) and chart interpretation tasks, compared to multi-crop variants. For instance, dynamic-resolution encoders with the highest token counts improve performance on the ScreenSpot-Pro benchmark to 17.5, compared to 10.6 for multi-crop approaches at lower resolutions.

This suggests that accurate, high-resolution features are critical for tasks requiring fine spatial grounding. The approach introduces a quadratic compute trade-off in token length, but analysis indicates that this is justified by gains in perceptual accuracy and downstream reasoning.

Method	Max Visual Tokens	ScreenSpot-Pro
Multi-crop + S²	2,048	10.6
Dynamic-res (3,600)	3,600	17.5

3. Explicit Reasoning Modes: Mode Tokens and Instruction Tuning

Phi-4-reasoning-vision-15B incorporates explicit control over reasoning processes via two dedicated tokens: > for chain-of-thought (CoT) reasoning and <nothink> for direct answers. During instruction tuning, 20% of data is presented in CoT format (primarily math and science), while 80% uses direct responses (captioning, OCR, GUI grounding). Users can override these defaults at inference time by prefixing prompts with the desired token, trading off latency and depth of reasoning as required.

Empirical results from ablations on the proportion of CoT versus direct samples indicate that a hybrid reasoning/non-reasoning approach—with systematic interleaving of modalities and mode tokens—outperforms fixed settings in the majority of evaluated benchmarks. For example, enforcing only direct answers on ChartQA leads to a 6.8-point drop in accuracy relative to the mixed mode.

4. Training Procedure: Three-Stage Curriculum and Objectives

The training regime consists of three sequential stages, each designed to progressively align perception and reasoning capabilities while optimizing for cross-entropy objectives over token sequences:

• Stage 1: Only the projection MLP is trained using high-quality image–caption pairs for vision-language alignment.
• Stage 2: All modules are trained (vision, MLP, LLM) using instruction-tuning data, including both CoT and direct responses, with increased sequence lengths.
• Stage 3: Long-context, multi-image, and refusal-augmentation instruction (RAI) samples are introduced, further tuning the model for complex, safety-sensitive generations.

All stages optimize cross-entropy loss over token outputs. The curriculum leverages targeted mixtures of dataset types and learning rates to promote stable convergence in fewer than 200 billion tokens, a significant reduction relative to the trillion-token scales standard in many large multimodal models.

Stage	Trainable Modules	Data	Samples	Seq. Len
1	MLP	Image–caption pairs	2 M	2,048
2	MLP, Vision, LLM	Single-image instruction	62.8 M	8,192
3	MLP, Vision, LLM	Long-context, multi-image, RAI	3.2 M	16,384

5. Data Curation, Filtering, and Synthetic Augmentation

Phi-4-reasoning-vision-15B’s high performance is attributed in large part to systematic data curation spanning three pillars: carefully filtered open-source datasets (70%), high-quality internal data (20%), and targeted acquisitions (10%). Each dataset is manually examined for 5–10 minutes, with excellent samples retained, programmatic correction of incorrect or misformatted entries performed using GPT-4o with verification pipelines, and low-quality segments re-generated or discarded.

Synthetic augmentation introduces sophisticated multimodal tasks and double-duty prompts integrating QA and instruction following. All spatial outputs (bounding boxes, GUI groundings) are normalized to $[0,1]$. Data-ratio ablations with a 5B model reveal that increasing the proportion of math samples by $3\times$ (while holding GUI data constant) improves performance in both domains, and that supplementing with Phi-Ground GUI data significantly boosts GUI-related metrics (ScreenSpot-V2 improves by ~12 points).

6. Performance Benchmarking and Efficiency Analysis

Phi-4-reasoning-vision-15B achieves leading accuracy among open-weight 12–15B models, scoring 5–10 points higher on math and GUI tasks than comparable architectures. On general VQA and chart QA, its performance is within 1–2 points of the best 32B-parameter models. Ten standard benchmarks—including AI2D, ChartQA, MathVista, MMMU, and ScreenSpot-V2—validate broad task generalization.

Inference was profiled on an NVIDIA H100 (one thread, batch size 1, greedy decoding, max output 4,096 tokens), with the following representative latency and output length (on a 400-sample subset):

• Phi-4-reasoning-vision-15B: ~0.36s per query, ~120 tokens output
• Qwen3-8B-4K: ~0.48s, ~180 tokens
• Qwen3-32B-4K: ~0.82s, ~200 tokens

These results position the model at the lower-compute, high-accuracy Pareto frontier for contemporary open-weight multimodal transformers.

7. Practical Recommendations and Methodological Lessons

Ablation studies and development experience from Phi-4-reasoning-vision-15B reinforce several methodological recommendations:

• Data quality and curation drive the majority of gains; systematic filtering and synthetic augmentation outweigh architectural variations.
• Dynamic-resolution vision encoders are integral for high-resolution grounding but their quadratic token-length cost requires judicious application; conditional patch selection based on text input is a potential future direction.
• Mixed reasoning/non-reasoning training with explicit mode tokens enables a single model to efficiently handle both low-latency and deep-reasoning tasks without requiring additional fine-tuning.
• The three-stage training recipe enables stable convergence with substantially fewer tokens than typical large multimodal models.
• Mid-fusion transformer design preserves unimodal strengths while avoiding the data and compute demands of early-fusion approaches.

Documentation, weights, code, and evaluation logs for Phi-4-reasoning-vision-15B are available via open-access repositories, with the stated intention of providing reproducible baselines and detailed ablation evidence for subsequent development and research in compact, multitask multimodal reasoning architectures (Aneja et al., 4 Mar 2026).