Phi-4-Multimodal-Instruct Overview

• Phi-4-Multimodal-Instruct is a configuration of the unified Phi-4-Multimodal model that integrates text, vision, and speech into a single system.
• It employs a frozen language model backbone combined with modality-specific LoRA adapters and projectors to incorporate visual and audio inputs without sacrificing text performance.
• Evaluations show competitive results across vision-language, audio, and multimodal benchmarks, serving as a non-thinking baseline compared to reasoning-enhanced variants.

Phi-4-Multimodal-Instruct is the instruction-tuned configuration of Phi-4-Multimodal, a single unified multimodal model built on the Phi-4-Mini backbone that adds vision and speech/audio capabilities without sacrificing the backbone’s text performance. In the Phi-4-Mini technical report, the term refers to the model after the supervised fine-tuning stages for multimodal instruction-following—vision-language, vision-speech, and speech-only—plus post-training safety alignment (Microsoft et al., 3 Mar 2025). In the later Phi-4-reasoning-vision-15B report, “Phi-4-Multimodal-Instruct” appears as a non-thinking, instruction-tuned baseline from the same family, aligned to visual inputs via instruction tuning but without the explicit mixed-mode reasoning design used in Phi-4-reasoning-vision-15B (Aneja et al., 4 Mar 2026).

1. Position within the Phi-4 family

Phi-4-Multimodal-Instruct is associated with the Phi-4-Multimodal checkpoint, reported as approximately 5.6B parameters, and is built on the Phi-4-Mini backbone, a 3.8B-parameter decoder-only transformer. The backbone has 32 layers, hidden size 3,072, tied input/output embeddings, and Group Query Attention. It uses the o200k_base tiktoken tokenizer with a 200,064 vocabulary and LongRoPE to support 128K context length. A fractional RoPE configuration is used such that 25% of attention head dimensions remain position-agnostic (Microsoft et al., 3 Mar 2025).

Within this family, Phi-4-Multimodal-Instruct is not a separate architectural lineage from Phi-4-Multimodal; rather, it is the instruction-tuned configuration of that unified multimodal system. The same checkpoint supports text-only, text+image, speech-only, and image+speech inputs. The base LM is frozen when modalities are added, and multimodal behavior is obtained through pretrained encoders, projectors into the LM embedding space, and modality-specific LoRA adapters (Microsoft et al., 3 Mar 2025).

The later Phi-4-reasoning-vision-15B report positions Phi-4-Multimodal-Instruct as a baseline for comparison against a reasoning-centric multimodal model. That report characterizes the instruct variant as a multimodal, instruction-tuned model without explicit mixed reasoning training. This comparison is central to understanding the model’s historical role: Phi-4-Multimodal-Instruct represents the family’s non-thinking multimodal instruction-following configuration, whereas Phi-4-reasoning-vision-15B introduces explicit mode tokens, a reasoning-capable backbone, and mixed reasoning/non-reasoning training (Aneja et al., 4 Mar 2026).

2. Architecture and modality integration

The architectural design is organized around a frozen language backbone and modality-specific extensions. Each new modality is integrated via a pretrained encoder and a small projector that maps modality features into the LM’s 3,072-dimensional embedding space. Multimodal behavior is enabled by attaching modality-specific LoRA adapters to the LM’s linear layers. The report describes this as a “mixture-of-LoRAs” design, intended to allow multiple inference modes to coexist without interference (Microsoft et al., 3 Mar 2025).

For vision, the model uses a SigLIP-400M image encoder, finetuned with LLM2CLIP. The default input resolution is 448×448, and the maximum image resolution in training is capped at 1344×1344. A 2-layer MLP projector maps vision features to the LM’s embedding space, and a vision LoRA adapter, denoted LoRA_V, is applied across the LM’s linear layers during supervised fine-tuning (Microsoft et al., 3 Mar 2025).

For speech/audio, inputs are represented as 80-dimensional log-Mel filter-bank features at a 10 ms frame rate. The speech encoder comprises 3 convolutional layers followed by 24 conformer blocks with attention dimension 1024, feed-forward dimension 1536, and 16 attention heads. The CNN sub-sampling rate is 8, yielding an 80 ms “speech token” rate to the LM, approximately 750 tokens per minute. A 2-layer MLP projector maps 1024-dimensional audio features to the LM’s 3,072-dimensional embedding space, and a speech/audio LoRA adapter, LoRA_A, is applied to all attention and MLP layers in the LM with rank 320 (Microsoft et al., 3 Mar 2025).

The report also describes modality-specific routers conceptually as part of the mixture-of-LoRAs approach. Although it does not provide explicit router equations or parameter counts, the stated intent is that routing ensures only the relevant adapter or adapters are active for a given input mode and task. A conceptual formulation consistent with the description is given in the report synthesis, but it is explicitly noted as not an explicit formula from the paper. This suggests that the router notion is architectural and operational rather than a formally elaborated new optimization objective (Microsoft et al., 3 Mar 2025).

Group Query Attention is a central efficiency mechanism. Each block has 24 query heads and 8 key/value heads, reducing KV cache memory to one-third of the standard multi-head size while preserving generation quality over long sequences. The model’s long-context readiness is further associated with LongRoPE and the fractional RoPE configuration (Microsoft et al., 3 Mar 2025).

3. Vision, audio, and multimodal training pipeline

The language backbone underlying Phi-4-Multimodal-Instruct is pretrained over a 5-trillion-token corpus emphasizing high-quality web and synthetic data, with improved filtering, boosted math/coding, and reasoning-heavy synthetic data. Post-training adds instruction-following, function calling, summarization, and code completion datasets. Modality training is then staged on top of the frozen backbone (Microsoft et al., 3 Mar 2025).

For vision-language training, the report specifies pretraining data consisting of interleaved image-text documents, image-text pairs, image grounding, OCR from PDFs and realistic images, and chart synthesis, totaling approximately 0.5T tokens. The loss is next-token prediction over text tokens only; image tokens incur no loss. The supervised fine-tuning data contributes approximately 0.3T tokens and combines public multimodal instruction-tuning sets with large in-house datasets spanning general natural images, charts, tables, diagrams, PowerPoint analysis, OCR, multi-image comparison, video summarization, and safety (Microsoft et al., 3 Mar 2025).

The vision-language pipeline has four stages:

Stage	Operation	Stated purpose
1	Train projector only on captions with image encoder frozen	Align vision with text
2	Train encoder + projector jointly on full vision pretraining data	OCR and dense understanding
3	Deploy LoRA_V and train with encoder + projector on curated single-frame SFT data	Generative vision-language abilities
4	Train on multi-frame data with vision encoder frozen	Extend context coverage to 64k for temporal/multi-image understanding

The vision path also introduces a dynamic multi-crop strategy. Crops are computed as $\lceil H/C \rceil \times \lceil W/C \rceil$, where $H$ and $W$ are image size and $C$ is crop size. If total crops are at most 16 in pretraining or at most 36 in SFT, the image is slightly resized to match the crop grid; otherwise, the InternVL2 aspect-ratio matching strategy is used, but without inflating small images to unreasonable sizes. Multi-frame training extends to 64k context coverage for multi-image and video use (Microsoft et al., 3 Mar 2025).

Speech/audio training is likewise staged. In pretraining for ASR alignment, the audio encoder is initialized from an AED ASR model, and the audio encoder plus projector are trained with the LM frozen using a learning rate of $4\times 10^{-5}$ for 50k steps on approximately 2M hours of anonymized in-house speech-text pairs across eight languages. After this stage the model supports ASR only. In post-training for instruction-following, the audio encoder is frozen and the audio projector plus LoRA_A are trained with learning rate $1\times 10^{-4}$ for another 50k steps on approximately 100M weighted speech/audio SFT samples spanning ASR, AST, Speech QA, Spoken Query QA, speech summarization, and audio understanding (Microsoft et al., 3 Mar 2025).

After completing vision and speech post-training, the system undergoes vision-speech joint training. In that stage, the base LM, audio encoder, and audio projector are frozen, while LoRA_V, the vision encoder, and the vision projector are fine-tuned on vision-speech SFT data, with language and vision post-training data mixed in to maintain quality (Microsoft et al., 3 Mar 2025).

4. Inference modes, input formats, and operational behavior

Phi-4-Multimodal-Instruct is designed as a single-checkpoint system with multiple inference modes: text-only, vision + language, speech/audio, and vision + speech. The report emphasizes that these modes coexist in one checkpoint because the LM is frozen and modality adaptation occurs in separable LoRA branches trained on curated data for each mode (Microsoft et al., 3 Mar 2025).

For speech/audio supervised fine-tuning and inference, the report provides a concrete input format:

<|user|><audio>{task prompt}<|end|><|assistant|>{label}<|end|>

The task prompt may be empty for Spoken Query QA. For ASR, AST, Speech QA, speech summarization, and audio understanding, it contains natural-language instructions for the assistant. In AST chain-of-thought decoding, the prompt requests “transcribe then translate,” with a separator token such as <sep> parsed for evaluation (Microsoft et al., 3 Mar 2025).

Images and interleaved multimodal inputs are handled through the training regime of interleaved image-text documents and the runtime provision of image or images plus text, and optionally audio, in the same conversation turn. The report does not standardize a particular image token name, but the pipeline expects image embeddings from the SigLIP encoder via the projector, then LM generation conditioned on the multimodal context (Microsoft et al., 3 Mar 2025).

The model supports 128K LM context. On the audio side, the tokenization rate of approximately 750 tokens per minute implies that inference theoretically supports approximately 2.8 hours of audio within the 128K context, although the report states that the system is not fine-tuned for such extremes. In supervised fine-tuning, maximum audio lengths are up to 30 minutes for summarization and 30 seconds for other tasks (Microsoft et al., 3 Mar 2025).

A useful contrast appears in the Phi-4-reasoning-vision-15B report. There, Phi-4-Multimodal-Instruct is described as a non-thinking baseline that does not incorporate explicit > and <nothink> mode control. This suggests that its multimodal instruction-following is conventional relative to the later mixed-mode reasoning system: it aligns a Phi-4 LLM with visual inputs via instruction tuning, but does not expose the explicit reasoning-mode mechanism that became central to Phi-4-reasoning-vision-15B (Aneja et al., 4 Mar 2026).

5. Evaluation profile

The Phi-4-Mini technical report evaluates Phi-4-Multimodal across vision-language, vision-speech, and speech/audio tasks. Across 13 single-image vision-language and 2 multi-image/video benchmarks, the 5.6B model averages 72.0. The report highlights results including DocVQA test 93.2, OCRBench 84.4, ChartQA test 81.4, TextVQA test 75.6, MathVista testmini 62.4, MMMU val 55.1, BLINK test 61.3, and VideoMME-16Frame test 55.0 (Microsoft et al., 3 Mar 2025).

On the ShareGPT4o vision-speech suite, the reported results are ShareGPT4o_AI2D 68.9, ShareGPT4o_ChartQA 69.0, ShareGPT4o_DocVQA 87.3, and ShareGPT4o_InfoVQA 63.7, for an average of 72.2. The report states that Phi-4-Multimodal significantly outperforms InternOmni-8.7B and Gemini-2.0-Flash variants on this suite (Microsoft et al., 3 Mar 2025).

Speech/audio evaluation spans ASR, AST, Spoken Query QA, speech summarization, and audio understanding. For ASR, the report states that Phi-4-Multimodal ranks No. 1 on the Hugging Face OpenASR leaderboard and is approximately 5.5% relatively better in WER than NVIDIA Canary-1B, with CommonVoice v15 average 6.80, FLEURS average 4.00, and OpenASR average 6.14. For AST, reported averages include CoVoST2 X→EN 39.33 in 0-shot and 40.76 in CoT, CoVoST2 EN→X 37.82 and 38.73, FLEURS X→EN 29.86 and 32.35, and FLEURS EN→X 32.15 and 33.56 (Microsoft et al., 3 Mar 2025).

For Spoken Query QA, the reported MT-Bench score is 7.05 and MMMLU accuracy is 38.50. For speech summarization on English meetings, the report gives Golden3 overall 6.28 with hallucination 0.14 and adherence 5.87, and AMI overall 6.29 with hallucination 0.13 and adherence 6.50. For audio understanding, AIRBench-chat is 6.98 and MMAU accuracy is 55.56 (Microsoft et al., 3 Mar 2025).

The later reasoning-vision report provides a narrower but historically important comparison by explicitly naming Phi-4-Multimodal-Instruct in Table 1 as a non-thinking baseline. It reports the following benchmark gaps between Phi-4-Multimodal-Instruct and Phi-4-reasoning-vision-15B: AI2D TEST 68.6 versus 84.8, ChartQA TEST 23.5 versus 83.3, MathVista MINI 50.5 versus 75.2, MMMU VAL 42.3 versus 54.3, and ScreenSpot v2 28.5 versus 88.2. The report interprets these gaps as illustrating the value of a reasoning-capable backbone, mixed reasoning/non-reasoning training via mode tokens, and high-/dynamic-resolution perception (Aneja et al., 4 Mar 2026).

6. Safety, limitations, and relationship to later reasoning models

Safety alignment in Phi-4-Multimodal-Instruct follows the broader Phi-4-Multimodal post-training regime. Text safety alignment extends the Phi-3 approach to Tier-1 languages via machine translation checked by GPT-4o-mini, and uses helpfulness/harmlessness preference datasets with in-house data targeting RAI harm categories. Independent red-teaming is reported. On English automated evaluations without jailbreaks, Phi-4-Multimodal has a 4.0% Defect Rate, with Violence 7%, Sexual 6%, Self-harm 0%, and Hateful 3%. With jailbreaks, the average Defect Rate is 2.25%. Refusal metrics on XSTest are IPRR 92% and VPRR 26.4% (Microsoft et al., 3 Mar 2025).

For multilingual harms, the average defect rate across German, French, Spanish, Italian, Portuguese, Chinese, Japanese, and English is 5.06%. For audio safety, evaluation with the Azure AI Evaluation SDK gives an average Defect Rate of 3.25%, and the report states that speech fairness checks show no egregious subgroup degradation across gender and age groups and locales. Sensitive attribute inference can be largely mitigated via system prompts, down to approximately 0.4%. On vision safety benchmarks, Phi-4-Multimodal scores favorably versus similar-size open models and close to GPT-4V on some tests (Microsoft et al., 3 Mar 2025).

The report also notes limitations. As a compact model, it may miss fine-grained factual memory, and multilingual capability is constrained by parameter budget and data mixture emphasis, with non-English performance lagging English. Long-audio processing beyond 2 hours is theoretically supported but not fine-tuned (Microsoft et al., 3 Mar 2025).

The relationship between Phi-4-Multimodal-Instruct and Phi-4-reasoning-vision-15B clarifies a common misconception. Phi-4-Multimodal-Instruct is not described as a reasoning-first multimodal model; in the later report it is explicitly a non-thinking, instruction-tuned baseline. Phi-4-reasoning-vision-15B instead introduces a reasoning-centric language backbone, a high-quality vision stack, high-resolution and dynamic-resolution encoding, and explicit <think> / <nothink> mode control. A plausible implication is that Phi-4-Multimodal-Instruct marks an earlier design point in the Phi-4 multimodal line: a unified, frozen-backbone, mixture-of-LoRAs model for multimodal instruction-following, later superseded in the vision-language branch by a design that makes reasoning behavior an explicit part of the training objective and interface (Aneja et al., 4 Mar 2026).