InfiMed-Foundation-4B: Multimodal Medical LLM
- The paper introduces a medical-specific model leveraging a 4B-parameter language backbone with a SigLIP vision encoder for robust image-text understanding.
- It employs a compute-efficient pretraining strategy with a low-to-high resolution curriculum and sequence packing, followed by a three-stage supervised fine-tuning pipeline.
- The model demonstrates superior performance on diverse medical benchmarks, outperforming several baselines in visual question answering and pathology interpretation.
InfiMed-Foundation-4B is a medical-specific multimodal LLM introduced in "InfiMed-Foundation: Pioneering Advanced Multimodal Medical Models with Compute-Efficient Pre-Training and Multi-Stage Fine-Tuning" (Zhu et al., 26 Sep 2025). It is the larger of two released variants, alongside InfiMed-Foundation-1.7B, and is built around a 4B-parameter language backbone, specifically Qwen3-4B-Instruct, combined with a SigLIP-so400m-384px vision encoder and a lightweight multimodal projector. The model is designed for image-text medical understanding, with emphasis on medical visual question answering, diagnostic support, radiology-related reasoning, pathology/image interpretation, pharmacology and broader domain-specific medical knowledge tasks mentioned in the motivation, and multimodal medical reasoning across heterogeneous clinical data. The paper attributes its performance to three main ingredients: a five-dimensional quality assessment framework for multimodal medical data curation, compute-efficient continual pretraining via low-to-high image resolution and multimodal sequence packing, and a three-stage supervised fine-tuning pipeline. On MedEvalKit, InfiMed-Foundation-4B reaches a 56.4 average across seven medical benchmarks and is reported to surpass HuatuoGPT-V-7B and MedGemma-27B-IT on the aggregate score (Zhu et al., 26 Sep 2025).
1. Model identity and problem setting
InfiMed-Foundation-4B is presented as a domain-adapted medical MLLM rather than a general-purpose assistant later repurposed for medicine. The motivating claim is that general-purpose MLLMs often lack the specialized knowledge required for medical tasks and can produce uncertain or hallucinatory responses, while knowledge distillation from advanced models struggles to capture domain-specific expertise in radiology and pharmacology. The paper therefore frames medical multimodal modeling as a problem of domain-specific knowledge acquisition, multimodal grounding, and compute-efficient continual pretraining rather than prompt-only specialization (Zhu et al., 26 Sep 2025).
The model’s intended scope is broader than a single benchmark family. The paper associates it with medical visual question answering, diagnostic tasks, radiological interpretation, pathological analysis, and clinically grounded image-text reasoning. At the same time, it does not describe InfiMed-Foundation-4B as a foundation model trained from scratch. The architecture inherits a pre-existing general LLM backbone and a frozen visual encoder, then specializes through curated continual pretraining and staged supervised fine-tuning. A common misconception is therefore that the main contribution is a novel base architecture; the paper instead emphasizes training recipe, data curation, and efficiency.
The 4B variant is the flagship result of the paper. The 1.7B variant is used to show that the recipe scales down effectively, but the 4B system is the one highlighted as surpassing HuatuoGPT-V-7B and MedGemma-27B-IT on the reported MedEvalKit macro-average, and it is the publicly released checkpoint linked by the authors (Zhu et al., 26 Sep 2025).
2. Architecture and compute-efficient pretraining
InfiMed-Foundation-4B consists of three components: the Qwen3-4B-Instruct LLM, the SigLIP-so400m-384px vision encoder, and a visual projector composed of Adaptive Average Pooling plus a two-layer MLP (Zhu et al., 26 Sep 2025).
| Component | Specification | Role |
|---|---|---|
| LLM | Qwen3-4B-Instruct | Text backbone |
| Vision encoder | SigLIP-so400m-384px | Visual feature extraction |
| Visual projector | Adaptive Average Pooling + two-layer MLP | Maps visual features into LLM token space |
The paper does not introduce a separate fusion transformer or cross-attention module. Instead, it follows the standard MLLM design pattern in which image features are encoded by the frozen vision backbone, projected into the LLM token space, and processed as part of the joint multimodal sequence. The vision encoder is frozen during both pretraining and fine-tuning. SigLIP produces 729 visual patches before pooling; during pretraining these are downsampled to 144 visual tokens per image, while supervised fine-tuning uses the full 729 visual tokens. The context length is 4096 tokens (Zhu et al., 26 Sep 2025).
The compute-efficiency strategy has two explicit parts. First, the paper describes a low-to-high image resolution strategy, operationally realized as a low-token-count to high-token-count visual curriculum: 144 visual tokens during continual pretraining, then 729 during SFT. The stated rationale is that reduced visual token count preserves essential visual features while lowering multimodal attention cost, whereas restoration to full token count during SFT recovers fine-grained visual cues important for medical interpretation. Second, the paper applies multimodal sequence packing. It computes the multimodal length of each sample, sorts samples by descending length, applies First-Fit-Decreasing bin packing, and packs multiple samples into bins that do not exceed the 4096-token context window. The packed bins are saved into JSON with "data" and "lengths" fields (Zhu et al., 26 Sep 2025).
The pretraining configuration reported in the appendix uses 32 NVIDIA H800 80GB GPUs, one epoch, AdamW, learning rate , weight decay 0.01, warmup ratio 0.03, global batch size 256, sequence packing enabled, 144 visual tokens per image, and 4096-token context length. The SFT stages use 16 GPUs, one epoch per stage, AdamW, learning rate , warmup ratio 0.03, global batch size 128, 729 visual tokens per image, and no sequence packing. The paper notes an internal inconsistency for SFT weight decay: the appendix table reports 0.01, while the implementation paragraph reports 0.1 (Zhu et al., 26 Sep 2025).
3. Data curation and the five-dimensional quality framework
A central claim of the work is that performance gains depend on curated mixtures of general-purpose and medical multimodal data rather than on scale alone. The general multimodal pool includes DataComp, CCS, OBELICS, MMC4, and MAmmoTH-VL-10M. The medical caption datasets listed include IU-Xray, LLaVA-Med, LLaVA-Med-60K-IM-text, MedTrinity-25M, MedPix-2.0, PMC-OA, PubMedVision, ROCO, and ROCOv2. The medical instruction datasets listed include LLaVA-Med, Path-VQA, PMC-VQA, PubMedVision, SLAKE, VQA-Med-2019, and VQA-RAD. After quality assessment, the paper states that IU-Xray, MedPix-2.0, PMC-OA, and VQA-Med-2019 were excluded (Zhu et al., 26 Sep 2025).
The filtering protocol is organized around a five-dimensional quality assessment framework developed with medical professionals. The dimensions are Medical Information Accuracy, Language Clarity and Fluency, Caption/Dialogue Completeness, Medical Imaging Relevance, and Practicality. The authors sampled 500 samples from each dataset and used GPT-o3 as an automated evaluator under criteria designed by medical experts. The appendix also defines an Overall score from 1 to 5 based on holistic quality judgment rather than simple averaging. No explicit weighted aggregation formula is provided (Zhu et al., 26 Sep 2025).
The scoring protocol is JSON-structured rather than equation-based. Each sample is rated along the five dimensions and an overall dimension, with a score and comment for each field. The framework is notable because it treats dataset quality as clinically structured rather than purely syntactic or benchmark-driven. It also encodes a strict rule for missing visual evidence: if no image is provided, Medical Imaging Relevance receives a score of 1 by default. This suggests that the curation process was designed to favor genuine image-text grounding rather than text-only medical answer quality.
The continual pretraining corpus is reported as approximately 30B tokens in total, comprising about 25B general-domain multimodal tokens and about 5B medical tokens. For supervised instruction tuning, Stage 1 uses approximately 10.1M samples from MAmmoTH-VL-10M; Stage 2 uses a roughly 11.69M-sample mixture of filtered general instruction data and medical instruction datasets; Stage 3 uses a balanced, downsampled cross-distribution set of approximately 304K samples. The paper also notes a specific filter on MAmmoTH-VL-10M-filtered that removes safety refusal responses such as "Sorry, I can’t..." (Zhu et al., 26 Sep 2025).
4. Three-stage supervised fine-tuning curriculum
The supervised fine-tuning pipeline is explicitly staged. Stage 1, General Instruction Following, uses MAmmoTH-VL-10M to build multimodal instruction-following and reasoning ability before medical specialization. The paper argues that many instruction datasets encourage only short phrase-level answers, whereas MAmmoTH-VL-10M supports reasoning-intensive responses with detailed rationales. Stage 2, Medical Instruction Following, mixes filtered general instruction data with LLaVA-Med-Instruct, Path-VQA, PMC-VQA, PubMedVision-Instruct Tuning, SLAKE, and VQA-RAD to inject medical QA capability and domain-specific reasoning while reducing catastrophic forgetting. Stage 3, Cross-distribution Instruction Adaptation, uses a balanced 304K-sample mixture to avoid domination by larger sources and improve generalization across heterogeneous datasets (Zhu et al., 26 Sep 2025).
The three stages form a curriculum in a precise sense. Stage 1 teaches general multimodal reasoning and instruction compliance. Stage 2 adds medical specialization. Stage 3 rebalances the training signal across distributions. The paper’s ablation table shows that all three stages are complementary rather than redundant. Stage 2 is particularly associated with gains on SLAKE, PMC-VQA, and OmniMedVQA, while Stage 3 produces especially large improvements on VQA-RAD, SLAKE, and PathVQA relative to the Stage 1+2 configuration (Zhu et al., 26 Sep 2025).
The reported ablation results are as follows. Stage 1 alone achieves an average of 51.2. Stage 1 plus Stage 2 reaches 53.4. Stage 1 plus Stage 3 reaches 53.8. The full three-stage pipeline reaches 56.4. In task-specific terms, adding Stage 3 on top of Stages 1 and 2 increases VQA-RAD from 53.0 to 57.9, SLAKE from 73.1 to 77.7, and PathVQA from 50.5 to 63.4. This is one of the clearest empirical signals in the paper that balanced cross-distribution adaptation materially affects generalization (Zhu et al., 26 Sep 2025).
The paper does not explicitly print the SFT objective function. A plausible implication is that the training uses standard autoregressive next-token cross-entropy over multimodal prompts and responses, but the paper itself does not give a formal loss equation. This absence matters because the work’s main novelty lies less in loss design than in curriculum design, data mixture selection, and compute allocation.
5. Evaluation and empirical performance
Evaluation is conducted with MedEvalKit, using standardized preprocessing and postprocessing, rule-based scoring for closed-ended questions, LLM-as-a-Judge for open-ended questions, and vLLM for accelerated inference. The benchmark suite includes MMMU-Med, VQA-RAD, SLAKE, PathVQA, PMC-VQA, OmniMedVQA, and MedXpertQA-Multimodal. These benchmarks span modalities including CT, dermoscopy, X-ray, and microscopy (Zhu et al., 26 Sep 2025).
| Benchmark | Score |
|---|---|
| MMMU-Med | 43.3 |
| VQA-RAD | 57.9 |
| SLAKE | 77.7 |
| PathVQA | 63.4 |
| PMC-VQA | 56.6 |
| OMVQA | 76.8 |
| MedXVQA | 21.9 |
| Avg. | 56.4 |
The headline comparison is the macro-average of 56.4. The paper reports HuatuoGPT-V-7B at 54.2, MedGemma-27B-IT at 55.4, Qwen2.5VL-7B at 52.0, Qwen2.5VL-3B at 49.2, MedGemma-4B-IT at 51.5, and LLaVA-Med-7B at 37.8. InfiMed-Foundation-4B is therefore reported as over HuatuoGPT-V-7B, over MedGemma-27B-IT, over Qwen2.5VL-7B, over Qwen2.5VL-3B, and over MedGemma-4B-IT on the average metric (Zhu et al., 26 Sep 2025).
Its performance profile is uneven in a diagnostically informative way. The model is especially strong on SLAKE, PathVQA, PMC-VQA, and OmniMedVQA, with PathVQA highlighted as the best among the listed open-source medical baselines in the table. Radiology, however, is not its strongest evaluated axis: on VQA-RAD it scores 57.9, below HuatuoGPT-V-7B at 67.0, Lingshu-7B at 67.9, InternVL3-8B at 65.4, and MedGemma-27B-IT at 62.3. MedXVQA remains difficult at 21.9, and the paper explicitly notes that this benchmark remains challenging for open models generally. The result is therefore best understood as broad medical multimodal competence with notable strength in medical VQA and pathology-oriented settings rather than uniform dominance across all subdomains.
The paper also reports qualitative case studies comparing Qwen2.5-VL-7B with InfiMed-Foundation-4B. In those examples, Qwen2.5-VL-7B is described as sometimes unable to make a judgment due to lack of medical knowledge, making an incorrect lesion size estimate, or producing an erroneous judgment, whereas InfiMed-Foundation-4B performs a systematic analysis and gives more accurate and reliable answers. These examples do not constitute a formal hallucination metric, but they are presented as evidence that the model’s medical specialization reduces knowledge-gap failures (Zhu et al., 26 Sep 2025).
6. Relation to adjacent InfiMed work, deployment status, and limitations
InfiMed-Foundation-4B is part of a broader 4B-scale medical modeling context, but adjacent papers should not be conflated with its model description. "InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training" introduces ORBIT as a rubric-based RL framework applied to Qwen3-4B-Instruct and SFT-derived 4B variants for medical consultation, not a direct release paper for InfiMed-Foundation-4B. The exact string "InfiMed-Foundation-4B" does not appear there. A plausible implication is that ORBIT should be read as a companion post-training and alignment method for the broader InfiMed project line rather than as the foundation-model pretraining reference itself (Wang et al., 17 Oct 2025).
A second relevant point of context is the controlled 4B-scale comparison between domain fine-tuning and retrieval-augmented generation for medical multiple-choice QA. That study reports that domain fine-tuning yields a percentage-point gain in majority-vote accuracy over a general 4B baseline on MedQA-USMLE, while the tested RAG pipeline does not produce a statistically significant gain. It concludes that, at that scale and on that benchmark, domain knowledge encoded in weights dominates domain knowledge supplied in context. This does not evaluate InfiMed-Foundation-4B directly, but it contextualizes the design logic of a medically specialized 4B model whose primary capability claim rests on medical adaptation in the parameters rather than on an external retrieval layer (Buskila, 26 Apr 2026).
The paper explicitly states that the InfiMed-Foundation-4B checkpoint is publicly available at https://huggingface.co/InfiX-ai/InfiMed-Foundation-4B (Zhu et al., 26 Sep 2025). It also provides several practical model settings: 4096-token context length, SigLIP-so400m-384px as the vision encoder, 729 visual tokens in SFT and inference-style use, and an MLP-based visual projection path. However, it does not state a license, VRAM requirement, quantization recipe, official inference memory budget, or code repository URL in the provided text.
Several limitations are explicit or strongly implied. The paper does not claim autonomous clinical deployment, so the model is best read as an assistive system rather than a diagnostic replacement. It is designed to reduce hallucinations through domain data quality and training strategy, but it does not claim hallucinations are eliminated. Dataset bias remains a concern because open medical corpora are skewed toward public, de-identified, and benchmark-friendly sources. The model also remains weak on MedXVQA, which indicates incomplete expert-level reasoning or out-of-distribution robustness. More broadly, the architecture is technically conservative: its significance derives less from architectural novelty than from medically supervised dataset curation, compute-efficient pretraining, and staged adaptation.