DSMTree: Advanced Medical Vision-Language Model

• DSMTree is a cutting-edge medical vision-language model that fuses a transformer-based text decoder with a specialized vision encoder to process high-dimensional medical images and texts.
• It employs interleaved image-patch and text-token embeddings with cross-attention to enable structured output and domain-adapted reasoning for varied clinical applications.
• Benchmarking reveals significant improvements in tasks such as 3D MRI classification and CXR anatomy localization, while supporting efficient, on-premise, privacy-preserving deployments.

MedGemma 1.5 (4B): Medical Vision-Language Modeling Foundation

MedGemma 1.5 (also referenced as MedGemma-4B, google/medgemma-4b-it in community models) is a 4-billion-parameter, transformer-based medical vision-LLM (VLM) developed by Google Health. Architecturally, it combines a clinically-adapted Gemma 3 backbone (decoder-only transformer) for language processing and a MedSigLIP vision encoder for medical image understanding. MedGemma 1.5 extends the general-purpose multimodal capabilities of the foundational Gemma 3 models with specialized support for high-dimensional medical imaging modalities, structured document parsing, and domain-adapted reasoning. Model weights, code, and fine-tuning recipes are available under permissive licenses, enabling both clinical benchmarking and downstream adaptation (Sellergren et al., 7 Jul 2025, Sellergren et al., 6 Apr 2026, Carrillo-Larco et al., 15 Sep 2025, Zun et al., 17 Oct 2025, Lawal et al., 23 Jun 2026, Buskila, 12 Apr 2026).

1. Architecture and Model Components

MedGemma 1.5 consists of two principal modules: a 4B-parameter transformer-based text decoder (Gemma3 lineage) and a 400M-parameter MedSigLIP convolutional neural network acting as vision encoder. Visual and textual input streams are fused by interleaving MedSigLIP-generated image-patch embeddings with text-token embeddings, processed jointly via cross-attention in the language decoder.

Key architectural features:

• Text decoder: 24 transformer layers (L ≈ 24), d_model ≈ 2048, standard autoregressive causal masking.
• Vision encoder: MedSigLIP, a medically fine-tuned derivative of SigLIP (CLIP-style), producing patch-level embeddings at 896×896 resolution.
• Fusion: No separate modality-specific adapter modules; modalities share the token-embedding dimension and are attended jointly.
• Structured output: Supports both free-form answers and structured outputs (e.g., anatomical localization as normalized bounding boxes in JSON).
• Context window: Capable of processing sequences up to ~32K tokens (vision+text) in MedGemma 1.5, with the base Gemma 3 supporting up to 128K.
• Quantization and inference: Supports 8-bit and 4-bit quantized inference, bfloat16 activations; validates on single-GPU (16–24GB VRAM) and TPU.

The model’s vision encoder is frozen after pretraining; only the language decoder and cross-modal layers are further trained in subsequent adaptation steps (Sellergren et al., 7 Jul 2025, Sellergren et al., 6 Apr 2026).

2. Training Regimen and Data

MedGemma 1.5 is trained in several stages:

1. Pretraining: Begins with Gemma 3’s general-domain pretraining on web-scale corpora and mixed image–text pairs.
2. Vision encoder adaptation: MedSigLIP fine-tuned with 2% medical images/texts (radiology, dermatology, histopathology, ophthalmology; ∼33M pairs).
3. Multimodal supervised learning: Additional medical image–text pair datasets (radiology reports, CT/MRI/WSI, EHRs, clinical lab reports) are used for supervised next-token generation and classification.
4. Distillation: Cross-entropy on soft teacher logits from both generalist and modality-specific teachers.
5. Reinforcement learning: Rewarding token-level ROUGE-L (image-to-text) and classification benchmarks, maintaining general capability through mixing with non-medical data.

Representative datasets include CXR-IND1 (chest X-rays), CT/MRI and WSI cohorts, ISIC and internal dermatology sets, EHRQA, EHR Datasets 2–5, and Mendeley Clinical Laboratory Test Reports. Multimodal pretraining gives particular attention to 3D medical scans (CT/MRI) via axial slicing, and WSIs through patch-based sampling at multiple magnifications. Document images (multi-page PDFs) are rendered and processed into patch grids (Sellergren et al., 6 Apr 2026, Sellergren et al., 7 Jul 2025).

3. Multimodal Integration and Preprocessing Strategies

MedGemma 1.5 introduces preprocessing pipelines enabling robust support of high-dimensional and heterogeneous modalities:

• 3D volumetric imaging: Decomposition into 2D slices (up to 85 axial slices per study), each treated as a 2D image patch embedding. This yields up to 21,760 visual tokens per CT/MRI scan, with three Hounsfield window mappings for CT and min-max normalization for MRI.
• Whole-slide pathology: Tissue-masked, multi-scale sampling (5×, 10×, 20×) with up to 126 896×896-pixel patches per slide, preserving spatial ordering to maintain contextual integrity across ~32K vision tokens.
• Long-sequence capping/subsampling: Enforced to remain within computational resource limits.
• Anatomical localization: JSON-structured format for bounding boxes, fully handled by the LLM without dedicated detection heads.
• Textual document ingestion: PDFs rendered as images, then divided into overlapping crops for tokenization and downstream extraction.

No explicit modality adapters are introduced; all adaptation is via cross-attention. This design permits both generative and retrieval-augmented clinical reasoning (Sellergren et al., 6 Apr 2026).

4. Benchmarking and Evaluation

MedGemma 1.5 demonstrates robust performance gains across diverse medical benchmarks:

• 3D MRI condition classification (accuracy): 51.3% (MedGemma 1) → 64.7% (1.5)
• Whole-slide pathology (macro F1): 2.2% → 49.4%
• CXR anatomy localization (mean IoU): 3.1% → 38.0%
• MedQA accuracy: 64.4% → 69.1%
• EHRQA accuracy: 67.6% → 89.6%
• Lab report extraction (macro F1): 59.5% → 77.5%
• Medical VQA: SLAKE F1 72.3 (MedGemma 4B) vs. Gemma 3 4B 40.2; VQA-RAD F1 49.9 vs. 33.6
• Chest X-ray findings: CheXpert macro F1 48.1 vs. Gemma 3 32.6

In medical MCQA for Peruvian specialty exams (PeruMedQA) (Carrillo-Larco et al., 15 Sep 2025), instruction-tuned MedGemma-4B ("-it" variant) achieves ~47% zero-shot accuracy, rising to ~67% with LoRA fine-tuning on Spanish-language data (LoRA rank 16, α=16, p=0.05, lr=5e-5, 10 epochs), matching or exceeding larger models for some exams. The medgemma-27b-text-it outperforms all smaller models, exceeding 90% accuracy in several domains, and is the recommended choice where maximal accuracy is required.

Hallucination detection studies in gastrointestinal endoscopy VQA (Lawal et al., 23 Jun 2026) show that MedGemma-4B, when evaluated with nine established detectors, is especially amenable to white-box hallucination detection based on internal hidden states (ReXTrust, AUC=92.99); gray-box and black-box detectors lag substantially (MaxEnt AUC=59.46).

5. Fine-Tuning and Adaptation Techniques

MedGemma 1.5 supports extensive parameter-efficient adaptation:

• Instruction tuning (SFT): Used to align the model to question-answering or structured completion formats.
• Low-Rank Adaptation (LoRA/PEFT): Efficient fine-tuning by updating low-dimensional projections (typical r=16 for MCQA; QLoRA r=4 for captioning).
• Distillation from external or synthetic teachers: Synthetic data (e.g., knowledge-distilled GPT-5 captions) is filtered for clinical correctness, and used as a fine-tuning corpus in small-batch PEFT setups (Zun et al., 17 Oct 2025).
• QLoRA for quantized adaptation: Quantize to 4 bits, with adapters inserted in each transformer layer. Used for fine-tuning clinical caption generators (batch size=16, epochs=10, AdamW optimizer, lr=2e-4, with gradient checkpointing).

The fine-tuned MedGemma-1.5 can be highly effective for resource-constrained deployments, eradicating hallucinated outputs in MCQA (invalid answer rate drops from 0.14% to 0%), and yielding substantial uplifts in both structured clinical captioning metrics and RAGAS-measured caption faithfulness and correctness.

6. Practical Implications and Deployment Considerations

MedGemma 1.5 is optimized for on-premise, privacy-preserving inference:

• Inference speed: ~28.7 tokens/second on benchmark hardware (Buskila, 12 Apr 2026).
• Resource requirements: Runs on single GPUs (16–24 GB VRAM); large-scale inference is efficient with fixed vision-token caps.
• Privacy: Fully offline operation supports HIPAA/PHI contexts.
• System integration: Structured output formats and multimodal fusion enable usage in retrieval-augmented generation (RAG), automated report generation, structured data extraction, and agentic clinical decision support.
• Fine-tuning guidance: For institutional workflows, supervised SFT on small in-domain datasets typically suffices. Cap/adjust long context sizes for WSI/PDFs to fit hardware constraints.
• Community resources: Model checkpoints, fine-tuning recipes, and benchmarks are available via https://goo.gle/MedGemma and associated repositories (Sellergren et al., 6 Apr 2026, Sellergren et al., 7 Jul 2025).

7. Limitations, Performance Gaps, and Failure Modes

Despite strong benchmarking results, several constraints and failure modes are noted:

• Scale limitations: MedGemma 1.5 4B, while efficient, lags larger general models (Llama 3.1 8B, Gemma 3 12B) on MedQuAD MCQA (BERTScore F1 ~0.848 vs 0.852; LLM-judge 0.459 vs 0.592/0.600) and reproducibility metrics (self-agreement 0.146; uniqueness 0.936) (Buskila, 12 Apr 2026).
• Benchmark confounding: Comparisons between clinically-tuned and general models are confounded by parameter count; a general-purpose 4B model would be needed for true ablation.
• Hallucination: All evaluated hallucination detectors on MedGemma-4B are susceptible to “confident confabulation,” with up to 15.3% of hallucinated outputs exhibiting high consistency/agreement yet remaining factually incorrect. Consistency- and uncertainty-based detectors systematically miss these false negatives (Lawal et al., 23 Jun 2026).
• Future extensions: Native support for volumetric (3D) imaging and genomic modalities remains an open development area; ongoing work seeks more robust, bias-resistant, and uncertainty-aware adaptation.
• Safety: Low-temperature deterministic decoding does not guarantee stable, reproducible results; clinical deployment necessitates ensembling, confidence gating, or mandatory human review for high-stakes outputs (Buskila, 12 Apr 2026).

Table: Performance Summary on Key Tasks

Task / Metric	MedGemma 1.0	MedGemma 1.5	Δ (Absolute)
3D MRI Accuracy	51.3%	64.7%	+13.4 pp
Whole-Slide Path (macro F1)	2.2%	49.4%	+47.2 pp
CXR Anatomy Loc (mean IoU)	3.1%	38.0%	+34.9 pp
MedQA Accuracy	64.4%	69.1%	+4.7 pp
EHRQA Accuracy	67.6%	89.6%	+22.0 pp
PeruMedQA (MCQA, LoRA FT)	~47% (van.)	~67%	+20 pp
Halluc. Detect. (AUC, ReXTrust / MaxEnt)	92.99 / 59.46		+33.53 (AUC)

Domain-level metrics and detailed equations are documented in primary sources (Sellergren et al., 6 Apr 2026, Lawal et al., 23 Jun 2026, Sellergren et al., 7 Jul 2025, Carrillo-Larco et al., 15 Sep 2025).

• (Sellergren et al., 7 Jul 2025) MedGemma Technical Report
• (Sellergren et al., 6 Apr 2026) MedGemma 1.5 Technical Report
• (Carrillo-Larco et al., 15 Sep 2025) PeruMedQA: Benchmarking LLMs on Peruvian Medical Exams
• (Zun et al., 17 Oct 2025) Fine-Tuning MedGemma for Clinical Captioning to Enhance Multimodal RAG over Malaysia CPGs
• (Lawal et al., 23 Jun 2026) A Benchmark for Hallucination Detection in VLMs for Gastrointestinal Endoscopy
• (Buskila, 12 Apr 2026) Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework