OctoMed: Multimodal Medical Reasoning

Updated 4 July 2026

OctoMed is a multimodal medical reasoning system that integrates text-only, multimodal reasoning, and classification tasks using a curated supervised fine-tuning recipe.
The system leverages rejection sampling on teacher-generated reasoning traces and heterogeneous data sources to boost generalization and benchmark performance.
Evaluation reveals OctoMed achieves state-of-the-art scores on multiple medical benchmarks, rivaling larger models while focusing on robust, out-of-distribution performance.

OctoMed is a multimodal medical reasoning system and a data-centric supervised fine-tuning recipe built on Qwen2.5-VL-7B-Instruct, with the released model obtained by full fine-tuning on over 8 million examples and 6.8 billion response tokens. It is designed for broad out-of-distribution performance across text-only medical reasoning, multimodal medical reasoning, and multimodal medical classification, rather than for direct clinical deployment. Its central claim is that, in medical vision-language modeling, the composition and curation of supervised reasoning data can be more decisive than architectural novelty for generalization and robustness (Ossowski et al., 28 Nov 2025).

1. Model identity and intended scope

OctoMed is presented not as a new backbone but as a medically adapted 7B-parameter multimodal vision-language reasoner. The underlying base model is Qwen2.5-VL-7B-Instruct; the distinctive contribution lies in the post-training data recipe, especially the design of the supervised fine-tuning mixture, the use of structured reasoning traces, and the scale of curation. The intended capability profile includes medical exam-style question answering, image-grounded medical reasoning, and multimodal classification across modalities including chest X-ray, MRI, fundus, dermatology, and pathology (Ossowski et al., 28 Nov 2025).

The task space is explicitly tripartite. For text-only medical reasoning, the paper lists benchmarks such as MedQA, HeadQA, MedMCQA, MMLU-PRO health subset, and MedXpertQA-text. For multimodal reasoning, it targets settings in which image and text jointly determine the answer, including PMC-VQA, NEJM Image Challenge, MMMU-PRO medical subset, and MedXpertQA-multimodal. For multimodal classification, it evaluates image-centric diagnostic tasks such as MRI brain tumor classification, chest X-ray classification, diabetic retinopathy grading from fundus, and pathology tissue classification. This breadth is important because OctoMed is positioned as a robustness-oriented medical reasoner under both task shift and modality shift, not as a narrow single-benchmark specialist (Ossowski et al., 28 Nov 2025).

The paper is explicit that this is a benchmark-focused system. It reports strong out-of-distribution benchmark performance and clinically relevant reasoning behavior, but it does not frame OctoMed as a validated clinical decision-support system. That distinction matters because the model’s strongest evidence is comparative benchmark accuracy, not prospective clinical utility or deployment safety (Ossowski et al., 28 Nov 2025).

2. Distillation corpus and supervised fine-tuning recipe

The core of OctoMed is a supervised distillation pipeline that constructs a very large, mixed medical reasoning corpus. The training process begins from a dataset

$\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N,$

where each example consists of an input and gold answer. Teacher models then generate reasoning traces

$r_i = \left(r_i^{(1)}, \ldots, r_i^{(T_i)}\right),$

with final teacher prediction $\hat{y}_i$ . The acceptance rule is binary:

$S(x_i, r_i, y_i, \hat{y}_i) = \begin{cases} 1, & \text{if } \hat{y}_i = y_i, \ 0, & \text{otherwise.} \end{cases}$

Only correct traces are retained:

$\mathcal{R}^+ = \{ (x_i, y_i, r_i) \mid S(x_i, r_i, y_i, \hat{y}_i) = 1 \}.$

This formalism captures the paper’s central curation principle: rejection sampling over teacher-generated rationales, keeping only those whose final answer matches ground truth (Ossowski et al., 28 Nov 2025).

Teacher choice is modality-dependent. DeepSeek-R1 is used for text-only tasks, while GPT-4o is used for multimodal tasks because it accepts image inputs. The multimodal distillation prompt asks for detailed thoughts inside > ... and the final choice inside <answer> ... </answer>, whereas the DeepSeek-R1 text prompt is simpler and asks for the final answer letter within <answer></answer> tags. At evaluation time, OctoMed itself is prompted in a chain-of-thought style format: “Please reason step-by-step, and put your final answer within \boxed{}.” The model is thus trained and evaluated in a reasoning-explicit regime, but with teacher styles that are heterogeneous enough to induce trace diversity (Ossowski et al., 28 Nov 2025).

The mixture spans three source categories: text-only exam-style reasoning, multimodal reasoning, and multimodal classification. The paper emphasizes that multimodal benchmarks often lack train splits, so a subset of the PMC-VQA train split becomes a major source of distilled multimodal reasoning data. For modality and anatomy metadata that were absent in large sources such as PMC-VQA, the authors infer metadata with GPT-4.1-mini. Decontamination is handled by 16-gram deduplication for text benchmarks and exact image deduplication by hashing image bytes. Images are preprocessed to a maximum resolution of 262,144 pixels (512 × 512) using a smart resize procedure. For classification tasks, stratified sampling is used to balance classes (Ossowski et al., 28 Nov 2025).

Training is by full fine-tuning rather than parameter-efficient adaptation. The final run is reported as 3 epochs on the 8M-trace corpus, with effective batch size 512, learning rate 5e-5, cosine scheduler, warmup ratio 0.1, and LlamaFactory. The paper notes a smaller-scale experimental setup elsewhere with batch size 128 and warmup ratio 0.01, indicating that recipe experiments and the final large run used different optimization settings (Ossowski et al., 28 Nov 2025).

3. Evaluation protocol and headline results

Evaluation is averaged over 5 independent runs with different sampling seeds, using temperature 0.6, top-p 0.95, vLLM for inference, and a maximum response length of 8192 tokens, plus image tokens for up to 10 multimodal images at maximum resolution 262,144 pixels. For think-first baselines that fail to terminate, the paper uses a forced exit mechanism from S1 that appends an end-of-think token and forces answering (Ossowski et al., 28 Nov 2025).

The main reported result is that OctoMed achieves state-of-the-art performance among open-source sub-10B models across all three benchmark categories, and remains competitive with much larger proprietary systems. Its category-level averages and representative scores are as follows:

Category	OctoMed average	Representative scores
Text-only reasoning	67.83	MedQA 90.81, HeadQA 82.36, MedMCQA 72.70
Multimodal reasoning	50.36	MMMU-PRO (H) 42.52, NEJM 61.14, PMC-VQA 61.13
Multimodal classification	67.29	Brain Tumor 80.86, CoronaHack 71.22, Aptos 75.43

These gains are large relative to the untuned base checkpoint. The paper reports Qwen2.5-VL-7B-Instruct category overalls of 48.10 for text-only tasks, 37.19 for multimodal reasoning, and 30.49 for multimodal classification, versus OctoMed’s 67.83 / 50.36 / 67.29. The paper further highlights that OctoMed exceeds MedGemma-27B in overall text-only average (67.83 vs 66.56) and overall multimodal reasoning average (50.36 vs 44.25), while its multimodal classification average 67.29 exceeds GPT-4o’s 53.96 on that category (Ossowski et al., 28 Nov 2025).

The reported MedQA score of 90.81 is especially prominent. It is obtained with a 10-sample majority vote ensemble, and is described as higher than MedGemma-27B (85.17) and slightly above GPT-4o (90.72), though still below DeepSeek-R1 (93.16). This placement is central to the paper’s claim that large-scale, curated medical reasoning SFT can allow a relatively modest 7B model to rival much larger systems on medical reasoning benchmarks (Ossowski et al., 28 Nov 2025).

4. Ablative evidence and reasoning-length behavior

OctoMed’s ablations are organized around the proposition that data recipe design is the dominant lever. One of the clearest studies compares direct versus chain-of-thought prompting during fine-tuning on a 100k subset. The results are sharply task-dependent: for multimodal classification overall, Direct 65.46 exceeds CoT 63.33; for multimodal reasoning overall, CoT 38.15 exceeds Direct 23.08; and for text-only overall, CoT 52.19 exceeds Direct 29.23. The final recipe therefore adopts CoT prompting, not because it is uniformly superior, but because it is markedly better on reasoning-heavy tasks (Ossowski et al., 28 Nov 2025).

A second important axis is question-source composition. Training on a single source category primarily improves downstream tasks of the same type, whereas mixed-source training improves overall robustness without causing harmful interference. This is one of the paper’s main arguments for combining text-only reasoning, multimodal reasoning, and multimodal classification within the same SFT mixture (Ossowski et al., 28 Nov 2025).

A third axis is the number of valid reasoning traces per question. On MedQA, the paper reports 75.16 for 1 rejection sample/question, 3 epochs, 76.50 for 4 samples/question, 1 epoch, and a best peak accuracy of 85.01 for 16 samples/question. The interpretation offered is that multiple correct traces act as a form of regularization, improving robustness and peak generalization more effectively than merely reusing the same question-answer pairs for more epochs (Ossowski et al., 28 Nov 2025).

The paper also studies question filtering, including student-model proportion filtering, teacher-model proportion filtering, and GPT-4.1-mini difficulty scoring. Filtering improves sample efficiency early in training, but all methods converge to similar final performance as no filtering. The final OctoMed recipe therefore uses no question filtering, preferring maximum coverage and relying on rejection sampling for quality control (Ossowski et al., 28 Nov 2025).

One of the more distinctive analyses concerns reasoning trajectory length. Because the distilled traces vary in style and length across teachers and tasks, OctoMed is reported to show an emergent ability to self-calibrate reasoning trajectory lengths. The paper notes around 320 reasoning tokens on PMC-VQA, interpreted as a relatively straightforward task, and substantially longer traces on more difficult settings such as MedXpertQA and MMMU-PRO. This suggests that the model learns task-aware allocation of its reasoning budget without an explicit controller or explicit supervision on trajectory length (Ossowski et al., 28 Nov 2025).

5. OctoMed in the SFT-to-RL pipeline

A later paper studies when reinforcement learning helps medical vision-LLMs and uses OctoMed as the medical-SFT checkpoint, denoted $M_{\text{SFT}}$ , with Qwen2.5-VL-7B-Instruct as $M_{\text{Base}}$ and QoQ-Med as $M_{\text{RL}}$ . In that framing, OctoMed is not treated as a new RL architecture; it is treated as the support-expanding bridge model that makes RL worthwhile. The paper defines

$S_K(\mathcal{D}) = \mathrm{Pass@K}(\mathcal{D}), \qquad A(\mathcal{D}) = \mathrm{Acc@1}(\mathcal{D}), \qquad G_K(\mathcal{D}) = S_K(\mathcal{D}) - A(\mathcal{D}),$

where $S_K$ is latent support, $r_i = \left(r_i^{(1)}, \ldots, r_i^{(T_i)}\right),$ 0 is greedy decoding behavior, and $r_i = \left(r_i^{(1)}, \ldots, r_i^{(T_i)}\right),$ 1 is the support gap. Its main conclusion is that RL is most effective when the model already has non-trivial support: supervised medical fine-tuning expands support, and RL then sharpens the output distribution (Jeddi et al., 1 Mar 2026).

In controlled MedMNIST-v2 experiments, OctoMed improves both frozen visual separability and VLM decision quality relative to the base model. In the frozen vision probe, OCT rises from 84.30 to 91.10 and Retina from 44.00 to 52.00. In greedy VLM Accuracy@1, OCT rises from 32.62 to 82.78, Path from 26.48 to 64.53, and Blood from 13.16 to 76.00. The paper interprets these changes as evidence that medical SFT improves not only language-side alignment but also useful visual representations and, more importantly, expands answer support enough for RL to become effective (Jeddi et al., 1 Mar 2026).

The same paper instantiates a boundary-aware recipe by starting from OctoMed-7B and applying RL post-training on 8,000 multiple-choice questions sampled from PMC-VQA. On six medical VQA benchmarks, OctoMed-7B achieves 62.16 average, while the OctoMed-initialized RL model reaches 64.91, a +2.75 average-point improvement over OctoMed. The gains are selective rather than universal: PMC improves from 55.50 to 59.00, MMMU from 56.47 to 62.94, PathVQA from 63.00 to 65.50, and SLAKE from 84.00 to 88.0; VQA-Rad remains 79.00; MedX-M declines slightly from 35.00 to 34.50. The paper’s verdict is therefore not that RL supersedes OctoMed, but that OctoMed-style medical SFT is the foundation and RL is a secondary sharpening stage (Jeddi et al., 1 Mar 2026).

6. Limitations, risks, and broader ecosystem

The OctoMed paper names or implies several limitations. It is not a clinical deployment paper; strong benchmark performance does not establish clinical readiness. Reasoning faithfulness is not guaranteed, because rejection sampling verifies final answers rather than the full causal validity of every intermediate step. Hallucination and robustness risks remain, and the distilled traces inherit the priors and possible errors of DeepSeek-R1 and GPT-4o, as well as biases in the underlying medical benchmarks. Much of the pipeline depends on verifiable multiple-choice settings, which are easier to score and filter than open-ended clinical reasoning. The paper also notes broader concerns about possible contamination in the Qwen2.5 family and therefore tests transfer to InternVL3.5-8B and Qwen3-VL-8B-Instruct, finding that the data recipe transfers across model families (Ossowski et al., 28 Nov 2025).

In adjacent literature, the name OctoMed also functions as a reference point for a broader multimodal medical AI stack. Prior-AttUNet is described as a potentially valuable inclusion in an OCT-focused medical AI toolkit or benchmark collection like OctoMed, particularly as an anatomy-prior-guided RETOUCH baseline for cross-device retinal OCT fluid segmentation (Yang et al., 25 Dec 2025). OMT-SAM is similarly framed as relevant to OctoMed-like multimodal medical foundation models, because it shows how a medical promptable segmenter can be extended with CLIP-based image-text prompting for organ-aware segmentation (Zhang et al., 18 Mar 2025). A plausible implication is that OctoMed is being read not only as the name of a 7B medical reasoning model, but also as a locus for a wider ecosystem of medical multimodal methods spanning reasoning, segmentation, and modality-specific analysis.

A further, more architectural antecedent can be seen in the earlier proposal of “A global physician-oriented medical information system”, which described a centralized, physician-facing platform for structured patient intake, diagnostic support, treatment ranking, and feedback-driven improvement based on outcome data (0810.1991). This suggests a longer lineage in which present-day OctoMed-related work can be interpreted as part of a broader aspiration toward continuously learning, multimodal clinical intelligence systems—although OctoMed itself, as documented in the 2025 and 2026 papers, remains a benchmark-centered medical vision-language reasoner rather than a deployed clinical infrastructure.