Med-Evo: Test-time Self-evolution for Medical Multimodal Large Language Models

Abstract: Medical Multimodal LLMs (MLLMs) have demonstrated remarkable capabilities across diverse healthcare tasks. However, current post-training strategies, such as supervised fine-tuning and reinforcement learning, heavily depend on substantial annotated data while overlooking the potential of unlabeled test data for model enhancement. This limitation becomes particularly pronounced in medical domains, where acquiring extensive labeled medical data is difficult due to the strict data sensitivity and annotation complexity. Moreover, leveraging test data poses challenges in generating reliable supervision signals from unlabeled samples and maintaining stable self-evolution. To address these limitations, we propose Med-Evo, the first self-evolution framework for medical MLLMs that utilizes label-free reinforcement learning to promote model performance without requiring additional labeled data. Our framework introduces two key innovations: $1)$ Feature-driven Pseudo Labeling (FPL) that identifies semantic centroids from all heterogeneous candidate responses to select pseudo labels in each rollout, and $2)$ Hard-Soft Reward (HSR) that combines exact match with token-level assessment and semantic similarity to provide hierarchical reward. Experiments on three medical VQA benchmarks and two base MLLMs show clear advantages of our approach over SOTA methods, with significant improvements of 10.43\% accuracy and 4.68\% recall on the SLAKE dataset using Qwen2.5-VL, showing the effectiveness of our method.

• The paper demonstrates a test-time reinforcement learning framework (Med-Evo) that enhances medical multimodal LLM's without human annotations. This increases the model’s accuracy by up to 10.43%.
• Med-Evo utilizes a unique semantic-centroid pseudo labeling method to overcome challenges in closed-ended questions & open-ended answers, effectively addressing lexical variations in medical phrasings to mitigate suboptimal convergence.
• The paper reveals that the most substantial improvements lie with the VQA benchmarks; however, the study notes potential limitations on scalability, encoder choice, and long-term stability.

Motivation and problem setting

Medical multimodal LLMs (MLLMs) are typically improved through supervised fine-tuning or reinforcement learning applied to labeled training corpora. The authors argue that this paradigm underuses a readily available resource in deployment: the unlabeled test data itself. In medical domains, the barrier to conventional post-training is acute—data sensitivity and annotation complexity make large labeled collections difficult to obtain, while clinical heterogeneity demands models capable of continuous, adaptive improvement. Med-Evo addresses this gap as a test-time training (TTT) framework that converts the model's own test-time responses into supervision signals and iteratively refines the policy through a closed loop of rollout generation, pseudo labeling, reward computation, and GRPO-based optimization.

The paper identifies two failure modes in existing TTT methods when applied to medical VQA. First, majority voting over sampled rollouts is unreliable when medical reasoning produces heterogeneous candidate answers with no dominant lexical form. Second, binary exact-match rewards and entropy minimization discard learning signal from semantically equivalent or partially correct open-ended answers, leading to suboptimal convergence.

Method

Med-Evo operates per test instance in four stages. Given an image–query pair $x$, the current policy $\pi_{\theta_{old}}$ samples $N$ candidate responses (rollouts). A semantic encoder $\mathbf{E}$ embeds each response, and Feature-driven Pseudo Labeling (FPL) selects the pseudo label $\bar{y}$ as the candidate whose embedding is closest to the rollout's semantic centroid $c = \frac{1}{N}\sum_i f_i$. This replaces string-matching majority voting with clustering in feature space, which is more robust to lexical variation across medically equivalent phrasings.

Supervision is then provided by the Hard-Soft Reward (HSR), a hierarchical composite:

$$r_{\text{ours}} = \alpha \cdot r_{\text{binary}} + \beta \cdot r_{\text{jaccard}} + (1-\alpha-\beta) \cdot r_{\text{semantic}}$$

The hard component is binary exact match; the soft components are token-level Jaccard similarity and encoder-derived semantic similarity, the latter normalized by the maximum distance within the rollout so it is bounded in $[0,1]$ and adapts its discriminativeness to candidate diversity. An adaptive rule applies the binary reward alone for closed-ended questions and the full composite for open-ended ones. Rewards feed into Group Relative Policy Optimization (GRPO) with group-normalized advantages, clipped surrogate objectives, and KL regularization against the reference policy, after which the reference model is refreshed to enable progressive self-evolution.

Experimental results

The evaluation covers three English medical VQA benchmarks—SLAKE, VQA-Rad, and VQA-Med—with two base models: Qwen2.5-VL-3B-Instruct (general-purpose) and MedVLM-R1 (medical-specialized). Baselines are EN-INF (entropy minimization), TTRV (frequency/entropy-based unsupervised rewards), and TTRL (majority-voting TTT), plus the unaugmented base model. Training uses 32 rollouts per instance with 16 used for optimization, RoBERTa as the semantic encoder, $\alpha=0.85$, $\beta=0.05$, on 4 RTX A6000 GPUs.

Base model	Dataset	Metric	Base	Best baseline	Med-Evo
Qwen2.5-VL-3B	SLAKE	Acc	68.73	72.68 (TTRL)	78.87
Qwen2.5-VL-3B	SLAKE	Recall	34.70	35.55 (EN-INF)	39.38
Qwen2.5-VL-3B	VQA-Rad	Acc	68.53	68.53	69.32
Qwen2.5-VL-3B	VQA-Med	Acc	56.88	56.88	57.79
MedVLM-R1	SLAKE	Acc	61.41	71.27 (TTRL)	71.55
MedVLM-R1	VQA-Rad	Acc	49.00	59.36 (TTRV)	61.75
MedVLM-R1	VQA-Med	Acc	44.04	46.79 (EN-INF)	48.62

The headline result is a 10.43% accuracy improvement over the base model on SLAKE with Qwen2.5-VL-3B, alongside consistent gains on recall and ROUGE across all datasets and both base models. Notably, several baselines degrade below the base model on some settings (e.g., TTRL drops VQA-Med accuracy from 44.04% to 38.53% with MedVLM-R1), whereas Med-Evo improves over the base model everywhere—a contrast the authors use to argue that their pseudo-labeling and reward design stabilize self-evolution where competing signals do not.

Ablations attribute gains to both components individually: FPL consistently achieves higher pseudo-label hit rates than majority voting at pass@8 and pass@16, and the full framework outperforms partial configurations combining only one innovation. Training curves show reward score correlating positively with closed-ended accuracy and open-ended recall, supporting HSR as an effective optimization signal.

Limitations and open questions

Several constraints qualify these results. Evaluation is restricted to three VQA benchmarks and two relatively small (3B-scale) base models; scaling behavior to larger MLLMs is untested. Gains on VQA-Med are modest relative to SLAKE (e.g., +4.58% accuracy with MedVLM-R1), suggesting dataset difficulty or distribution shift affects self-evolution efficacy. The method depends on the quality of the semantic encoder (RoBERTa here); whether domain-specific encoders would further improve FPL hit rates is not examined. Hyperparameters $\pi_{\theta_{old}}$0 and $\pi_{\theta_{old}}$1 were fixed rather than tuned per dataset, leaving their sensitivity uncharacterized. Finally, because pseudo labels derive from the model's own outputs, the framework inherits a potential confirmation-bias risk—the paper does not measure how often consensus among rollouts is systematically wrong, nor how performance evolves under prolonged self-training beyond the reported horizon.

Conclusion

Med-Evo demonstrates that label-free test-time reinforcement learning can yield substantial improvements for medical MLLMs by replacing majority voting with semantic-centroid pseudo labeling and binary rewards with a hierarchical hard-soft reward. Consistent gains across two base models and three benchmarks, including large improvements where prior TTT methods regress, indicate that reliable pseudo-supervision is the central bottleneck for test-time adaptation in medical VQA. The approach offers a practical path to continual model improvement in annotation-scarce clinical settings, contingent on resolving the open questions around scale, encoder choice, and long-run stability of self-generated supervision.