VisPlay: Self-Evolving VLM Framework

• VisPlay is a self-evolving reinforcement learning framework that improves vision-language models by autonomously generating challenging visual questions and refining reasoning abilities.
• Its architecture decomposes a pretrained VLM into an Image-Conditioned Questioner and a Multimodal Reasoner that interact via iterative self-play and Group Relative Policy Optimization.
• Empirical evaluations demonstrate notable gains in reasoning accuracy, generalization, and reduced hallucinations across diverse multimodal benchmarks.

VisPlay is a self-evolving reinforcement learning (RL) framework designed to improve vision-LLMs (VLMs) on complex visual reasoning tasks using only large-scale unlabeled image data. By decomposing a pretrained VLM into two specialized roles—an image-conditioned Questioner and a multimodal Reasoner—VisPlay enables autonomous improvement through iterative role-based self-play. Joint training is performed via Group Relative Policy Optimization (GRPO), which balances the generation of challenging, diverse questions with the quality of model-generated ("silver") answers. Empirical evaluation demonstrates consistent gains in visual reasoning, generalization, and hallucination reduction across multiple benchmarks and VLM model families, illustrating a scalable, label-efficient path toward self-improving multimodal intelligence (He et al., 19 Nov 2025).

1. Conceptual Overview and Motivation

The motivation for VisPlay arises from the limitations of prior RL schemes for VLM improvement, which typically require costly human-annotated labels or heuristic and task-specific reward functions. VisPlay addresses scalability by enabling reasoning enhancement in VLMs solely from unlabeled imagery. The core insight is the construction of a self-evolving curriculum in which the model generates its own visual questions and supervisory signals, obviating the need for external annotations.

In VisPlay, a single pretrained VLM is assigned, per image, two interacting roles:

• Image-Conditioned Questioner ($Q_\theta$): Proposes diverse, challenging, and valid questions about an input image to probe the model's reasoning frontier.
• Multimodal Reasoner ($S_\phi$): Attempts to answer the question-image pair, generating answer candidates whose consensus serves as a “silver” supervisory label.

Through repeated role interaction and joint training, both the inquisitiveness of the Questioner and the reasoning capacity of the Reasoner increase over iterations, establishing a fully self-supervised training loop.

2. Architectural Framework

VisPlay is realized by decomposing a VLM into the following roles—both derived from a shared backbone and updated cooperatively:

Image-Conditioned Questioner ($Q_\theta$):

• Input: Raw image $I$.
• Output: Group $\{x_i\}_{i=1}^G$ of distinct and difficult visual questions.
• Purpose: Explore the model’s reasoning limits and induce learning by questioning.

Multimodal Reasoner ($S_\phi$):

• Input: Image–question pair $(I, x)$.
• Output: Set of answer candidates $\{y_j\}_{j=1}^G$.
• Purpose: Solve automatically generated questions and provide “silver” labels (majority-vote answers).

The two agents, Questioner and Reasoner, are updated iteratively, with each self-play episode gradually escalating question difficulty and answer accuracy. This co-evolution is central to the VisPlay approach.

3. Training Protocol and Group Relative Policy Optimization

The joint training in VisPlay is realized by casting both roles as stochastic policies optimized under a Group Relative Policy Optimization (GRPO) objective. GRPO eliminates the need for classic value function learning by using groupwise comparisons of reward signals among samples.

GRPO Objective

Given old policy $\pi_{\theta_{\mathrm{old}}}$, current policy $\pi_\theta$, and $G$ group samples $\{x_i\}$ with associated scalar rewards $\{r_i\}$, the normalized advantage for each sample is: $$\hat A_i = \frac{r_i - \mu_r}{\sigma_r + \varepsilon}, \quad \mu_r = \frac{1}{G} \sum_{k=1}^G r_k, \quad \sigma_r = \sqrt{\frac{1}{G} \sum_{k=1}^G (r_k - \mu_r)^2}$$ The GRPO surrogate loss is minimized as: $$\mathcal{L}_{\mathrm{GRPO}}(\theta) = -\frac{1}{G} \sum_{i=1}^G \min \left( \rho_i(\theta) \hat A_i, \;\mathrm{clip}\left( \rho_i(\theta), 1 - \epsilon, 1 + \epsilon \right) \hat A_i \right) + \beta \, \mathrm{KL}\left( \pi_{\theta_{\mathrm{old}}} \parallel \pi_\theta \right),$$ where $$\rho_i(\theta) = \frac{\pi_\theta(x_i)}{\pi_{\theta_{\mathrm{old}}}(x_i)}$$ and $\epsilon, \beta$ are hyperparameters.

Reward Structure

Rewards for sample quality, question diversity, and answer uncertainty are rigorously defined to promote a balanced curriculum:

• For the Questioner:
  • Pseudo-label confidence: $c = \max_y \hat p(y|x, I)$, with $\hat p(y|x,I)=\frac{1}{m}\sum_{j=1}^m \mathbbm{1}\{y_j=y\}$.
  • Uncertainty (difficulty) reward: $r_{\mathrm{unc}}(x,I) = 1 - |2c - 1|$, maximized when answer consensus is ambiguous.
  • Diversity penalty: $r_{\mathrm{div}}(x_i, I) = \lambda \frac{|C_k|}{G}$, penalizing redundancy via clustering (e.g., BLEU similarity).
  • Formatting validity: $\mathbbm{1}_{\mathrm{valid}}(x)$ ensures questions meet format constraints.
  • Combined reward: $r_i = \mathbbm{1}_{\mathrm{valid}}(x_i) \cdot \mathrm{ReLU}(r_{\mathrm{unc}}(x_i, I) - r_{\mathrm{div}}(x_i, I))$.
• For the Reasoner:
  • On curated dataset $\mathcal{S} = \{(I, x, \tilde y)\}$, with majority vote answer $\tilde y$ and thresholded confidence $c$:

  $$r_j = \begin{cases} 1, & y_j = \tilde y \ 0, & \text{otherwise} \end{cases}$$

Algorithmic Steps

Self-play proceeds over $K$ iterations as follows:

1. Questioner Update: Sample groups of questions per image; compute rewards; update $\theta$ via $\mathcal{L}_{\mathrm{GRPO}}$.
2. Dataset Construction: For each image, retain questions resulting in moderate answer confidence ($c \in [\tau_{\mathrm{low}},\,\tau_{\mathrm{high}}]$).
3. Reasoner Update: Sample answers for curated question–image pairs; update $\phi$ via same GRPO objective.

4. Implementation, Hyperparameters, and Computational Setup

VisPlay has been validated on multiple model families and large-scale datasets. Key components include:

Hyperparameter	Value/Setting	Comment
Model architectures	Qwen2.5-VL-3B/7B, MiMo-VL-7B	All instruction-tuned
Unlabeled dataset	Vision-47K (~47K web images)	Diverse domains (charts, medical, etc.)
Group size $G$	8	Number of samples per batch
Reasoner samples $m$	4	Votes per question for majority label
Question budget $N$	Few hundred per image	Generates wide candidate set
Confidence thresholds	$\tau_{\mathrm{low}}=0.25$, $\tau_{\mathrm{high}}=0.75$	Intermediate-difficulty supervision
Diversity weight $\lambda$	≈ 0.1	Penalty scaling for question similarity
Compute	4 A100 GPUs, micro-batch size 1	Long chain-of-thought support
Training schedule	3–5 self-play iterations; 10 epochs/iteration	Reasoner re-trained for 1 epoch

Training progresses with iterative updates to both policies, typically through 3–5 self-play cycles.

5. Empirical Findings and Evaluation

VisPlay achieves significant improvements over pretrained VLM baselines across a suite of multimodal understanding and reasoning tasks. Selected empirical results include:

Model	Benchmark Suite	Baseline (%)	VisPlay, Iter 3 (%)	Gain
Qwen2.5-3B	MMMU, MM-Vet, RealWorldQA, VisNumBench	30.6	47.3	+16.7
Qwen2.5-7B	Same	40.4	48.6	+8.2
MiMo-7B	Same	43.6	45.7	+2.1

Further, VisPlay achieves gains of 8–14 points on visual mathematics (MathVerse, MATH-Vision), and on hallucination detection (HallusionBench), Qwen2.5-3B increases accuracy from 32.8 to 94.9 at iteration 2. Ablation against human-labeled GRPO shows that VisPlay (iteration 3) reaches competitive overall accuracy (47.3% vs. 47.1% on Qwen2.5-3B) with much lower hallucination rates.

6. Co-evolution Dynamics and Analysis

A hallmark of VisPlay is the co-evolutionary “bootstrapping” dynamic: as the Questioner increases the difficulty (quantified by uncertainty rewards $r_{\mathrm{unc}}$), the Reasoner’s accuracy concurrently rises. Visualizations illustrate a trajectory from early-stage simple queries (e.g., object counting) to late-stage compositional and comparative reasoning. This demonstrates mutual escalation in task complexity and reasoning capabilities over self-play iterations.

A plausible implication is that this self-sustaining loop—posing increasingly complex questions and synthesizing supervised signals via model-generated consensus—extends the feasible frontiers of VLM performance without reliance on curated data.

7. Significance and Outlook

VisPlay presents a framework in which vision-language modeling can be scaled without human annotation by leveraging self-play and group-based RL optimization. Its capacity to induce curriculum learning, control for diversity, and generate high-quality supervision from unlabeled data sets a precedent for self-supervised multimodal intelligence research. The results highlight state-of-the-art gains across eight multimodal benchmarks and point to a scalable paradigm for future advances in multimodal autonomous learning (He et al., 19 Nov 2025).