TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RL

Abstract: Reinforcement learning (RL) for visual reasoning needs scalable, verifiable, and controllable training signals. Existing visual RL post-training trains on static curated datasets, with fixed image-question-answer samples bounded by their collection budget. In this work, we introduce TRON (Targeted, Rule-verifiable Online eNvironments), an online environment substrate: a training rollout is generated on demand by a controllable generator-verifier program that samples a fresh latent visual state, renders an image, asks a question, and exactly verifies the answer. A single run can therefore draw an unbounded stream of fresh instances at the difficulty level required by the current curriculum. The current TRON suite contains 520 environments organized into five ability buckets (spatial, mathematical, diagram, pattern/logic, and counting); the same substrate supports both a single full model trained on all buckets and per-bucket ability-specialist models, with no additional data collection. We also introduce a substrate analysis covering generation reliability, instance and level diversity, cross-environment near-duplicates, and base-model pass rate by difficulty level. RL post-training with METHOD consistently improves performance on ten external multimodal reasoning benchmarks across Qwen3-VL-4B, Qwen2.5-VL-7B, and MiMo-VL-7B-SFT.

• The paper presents a procedural generator-verifier substrate that offers scalable, curriculum-controlled visual reasoning tasks via 520 rigorously audited environments.
• It categorizes tasks into five ability buckets with precise difficulty calibration, ensuring adaptive challenge progression and improved training signals.
• The study demonstrates robust improvements on external multimodal benchmarks and emphasizes that capability coverage is key to effective RL transfer.

TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RL

Motivation and Problem Statement

Visual reasoning in RL settings is bottlenecked by the lack of scalable, verifiable, and curriculum-controllable training signals. Prevailing approaches utilize static, human-annotated or synthetically compiled datasets comprising fixed image-question-answer tuples. These are inadequate for RL optimization due to their bounded size, lack of curriculum control, and high pretraining overlap with current VLMs, which diminishes their utility for post-training. Furthermore, generating diverse, on-demand, and verifiable environments for visual reasoning presents unique technical challenges compared to symbolic domains such as mathematics or code, where exact rewards can be programmatically derived or verified.

TRON Substrate: Architecture and Environment Design

TRON introduces a procedural generator-verifier substrate encompassing 520 environments, systematically categorized into five ability buckets: spatial, mathematical, diagram, pattern/logic, and counting reasoning (Figure 1). Each environment is defined by a generator-verifier pair—$G$, a generator, samples latent visual states and renders images with associated questions and ground-truth answers; $V$, a verifier, deterministically maps model predictions to a rule-based scalar reward by comparing predictions against the ground truth using task-specific matching (exact, set, ordered, or custom logic). The environments are strictly synthetic, enabling precise control over instance diversity and difficulty calibration. Difficulty progression within each environment is parameterized by a per-sample level $\ell \in \{0, \ldots, 9\}$, controlling the compositional or perceptual complexity of the generated tasks.

Figure 1: Overview of the TRON substrate: 520 rule-verifiable generator-verifier environments, each organized by targeted ability, producing on-demand, difficulty-calibrated visual reasoning tasks.

The procedural approach exposes three diversity axes: (1) environment class (the reasoning mechanism targeted), (2) intra-environment instance diversity (via latent state randomization), and (3) difficulty-driven generator variability. This enables on-demand sampling of unbounded novel instances at varying challenge levels and supports both monolithic and ability-specialized RL curricula.

Fine-Grained Environment Taxonomy

Each ability bucket contains a diverse array of reasoning mechanisms, with representative generator-verifier programs for each (Figures 2–6):

• Spatial: 3D mental rotation, pathfinding, perspective taking, mechanical diagrams.
• Mathematical: Compositional geometry, coordinate reasoning, graph calculus, symbolic arithmetic, visual probability/statistics.
• Diagram: Chart/figure parsing, table/join operations, graph/process following, scientific visualizations.
• Pattern/Logic: Matrix and sequence pattern induction, symbolic logic, constraint puzzles, algorithmic search.
• Counting/Quantitative: Object and region counting, path enumeration, 3D hidden block inference, robust feature estimation.

Immediately below is an example of spatial reasoning generators, showcasing instance and difficulty scaling.

Figure 2: Spatial Reasoning problems: maze navigation and cube-net face reasoning; difficulty progresses by compositionality and perceptual complexity.

Subsequent figures illustrate mathematical, diagram, pattern/logic, and counting environments; each portrays consistent scaling along their canonical task axes.

Figure 3: Mathematical Reasoning: levels progress from basic geometric queries to multi-step angle chase and high-entropy probability trees.

Figure 4: Visual Diagram Understanding: scientific graph tasks and circuit prediction; harder levels compound plots, interpolation, and circuit depth.

Figure 5: Visual Pattern and Logical Reasoning: pattern grid induction, matrix completion; increased grid size, rule complexity, and invariance.

Figure 6: Counting and Quantitative Estimation: occlusion, clutter, missing-cell estimation; higher levels escalate objects and visual distractors.

Training Protocol and Curriculum Control

TRON operates within a DAPO-style RL post-training regime. Training data are generated online: sampling $(\text{env}, \ell, \text{seed})$ tuples yields fresh (image, question, reference answer) triples. Images are further augmented by noise/rotation/jitter to regularize the policy. The generator’s difficulty ladder is advanced in a curriculum fashion: each environment tracks its rolling success and self-promotes to higher $\ell$ only after consistent model mastery at the current level, ensuring adaptive, continual challenge during RL training.

An 8-response rollout per prompt is scored by the exact, deterministic verifier; rewards are optimized using prompt-grouped advantages with group filtering, KL regularization, and entropy penalties. Hardware is set for scalable parallelization (4×H100 80GB with tensor parallelism). The substrate supports both single-model joint training and ability-specialist configurations via sampler control.

Substrate Audit: Quality, Diversity, and Difficulty

Prior to any RL, a comprehensive audit evaluates each generator-verifier environment across three axes:

1. Quality: Generation success rate, render validity, field completeness, and verifier soundness (minimum 99.1% overall yield).
2. Difficulty Axis: Empirically, for base Qwen3-VL-4B, mean pass rates monotonically drop from 72.8% ($\ell=0$) to 41.3% ($\ell=9$),. This validates that difficulty progression induces genuine performance challenges, not superficial changes.
3. Diversity: Intra-environment seed/prompt/answer entropy, inter-level visual and question shifts, and cross-environment near-duplicate detection. The majority (83.7%) of environments score grade A/B for diversity, with outliers iteratively re-authored for coverage.

Impact on External Benchmarks

RL post-training with TRON yields consistent and robust improvements across external multimodal reasoning benchmarks (WeMath, MM-HELIX, MME-Reasoning, SpatialEval, LogicVista, CharXiv, MathVerse, DynaMath, PuzzleVQA, ChartQAPro). Gains are observed for all three backbones, with improvements observed in both mean and individual benchmark scores and not concentrated in any specific benchmark type.

Strongest gains are observed on structured-reasoning benchmarks (e.g., MM-HELIX, SpatialEval): tasks with rigorous operations, grid or graph structure, and deterministic verifiability align closely with the kind of procedural mechanisms encoded by TRON.

Ability-Specialist Analysis and Transfer

A major empirical finding is the separation between visual format (image category) and underlying capability (skills/mechanisms required). Ability-specialist models (trained on one bucket only) exhibit:

• Within-bucket transfer: Specialists substantially outperform base models on external tasks whose visual format matches their training bucket.
• Cross-bucket transfer: Explicit ability transfer is observed when the underlying reasoning mechanism is shared, even when the visual format diverges; e.g., mathematical specialists improve maze navigation and positional inference.
• No sufficient gain from format alone: Visual format alignment does not guarantee best-in-domain performance unless the actual required reasoning skills are also present.

The practical implication is that capability coverage—not visual surface diversity alone—is the key driver for robust out-of-distribution (OOD) and cross-domain transfer in visual reasoning RL.

Figure 7: Training dynamics for ability specialists: consistent validation accuracy improvement and curriculum escalation across buckets; overfitting controlled for small buckets.

Discussion and Future Implications

TRON provides a methodology enabling RL for visual reasoning tasks to scale without dataset collection bottlenecks and to maintain continual curriculum advancement. The procedural, rule-verifiable substrate introduces auditability and precise measurement of quality, difficulty, and diversity, facilitating reproducible research and ablation studies.

From a theoretical standpoint, the findings validate that structured environmental diversity and fine-grained ability decomposition—rather than dataset size or visual variety alone—are essential for robust visual reasoning generalization. In practice, such substrates can underpin modular RL pipelines for next-generation VLMs, supporting both foundation and specialist models.

Future directions would include expanding visual style and linguistic coverage to further bridge the sim2real gap, integrating more realistic scientific/photographic domains, and developing adaptive curriculum mechanisms based directly on dynamically inferred skills gaps. Ongoing work will also seek to further decouple and measure the interplay between surface modality and underlying reasoning mechanisms in transfer learning for VLMs.

Conclusion

TRON constitutes an extensible, highly diverse online generator-verifier substrate for RL-based visual reasoning, validated with comprehensive audits and producing measurable gains in SOTA VLMs on standardized external evaluations. The procedural, difficulty-laddered environments and their ability-driven organization set a new standard for visual reasoning post-training, enabling adaptive curriculum-driven model improvement untethered from static dataset limitations.