Visually-Anchored Policy Optimization (VAPO)

• VAPO is a paradigm that couples policy updates with visual evidence, ensuring actions or tokens are grounded in image-based signals.
• In robot manipulation, VAPO employs visual affordance maps for planning, achieving up to 90% success and 4x faster training compared to baselines.
• In multimodal tasks, explicit visual anchors counteract visual forgetting and improve chain-of-thought reasoning and transcription fidelity.

Visually-Anchored Policy Optimization (VAPO) denotes a set of policy-optimization schemes in which learning updates are tied to visual evidence, but the acronym is not stable across the arXiv literature. In the supplied corpus, it names a robot-manipulation method built around visual affordances, a multimodal reasoning method that counteracts visual forgetting, and a SlideASR post-training method that enforces “Look before Transcription”; by contrast, a separate long-chain-of-thought RL line uses the same acronym for Value-based Augmented Proximal Policy Optimization, a purely text-only method with no visual or multimodal component (Borja-Diaz et al., 2022, Tian et al., 30 Sep 2025, Hu et al., 8 Oct 2025, Yue et al., 7 Apr 2025). This suggests a broader visually anchored policy-optimization family whose common principle is to make policy improvement depend not only on task reward, but also on signals that measure whether actions, tokens, or trajectories are grounded in visual input.

1. Terminology and scope

The main source of ambiguity is terminological rather than methodological. Several papers use closely related labels—Visual Affordance-guided Policy Optimization, Vision-Anchored Policy Optimization, Visually-Anchored Policy Optimization, Visually-Guided Policy Optimization, Perception-Grounded Policy Optimization, and Reflection-Anchor Policy Optimization—for methods that explicitly couple optimization to visual evidence. A different 2025 reasoning paper explicitly states that VAPO stands for Value-based Augmented Proximal Policy Optimization, not “Visually-Anchored Policy Optimization”, and further states that the method is a purely text-only RL framework for long chain-of-thought reasoning in LLMs (Yue et al., 7 Apr 2025).

Usage of the acronym or closely related label	Core mechanism	Representative paper
Visual Affordance-guided Policy Optimization	Self-supervised affordance model from teleoperated play, model-based planning, local SAC	(Borja-Diaz et al., 2022)
Vision-Anchored Policy Optimization	Visual claims inserted as anchors along multimodal reasoning trajectories	(Tian et al., 30 Sep 2025)
Visually-Anchored Policy Optimization for SlideASR	> <answer> reasoning with format, OCR, ASR, and visual anchoring rewards	(Hu et al., 8 Oct 2025)
>	Value-based Augmented Proximal Policy Optimization	Long-CoT, text-only RL with value pretraining, decoupled GAE, and length-adaptive GAE

Within the visually anchored sense, the central idea is consistent: the policy is not optimized solely from end-task correctness, but from intermediate signals that quantify whether the model is actually using the image, slide, or affordance-bearing region. The anchor can be a spatial affordance map, a token-level visual-dependence score, a masked-versus-unmasked KL divergence, or a structured intermediate output whose fidelity can be verified against the visual input.

2. Affordance-guided robot control

A 2022 robotics formulation proposed Visual Affordance-guided Policy Optimization (VAPO) for manipulation in human-centered environments (Borja-Diaz et al., 2022). The method learns a self-supervised visual affordance model from human teleoperated play data and then uses that model both for motion planning and for local model-free RL. The affordance model predicts a binary affordance mask $A \in \mathbb{R}^{H \times W}$ and a direction field $V \in \mathbb{R}^{H \times W \times 2}$ whose vectors point affordance pixels toward affordance centers. Hough voting converts these dense predictions into actionable centers in image space (Borja-Diaz et al., 2022).

The control stack is explicitly hybrid. A static-camera affordance model supports model-based free-space planning toward an affordance center, while a gripper-camera affordance model supports local contact-rich manipulation under Soft Actor-Critic. The overall policy is a state-dependent mixture,

$$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$

so that model-based control dominates far from the target region and RL dominates once the end effector is near the affordance center (Borja-Diaz et al., 2022).

The reward is visually anchored through the distance to the affordance center: $$r(s,a) = \lambda_1 R_{\text{succ}} + \lambda_2 R_{\text{aff}} + \lambda_3 R_{\text{out}}.$$ Here $R_{\text{succ}}$ rewards task completion, $R_{\text{aff}}$ increases as the end effector gets closer to the affordance center, and $R_{\text{out}}$ penalizes leaving the local neighborhood (Borja-Diaz et al., 2022). In this formulation, “anchoring” is literal: the learned policy is biased toward the same object regions favored by people during play.

Empirically, this affordance-guided design produced large sample-efficiency and generalization gains. In simulated grasping, VAPO reached success rate $\approx 0.6$ after 100k steps, whereas the local-SAC baseline required 400k steps to reach the same level; after 400k steps, VAPO reached about 0.90 overall success and the baseline saturated around 0.60 (Borja-Diaz et al., 2022). In zero-shot evaluation on objects unseen by both the affordance model and RL, VAPO succeeded on 13/15 objects versus 8/15 for the baseline, and on drawer opening it achieved 0.84 success over 100 episodes versus 0.52 for local-SAC (Borja-Diaz et al., 2022). The paper reports that the policies train 4x faster than the baselines and generalize better to novel objects because the visual affordance model can anticipate their affordance regions (Borja-Diaz et al., 2022).

3. Vision-anchored reasoning in large vision-LLMs

A later LVLM line uses Vision-Anchored Policy Optimization (VAPO) to address what it terms the “dual nature” of multimodal reasoning: RL-based reasoning improves logic, but extended chain-of-thought can degrade perceptual grounding (Tian et al., 30 Sep 2025). The paper operationalizes this through an early-decision analysis and attention tracing. Accuracy rises as the first portions of the chain are generated, then later reasoning steps can hurt accuracy; the degradation is especially pronounced on vision-heavy benchmarks such as MMStar and HallusionBench (Tian et al., 30 Sep 2025). The paper attributes this to visual forgetting, defined operationally as progressively downweighting or ignoring image tokens during autoregressive generation. In its error analysis, perception errors account for 55.23% of all failures under full reasoning, and 32.35% of these are recoverable via early decision (Tian et al., 30 Sep 2025).

The method introduces visual anchors by generating balanced true/false visual claims with GPT-5, inserting them at randomly chosen positions along the reasoning trajectory, and asking the model to answer each claim with a binary judgment. If $s_k$ denotes whether anchor $k$ is answered correctly, then the perception reward is

$V \in \mathbb{R}^{H \times W \times 2}$0

with default $V \in \mathbb{R}^{H \times W \times 2}$1 and $V \in \mathbb{R}^{H \times W \times 2}$2 (Tian et al., 30 Sep 2025). This is then gated by answer correctness: $V \in \mathbb{R}^{H \times W \times 2}$3 with default $V \in \mathbb{R}^{H \times W \times 2}$4 (Tian et al., 30 Sep 2025). The gating is intended to prevent reward hacking by ensuring that perceptual fidelity is rewarded only when the final task answer is correct.

Training uses ViRL39K and the GRPO machinery, but replaces plain answer-only reward with the vision-anchored reward above. The resulting VAPO-Thinker-7B improves both math-heavy and vision-heavy performance. On the general-purpose and vision-heavy set MMMU, MMStar, HallusionBench, and MMVet, the best 7B baseline averages 59.9%, whereas VAPO-Thinker-7B reaches 63.1%; on HallusionBench specifically, the score rises from 49.5 to 57.4 (Tian et al., 30 Sep 2025). The paper further shows that VAPO reduces visual forgetting in attention analyses and makes accuracy as a function of reasoning progress monotone increasing or non-decreasing, rather than peaking early and falling later (Tian et al., 30 Sep 2025).

A central significance of this formulation is that it turns visual grounding into an explicit RL target without modifying the forward architecture. The visual anchor is neither a new module nor a supervised chain-of-thought; it is a reward-carrying probe inserted into the trajectory so that longer reasoning is incentivized to remain visually accountable.

4. Related visually anchored optimization mechanisms

Several adjacent methods make the same design move—reweighting optimization toward visually sensitive tokens or trajectories—even when they do not use the exact VAPO name. This suggests a broader visually anchored policy-optimization family.

Token Preference Optimization with self-calibrated visual-anchored rewards defines a token-level visual-anchored reward as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one, then feeds that reward into a DPO-style preference objective (Gu et al., 2024). For token $V \in \mathbb{R}^{H \times W \times 2}$5,

$V \in \mathbb{R}^{H \times W \times 2}$6

and the self-calibrated reward is

$V \in \mathbb{R}^{H \times W \times 2}$7

with $V \in \mathbb{R}^{H \times W \times 2}$8 in the main experiments (Gu et al., 2024). On LLaVA-1.5-7B, TPO raises AMBER F1 from 74.3 to 85.0, improves MMHal score from 2.01 to 2.47 while reducing hallucination rate from 61.46 to 51.04, and raises HallusionBench Hard from 41.16 to 48.37 (Gu et al., 2024).

Visual-Advantage On-Policy Distillation (VA-OPD) introduces token-level visual advantage

$V \in \mathbb{R}^{H \times W \times 2}$9

then uses VA both for rollout-level reweighting and for token-level KL averaged separately within high-VA and low-VA groups (Liu et al., 21 May 2026). The paper reports that the top 10% of tokens by VA account for ~93% of the total VA mass, indicating that visual supervision is concentrated in a small minority of positions (Liu et al., 21 May 2026). In the 8B$$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$02B Geometry3K setting, Math Avg rises from 45.4 under standard OPD to 48.3 under VA-OPD, and Visual Avg rises from 64.6 to 66.1 (Liu et al., 21 May 2026).

Visually-Guided Policy Optimization (VGPO) targets temporal visual forgetting through a Visual Attention Compensation mechanism and dual-grained advantage re-weighting (Wang et al., 10 Apr 2026). It computes token-level visual similarity to a visual prototype, boosts late-stage visually activated tokens, and then reshapes advantages at both the intra-trajectory and inter-trajectory levels. On Qwen2.5-VL-7B, VGPO reaches Avg-Math = 66.6 and Avg-Vision = 63.3, compared with 63.8 and 59.6 for DAPO in the same setting (Wang et al., 10 Apr 2026).

Perception-Grounded Policy Optimization (PGPO) formalizes token visual dependency as

$$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$1

then uses a threshold-gated, mass-conserving mechanism to amplify advantages on visually dependent tokens and suppress others (Ye et al., 2 Apr 2026). The paper reports an average gain of 18.7% across seven multimodal reasoning benchmarks and emphasizes that mass conservation is required to avoid variance explosion and training collapse (Ye et al., 2 Apr 2026).

Reflection-Anchor Policy Optimization (RAPO) studies the problem from an information-theoretic standpoint. It derives a lower bound on downstream visual gain that highlights two factors: local branching room, measured through token entropy, and downstream visual propagation potential, measured through suffix divergence from a vision-marginalized reference (Gong et al., 10 May 2026). Guided by this analysis, RAPO selects high-entropy reflection anchors and optimizes a chain-masked finite-window KL surrogate for downstream visual dependence (Gong et al., 10 May 2026).

Taken together, these papers support a common interpretation: visually anchored optimization is less a single algorithm than a research direction in which policy updates are deliberately concentrated on the small set of steps whose predictive distribution materially changes when visual evidence is altered, masked, or removed.

5. SlideASR and “Look before Transcription”

A distinct 2025 use of the term appears in “Look before Transcription: End-to-End SlideASR with Visually-Anchored Policy Optimization” (Hu et al., 8 Oct 2025). Here VAPO is a reinforcement-learning-based post-training method for omni-modal LLMs that use both audio and slide images. The motivating failure mode is specific: naïve omni-modal models often degenerate into OCR systems, copying slide text instead of transcribing speech. On SlideSpeech, the paper reports that this OCR-like behavior occurs in 13–63% of samples depending on the OLLM (Hu et al., 8 Oct 2025).

The method imposes a structured output format: $$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$7 and optimizes this reasoning process with four rewards: Format, OCR, ASR, and Visual Anchoring (Hu et al., 8 Oct 2025). The total reward is

$$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$2

with default $$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$3 (Hu et al., 8 Oct 2025). The visual anchoring reward uses entities correctly recognized in <think> and measures how faithfully they are reused in <answer> through an F1 score (Hu et al., 8 Oct 2025).

The benchmark introduced with the paper, SlideASR-Bench, contains SlideASR-S with 6,413 training samples, 44,240 entities, and 67.3 hours, plus 2,054 test samples, 13,895 entities, and 18.5 hours; it also contains SlideASR-R, a real set of 60 samples, 200 domain-specific entities, and 0.35 hours (Hu et al., 8 Oct 2025). On SlideSpeech test, VAPO-7B achieves WER 10.31, B-WER 2.87, U-WER 10.84, and Recall 97.32 (Hu et al., 8 Oct 2025). On SlideASR-S, VAPO-7B reaches WER 4.60, NE-WER 2.83, NE-FNR 2.97 for English and WER 2.13, NE-WER 3.78, NE-FNR 1.36 for Chinese; on SlideASR-R it achieves NE-WER 26.48 and NE-FNR 15.35 (Hu et al., 8 Oct 2025). On ChineseLips, VAPO-7B attains CER 1.298, outperforming contextless and naïve image-conditioned baselines (Hu et al., 8 Oct 2025).

This SlideASR formulation broadens the meaning of visually anchored policy optimization beyond vision-language question answering. The anchor is not a token-level KL or visual claim, but an explicitly supervised intermediate reasoning block whose correctness and reuse can be measured against the slide image. The method is therefore a process-control form of VAPO: RL optimizes not only the final transcript, but also a visually accountable intermediate reasoning protocol.

6. Naming collision, theoretical limits, and unresolved issues

A major misconception is to treat all “VAPO” papers as variants of the same method. In the long-CoT LLM literature, VAPO is explicitly defined as Value-based Augmented Proximal Policy Optimization, not visually anchored policy optimization (Yue et al., 7 Apr 2025). That paper describes a text-only RL framework for long chain-of-thought reasoning with offline value pretraining on Monte-Carlo returns, decoupled GAE, length-adaptive GAE, token-level PPO loss, asymmetric clipping (“Clip-Higher”), positive-example LM loss, and group sampling (Yue et al., 7 Apr 2025). On AIME 2024 with Qwen2.5-32B, it reports 60.4 avg@32, exceeding DeepSeek-R1-Zero-Qwen-32B at 47 and DAPO at 50, and reaching state-of-the-art performance in about 5,000 gradient update steps (Yue et al., 7 Apr 2025). Despite the acronym overlap, this line does not belong to the visually anchored family in the ordinary multimodal sense.

The accompanying theoretical critique argues that this value-based VAPO faces structural limitations in long-term credit assignment, value function representational capacity, and the translation of global value signals into local policy improvements, especially under sparse rewards (Shao et al., 3 Jun 2025). The analysis emphasizes that Monte Carlo targets are unbiased but high-variance, that early-step TD errors can be tiny and noisy when all steps share the same distant terminal reward, and that scalar value baselines are a blunt instrument for token-level reasoning decisions (Shao et al., 3 Jun 2025). These arguments are about long-horizon RL generally, but they also clarify why later visually anchored work moved toward richer intermediate signals, token-level dependence measures, and explicit anchors.

Across the visually anchored line itself, the unresolved issues are more heterogeneous. Vision-anchored reasoning depends on the quality of visual claims and on hyperparameters such as $$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$4, $$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$5, and $$\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),$$6 (Tian et al., 30 Sep 2025). SlideASR VAPO is tailored to textual slide content, incurs slower inference because of <think> + <answer>, and can fail if the internal OCR step is wrong (Hu et al., 8 Oct 2025). TPO notes that its visual corruption is global and suggests future object-specific corruption (Gu et al., 2024). RAPO emphasizes that its lower bound is local and that the training objective is only a surrogate for exact downstream visual gain (Gong et al., 10 May 2026). PGPO requires an additional masked forward pass and is only evaluated up to 7B models (Ye et al., 2 Apr 2026). A plausible implication is that visually anchored policy optimization is converging on a shared diagnosis—final-answer reward is too coarse for multimodal reasoning—but has not converged on a single best anchoring signal or a unified theory of when visual dependence should be injected, amplified, or regularized.