---
title: Visually-Anchored Policy Optimization (VAPO)
url: https://www.emergentmind.com/topics/visually-anchored-policy-optimization-vapo
type: topic
---

# Visually-Anchored Policy Optimization (VAPO)

Visually-Anchored Policy Optimization (VAPO) denotes a set of policy-optimization schemes in which learning updates are tied to visual evidence, but the acronym is not stable across the arXiv literature. In the supplied corpus, it names a robot-manipulation method built around visual affordances, a multimodal reasoning method that counteracts visual forgetting, and a SlideASR post-training method that enforces “Look before Transcription”; by contrast, a separate long-chain-of-thought RL line uses the same acronym for **Value-based Augmented Proximal Policy Optimization**, a purely text-only method with no visual or multimodal component [2203.00352][2509.25848][2510.08618][2504.05118]. This suggests a broader visually anchored policy-optimization family whose common principle is to make policy improvement depend not only on task reward, but also on signals that measure whether actions, tokens, or trajectories are grounded in visual input.

## 1. Terminology and scope

The main source of ambiguity is terminological rather than methodological. Several papers use closely related labels—*Visual Affordance-guided Policy Optimization*, *Vision-Anchored Policy Optimization*, *Visually-Anchored Policy Optimization*, *Visually-Guided Policy Optimization*, *Perception-Grounded Policy Optimization*, and *Reflection-Anchor Policy Optimization*—for methods that explicitly couple optimization to visual evidence. A different 2025 reasoning paper explicitly states that **VAPO stands for Value-based Augmented Proximal Policy Optimization, not “Visually-Anchored Policy Optimization”**, and further states that the method is a purely text-only RL framework for long chain-of-thought reasoning in LLMs [2504.05118].

| Usage of the acronym or closely related label | Core mechanism | Representative paper |
|---|---|---|
| Visual Affordance-guided Policy Optimization | Self-supervised affordance model from teleoperated play, model-based planning, local SAC | [2203.00352] |
| Vision-Anchored Policy Optimization | Visual claims inserted as anchors along multimodal reasoning trajectories | [2509.25848] |
| Visually-Anchored Policy Optimization for SlideASR | `<think><answer>` reasoning with format, OCR, ASR, and visual anchoring rewards | [2510.08618] |
| Value-based Augmented Proximal Policy Optimization | Long-CoT, text-only RL with value pretraining, decoupled GAE, and length-adaptive GAE | [2504.05118] |

Within the visually anchored sense, the central idea is consistent: the policy is not optimized solely from end-task correctness, but from intermediate signals that quantify whether the model is actually using the image, slide, or affordance-bearing region. The anchor can be a spatial affordance map, a token-level visual-dependence score, a masked-versus-unmasked KL divergence, or a structured intermediate output whose fidelity can be verified against the visual input.

## 2. Affordance-guided robot control

A 2022 robotics formulation proposed **Visual Affordance-guided Policy Optimization (VAPO)** for manipulation in human-centered environments [2203.00352]. The method learns a self-supervised visual affordance model from human teleoperated play data and then uses that model both for motion planning and for local model-free RL. The affordance model predicts a binary affordance mask \(A \in \mathbb{R}^{H \times W}\) and a direction field \(V \in \mathbb{R}^{H \times W \times 2}\) whose vectors point affordance pixels toward affordance centers. Hough voting converts these dense predictions into actionable centers in image space [2203.00352].

The control stack is explicitly hybrid. A static-camera affordance model supports model-based free-space planning toward an affordance center, while a gripper-camera affordance model supports local contact-rich manipulation under Soft Actor-Critic. The overall policy is a state-dependent mixture,
\[
\pi(a|s) = (1 - \alpha(s)) \cdot \pi_{\text{mod}}(a|s) + \alpha(s) \cdot \pi_{\text{rl}}(a|s),
\]
so that model-based control dominates far from the target region and RL dominates once the end effector is near the affordance center [2203.00352].

The reward is visually anchored through the distance to the affordance center:
\[
r(s,a) = \lambda_1 R_{\text{succ}} + \lambda_2 R_{\text{aff}} + \lambda_3 R_{\text{out}}.
\]
Here \(R_{\text{succ}}\) rewards task completion, \(R_{\text{aff}}\) increases as the end effector gets closer to the affordance center, and \(R_{\text{out}}\) penalizes leaving the local neighborhood [2203.00352]. In this formulation, “anchoring” is literal: the learned policy is biased toward the same object regions favored by people during play.

Empirically, this affordance-guided design produced large sample-efficiency and generalization gains. In simulated grasping, VAPO reached success rate \(\approx 0.6\) after **100k steps**, whereas the local-SAC baseline required **400k steps** to reach the same level; after 400k steps, VAPO reached about **0.90** overall success and the baseline saturated around **0.60** [2203.00352]. In zero-shot evaluation on objects unseen by both the affordance model and RL, VAPO succeeded on **13/15** objects versus **8/15** for the baseline, and on drawer opening it achieved **0.84** success over 100 episodes versus **0.52** for local-SAC [2203.00352]. The paper reports that the policies train **4x faster** than the baselines and generalize better to novel objects because the visual affordance model can anticipate their affordance regions [2203.00352].

## 3. Vision-anchored reasoning in large vision-language models

A later LVLM line uses **Vision-Anchored Policy Optimization (VAPO)** to address what it terms the “dual nature” of multimodal reasoning: RL-based reasoning improves logic, but extended chain-of-thought can degrade perceptual grounding [2509.25848]. The paper operationalizes this through an early-decision analysis and attention tracing. Accuracy rises as the first portions of the chain are generated, then later reasoning steps can hurt accuracy; the degradation is especially pronounced on vision-heavy benchmarks such as MMStar and HallusionBench [2509.25848]. The paper attributes this to **visual forgetting**, defined operationally as progressively downweighting or ignoring image tokens during autoregressive generation. In its error analysis, **perception errors account for 55.23% of all failures under full reasoning**, and **32.35%** of these are recoverable via early decision [2509.25848].

The method introduces **visual anchors** by generating balanced true/false visual claims with GPT-5, inserting them at randomly chosen positions along the reasoning trajectory, and asking the model to answer each claim with a binary judgment. If \(s_k\) denotes whether anchor \(k\) is answered correctly, then the perception reward is
\[
w_k = \exp\left(\beta \cdot \frac{a_k}{T}\right), \qquad
R_{\rm perc} = \frac{\sum_{k=1}^{K} w_k s_k}{\sum_{k=1}^{K} w_k},
\]
with default \(K=20\) and \(\beta = 1.5\) [2509.25848]. This is then gated by answer correctness:
\[
R_{i} = R_{\rm acc} + R_{\rm fmt} + \gamma \cdot \mathbf{1}[R_{\rm acc} = 1] \cdot R_{\rm perc},
\]
with default \(\gamma = 0.1\) [2509.25848]. The gating is intended to prevent reward hacking by ensuring that perceptual fidelity is rewarded only when the final task answer is correct.

Training uses ViRL39K and the GRPO machinery, but replaces plain answer-only reward with the vision-anchored reward above. The resulting **VAPO-Thinker-7B** improves both math-heavy and vision-heavy performance. On the general-purpose and vision-heavy set MMMU, MMStar, HallusionBench, and MMVet, the best 7B baseline averages **59.9%**, whereas VAPO-Thinker-7B reaches **63.1%**; on HallusionBench specifically, the score rises from **49.5** to **57.4** [2509.25848]. The paper further shows that VAPO reduces visual forgetting in attention analyses and makes accuracy as a function of reasoning progress monotone increasing or non-decreasing, rather than peaking early and falling later [2509.25848].

A central significance of this formulation is that it turns visual grounding into an explicit RL target without modifying the forward architecture. The visual anchor is neither a new module nor a supervised chain-of-thought; it is a reward-carrying probe inserted into the trajectory so that longer reasoning is incentivized to remain visually accountable.

## 4. Related visually anchored optimization mechanisms

Several adjacent methods make the same design move—reweighting optimization toward visually sensitive tokens or trajectories—even when they do not use the exact VAPO name. This suggests a broader visually anchored policy-optimization family.

**Token Preference Optimization with self-calibrated visual-anchored rewards** defines a token-level visual-anchored reward as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one, then feeds that reward into a DPO-style preference objective [2412.14487]. For token \(y_i\),
\[
s_{y_i}= p_{log}(y_i | x,v,y_{<i}) - p_{log}(y_i | x,v_{c},y_{<i}),
\]
and the self-calibrated reward is
\[
c_{y_i}=\begin{cases}
a+\text{sigmoid}(s_{y_i}) & \text{if }y_i \in y_w\\
a+1-\text{sigmoid}(s_{y_i}) & \text{if }y_i \in y_l
\end{cases}
\]
with \(a=0.5\) in the main experiments [2412.14487]. On LLaVA-1.5-7B, TPO raises AMBER F1 from **74.3** to **85.0**, improves MMHal score from **2.01** to **2.47** while reducing hallucination rate from **61.46** to **51.04**, and raises HallusionBench Hard from **41.16** to **48.37** [2412.14487].

**Visual-Advantage On-Policy Distillation (VA-OPD)** introduces token-level visual advantage
\[
a_t = \max\!\left(\log p(y_t \mid v, q, y_{<t}) - \log p(y_t \mid \tilde{v}, q, y_{<t}),\; 0\right),
\]
then uses VA both for rollout-level reweighting and for token-level KL averaged separately within high-VA and low-VA groups [2605.21924]. The paper reports that the top **10%** of tokens by VA account for **~93%** of the total VA mass, indicating that visual supervision is concentrated in a small minority of positions [2605.21924]. In the 8B\(\rightarrow\)2B Geometry3K setting, Math Avg rises from **45.4** under standard OPD to **48.3** under VA-OPD, and Visual Avg rises from **64.6** to **66.1** [2605.21924].

**Visually-Guided Policy Optimization (VGPO)** targets temporal visual forgetting through a Visual Attention Compensation mechanism and dual-grained advantage re-weighting [2604.09349]. It computes token-level visual similarity to a visual prototype, boosts late-stage visually activated tokens, and then reshapes advantages at both the intra-trajectory and inter-trajectory levels. On Qwen2.5-VL-7B, VGPO reaches **Avg-Math = 66.6** and **Avg-Vision = 63.3**, compared with **63.8** and **59.6** for DAPO in the same setting [2604.09349].

**Perception-Grounded Policy Optimization (PGPO)** formalizes token visual dependency as
\[
\mathcal{S}(s_t, I) := D_{\text{KL}\left( \pi_\theta(\cdot \mid s_t, I) \; \Vert \; \pi_\theta(\cdot \mid s_t) \right),
\]
then uses a threshold-gated, mass-conserving mechanism to amplify advantages on visually dependent tokens and suppress others [2604.01840]. The paper reports an average gain of **18.7%** across seven multimodal reasoning benchmarks and emphasizes that mass conservation is required to avoid variance explosion and training collapse [2604.01840].

**Reflection-Anchor Policy Optimization (RAPO)** studies the problem from an information-theoretic standpoint. It derives a lower bound on downstream visual gain that highlights two factors: local branching room, measured through token entropy, and downstream visual propagation potential, measured through suffix divergence from a vision-marginalized reference [2605.09614]. Guided by this analysis, RAPO selects high-entropy reflection anchors and optimizes a chain-masked finite-window KL surrogate for downstream visual dependence [2605.09614].

Taken together, these papers support a common interpretation: visually anchored optimization is less a single algorithm than a research direction in which policy updates are deliberately concentrated on the small set of steps whose predictive distribution materially changes when visual evidence is altered, masked, or removed.

## 5. SlideASR and “Look before Transcription”

A distinct 2025 use of the term appears in **“Look before Transcription: End-to-End SlideASR with Visually-Anchored Policy Optimization”** [2510.08618]. Here VAPO is a reinforcement-learning-based post-training method for omni-modal LLMs that use both audio and slide images. The motivating failure mode is specific: naïve omni-modal models often degenerate into OCR systems, copying slide text instead of transcribing speech. On SlideSpeech, the paper reports that this OCR-like behavior occurs in **13–63%** of samples depending on the OLLM [2510.08618].

The method imposes a structured output format:
```text
<think> [internal OCR of slide] </think> <answer> [final transcription of speech, using entities from think] </answer>
```
and optimizes this reasoning process with four rewards: **Format**, **OCR**, **ASR**, and **Visual Anchoring** [2510.08618]. The total reward is
\[
R_{\text{total}} = \lambda_1 R_{\text{Format}} + \lambda_2 R_{\text{OCR}} + \lambda_3 R_{\text{ASR}} + \lambda_4 R_{\text{VA}},
\]
with default \(\lambda_1 = \lambda_2 = \lambda_3 = \lambda_4 = 1\) [2510.08618]. The visual anchoring reward uses entities correctly recognized in `<think>` and measures how faithfully they are reused in `<answer>` through an F1 score [2510.08618].

The benchmark introduced with the paper, SlideASR-Bench, contains **SlideASR-S** with **6,413** training samples, **44,240** entities, and **67.3 hours**, plus **2,054** test samples, **13,895** entities, and **18.5 hours**; it also contains **SlideASR-R**, a real set of **60** samples, **200** domain-specific entities, and **0.35 hours** [2510.08618]. On SlideSpeech test, **VAPO-7B** achieves **WER 10.31**, **B-WER 2.87**, **U-WER 10.84**, and **Recall 97.32** [2510.08618]. On SlideASR-S, VAPO-7B reaches **WER 4.60, NE-WER 2.83, NE-FNR 2.97** for English and **WER 2.13, NE-WER 3.78, NE-FNR 1.36** for Chinese; on SlideASR-R it achieves **NE-WER 26.48** and **NE-FNR 15.35** [2510.08618]. On ChineseLips, **VAPO-7B** attains **CER 1.298**, outperforming contextless and naïve image-conditioned baselines [2510.08618].

This SlideASR formulation broadens the meaning of visually anchored policy optimization beyond vision-language question answering. The anchor is not a token-level KL or visual claim, but an explicitly supervised intermediate reasoning block whose correctness and reuse can be measured against the slide image. The method is therefore a process-control form of VAPO: RL optimizes not only the final transcript, but also a visually accountable intermediate reasoning protocol.

## 6. Naming collision, theoretical limits, and unresolved issues

A major misconception is to treat all “VAPO” papers as variants of the same method. In the long-CoT LLM literature, **VAPO** is explicitly defined as **Value-based Augmented Proximal Policy Optimization**, not visually anchored policy optimization [2504.05118]. That paper describes a text-only RL framework for long chain-of-thought reasoning with offline value pretraining on Monte-Carlo returns, decoupled GAE, length-adaptive GAE, token-level PPO loss, asymmetric clipping (“Clip-Higher”), positive-example LM loss, and group sampling [2504.05118]. On AIME 2024 with Qwen2.5-32B, it reports **60.4** avg@32, exceeding DeepSeek-R1-Zero-Qwen-32B at **47** and DAPO at **50**, and reaching state-of-the-art performance in about **5,000** gradient update steps [2504.05118]. Despite the acronym overlap, this line does not belong to the visually anchored family in the ordinary multimodal sense.

The accompanying theoretical critique argues that this value-based VAPO faces structural limitations in long-term credit assignment, value function representational capacity, and the translation of global value signals into local policy improvements, especially under sparse rewards [2506.03038]. The analysis emphasizes that Monte Carlo targets are unbiased but high-variance, that early-step TD errors can be tiny and noisy when all steps share the same distant terminal reward, and that scalar value baselines are a blunt instrument for token-level reasoning decisions [2506.03038]. These arguments are about long-horizon RL generally, but they also clarify why later visually anchored work moved toward richer intermediate signals, token-level dependence measures, and explicit anchors.

Across the visually anchored line itself, the unresolved issues are more heterogeneous. Vision-anchored reasoning depends on the quality of visual claims and on hyperparameters such as \(K\), \(\beta\), and \(\gamma\) [2509.25848]. SlideASR VAPO is tailored to textual slide content, incurs slower inference because of `<think> + <answer>`, and can fail if the internal OCR step is wrong [2510.08618]. TPO notes that its visual corruption is global and suggests future object-specific corruption [2412.14487]. RAPO emphasizes that its lower bound is local and that the training objective is only a surrogate for exact downstream visual gain [2605.09614]. PGPO requires an additional masked forward pass and is only evaluated up to 7B models [2604.01840]. A plausible implication is that visually anchored policy optimization is converging on a shared diagnosis—final-answer reward is too coarse for multimodal reasoning—but has not converged on a single best anchoring signal or a unified theory of when visual dependence should be injected, amplified, or regularized.

Source: https://www.emergentmind.com/topics/visually-anchored-policy-optimization-vapo