Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment

Abstract: Vision-LLMs (VLMs) struggle to generalize in interactive physical reasoning, particularly under unseen tasks and environments. Two key failure modes are prominent: hallucinated chain-of-thought (CoT) reasoning that contradicts physical reality, and misalignment between the model's reasoning and actions. We present VAORA (Visual Action Outcome Reasoning Alignment), a novel reward design that directly addresses both issues. VAORA introduces two complementary rewards: Visual Alignment Reward, which anchors VLM reasoning to the visual context independent of the agent action itself, and Visual-Action Alignment Reward, which grounds reasoning in the visual outcome induced by the model's action. Together, these rewards suppress hallucinated CoT and reduce the gap between reasoning and behavior. To improve training stability, we further employ smooth, dense rewards by estimating success probabilities using a pre-trained in-domain expert agent. Experiments on PHYRE and Virtual Tool support our performances across novel-task and unseen-environment settings, confirming that grounded and generalizable physical intelligence can be induced through VAORA.

• The paper introduces VAORA, a reward framework that aligns a vision-language model’s reasoning with scene structure, collision events, and post-action object placement using symbolic visual-action consistency.
• VAORA improves PHYRE Testing Set 1 performance from 0.176 to 0.382 Pass@1 over SFT, surpasses the DQN expert on most metrics, and avoids the severe degradation caused by sparse-reward GRPO.
• The method transfers zero-shot from PHYRE to Virtual Tool and improves adapted CRAFT VQA accuracy from 44.14 to 46.06, indicating that outcome-grounded reasoning supports broader causal and physical understanding.

Motivation and problem statement

This paper addresses the failure of vision-LLMs (VLMs) to generalize in interactive physical reasoning, where an agent must intervene in a physical scene—placing or launching objects so that the resulting dynamics satisfy a goal. The authors characterize generalization along two axes: cross-task transfer within the same environment and cross-environment transfer across distinct physics simulators. They argue that conventional non-VLM agents, which collapse perception and decision-making into a single visual-encoder-to-MLP mapping, are architecturally incapable of interpretable reasoning and are prone to exploiting spurious correlations rather than transferable physical principles.

The paper's central diagnostic claim is that both dominant VLM training paradigms fail to supervise the connection between chain-of-thought (CoT) reasoning and physical reality, producing two distinct failure modes: hallucinated CoT, in which the model produces physically incorrect reasoning that leads to wrong actions, and misaligned action, in which the model bypasses its own reasoning via a visual shortcut and plans incorrectly despite plausible-sounding traces. Supervised fine-tuning (SFT) teaches imitation of the linguistic form of expert reasoning without grounding; success-driven reinforcement learning (RL) encourages shortcut behavior that bypasses CoT entirely. Both leave the reasoning trace unsupervised against the actual outcome of the agent's intervention.

The VAORA reward design

The proposed method, VAORA (Visual Action Outcome Reasoning Alignment), is a reward framework that projects the model's reasoning trace and the environment's observations into a shared symbolic space via domain-specific transformations $\phi(\cdot)$ and $\psi(\cdot)$, then measures their consistency. Three rewards are defined under two categories:

Visual Alignment Reward ($r_G$) anchors reasoning to the initial scene independent of the agent's action. The model is prompted to emit structured object tuples—spatial label from a $3\times3$ grid of nine regions, normalized coordinates, and size—which are scored against ground-truth tuples extracted from the simulator using a soft grid score plus a numerical deviation term, with penalties for hallucinated, duplicated, or malformed objects.

Visual-Action Alignment Rewards ground reasoning in the post-interaction outcome. The collision reward $r_C$ scores predicted collision events (object identities, action type, contact point, motion direction) against events extracted from the post-interaction simulation. The placement reward $r_P$ scores the model's stated placement descriptions against the actual position of the placed ball in the first post-interaction frame, using object-relative spatial labels. Both apply per-event penalties for unmatched predictions.

To stabilize training under sparse binary success signals—a known cause of policy collapse in continuous-action settings—the authors replace the environment's binary reward with a smooth success-probability estimate $r_{\mathrm{DQN}}$ from a frozen, pre-trained DQN expert trained on the same task distribution. The two visual-action alignment rewards are gated by this expert signal, so that reasoning quality is rewarded only when the action itself is physically plausible; the grounding reward is optimized outside the gate. Optimization uses GDPO over the composed reward.

A notable design constraint: the collision reward models only the first object struck by the placed ball, which restricts which PHYRE task types are eligible for held-out evaluation (10 of 25 types qualify). This is disclosed explicitly and shapes the experimental protocol.

Cross-task generalization on PHYRE

Using Qwen3-VL-8B-Instruct as the base model, SFT on Gemini-3.1-Flash-generated traces, and three disjoint held-out splits of 5 task types each, VAORA outperforms all open-source baselines across every testing set and metric, and surpasses the DQN expert—the very source of its reward signal—on nearly all metrics. Representative Pass@1/Pass@5 figures on Testing Set 1:

Method	Pass@1	Pass@5
Qwen3-VL-8B (SFT)	0.176	0.470
Qwen3-VL-8B (GRPO)	0.006	0.062
Gemini-3.1-Pro	0.278	0.552
+EG	0.270	0.512
+EG+VAORA	0.382	0.594

Two findings deserve emphasis. First, GRPO and SFT+GRPO with only the native binary success signal degrade severely below the SFT baseline, providing direct empirical evidence of training instability under sparse rewards in continuous action spaces. Second, surpassing the DQN expert on unseen tasks implies that symbolically grounded reasoning confers a genuine generalization advantage over direct visual-to-action mappings, even when the expert provides the dense reward signal.

Cross-environment transfer and broader physical understanding

On Virtual Tool, evaluated zero-shot after training exclusively on PHYRE (with protocol adaptations: tool selection removed, colors remapped), VAORA achieves the best Pass@1 (0.167 vs. 0.111 for both Gemini-3.1-Pro and Flash) and remains competitive on Pass@3 and Pass@5. In sharp contrast, the DQN expert achieves near-zero performance (Pass@1 = 0.000), underscoring that non-VLM agents do not transfer beyond their training distribution at all. One caveat bears noting: the evaluation protocol was modified from the original Virtual Tool benchmark to enable direct transfer, so these numbers are not directly comparable to prior results on the unmodified benchmark.

On CRAFT VQA (adapted to single-image input), VAORA improves overall accuracy from 44.14 (SFT) to 46.06, with the largest gains in counterfactual (+1.3 over SFT) and causal categories—the reasoning types that require understanding interactive physical consequences. This indicates that training a model to reason about the outcomes of its own actions induces causal understanding that transfers beyond action selection, though the absolute margins here are modest relative to the PHYRE gains.

Why alignment matters: reward breakdown

An analysis decomposing performance into grounding, placement, and collision rewards explains the mechanism behind the headline results. SFT transfers static scene grounding (a capability requiring no interaction) but consistently fails on placement and collision rewards, confirming that imitating reasoning traces supervises nothing about post-action consequences. Expert-guided RL alone exhibits the opposite pathology: grounding degrades substantially and placement collapses entirely on one split, showing that success-driven optimization decouples action quality from meaningful reasoning. Only the full VAORA combination recovers grounding while simultaneously improving placement and collision rewards across all splits. An ablation further shows each component contributes complementary gains, with the full combination ($r_G$, $r_P$, $r_C$) reaching Pass@1 = 0.366 versus 0.272 for grounding alone on Testing Set 3.

Limitations and open questions

The authors concede that the approach operates in a single-turn setting: the model commits to one action without observing outcomes of prior attempts. When test-time physical concepts differ substantially from training, or when dynamics deviate significantly (e.g., different gravitational constants or friction coefficients), the model has no mechanism to detect and recover from errors. Additional constraints implicit in the method include the reliance on a domain-specific symbolic parser ($\psi(\cdot)$0, $\psi(\cdot)$1) that must be hand-instantiated per environment, the dependence on a pre-trained in-domain expert for reward gating, and the first-collision-only modeling of interaction chains. The open question the paper identifies is whether test-time adaptive, multi-turn interaction—in which the model updates its physical understanding from environmental feedback across successive attempts—can extend these gains to settings with substantial distributional shift.

Conclusion

VAORA demonstrates that directly supervising the alignment between a VLM's reasoning trace and the visual outcome of its actions—stabilized by dense expert-derived success probabilities—yields consistent gains in cross-task generalization, zero-shot cross-environment transfer, and causal physical understanding, while diagnosing precisely why SFT and sparse-reward RL each fail. The result supports the paper's central claim that grounded, outcome-aligned reasoning, rather than larger scale alone, is what enables generalizable physical intelligence in compact 8B models.