Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

Abstract: Humans can reproduce the viewpoint specified by a target image through active head and body motion, yet spatial intelligence in foundation models has largely been studied as passive understanding of pre-collected observations. We introduce Target Viewpoint Reproduction (TVR) -- an active task where an agent adjusts its viewpoint in a 3D environment until its observation matches a given target image -- and TVRBench, an indoor-simulation benchmark spanning scene scale and target-view visual richness. TVR is far from solved: on the evaluation split, the strongest open-source and closed-source models reach only 7.8% and 12.0% success. Fine-grained analysis identifies two consistent bottlenecks: off-the-shelf models struggle with multi-turn visual history, and performance drops sharply when viewpoint reproduction requires body translation rather than in-place rotation, exposing a gap in mapping spatial discrepancies to embodied movement. To study reducing this gap, we build a unified TVR post-training framework covering expert-trajectory SFT, rationale-supervised CoT-SFT, offline Single-turn GRPO, and on-policy Multi-turn GRPO from live simulator rollouts. Visual-action SFT supplies the main gain, raising a 9B open-source model to 50.8% success; Multi-turn GRPO provides targeted multi-room refinement and reaches 51.4% overall, while CoT supervision and Single-turn GRPO degrade closed-loop performance. These results establish TVRBench as a testbed for measuring and training foundation models that actively perceive and act in 3D environments. Our code, data, and models are available at https://github.com/aim-uofa/TVRBench.

• The paper introduces the TVR task to assess precise active viewpoint reproduction and diagnose spatial memory and translation in foundation models.
• It reveals a significant performance gap between models and humans, with body translation emerging as a critical bottleneck in active navigation.
• Post-training methods like VA-SFT and Multi-turn GRPO significantly improve success by leveraging comprehensive visual-action memory.

Target Viewpoint Reproduction: Benchmarking Active Spatial Intelligence in Foundation Models

Introduction and Motivation

The ability to actively match a specified viewpoint in a 3D environment, given only a target image, probes a core aspect of spatial intelligence: closing the perception–reasoning–action loop for embodied agents. While prior work has primarily focused on passive visual understanding or navigation to a spatial region, it remains unclear whether foundation models, especially MLLMs, can perform precise, active viewpoint reproduction, mapping visual discrepancy directly to egocentric movement. This paper introduces the Target Viewpoint Reproduction (TVR) task and TVRBench, a comprehensive benchmark designed to diagnose exploration efficiency, spatial memory, and embodied action mapping in both simulated single-room and multi-room environments.

Figure 1: TVRBench task structure: agents must reproduce a target viewpoint through active exploration across single- and multi-room indoor environments, varying in visual richness and spatial complexity.

TVR Task Definition and Benchmark Construction

TVR tasks require an agent to operate in a simulated indoor environment, taking as input a target RGB image $I^\star$ and its current first-person observation $I_t$. The agent must sequentially execute discrete actions—translations, rotations, head pitch adjustments, or Stop—with the goal of terminating at a pose exactly matching the target viewpoint.

The architecture of TVRBench emphasizes two axes of difficulty: (1) scene scale (single vs. multi-room) and (2) target-view visual richness (object-rich vs. object-sparse). Each episode terminates either upon the agent invoking the Stop action or exhausting a step limit. Success is ascribed only to cases where the final agent pose is precisely coincident with that of $I^\star$, enforcing a stringent observation-level match.

The action space is granular, with translations of $0.25\,\text{m}$, rotations in $45^\circ$ increments, and discrete horizon adjustments, systematically probing the agent’s ability to make fine-grained orientation and position judgments. TVRBench supports both action-only (AO) and visual-action (VA) episodic memory representations, investigating the impact of memory design on long-horizon exploration.

Baseline Evaluation: Foundation Models on TVRBench

A thorough evaluation of five open-source (Qwen3.5/3.6 variants) and three closed-source (GPT-4o, GPT-5, Gemini-3.1-Pro) foundation models reveals substantial performance limitations in TVR. The highest performing configuration, Gemini-3.1-Pro (AO), achieves only 12.0% overall success, while the best open-source model configuration attains 7.8%. Human subjects, by contrast, achieve a 93.0% success rate.

Every open-source baseline performs better with AO than VA memory, with a mean advantage of $+3.8$ percentage points, suggesting models are not leveraging raw visual-history context efficiently. The distribution of failure modes is dominated by repeated pose revisits and local cyclical behaviors, where agents either rotate endlessly or move in confined loops without closing the target-view gap.

Figure 2: Failure modes in untrained Qwen3.5-9B: episodes show excessive revisiting and cycling between a limited set of positions or heading directions, and models over-prefer rotation over translation; ablations confirm translation as the dominant bottleneck.

Critical controlled ablations reveal that body translation is the limiting factor: removing body translation elevates Qwen3.5-9B to 80.5% on rotational-only variants, while translation-only preserves poor performance at 10.0%.

Behavioral Analysis: Failure Modes and Memory

Detailed qualitative trajectory analysis further substantiates the translation bottleneck. With AO memory, the untrained Qwen3.5-9B repeatedly issues actions that only rotate the agent in place, unable to infer that all plausible yaws have already been explored from the current position.

Figure 4: Failure case—agent performs multiple in-place rotations after an initial translation, exhausting the episode budget without progress; action-only memory obscures evidence of past positional attempts.

Similarly, agents often oscillate between a small subset of positions without progress towards the target pose, reiterating the inadequacy of shallow action histories.

Figure 6: Failure case—agent walks between a handful of grid positions without reaching the target, never registering that these cells have already been visited due to limited action memory.

Post-Training Framework: SFT and RL for Active Viewpoint Control

To systematically address these challenges, the paper introduces a unifying TVR post-training pipeline encompassing four regimes: supervised fine-tuning (SFT) with and without per-step Chain-of-Thought (CoT) rationales, Single-turn GRPO (per-step offline RL), and Multi-turn GRPO (on-policy RL with trajectory-level feedback).

Figure 7: TVRBench post-training schemes: SFT on rule-based trajectories (with/without CoT), offline Single-turn RL, and online Multi-turn RL; the latter allows optimization over full exploration horizons.

Visual-action SFT on the 9B model backbone boosts overall success to 50.8%, far outperforming both baseline and closed-source models. Multi-turn GRPO built atop VA-SFT yields a marginal further increase to 51.4%, with improvement concentrated in the multi-room (long-horizon) splits. Notably, SFT with VA context consistently outperforms AO variants, and CoT rationales actively degrade closed-loop performance in all settings evaluated.

Figure 9: Single-room VA-SFT + Multi-turn GRPO: efficient translation and rotation to target pose with precise Stop action and no unnecessary repetition.

In contrast, both AO-SFT and Single-turn GRPO fail to close the body translation gap or support robust stop calibration. Single-turn GRPO consistently regresses below SFT-initialized policies, confirming that per-step action-matching RL does not capture the closed-loop, multi-step reasoning necessary for TVR.

Figure 3: Multi-room VA-SFT + Multi-turn GRPO: policy successfully traverses complex environments and aligns with the required target viewpoint before termination.

Implications and Research Outlook

The TVRBench results illuminate a significant disparity between passive understanding and active spatial intelligence in current generation foundation models. The findings are notable for several reasons:

• Explicit viewpoint reproduction (not region navigation) exposes a unique failure mode in foundation models: translation control and spatial memory, not visual appearance matching per se, are major obstacles.
• Per-step, action-matching objectives—whether via supervised learning or RL—do not transfer to active, closed-loop spatial control tasks, underscoring the need for trajectory-level or high-level objective optimization.
• Visual-action episodic memory substantially improves policy learning, confirming the importance of full observation-action histories for tackling exploration and loop-closure in visual navigation.
• Chain-of-Thought rationale annotation, as currently instantiated, impedes rather than facilitates action learning for embodied control, suggesting the need for new forms of intermediate supervision more attuned to the structure of the task.

From a theoretical perspective, TVRBench and its post-training analysis delineate a clear agenda for spatial reasoning in MLLMs: establishing mechanisms for mapping visual discrepancy to reliable egocentric movement, memory architectures supporting long-horizon history, and reward schemes that incentivize efficient, closed-loop exploration.

Practically, the TVR approach provides a scalable, controlled testbed for evaluating and improving foundation models' ability to act in 3D. However, the results are necessarily bounded by simulation and discrete action spaces; extension to continuous-control and physical platforms remains a future direction. Further, success is highly sensitive to episodic memory representation and the alignment between training reward and deployment criterion.

Conclusion

The TVR and TVRBench paradigm provides a fine-grained diagnostic for embodied spatial intelligence in foundation models. Strong numerical evidence indicates that existing MLLMs, even at the largest scales, do not generalize to active viewpoint reproduction, with explicit body translation control representing the principal failure mode. Gains from post-training are contingent on trajectory-level optimization and full visual-action memory, while naive application of per-step CoT supervision or RL can degrade performance. TVRBench, therefore, serves both as a rigorous evaluation and a compact sandbox for future advances in active spatial reasoning, with open implications for navigation, robotics, and interactive AI systems.

---

Reference: "Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?" (2606.01247)