RoboFarseer for Robotic Manipulation Planning

• The paper introduces RoboFarseer, a dual-role vision-language model that integrates chain-of-thought reasoning with verifiable rewards to produce structured multi-step skill plans.
• It employs a hierarchical planner-monitor architecture that generates and verifies atomic manipulation tasks, ensuring temporal coherence and physical plausibility in open-world scenes.
• Empirical results demonstrate a significant boost in task success—up to a 60% improvement over baseline controllers—validating its effectiveness against larger proprietary models.

RoboFarseer is a trained vision-LLM for real-world, long-horizon robotic manipulation planning, introduced within the REVER framework for embodied planning with verifiable reward. It is designed to generate multi-step skill plans from free-form language instructions and to verify step-wise completion during execution, with the stated objective of producing plans that are physically plausible, temporally coherent, and robust to execution failures in open-world household scenes (Bo et al., 30 Sep 2025).

1. Problem setting and system identity

RoboFarseer addresses a specific limitation in embodied AI: large vision-LLMs can perform high-level perception and language understanding, but in robotic planning they often hallucinate, violate physical constraints, and fail to recover when the environment changes. The paper further argues that supervised imitation on human plan traces is limited because it teaches fixed sequences without an explicit notion of reward or recovery, while reinforcement learning for robotic planning has been constrained by the lack of dense, interpretable rewards (Bo et al., 30 Sep 2025).

Within that framing, RoboFarseer is not presented as a generic LLM, a low-level controller, or a one-shot planner. It is the fine-tuned VLM produced by REVER, and its role is explicitly dual: it plans and it monitors. The model is based on Qwen2.5-VL-7B and is trained to emit explicit chain-of-thought reasoning together with structured skill plans. A central design claim is that a relatively small 7B model, when trained on real-world sequential manipulation data and optimized with a verifiable reward, can compete with much larger proprietary models on planning tasks (Bo et al., 30 Sep 2025).

A common misunderstanding would be to treat RoboFarseer as a replacement for execution or control. The reported system is instead hierarchical. RoboFarseer operates above a low-level controller, generating a skill sequence, checking whether the current sub-task has been completed, and replanning when execution fails or the world state drifts from the original plan.

2. Data pipeline and task representation

The data pipeline begins with the Universal Manipulation Interface (UMI), which is used to capture hardware-agnostic kinesthetic demonstrations “in the wild.” The paper first collects a library of atomic manipulation skills,

$\mathcal{S} = \{s_1, s_2, \ldots, s_N\},$

and then synthesizes long-horizon tasks by composing them into ordered sequences,

$\mathcal{T} = (s^{(1)}, s^{(2)}, \ldots, s^{(K)}).$

This composition-based formulation is central to the system’s notion of long-horizon planning because it reduces complex tasks to reusable skill vocabularies rather than unconstrained free-form action generation (Bo et al., 30 Sep 2025).

The annotation pipeline automatically converts demonstrations into two supervision targets. For planning, the label is the full skill sequence associated with an initial scene observation and a user instruction. For completion verification, the label is binary, determined from before-and-after observations of a sub-task. Positive examples come from successful executions, while negative examples are created by pairing the start frame with an intermediate frame that does not yet complete the sub-task. Each data point is represented as a triplet $d = (q, \mathcal{O}, y)$, where $q$ is the prompt, $\mathcal{O}$ is the observation set, and $y$ is the ground-truth answer.

The resulting LEAP dataset contains 12,000 tasks with a 9:1 train/test split and is organized into LEAP-L for long-horizon sequential planning and LEAP-U for instruction-aligned planning. The final fine-tuning corpus combines LEAP-L, LEAP-U, EgoPlan, ShareRobot, and ERQA planning data, yielding 57,000 cleaned triplets total (Bo et al., 30 Sep 2025).

This data construction strategy suggests that the system’s scalability comes less from manually authored task programs and more from reusable atomic skills plus automated annotation. A plausible implication is that REVER treats long-horizon planning as a structured sequencing problem grounded in real manipulation traces rather than as unrestricted language generation.

3. Planner-monitor architecture and closed-loop execution

RoboFarseer is explicitly trained to occupy two roles. As a planner, it receives a high-level user request and a current scene image and generates a structured sequence of atomic skills. As a monitor or verifier, it receives the current sub-task together with a pair of images—the start image and the current image—and determines whether the sub-task has been completed (Bo et al., 30 Sep 2025).

At deployment, the model is embedded in a closed-loop hierarchical execution system. The planner first emits an initial skill list from the instruction and observation. The low-level controller then executes one skill at a time. Meanwhile, RoboFarseer checks progress at 5 Hz. If the current sub-task is complete, execution advances to the next one; otherwise, monitoring continues. If execution fails or the environment drifts too far from the plan, the current observation is fed back to RoboFarseer for replanning.

The low-level controller is a diffusion policy with a frozen language encoder, a visual encoder, FiLM conditioning, and UMI-derived end-effector trajectories from SLAM. The paper therefore positions RoboFarseer as a deliberative and verification layer over a separate motor execution stack, not as an end-to-end action generator.

This closed-loop formulation matters because it distinguishes RoboFarseer from open-loop plan generators. The model is trained to reason over sequential structure, but the reported system also uses the same model to supervise execution progress in situ. That design directly targets the mismatch between nominal plan generation and real-world task completion.

4. Verifiable reward and reinforcement-learning procedure

The paper’s main methodological contribution is a verifiable reward that is deterministic, interpretable, and cheap to compute. The total reward combines a format term and a content term. The format reward checks whether the output matches the required template with separated reasoning and answer sections, specifically the pattern > ... followed by <answer>...</answer>. This reward therefore directly incentivizes chain-of-thought style outputs with explicit structural separation (Bo et al., 30 Sep 2025).

The content reward measures agreement between the generated plan and the ground-truth skill sequence using maximum-weight bipartite matching. Step similarity is computed from action and object components, with $w_a = 0.3$ for action similarity and $w_o = 0.7$ for object or location similarity. Object similarity is defined by exact match, substring match, or membership in the same semantic set from ontology $\mathcal{C}$. A length penalty with $w_l = 0.1$ is then applied. For completion-verification examples, the content reward is simply 1 for a correct binary decision and 0 otherwise (Bo et al., 30 Sep 2025).

RoboFarseer is fine-tuned with GRPO. For each sampled response $\mathcal{T} = (s^{(1)}, s^{(2)}, \ldots, s^{(K)}).$0, the reward is normalized into an advantage,

$\mathcal{T} = (s^{(1)}, s^{(2)}, \ldots, s^{(K)}).$1

The policy is then optimized with a clipped PPO-style objective and a KL penalty relative to a reference model. The reported training configuration uses Qwen2.5-VL-7B as the base model, 8 H100 GPUs, approximately 28 hours per epoch, 3 epochs total, batch size 8 per device, gradient accumulation 12, learning rate $\mathcal{T} = (s^{(1)}, s^{(2)}, \ldots, s^{(K)}).$2 with cosine decay, KL coefficient $\mathcal{T} = (s^{(1)}, s^{(2)}, \ldots, s^{(K)}).$3, 8 sampled completions per prompt, and maximum output length 2048 tokens (Bo et al., 30 Sep 2025).

The paper explicitly attributes RoboFarseer’s reasoning behavior to this reward design and to prompts that require temporal and spatial decomposition under a constrained skill grammar. The intended result is executable, grammar-constrained output rather than free-form narrative planning.

5. Benchmark results and comparative performance

RoboFarseer is evaluated on five benchmarks: LEAP-L, LEAP-U, ShareRobot-Planning, EgoPlan-Bench2, and RoboVQA. Baselines include proprietary models such as Gemini-2.5-Pro, Gemini-2.5-Flash, GPT-4.1, GPT-4o, and Claude-Sonnet-4; open models such as Qwen2.5-VL-72B-Instruct and Qwen2.5-VL-32B-Instruct; and embodied-specific models such as RoboBrain2-7B and RoboBrain2-32B (Bo et al., 30 Sep 2025).

On top-1 planning accuracy, RoboFarseer-7B scores 59.3% on LEAP-L (MCQ), compared with 66.9% for Gemini-2.5-Pro and 64.1% for Claude-Sonnet-4. The paper also states that RoboFarseer beats every open-source competitor by at least 9 percentage points on LEAP-L. On LEAP-U, ShareRobot-Planning, and RoboVQA, RoboFarseer achieves the best score among all models. On EgoPlan2, it scores 56.4%, just behind RoboBrain2-32B at 57.2%, which the paper presents as evidence that a 7B model trained with REVER can rival a 32B embodied specialist (Bo et al., 30 Sep 2025).

The strongest reported advantage appears in open-ended plan generation. On LEAP-L, RoboFarseer reaches 76%, while Gemini-2.5-Pro gets 24% and RoboBrain2-32B gets 26%. Claude-Sonnet-4 performs near zero because it fails to conform to the required skill grammar. Similar trends are reported on LEAP-U. The paper additionally notes that with two in-context examples, planning performance improves for baselines, but RoboFarseer still maintains 92% exact-match, which is used to support the claim that reward-based fine-tuning outperforms simple prompting (Bo et al., 30 Sep 2025).

These results are the empirical basis for the paper’s broader claim that RoboFarseer matches or exceeds proprietary models that are orders of magnitude larger and surpasses the best baseline by more than 40% on open-ended planning. The reported gains are therefore not framed as generic language-model scaling effects, but as consequences of task-specific data and verifiable-reward optimization.

6. Real-world deployment, failure structure, and limitations

Real-world tests are conducted on a Dobot Nova5 6-DoF arm with a GoPro10 camera in three household scenes: Tidy up Desktop, Brew Tea, and Bring Food and Drinks. The evaluation covers 10 instructions total, with 10 episodes per instruction and randomized scene placements (Bo et al., 30 Sep 2025).

The reported results show that the Bring Food and Drinks family averages about 90% overall success, with step-level accuracy above 95%. Tidy up Desktop ranges from 60% to 90% overall success depending on object complexity. Brew Tea is identified as the hardest task, with overall success around 50% to 70%, due to precision pouring, difficult liquid localization, and fragile grasping of the lid. An ablation without the VLM planner, using only the control policy, averages 14% overall success and obtains 0% on tea tasks. The paper summarizes the complete system as boosting overall success by roughly 60% compared with the same low-level controller without the planner (Bo et al., 30 Sep 2025).

Failure analysis attributes 26.9% of failures to perception errors, 16.2% to reasoning errors, 43.6% to downstream control failure, and 3.3% to miscellaneous causes. This distribution is important because it indicates that the dominant bottleneck in the reported system is downstream execution rather than planning alone. A plausible implication is that further gains may depend at least as much on manipulation robustness and perception fidelity as on additional VLM scaling.

The paper also states several limitations directly. RoboFarseer still relies on a constrained skill grammar. Some tasks remain difficult because of visual ambiguity or fine motor precision. Future work is identified in reactive replanning, larger skill vocabularies, bimanual manipulation, and loco-manipulation. Taken together, these constraints place RoboFarseer within a specific research trajectory: real-world long-horizon manipulation planning supervised by explicit, verifiable rewards rather than by imitation alone or prompt engineering alone (Bo et al., 30 Sep 2025).