---
title: 'ViReSkill: Lifelong LLM Robot Manipulation'
url: https://www.emergentmind.com/topics/vireskill
type: topic
---

# ViReSkill: Lifelong LLM Robot Manipulation

ViReSkill is a lifelong robot learning framework for LLM-based manipulation planning that combines **vision-grounded replanning** with a **skill memory** for accumulation and reuse. It is designed to address two obstacles in direct LLM/VLM deployment for robot planning: symbolic plans are often insufficiently grounded in scene geometry and object physics, and model outputs can vary for identical prompts, undermining execution reliability. ViReSkill responds to these issues with a closed loop in which failed executions trigger visually grounded replanning from the observed state, while successful executions are stored as reusable skills that can later be replayed without additional LLM/VLM calls. In the paper introducing the method, ViReSkill is evaluated on LIBERO, RLBench, and a physical UR5 platform, where it is reported to improve average success from **45% to 78%** on LIBERO, from **47% to 82%** on RLBench, and from **30% to 75%** on the real robot [2509.24219].

## 1. Problem formulation and learning objective

ViReSkill is framed as a method for **lifelong manipulation learning**, in which a robot repeatedly encounters manipulation tasks specified in natural language and is expected to improve over time through experience rather than re-solve every task from scratch [2509.24219]. The notation in the paper defines the \(i\)-th task as \(\mathcal{T}_i\), its free-form language description as \(\mathcal{D}_i\), and its associated skill as \(\pi_i \equiv \mathcal{P}(\mathcal{T}_i)\), where a skill consists of both a high-level plan and the low-level control code required for execution. The evolving skill repertoire is denoted \(\mathcal{P} = \{\pi_i\}_{i=1}^M\).

Execution is mediated by an executor \(\mathcal{C}\), based on VoxPoser, under stochastic factors \(\xi\) such as pose variation or nondeterministic dynamics. The rollout model is

$$
\tau(\xi) = \mathcal{C}(\pi_i, \mathcal{T}_i, \xi),
$$

with binary task success \(r(\tau,\mathcal{T}_i)\in\{0,1\}\). The task success rate across \(N\) rollouts is defined as

$$
\mathrm{SR}(\pi_i, \mathcal{T}_i) = \frac{1}{N} \sum_{n=1}^{N} r\!\left(\tau(\xi_n), \mathcal{T}_i\right).
$$

The lifelong objective is to maximize the mean success rate over all tasks. This formalization is significant because ViReSkill optimizes not only one-shot plan quality, but the quality of the **evolving skill repertoire** itself. A successful plan is therefore treated as persistent task knowledge rather than a disposable inference artifact.

## 2. Planning stack and skill representation

ViReSkill inherits its planning abstraction from **VoxPoser**, where an LLM generates Python code that calls perception and motion APIs [2509.24219]. The pipeline has three stages. First, a **planner** decomposes a natural-language instruction \(\mathcal{D}_i\) into a sequence of subtask descriptions,
\[
Planner(\mathcal{D}_i) = (d_1, d_2, \dots, d_n).
\]
Second, a **composer** maps each subtask \(d_j\) into one or more Language Model Programs (LMPs) that may invoke object detection, affordance prediction, and motion generation. Third, the **executor** runs these LMPs to produce sub-trajectories \(\tau_j\), which concatenate into the full trajectory \(\tau = (\tau_1, \tau_2, \dots, \tau_n)\).

Within this formulation, a skill is represented as
\[
\pi_i = \left( Composer(d_1), Composer(d_2), \dots, Composer(d_n) \right).
\]
This is an important representational choice. In ViReSkill, a skill is **not** a latent policy learned by gradient descent; it is an **executable programmatic artifact** that contains both a symbolic decomposition and low-level control code [2509.24219]. That representation makes storage, direct replay, and code-level revision straightforward. It also explains why ViReSkill can reuse a successful skill without requerying the LLM: the reusable unit is already executable.

The paper situates this representation within a broader concern about LLM-based planning reliability. A purely symbolic plan can be linguistically plausible yet physically poor, for example by underspecifying placement geometry or selecting an implausible grasp. ViReSkill therefore treats the initial code generation stage as only a first pass, not a terminal plan.

## 3. Vision-grounded replanning and failure analysis

The central adaptive mechanism in ViReSkill is **vision-grounded replanning** [2509.24219]. After each rollout, the system evaluates the binary success signal and retains observations \(\mathcal{O}\), including execution video. If execution fails, the current control code is not merely retried; instead, ViReSkill performs a visually grounded diagnosis and generates a revised action sequence conditioned on the observed state.

The main diagnostic component is a **hierarchical video-based reflection mechanism**. Its design addresses three practical constraints identified in the paper: long-horizon executions are difficult for a VLM to reason over end-to-end, structured plan code is difficult for a VLM to read directly, and only a limited number of frames can be supplied. ViReSkill therefore segments both the plan and the video into logical chunks, summarizes each chunk into natural language, and asks a VLM to compare the summarized plan against the corresponding video segments. The chunking heuristic is to break at each **“open gripper”** action, which serves as an approximate action boundary in manipulation sequences.

This process supports two different diagnoses. In the common case, the VLM identifies the **first failed chunk** and produces a grounded explanation tied to the corresponding video segment. In a second case, each low-level action may execute correctly while the task still fails because the plan logic is wrong. The paper gives the example of grasping a new object without first releasing the object already held. In such cases, the system can determine that all individual actions executed successfully and trigger a separate **plan-level reflection** step for high-level logical revision.

The replanning prompt is conditioned on four inputs: the failed plan or code, the original task description, the current observations, and retrieved prior skills from memory. The paper also states that the prompts are specialized for **execution failures** versus **logical errors**, and that the repair process is inspired by DROC in grounding revisions on related-object properties such as proper offsets, obstacle avoidance strategy, and object scale [2509.24219]. This suggests that replanning is not intended as generic textual “retry,” but as a physically informed code repair step anchored in observed scene dynamics.

## 4. Skill memory, retrieval, and continual reuse

ViReSkill’s second defining component is a **task-specific skill memory** that stores successful high-level plans together with low-level control code [2509.24219]. The memory is initially empty. If no prior solution exists for a task, the system must rely on LLM planning from the task description. Once a task is solved, however, the successful control program is committed to memory and becomes directly reusable.

At the start of a subsequent episode, if a stored skill exists for the task, the robot initializes from that skill rather than invoking the LLM again. This reduces both computational cost and output variance. More importantly, memory is also used **for transfer**, not just exact replay. The retrieval mechanism combines two criteria: task-description similarity and plan-code line-level similarity. Specifically, the paper describes a mixed retrieval set consisting of the top \(k/2\) entries by cosine similarity between full task-description embeddings and the top \(k/2\) entries by average best-match similarity between subtask lines, with a task-similarity threshold of **0.5** [2509.24219]. This allows related skills to be retrieved during replanning even when task wording differs but control structure overlaps.

The training protocol is explicitly organized into **2 rounds** with **5 iterations per round**, yielding **10 memory checkpoints**. In the evaluation stage, each stored skill is tested over **5 rollouts**; if no skill exists for a task at a given checkpoint, the success rate is recorded as zero [2509.24219]. This protocol operationalizes the notion of lifelong learning as a monotone growth of usable experience: every successful execution may increase either exact-replay competence or transfer competence for related tasks.

Conceptually, the memory mechanism converts LLM/VLM planning from a stateless inference procedure into an accumulating capability. Each successful execution becomes a persistent programmatic prior. The paper emphasizes that this is not merely a cache, because retrieved skills also support repair on related tasks.

## 5. Empirical performance and ablations

The simulation experiments use **GPT-4.1-mini** as the LLM for code generation and replanning and **GPT-4o-mini** as the VLM for failure analysis, with a VoxPoser-based executor. The real-robot setup uses a **UR5** arm with a **Robotiq 2F gripper**, an **Intel RealSense RGB-D camera**, **LangSAM** for object segmentation, and a **Sentence Transformer** for retrieval [2509.24219].

On **LIBERO**, the final-iteration average success rates reported are **0.45** for VoxPoser, **0.46** for Retry, **0.71** for Reflexion, **0.66** for REFLECT, and **0.78** for ViReSkill. The largest relative advantage appears on **LIBERO-10**, where ViReSkill reaches **0.66**, compared with **0.48** for REFLECT and **0.46** for Reflexion [2509.24219]. The paper explicitly interprets this as evidence that video-grounded replanning is particularly useful on long-horizon and temporally dependent tasks.

On **RLBench**, across nine selected tasks, ViReSkill reaches an average of **0.82**, compared with **0.47** for VoxPoser, **0.58** for Retry, and **0.78** for Reflexion. It matches Reflexion on tasks such as **TakeLidOff**, **PushButton**, and **TakeUmbrellaOut**, and exceeds it on tasks including **CloseDrawer** and **BasketballInHoop** [2509.24219]. The paper attributes the improvement over Reflexion to memory-based reuse of verified skills, not only to reflection.

On the physical **UR5** system, ViReSkill is evaluated on four tasks: placing juice in a basket, placing a bowl on a plate, placing both cups on a tray, and placing a pen in a holder. The overall result is **75%** success for ViReSkill versus **30%** for VoxPoser, with the most pronounced difference on the two-cup tray task, where ViReSkill achieves **3/5** and VoxPoser **0/5** [2509.24219].

The skill-transfer ablation isolates the contribution of retrieval during replanning. Without skill transfer, ViReSkill achieves an average of **0.73** on LIBERO; with skill transfer, the average rises to **0.78**. The largest improvement is on **LIBERO-10**, from **0.50** without transfer to **0.66** with transfer [2509.24219]. The paper treats this as evidence that the memory is not only stabilizing exact replay, but also functioning as a genuine lifelong transfer mechanism.

## 6. Limitations, scope, and disambiguation

The limitations section of the ViReSkill paper is explicit on two points [2509.24219]. First, execution is **open-loop**: once a trajectory is generated, it is followed without real-time perceptual correction. Replanning occurs **between** executions rather than within an execution. The authors propose **Visual-Language Actions (VLA)** as a future direction for time-step-level closed-loop action generation. Second, **memory scaling** is identified as a future systems bottleneck, motivating directions such as skill merging, compression, and faster retrieval.

A plausible implication is that ViReSkill will be strongest when errors can be diagnosed from episode-level execution traces and repaired through code revision, rather than when success depends on high-frequency closed-loop correction. The paper also suggests likely degradation when failure localization is weak or when low-level controller errors dominate despite an improved high-level plan, but these are presented as implications rather than benchmarked failure modes [2509.24219].

The name should also be distinguished from several adjacent methods. **ViSkill** is a reinforcement-learning framework for long-horizon surgical robot tasks that uses a value function and chaining policy to select terminal states favorable for downstream subtasks; it does not use vision-grounded replanning with programmatic skill memory [2307.16503]. **VISUALSKILL** is a framework for computer-use agents in graphical interfaces that stores reusable skills as hierarchical multimodal topic guides with screenshots, rather than as robot-executable plans and control code [2606.18448]. These works overlap in their concern with reusable skills and long-horizon behavior, but they operate in different action spaces and with different notions of grounding.

Within the broader literature on skill reuse, ViReSkill is notable for pairing **failure-driven grounded repair** with **success-driven consolidation**. The framework’s distinctive contribution is not simply memory, nor simply VLM-based reflection, but the coupling of the two into a continual loop: **plan, execute, diagnose, replan, succeed, store, and replay** [2509.24219].

Source: https://www.emergentmind.com/topics/vireskill