VLM-Guided Autonomous Skill Acquisition

• VLM-Guided Autonomous Skill Acquisition is an emerging paradigm that leverages pretrained vision-language models to autonomously acquire and refine robotic skills through semantic grounding and closed-loop reasoning.
• It integrates high-level semantic reasoning with reinforcement and imitation learning to yield systems that are sample efficient, robust, and generalizable compared to traditional approaches.
• Empirical evaluations demonstrate significant improvements in success rates and sim-to-real transfer by combining semantic priors, structured planning, and specialized optimization criteria.

Vision-LLM (VLM)-Guided Autonomous Skill Acquisition is an emerging paradigm that leverages pretrained VLMs to accelerate, structure, and generalize robotic and agent skill acquisition through semantic grounding, closed-loop reasoning, and dynamic guidance during learning and execution. This approach integrates the interpretive and abstraction capabilities of large-scale VLMs with reinforcement learning (RL), imitation learning (IL), and hybrid pipelines, yielding systems that are more sample efficient, robust, and interpretable compared to purely end-to-end or manual-engineering approaches.

1. Integration of VLMs in Skill Acquisition Architectures

Contemporary VLM-guided skill acquisition frameworks employ vision-language foundation models as semantic reasoning or planning modules within broader agent architectures. These models process high-dimensional raw observations (e.g., multi-view RGB images, depth maps) to generate structured intermediate representations—typically semantic plans, skill priors, or direct action specifications. Notable system instantiations include:

• Critical Object-Oriented Reasoning with RL: COVLM-RL processes multi-view panoramic images using a chain-of-thought (CoT) VLM prompting pipeline (identification → prediction → planning). It encodes critical object-centric semantic priors which guide a PPO-style RL agent via compact meta-action vectors and dense embeddings, thus tightly coupling high-level semantic reasoning with continuous low-level control (Li et al., 10 Dec 2025).
• Skill Memory and Replanning Loops: ViReSkill interleaves LLM/VLM-based symbolic planning and diagnosis with a task-indexed “skill memory.” Upon plan failure, a VLM replanner interprets failures in physical terms and proposes corrected plans, guiding iterative skill acquisition and lifelong improvement (Kagaya et al., 29 Sep 2025).
• Hierarchical and Compositional Models: CLASP combines VLM-generated “tool” schemas (skill descriptions, parameters, preconditions, postconditions) with data-efficient, task-parameterized movement primitives. VLMs interpret user commands, select and compose skills, and identify capability gaps for on-demand demonstrations (Knauer et al., 6 Jun 2026).

System architecture typically follows an abstraction pipeline: perception → VLM-based plan/skill reasoning → semantic encoding or skill proposal → RL/IL policy module.

2. VLM Prompting and Semantic Reasoning Strategies

Skill acquisition pipelines using VLMs employ various advanced prompt engineering and semantic decoding procedures:

• Chain-of-Thought Prompting: In COVLM-RL, each consistent sequence of identification (“Which is the most critical object?”), prediction (“What will that object do next?”), and planning (“What should the ego-vehicle do?”) yields a structured semantic plan. This plan is parsed into actionable priors (one-hot meta-actions and dense embeddings) for downstream policy learning (Li et al., 10 Dec 2025).
• Plan Decomposition and Gap Detection: InSight segments demonstrations into primitive actions via VLM plan decomposition and uses VLMs to identify missing primitives, propose parametrized low-level heuristics, and verify acquisition outcomes, thus supporting continual, self-guided skill expansion (Wang et al., 23 Jun 2026).
• Guide-to-Skill Distillation: MMG2Skill compiles in-the-wild multimodal guides (HTML, images) into structured, editable skills using VLM-based normalization and step extraction. These skills are iteratively refined via trajectory-level root-cause analysis performed exclusively by fixed VLM policies without reward feedback (Che et al., 1 Jun 2026).

These reasoning mechanisms inject interpretable and compositional priors, reduce blind action-space exploration, and facilitate rapid adaptation to novel tasks.

3. Mathematical Formulations and Optimization Criteria

VLM-guided skill acquisition systems frequently augment standard RL/IL objectives with new loss components to integrate semantic guidance:

• Augmented Markov Decision Processes: For example, the COVLM-RL agent observes augmented states $S = O \times M$ where $O$ are physical/environment states and $M$ are high-level meta-actions and semantic VLM embeddings (Li et al., 10 Dec 2025).
• Semantic Consistency Regularization: COVLM-RL minimizes an RL surrogate loss (e.g., PPO) combined with a contrastive-style “consistency loss” to align continuous actions $a_t$ with the VLM semantic embedding $\omega_t$:

$$L_{\mathrm{cons}} = -\log \left[ \frac{\exp(\omega_t^\top a_t)}{\sum_{i=1}^K \exp(\omega_i^\top a_t)} \right]$$

The total training loss is $$L_{\mathrm{total}} = L_{\mathrm{RL}} + \lambda L_{\mathrm{cons}}$$.

• Skill Memory and Retrieval: ViReSkill leverages key–value skill libraries, with keys as concatenated embeddings of descriptions and code, enabling hybrid retrieval by task and code similarity (Kagaya et al., 29 Sep 2025).

Optimization thus enforces both reward maximization and semantic policy–plan alignment, stabilizing learning and enhancing interpretability.

4. Empirical Evaluation and Benefits

Systematic evaluations in simulation and real-world settings have demonstrated compelling benefits:

Framework	Domain/Platform	Baseline SR (%)	VLM-Guided SR (%)	Notable Gains
COVLM-RL (Li et al., 10 Dec 2025)	CARLA	40 (seen) / 20 (unseen)	70 / 70	+30 pp seen, +50 pp unseen
ViReSkill (Kagaya et al., 29 Sep 2025)	LIBERO/RLBench/UR5	45-47 (baselines)	78-82	+30–50 pp, robust sim-to-real transfer
CLASP (Knauer et al., 6 Jun 2026)	DLR SARA ARM	0% (5-demos To.5)	90–100 (CLASP)	90%+ with 2–5 demos, instant composition
MMG2Skill (Che et al., 1 Jun 2026)	GUI/Game/Card	Baselines vary	+12.8 — +25.3 pp	All VLMs, all domains improved
InSight (Wang et al., 23 Jun 2026)	Real Manipulation	0% (baseline OOD)	80–96%	High success with no human demos

Key observations:

• Semantic priors effect substantial dimensionality reduction (e.g., panoramic ResNet features $\sim$O(10^4) → (5-d one-hot + 256-d embedding) in COVLM-RL).
• Learning speed (reward convergence) is often doubled or more, with greatly increased stability when semantic regularizers are introduced.
• VLM-derived skill schemas and compositional plans enable data-efficient expansion and robust generalization to new environments, objects, and tasks.

5. Limitations, Ablations, and Failure Analyses

Despite significant progress, key limitations have been documented:

• Latency and Inference Cost: VLM queries may be computationally intensive. COVLM-RL demonstrates that querying every N steps (vs. every step) maintains near-optimal generalization at 5× lower compute (Li et al., 10 Dec 2025).
• Skill Memory Scalability: As memory size grows, retrieval and update latencies increase sublinearly. Compression and hierarchical memory structures are discussed as future directions (Kagaya et al., 29 Sep 2025).
• Dependence on Prompt Engineering: The quality of VLM reasoning and segmentation is sensitive to prompt phrasing and in-context exemplars (Wang et al., 23 Jun 2026, Li et al., 10 Dec 2025).
• Open-loop/Closed-loop Gaps: Some systems (e.g., ViReSkill, InSight) are predominantly open-loop, executing skills without intra-trial feedback, which limits adaptivity to unanticipated dynamics (Wang et al., 23 Jun 2026, Kagaya et al., 29 Sep 2025).
• Expressiveness of Primitives: In current frameworks, primitives are limited to single-axis motion or other restricted classes, constraining the richness of autonomously acquired behaviors (Wang et al., 23 Jun 2026).
• Error Modes: Dominant error sources include mis-identification of critical objects, erroneous axis selection for primitive execution, and premature/late primitive termination (Wang et al., 23 Jun 2026, Li et al., 10 Dec 2025).

Ablation results repeatedly confirm the necessity of both semantic guidance and skill revision. E.g., removing semantic regularization in COVLM-RL reduces SR by ∼15pp, and omitting trajectory-driven revision in MMG2Skill reduces domain improvements by over half.

6. Generalization, Continual Learning, and Sim-to-Real Transfer

VLM-guided architectures systematically demonstrate enhanced generalization and lifelong learning capabilities:

• Continual Skill Expansion: InSight and ViReSkill show how autonomous detection of missing skills or primitives, followed by VLM-orchestrated acquisition and integration, enables continual expansion of the agent’s behavioral repertoire (Wang et al., 23 Jun 2026, Kagaya et al., 29 Sep 2025).
• Sim-to-Real Transfer: Domain randomization, calibrated perception modules, and robust semantic priors deliver strong sim-to-real transfer, as shown by ViReSkill and CLASP on the UR5 and DLR SARA arms with minimal or no adaptation (Kagaya et al., 29 Sep 2025, Knauer et al., 6 Jun 2026).
• Hierarchical and Compositional Generalization: Structured semantic skills, compositional fusion (e.g., covariance-weighted TP-KMP in CLASP), and scenario-tailored plan generation support robust execution across novel objects, poses, and multi-step goals (Knauer et al., 6 Jun 2026).

Trained agents thus outperform purely supervised or RL-only baselines on diverse metrics (success rate, sample efficiency, trajectory quality) and exhibit resilience to task, scene, and embodiment shift.

7. Outlook and Research Directions

Persisting challenges and opportunities identified across VLM-guided skill acquisition research include:

• Closed-Loop Semantic Control: Next-generation frameworks aim to unify VLM-based per-step situational awareness with low-level control, supporting online adaptation and recovery (Kagaya et al., 29 Sep 2025, Wang et al., 23 Jun 2026).
• Curated Benchmarking: There is broad recognition of the need for richer benchmarks spanning perception, reasoning, imitation, and RL under standardized conditions (Xiang et al., 26 Jun 2025).
• Model Compression and Prompt Optimization: Efficient VLM distillation and robust prompt optimization are key for scalable, real-time deployment (Li et al., 10 Dec 2025, Kagaya et al., 29 Sep 2025).
• Hierarchical, Multi-Agent, and Modular Extensions: Methods for hierarchical plan refinement, multi-agent skill sharing, and modular curriculum learning are highlighted as promising extensions (Knauer et al., 6 Jun 2026, Kagaya et al., 29 Sep 2025).

VLM-guided skill acquisition rests on systematic semantic reasoning, plan regularization, memory-driven reuse, and continual expansion. These techniques collectively bridge semantic understanding and physical competence in embodied agents, yielding increasingly robust, interpretable, and generalizable skills across digital and real-world domains.