VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation

Abstract: Recent agent frameworks such as Claude Code, Codex, and OpenClaw are strong at tool use and orchestration, but whether they can handle long video generation, a long-horizon multimodal task, remains underexplored. Unlike earlier video agents whose pipeline is handcrafted, these frameworks can build and refine their own workflows. We introduce VideoWeaver, an agent harness and benchmark that evaluates and evolves skills for long video generation, where an agent turns a single instruction into a long video by composing foundation skills into its own workflow rather than following a predefined pipeline. The benchmark has 16 task categories and 285 cases, with references spanning text, image, audio, video, and their combinations. Because errors can arise at any stage and not just in the final video, we propose an agent-as-judge that inspects both the execution trace and the final video, grounding its scores in evidence such as metadata and intermediate files. Using this feedback, we further design a skill evolution algorithm that refines and merges the agent's skills. Across multiple frameworks and models, we find that an explicit composition skill improves the generation process over using foundation skills alone, that skill evolution further improves output quality, and that performance varies notably across harness and model choices. The proposed agent-as-judge also aligns well with human judgments, especially on process metrics. Code and dataset is available at https://github.com/JianhuiWei7/VideoWeaver

• The paper demonstrates that autonomous agents can evolve composition skills beyond rigid pipelines, outperforming foundation-only baselines.
• It introduces an agent-as-judge framework that evaluates intermediate artifacts and output metrics, ensuring robust and explainable performance.
• The study validates skill evolution through iterative feedback, achieving superior coherence and generalization in long video generation.

VideoWeaver: Agentic Long Video Generation via Skill Evolution and Evidence-Grounded Evaluation

Introduction and Motivation

Long video generation presents unique challenges in cross-clip consistency, multimodal orchestration, and error tracing. While recent generative models such as Sora and Veo advance short clip generation, and agentic systems like StoryAgent and Co-Director orchestrate workflows for complex multimodal scenarios, most remain either end-to-end or fixed in their pipeline logic. "VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation" (2606.08091) systematically investigates the ability of general-purpose agent frameworks—exemplified by Claude Code, Codex, and OpenClaw—to autonomously compose and evolve skills for long-horizon multimodal video generation.

Rather than relying on rigid, human-designed pipelines, VideoWeaver enables agents to synthesize procedural "composition skills" from a set of foundation skills, dynamically orchestrating video, audio, and image generation, as well as media processing tools, into self-devised workflows. The paper introduces not only a benchmark suite spanning 16 diverse categories and 285 cases—covering a breadth of modality configurations and durations—but also an "agent-as-judge" evaluation method, which grounds its evaluation in execution traces, intermediate artifacts, and final outputs, departing from the common single-output-centric assessment.

Dataset and Task Formalization

VideoWeaver’s dataset delineates 16 categories of long video tasks, extending across instructional videos, anime, product demos, cinematic storytelling, and more—each annotated with multimodal references (text, image, audio, video) and rigorous specification for output duration and resolution. The distribution of input modalities and target durations is visualized (Figure 1):

Figure 1: Distribution of dataset tasks by input modality (Text, Image, Video, Audio) and required output duration.

Tasks are split into training, test, and three held-out out-of-distribution (OOD) categories to assess both standard and generalization performance. Each task requires the agent, given a single high-level instruction, to perform multi-step planning and execution: decomposing objectives, managing artifacts, invoking relevant skills, and maintaining consistency across the full video.

Agent Harness and Skill Evolution

The agent harness pairs a backbone LLM (e.g., GPT-5.5, Seed2.0) with a rich suite of foundation skills for basic media generation, understanding, and processing. Foundation skills are self-contained capabilities (see Appendix for a skill list), but are insufficient for long-term workflow management. VideoWeaver, therefore, introduces "composition skills"—high-level policies specifying orchestrated invocation and sequencing of foundation skills over multiple steps—to achieve coherent, long-horizon video assembly. Composition skills themselves can be synthesized and refined by a "creator skill," which learns to construct effective orchestrations from available primitives and task exemplars.

Skill evolution proceeds through three agent-driven stages: task inference (initial skill creation and execution), optimization (iterative refinement of skills using feedback), and merging (integrating category-level skills into a universal creator). This pipeline is shown in Figure 2:

Figure 2: Skill evolution pipeline comprised of execution, optimization (with trace and output feedback), and merging into unified creator skills.

Evaluation Framework: Agent-as-Judge

Traditional video benchmarks focus evaluation on the end product, missing diagnostic opportunities for tracing failures in the generation pipeline. VideoWeaver’s agent-as-judge framework (Figure 3) expands this by leveraging an LLM equipped with evaluative skills to inspect not just the final video but also intermediary artifacts, execution logs, metadata, and tool invocation traces.

Figure 3: The agent-as-judge evaluation protocol, enabling multi-turn inspection of both process artifacts and outputs to provide evidence-grounded scores across process and output metrics.

Process metrics include error-free tool execution, correct input processing, planning, step following, and use of user assets. Output metrics assess format compliance, prompt fidelity, cross-clip visual and audio consistency, synchrony, logical coherence, and reference integration. Evaluation is binary (pass/fail) per metric, paired with evidentiary support and textual feedback. This fine-grained diagnostic feedback is used not only for benchmarking but as a direct optimization signal in skill evolution.

Experimental Results and Analysis

Harness and Model Comparison

Results across multiple harnesses and model backbones reveal marked variance in process and output quality. Notably, mature pairings such as Codex + GPT-5.5 and Claude Code + Opus 4.7 outperform others in both procedural (process metrics >0.95) and compositional (output average >0.60) criteria. Model-harness compatibility exerts a substantial influence: substituting different LLMs in the same harness settings leads to significant degradations in multi-stage performance metrics.

Impact of Composition Skill and Skill Evolution

Explicit composition skills—where orchestration is externalized rather than ad hoc—significantly surpass foundation-only baselines, especially on measures of input preprocessing, planning, and reference asset usage (scores up to 0.98). Iterative skill evolution further improves both process and output quality. Incorporating judge-generated feedback during evolution yields additional gains, especially in output metrics, showing that process-visible fixes are often addressed by self-evolution, while output-reflective improvements benefit from external feedback.

Comparative results indicate:

• Foundation-only agents are prone to omitting essential preparatory and coherence-preserving steps.
• Human-constructed ("Expert") composition skills remain the strongest baseline, but evolved skills close much of the gap.
• Evolved skills generalize beyond their training distribution, maintaining or exceeding performance when transferred to novel (OOD) task categories.

Generalization and Human Alignment

Evolved skills show positive win rates in pairwise comparisons with initial skills on both test and OOD categories (win/tie/loss rates: 25.3/58.2/16.5 for test, 23.8/59.7/16.5 for OOD), indicating substantial generalization in procedural orchestration.

The automatic agent-as-judge demonstrates high concordance with human annotators on process metrics (exact match >0.92, Kendall’s $\tau_b$ >0.76), and reasonable alignment on complex output metrics (e.g., format compliance, audio/video consistency). This suggests the viability of evidence-grounded LLM-based assessment in both pipeline diagnostics and quality control.

Resource Utilization

Advanced skill compositions require more tool invocations and LLM token consumption, reflected in higher relative generation times (e.g., RGT up to 65.94 for Expert compositions). However, better-orchestrated solutions exhibit decreased Vision Content RGT (i.e., more time in planning/reasoning versus raw media generation), indicating greater emphasis on procedural coordination for superior outcomes.

Representative Cases

Figure 4: Top: Long-video editing with image and video references; Bottom: Audio-driven story generation from text and audio inputs. These examples illustrate the agent's ability to coordinate multimodal inputs and maintain temporal/narrative coherence through evolved skill policies.

Discussion and Future Implications

VideoWeaver establishes a comprehensive paradigm for agentic long video generation built upon modular skill composition. By merging evidence-grounded agent evaluation with autonomous skill evolution, the framework provides a scalable approach to systematically improving agentic orchestration in complex, multi-stage, and multimodal content synthesis tasks.

Practically, this carries significant implications for accelerating multimodal generative applications—including instruction-driven media storytelling, educational content, and automated digital editing—in ways that are flexible to both new modalities and evolving toolsets. Theoretically, it reflects the broader trend of treating LLM agents not merely as planners but as meta-learners capable of skill evolution, self-debugging, and iterative process optimization. The combination of execution trace analysis and output review, along with fine-grained diagnostics, paves the way for explainable, robust, and generalizable agentic systems.

Future directions include scaling dataset scope, integrating broader and more diverse model/tool backends, and developing enhanced optimization protocols for better credit assignment and robust skill verification. Extensions to open-source toolchains and further alignment monitoring (including richer human-in-the-loop protocols) are proposed.

Conclusion

VideoWeaver provides an integrated benchmark, agent harness, and methodological contribution for agentic long video generation. Its explicit treatment of skill composition and evolution, combined with evidence-grounded process/output evaluation, delivers clear improvements in agent performance and generalization. The findings demonstrate the necessity of explicit orchestration skills and feedback-driven evolution for long-horizon multimodal generation, supported by a reliable LLM-based agent-as-judge that aligns with human judgment and operationalizes detailed diagnostic feedback.