CutClaw: Autonomous Video Editing

• CutClaw is an autonomous multi-agent video editing framework that transforms long footage into concise, narrative-driven outputs with precise music synchronization.
• It employs hierarchical semantic decomposition and multimodal LLMs to segment video and audio, ensuring high aesthetic quality and rhythmic alignment.
• Its multi-agent pipeline—featuring Playwriter, Editor, and Reviewer agents—enables rigorous shot selection, temporal precision, and robust protagonist focus, surpassing earlier frameworks.

CutClaw is an autonomous multi-agent framework for hours-long raw video editing, designed to generate concise narrative-driven videos with precise music synchronization. Leveraging multiple Multimodal LLMs (MLLMs), CutClaw integrates hierarchical semantic decomposition, music-anchored planning, and rigorous multi-pass visual curation to deliver broadcast-quality outputs with organic audio-visual harmony and explicit protagonist focus. Developed and evaluated by GVCLab, CutClaw substantially surpasses earlier automated editing frameworks in visual quality, fidelity to user instruction, and rhythmic alignment (Zhao et al., 31 Mar 2026).

1. Problem Definition and Objectives

CutClaw addresses the task of automated editing for hours-long video footage, aiming to produce short, meaningful videos driven by explicit user narrative instructions $\mathcal{I}$ and synchronized to a given music track $\mathcal{M}$. The system prioritizes four core requirements:

• Semantic preservation of source material with respect to $\mathcal{I}$.
• Precise alignment of cuts and transitions to the rhythm of $\mathcal{M}$, achieving minimal temporal asynchrony ($\Delta t \leq 0.1\,\mathrm{s}$ tolerance for cut alignment).
• High visual aesthetics (e.g., saliency, framing) and perceptual quality.
• Protagonist prominence, as determined by character identity tracking and MLLM-based semantic grounding.

2. Hierarchical Multimodal Decomposition

The first stage reduces the enormous search space of raw video and music into discrete, interpretable segments, thereby enabling tractable agentic reasoning and joint optimization.

2.1. Video: Shots to Scenes

• Shot Detection: A boundary detector (e.g., PySceneDetect) splits the input video $\mathcal{V}$ into atomic shots $\mathcal{S} = \{s_i\}$.
• Semantic Embedding: Each shot $s_i$ is embedded as a vector $\mathbf{v}_i$ (covering cinematography, character, and environmental attributes) via an MLLM encoder (e.g., Qwen3-VL).
• Scene Boundary Inference: Scene transitions are triggered when the similarity

$$\mathrm{Sim}(s_i,s_{i+1}) = \boldsymbol{\alpha}^\top \mathbf{v}_{i,i+1}$$

drops below threshold $\mathcal{M}$0 (where $\mathcal{M}$1 is the LLM embedding for the shot pair and $\mathcal{M}$2 is an attribute weight vector).

• Aggregation and Annotation: Shots are grouped into contiguous scenes $\mathcal{M}$3, which are enriched by character identities $\mathcal{M}$4 via ASR and LLMs, forming scene descriptions $\mathcal{M}$5.

2.2. Audio: Beats to Structural Units

• Keypoint Extraction: Downbeats $\mathcal{M}$6, pitch change points $\mathcal{M}$7, and spectral energy shifts $\mathcal{M}$8 are identified (e.g., using madmom), yielding a pool $\mathcal{M}$9 of salient moments.
• Filtering and Structuring: Deduplication and denoising produce a clean keypoint set $\mathcal{I}$0. The music is partitioned into macro-units $\mathcal{I}$1 (e.g., verse, chorus) via an LLM.
• Segment Boundary Scoring: Within each $\mathcal{I}$2, each candidate keypoint $\mathcal{I}$3 is scored as

$\mathcal{I}$4

where each $\mathcal{I}$5 term summarizes local saliency.

Each $\mathcal{I}$6 is annotated with captions describing local rhythmic and emotional character.

3. Multi-Agent Architecture

CutClaw's agentic structure is hierarchical and mirrors professional editing workflows.

3.1. Playwriter Agent

• Consumes scene set $\mathcal{I}$7, audio units $\mathcal{I}$8, and user instructions $\mathcal{I}$9.
• Generates a global shot plan $\mathcal{M}$0 for each audio macro-unit $\mathcal{M}$1, adhering to:
  • Non-overlap Constraint: $\mathcal{M}$2.
  • Duration Anchoring: $\mathcal{M}$3.
• Each shot specification $\mathcal{M}$4 provides duration, scene index, and an LLM-generated scene description $\mathcal{M}$5.

3.2. Editor Agent

• For each $\mathcal{M}$6, retrieves and trims candidate shots in $\mathcal{M}$7, expands to neighbors as needed.
• Conducts fine-grained search:

$\mathcal{M}$8

where $\mathcal{M}$9 is the MLLM-derived aesthetic score, $\Delta t \leq 0.1\,\mathrm{s}$0 is the protagonist proportion, and $\Delta t \leq 0.1\,\mathrm{s}$1 control term balance.

• Commits candidates to Reviewer or backtracks on rejection.

3.3. Reviewer Agent

• Validates each proposed clip on three axes:
  • Semantic Identity Verification: Ensures correct protagonist identity via MLLM sampling.
  • Temporal and Structural Integrity: Enforces non-overlap and exact alignment to audio keypoints.
  • Perceptual Quality: Rejects clips exhibiting artifacts (e.g., blur, poor resolution).
• Provides structured feedback to the Editor for iterative refinement.

4. Music Synchronization and Objective Formulation

The music synchronization module defines CutClaw’s “AV Harmony”:

$\Delta t \leq 0.1\,\mathrm{s}$2

• $\Delta t \leq 0.1\,\mathrm{s}$3 measures alignment of transition timestamps to audio keypoints.
• Other terms evaluate visual quality ($\Delta t \leq 0.1\,\mathrm{s}$4), narrative coherence ($\Delta t \leq 0.1\,\mathrm{s}$5), and semantic fidelity ($\Delta t \leq 0.1\,\mathrm{s}$6).

The Editor’s local optimization objective focuses on maximizing $\Delta t \leq 0.1\,\mathrm{s}$7 and $\Delta t \leq 0.1\,\mathrm{s}$8, as defined above.

5. Experimental Evaluation

CutClaw was benchmarked on 10 long-form videos (feature films, VLOGs; total $\Delta t \leq 0.1\,\mathrm{s}$924 hrs), 10 segmented music tracks from diverse genres, and 20 instruction scenarios (object-centric, narrative-centric). Experimental comparisons encompassed the following baselines:

• NarratoAI (subtitle-driven)
• UVCOM (highlight detection)
• Time-R1 (temporal grounding)

Key evaluation metrics included:

• Visual Quality (LLM-graded, scored 77.6 ± 0.8 vs. ~72 for baselines)
• Instruction Follow (70.0 vs. ~62)
• AV Harmony (beat alignment; 86.5 vs. ~79)
• User Study: $\mathcal{V}$02,000 votes; CutClaw led by a factor of over 2$\mathcal{V}$1 in Visual Quality, Instruction Follow, Audio-Visual Harmony, and Human-Likeness.

Ablation analyses revealed significant decrements in AV Harmony and semantic metrics upon removal of audio context and/or agentic procedures (Zhao et al., 31 Mar 2026).

Metric	CutClaw	Best Baseline
Visual Quality	77.6	~72
Instruction Follow	70.0	~62
AV Harmony	86.5	~79

6. Contributions, Limitations, and Future Directions

Major contributions include:

• Formalization of audio-driven video editing as joint optimization over visual, narrative, instructional, and rhythmic criteria.
• Hierarchical multimodal discretization for tractable agentic decision-making.
• Multi-agent MLLM-based pipeline enabling division of global planning (Playwriter), local retrieval/trimming (Editor), and cross-cutting validation (Reviewer).
• Comprehensive experimental validation demonstrating substantial gains over prior SOTA.

Limitations and future research:

Expressive visual hooks including generative effects and monologue highlights are unsupported. High inference latency remains a barrier, precluding real-time feedback. Robustness under extreme edit constraints or atypical source/musical material requires further investigation.

CutClaw provides a foundation for agent-powered, multi-stage, music-synchronized video editing, with demonstrated merits in both automated and human-centric evaluation settings (Zhao et al., 31 Mar 2026).