Crayotter: Open-Source Long-Form Video Editing

• Crayotter is an open-source multimodal multi-agent system that edits long-form videos from real footage while preserving narrative intent.
• Its artifact-centric workflow externalizes state through coverage reports, editing blueprints, and tool calls, ensuring traceability and reproducibility.
• The system employs a three-phase pipeline—material preparation, blueprint research, and timeline execution—to maintain coherence and enable targeted repairs.

Crayotter is an open-source multimodal multi-agent system for prompt-driven long-form video editing that operates on real, heterogeneous source footage rather than treating the task as single-shot video generation. It was proposed for settings in which an agent must preserve narrative intent across material preparation, timeline construction, post-production, and revision while also leaving enough visible evidence to diagnose failures. Its defining characteristic is an artifact-centric workflow in which coverage reports, multimodal analyses, editing blueprints, tool calls, timeline files, intermediate renders, and final exports are externalized as inspectable state, making editing runs traceable, replayable, diagnosable, and selectively revisable (Yan et al., 31 May 2026).

1. Problem setting and system rationale

Crayotter addresses long-form video editing as a long-horizon multimodal control problem. In the formulation given in the paper, the difficulty is not merely selecting plausible clips, but maintaining coherence over multiple coupled stages: material preparation, timeline construction, post-production, and revision. The motivating claim is that small errors propagate in this setting: wrong asset retrieval, poor in/out points, continuity-breaking transitions, narration that no longer matches visuals, and pacing drift can compromise the final video and, in many systems, force a full restart (Yan et al., 31 May 2026).

The system was proposed against two failure modes attributed to existing approaches. First, generation-first systems focus on creating clips but do not robustly coordinate editing operations over real footage. Second, practical tooling systems may be usable but often lack scientific traceability, making runs hard to replay, inspect, compare, or selectively repair. Crayotter’s response is to make production state visible, reusable, and correctable by externalizing artifacts rather than relying on hidden conversational state.

A central design principle is environment-grounded reflection. After tool execution, revision is driven by observable artifacts in the editing environment rather than by textual reasoning alone. This suggests that failure analysis is intended to occur at the level of concrete production evidence—such as a coverage gap, a mistimed segment, or a flawed transition—rather than at the level of opaque model internals.

2. Formal structure and artifact-centric representation

The overall workflow is formalized as a transformation

$\mathcal{F}: (q,\mathcal{A}_0) \rightarrow y,$

where $q$ denotes the user request, $\mathcal{A}_0$ the optional local asset set, and $y$ the final exported video. The process is factorized into three phases: retrieval produces the material pool $\mathcal{P}$, research produces the editing blueprint $\mathcal{B}$, and tool execution produces the final output $y$ (Yan et al., 31 May 2026).

The three key intermediate artifact types are defined as

$$\begin{aligned} \mathcal{P} &= (\mathcal{S},\Gamma,\mathcal{M}),\ \mathcal{B} &= \{b_m\}_{m=1}^{M_z},\ \mathcal{H} &= \{(s_k,a_k,s_{k+1},\zeta_k)\}_{k=1}^{K}. \end{aligned}$$

Here, $\mathcal{P}$ is the material pool, with $\mathcal{S}$ the selected source videos, $q$0 the coverage evidence or report, and $q$1 the multimodal analysis records. The blueprint $q$2 is a shot-level editing plan. The history $q$3 records execution as state/action transitions with diagnostics.

Source video and edit events are also formalized explicitly. Each source video is represented as

$q$4

where $q$5 is the visual stream, $q$6 the optional audio stream, and $q$7 the duration. The edited output is represented as

$q$8

with each editing event

$q$9

The interval $\mathcal{A}_0$0 denotes the in/out span, and $\mathcal{A}_0$1 is the editing operator. The paper lists operators including cut, transition, narration alignment, subtitle placement, loudness normalization, and export. This representation makes clear that Crayotter must choose source identity, temporal interval, and edit operation under a persistent narrative objective.

The artifact-centric architecture is significant because each phase emits inspectable products that serve simultaneously as downstream inputs and as objects for human inspection. This separates planning state from hidden chain-of-thought-like reasoning and anchors the workflow in persistent, testable artifacts.

3. Three-phase production pipeline

Crayotter uses a three-phase pipeline: coverage-aware material preparation, artifact-based editing research, and tool-grounded timeline execution (Yan et al., 31 May 2026).

Phase 1: coverage-aware material preparation

Phase 1 constructs a coverage-verified material pool,

$\mathcal{A}_0$2

The request $\mathcal{A}_0$3 is expanded into weighted tags,

$\mathcal{A}_0$4

with the PDF text malformed but the intended categories given as scene requirements, people or actions, affective style, narrative purpose or story, and shot evidence. Each tag $\mathcal{A}_0$5 receives an importance weight $\mathcal{A}_0$6.

Retrieval is iterative. At round $\mathcal{A}_0$7, a query $\mathcal{A}_0$8 is generated from the original request and the currently missing tags. A coarse retriever returns a high-recall candidate set via a combined text-relevance and metadata-relevance score. Crayotter then samples frames $\mathcal{A}_0$9 and temporal windows $y$0 for each candidate and estimates tag support using

$y$1

The reranking score balances request relevance against marginal coverage gain, and the selected pool is evaluated using normalized coverage,

$y$2

Missing tags $y$3 become the basis for the next query. The loop stops when the coverage target is reached or the retrieval budget is exhausted.

The inspectable outputs of this phase are coverage reports, candidate rankings, downloaded source pools, multimodal analysis JSONs, and tag-level evidence for covered or missing requirements. These artifacts make it possible to attribute later failures to retrieval insufficiency, weak verification, or source-pool contamination such as off-topic or vertical footage.

Phase 2: artifact-based editing research

Phase 2 converts the material pool into a time-grounded editing blueprint,

$y$4

Its central problem is temporal localization: not only determining which source clip is relevant, but also determining which exact interval best supports a narrative beat. For each selected video $y$5, Crayotter builds a time-marked version

$y$6

with the PDF syntax truncated but the intended meaning stated explicitly: human-readable time coordinates are overlaid on sampled frames or analysis windows. The paper states that this helps frozen multimodal LLMs localize events because temporal coordinates are rendered directly on the same frames that depict the events.

Narrative beats are denoted

$y$7

with examples including establishing shot, activity montage, close-up interaction, and closing scene. For each beat, the research agent predicts source and interval by parsing the output of a research model conditioned on time-marked views, the beat, and the coverage report. Each blueprint element is

$y$8

where $y$9 is the intended cinematic role, $\mathcal{P}$0 the transition or rhythm rationale, and $\mathcal{P}$1 the narration or subtitle intent.

The blueprint is therefore not free-form prose but a temporally addressable production plan. It records sequence of beats, source assignment, in/out points, clip order, pacing, transition rationale, and narration intent. The externalized outputs include a narrative blueprint, clip ordering plan, rhythm targets, transition plans, narration plans, and beat-to-source and beat-to-timestamp mappings.

Phase 3: tool-grounded timeline execution

Phase 3 executes the blueprint in an external editing environment and outputs timeline files, intermediate renders, narration and subtitle tracks, normalized audio, and the final exported video $\mathcal{P}$2. The environment is formalized as an artifact transition system:

$\mathcal{P}$3

where $\mathcal{P}$4 contains the current timeline, source assets, subtitle tracks, narration files, rendered previews, and tool logs, and $\mathcal{P}$5 is a tool action chosen from the tool set.

Crayotter exposes 21 registered tools divided into Phase 1 preparation tools and Phase 3 editing tools. The listed taxonomy includes search_video, rank_video_candidates, download_video, analyze_video, recall_semantic_segments, cut_video, batch_cut_video, merge_videos, inspect_video_duration, add_transition, plan_transition_timeline, list_transition_presets, score_cut_continuity, recommend_transition_for_cut, build_edit_timeline_from_segments, align_narration_to_timeline, validate_timeline_constraints, add_narration_segments, validate_narration_timeline, duck_background_audio, normalize_loudness, add_subtitles, and export_video. The paper simultaneously states that there are 21 registered tools and then lists 23 tool names; this discrepancy appears in the provided technical overview and should be read as a documentation inconsistency rather than a distinct architectural claim.

The transition subsystem supports 17 transition presets in three categories: basic, motion, and cinematic. Continuity diagnostics sample frame signatures around cuts, estimate visual gaps using brightness, saturation, and motion, and recommend hard cuts, dissolves, or fade-through-black. After each tool call, diagnostics

$\mathcal{P}$6

are computed over the previous state, action, new state, blueprint, and request. The verifier checks tool validity, timestamp accuracy, coverage preservation, narration alignment, transition smoothness, and render quality. If a problem is detected, Crayotter performs local repair rather than restarting the entire workflow.

A particularly important implementation detail is the mandatory composite re-analysis step before narration: once the merged video is assembled, the system re-analyzes the actual merged video with a multimodal vision model so that narration aligns with rendered content rather than only with source analyses.

4. Multi-agent organization, traceability, and selective revision

The implementation uses an explicit planner–researcher–executor separation. The planner interprets user intent, converts the prompt into phase goals, drives request decomposition, and manages phase routing. The researcher synthesizes retrieval evidence, multimodal analysis, temporal localization, and narrative structure into the blueprint $\mathcal{P}$7. The executor translates the blueprint into tool calls, manipulates the timeline, inserts transitions, aligns narration, generates subtitles, normalizes audio, exports previews and final outputs, and responds to diagnostics with local repair (Yan et al., 31 May 2026).

Communication among these agents is mediated through artifact contracts rather than free-form inter-agent chat. The stable contract spans input-side artifacts such as requests, retrieval candidates, and per-video analyses; planning artifacts such as blueprints, clip ordering, rhythm, transitions, and narration plans; and execution artifacts such as timeline files, intermediate renders, and tool logs. This makes inter-agent coordination inspectable and reduces dependence on ephemeral hidden reasoning.

Traceability is a central systems contribution. Across the three phases, Crayotter externalizes user request, retrieval candidates, local asset inventory, analysis outputs, coverage tags and weights, coverage reports $\mathcal{P}$8, multimodal analyses $\mathcal{P}$9, narrative beats, blueprint $\mathcal{B}$0, clip ordering plans, transition plans, rhythm targets, narration plans, tool calls and arguments, execution states and logs, timeline files, previews, subtitle tracks, narration tracks, loudness-adjusted audio, and final exports.

Because failures are attached to explicit artifacts, diagnosis can be localized. The paper gives examples: in the “Yili grassland” case, a damaged source interval was detected and replaced with a stable interval from the same source while still yielding a valid export; in “Roast-duck” and “campus-sports,” the traces exposed vertical-material leakage, missing requested sub-events, and weak documentary continuity. The significance lies less in the existence of failures than in the fact that they can be assigned to a phase and a segment and then revised locally.

The paper also notes runtime utilities: phase routing modes including full pipeline, skip research, local-only, or local-first sufficiency check; bounded reference-only memory for reusable tool patterns, failure guards, and quality checklists without storing task-specific creative content; and an agent stall watchdog. These utilities are presented as mechanisms for robustness and replayability.

5. Replayable trajectory schema and verifiable reward design

Crayotter includes a replayable trajectory schema and a verifiable reward design intended to prepare the workflow for future policy optimization. Phase 3 is framed as an episodic MDP,

$\mathcal{B}$1

with the PDF text corrupted but the intended semantics stated: observable artifact states, tool actions, environment transition, reward, and discount factor (Yan et al., 31 May 2026).

Actions are structured tool calls,

$\mathcal{B}$2

where $\mathcal{B}$3 is the tool identity and $\mathcal{B}$4 the JSON arguments. The policy $\mathcal{B}$5 is optimized with GRPO over full trajectories

$\mathcal{B}$6

to maximize discounted return. Step reward is decomposed into execution validity, artifact production validity, alignment to blueprint and constraints, and successful local repair or reflection. One explicit alignment term is

$\mathcal{B}$7

where $\mathcal{B}$8 is the current rendered preview, $\mathcal{B}$9 the target duration, and $y$0 the hard-violation count.

Episode-level quality is scored by a judge reward that averages five evaluation dimensions: theme, richness, narrative, smoothness, and visual. Total return combines step rewards, judge reward, and optional human preference. The GRPO formulation uses group-normalized advantage,

$y$1

and a clipped objective with KL regularization to a reference policy.

The important architectural point is that the RL state is observable artifact state rather than hidden dialogue state. This suggests that the system is designed to support reinforcement learning over workflow execution, imitation learning from successful traces, and failure mining targeted at subskills such as timing, transitions, or narration alignment. The paper presents this as preparation for future policy optimization rather than as a completed RL training study.

6. Evaluation, implementation details, limitations, and nomenclature

Crayotter is evaluated on a 23-theme editing pack against CapCut-Mate and CutClaw. The 23 cases span five scenario families: pet with 3 cases, campus with 5, travel with 5, scenery with 5, and food with 5. Human evaluation uses three annotators and a 1–5 scale over theme alignment, content richness, narrative coherence, editing smoothness, and visual quality; the overall score is the mean of the five dimensions, averaged first across annotators for each theme-method pair and then across all 23 themes. The paper also reports AI-based judgments using the same multidimensional rubric (Yan et al., 31 May 2026).

Method	Human overall	AI overall
Crayotter	3.40 ± 0.59	2.39 ± 0.62
CapCut-Mate	2.44 ± 0.72	2.10 ± 0.64
CutClaw	1.70 ± 0.62	1.57 ± 0.40

In human evaluation by dimension, Crayotter scores 3.59 on Theme, 3.35 on Richness, 3.22 on Narrative, 3.29 on Smoothness, 3.54 on Visual, and 3.40 Overall. CapCut-Mate scores 2.59, 2.71, 2.01, 2.13, 2.74, and 2.44 respectively, while CutClaw scores 1.59, 1.64, 1.72, 1.86, 1.67, and 1.70. The paper highlights the clearest gains for Crayotter in theme alignment, narrative coherence, and editing smoothness, while visual quality remains the closest category. In AI evaluation by scenario family, Crayotter is strongest on scenery at 2.88, whereas travel is the one category in which CapCut-Mate’s AI score of 2.64 exceeds Crayotter’s 2.44.

The paper also reports trace and runtime characteristics from released traces: all 23 Crayotter runs complete; each stages 8 local source files under a no-download protocol; average trace length is 221 observable events; and average runtime is 1243 seconds. The core stack used in each run includes source analysis, semantic segment recall, batch cutting, timeline merging, transition planning, narration alignment, subtitle insertion, loudness normalization, duration inspection, and export. For reproducibility, code, traces, and examples are publicly available, trajectories can be exported in verl-compatible JSONL, rollout uses an sglang multi-turn loop, tools are invoked via subprocess-backed tool calls, and training episodes are specified by fixtures containing user request, target duration, allowed tools, seeded workspace files, and optional scripted warm-up turns.

The limitations are stated explicitly. The evaluation is system-level rather than a module ablation study. Large-scale policy optimization is not performed. Broader human preference studies are left for future work. Food and pet scenarios remain difficult. Close-up continuity and fine-grained action coverage remain weak points. Vertical or off-topic clip leakage can still occur, and documentary continuity can remain weak in some scenarios. The system is therefore presented as an improvement in traceability and outcome quality rather than as a solved account of long-form editing.

A terminological clarification is warranted because the string “Crayotter” can be mistaken for cryotron- or nanocryotron-related terminology. In the cited literature, cryotron-derived devices refer to superconducting switching and cryogenic memory architectures, such as the hybrid “nanocryotron-driven charge configuration memristor” (Mraz et al., 2022). Crayotter, by contrast, denotes a multimodal multi-agent video editing system. The similarity is orthographic rather than conceptual.