Timeline-Bench: Evaluating Agents on Realistic Video-Editing Tasks, from Raw Footage to Final Cut

Abstract: AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable. To this end, we introduce Timeline-Bench, a benchmark of 56 real video-editing tasks, each asking an agent to turn raw production material into a finished video. Tasks range from selecting dialog takes and shaping interview footage into a story to cutting commercials from product shots, voiceovers and graphics. Every task provides a brief, source assets, a container and a set of tests. A task is resolved when the output passes every test. The tests check the delivery format, the content and the brief's explicit requirements, and include a quality test calibrated on 2,582 blind judgments by 43 video editors. We evaluate 16 agents that pair frontier models with coding-agent harnesses such as Codex, Claude Code and OpenCode. The best, GPT-6 Astra in Codex with curated editorial guidance, resolves only 15 of the 56 tasks (26.8%), and the average agent resolves 14.0%. Human editors prefer the reference edit in 83.5% of judgments. Most unresolved runs (562 of 771) fail only the quality test: agents perceive footage through stills and transcripts and check their renders for defects, not craft. We release the tasks, verifier and per-run results at https://timelinebench.tensortest.com.

• The paper evaluates a benchmark dataset of 56 professionally varied, complex video editing tasks using 16 agents, with a primary focus on gaps between completed, correctly formatted tasks and professional- quality projects, such as assessments on story, pacing, photo, sound, and graphics, where professional accuracy significantly outweighs agent accuracy; agent accuracy leads to consistent gaps in AI comprehension of tasks.
• The study analyzed the failure models of 16 agents, revealing that while 562 out of 771 uncompleted tasks pass technical requirements, they fall short in quality standards, highlighting AI's inability to produce professional-level editorial decisions in video editing.
• The work details that the Keypoint 3 highlights how each of the 16 agents struggled at aligning with major quality components such as story assembly, pacing, sound and more where human decision-making wins 83.5% of the time.
• follow-up_questions

Benchmark scope and motivation

“Timeline-Bench: Evaluating Agents on Realistic Video-Editing Tasks, from Raw Footage to Final Cut” (2609.35143) evaluates whether AI agents can complete end-to-end professional video-editing assignments rather than isolated operations such as shot selection, temporal localization, or GUI manipulation. The benchmark is motivated by a substantive distinction between satisfying explicit production requirements and producing an edit that experienced editors regard as professionally successful. Video editing is particularly suitable for exposing this distinction because editorial decisions are temporally interdependent: shot selection affects narrative interpretation, duration affects pacing, and changes to picture can require corresponding revisions to dialogue, music, ambience, graphics, and sound design.

The benchmark contains 56 assignments drawn from 11 licensed EditStock packages, 15 Cinestudy projects, and 30 commissioned projects divided between user-generated-content and commercial collections. Each task supplies a brief, raw audiovisual assets, supporting paperwork, a reproducible execution environment, and delivery tests. The reference edit is held out from the agent. Tasks include narrative scenes, interview-based stories, documentaries, trailers, advertisements, personal-branding videos, and travel films. Collectively, they contain approximately 33 hours of primary source footage, with a median of 12.4 minutes per task and delivery durations ranging from 25 to 315 seconds. The dataset includes both landscape and portrait outputs and requires a broad combination of editorial skills, including scripting, take selection, music, dialogue editing, captions, graphics, color matching, reframing, and, in selected cases, dual-system synchronization and compositing.

This design places the benchmark closer to outcome-graded professional-work evaluations than to conventional video-understanding benchmarks. It also extends related agent benchmarks that evaluate software artifacts or knowledge-work deliverables by requiring a finished audiovisual artifact whose quality cannot be fully specified as a binary checklist. The benchmark’s central methodological contribution is therefore not only the task collection but the integration of mechanical verification with human-calibrated quality assessment.

Task construction and evaluation protocol

A task is resolved only if the rendered video passes every required test. The evaluation pipeline separates technical delivery, explicit brief compliance, generic content integrity, and editorial quality. Four delivery tests verify the existence of the output, codec and frame rate, duration, and audio properties. Six shared content tests reject degenerate outputs, including excessive silence, frozen imagery, repeated footage, dead air, channel imbalance, and excessive face cropping. Brief tests check requirements such as scene ordering, use of supplied voice-over, inclusion of specified content, and required on-screen text.

Most brief tests are programmatic. The verifier uses frame and soundtrack matching, OCR, and speech transcription to establish whether required source material, text, and spoken content appear in the appropriate form. The remaining visible-content tests are decided by a two-of-three majority among multimodal model judges. The authors validate the tests against reference edits and deliberately defective controls, such as muted audio, reordered sections, black openings, and frozen frames. This is important because a benchmark can otherwise produce misleading failure rates through under-specified or over-permissive verifiers.

Quality is evaluated separately from compliance. Three multimodal model judges score each edit on story and assembly, pacing, picture, sound, and graphics, together with an overall rating. One judge watches the video with sound; two judges receive contact sheets and file-level measurements rather than continuous audiovisual playback. Scores are standardized within judge and aggregated into a panel score. An agent output passes the quality test only when its panel score exceeds that of the held-out reference edit by a collection-specific tie margin. The margins are calibrated against 2,582 assessable judgments from 43 professional video editors, who performed blind pairwise comparisons between agent outputs and reference edits.

This procedure has a deliberate asymmetry. The reference edit is not treated as unique ground truth, since multiple valid editorial solutions may satisfy a brief. Instead, human preference establishes an empirical tolerance for the model-based quality test. The calibration is effective at the aggregate agent level: quality-test resolution rates correlate strongly with human win-or-tie rates across agents, with Spearman correlation $\rho = 0.93$ and Pearson correlation $r = 0.93$. However, the paper reports substantially weaker validity at the individual-edit level: agreement with the human majority is only $\kappa = 0.15$, and per-task correlations reach only $\rho = 0.32$. Thus, the quality test is suitable for comparing agents over many tasks, not for treating every individual pass or failure as a reliable human-equivalent judgment.

Agents and experimental conditions

The study evaluates 16 agents formed from frontier models and three execution settings. Fifteen are coding or terminal agents operating in Linux containers with FFmpeg, Python, Node.js, Remotion, HyperFrames, OCR, transcription, and other media-processing utilities. The models are run through OpenCode, Codex CLI, or Claude Code. A sixteenth condition uses GPT-6. Astra in Codex computer-use mode with DaVinci Resolve on macOS, without shell access.

Each agent receives one attempt on each of the 56 tasks, producing 896 runs in total. Coding runs have a 300-minute limit, 32 virtual CPUs, and 256 GB of memory. The benchmark therefore evaluates sustained planning, asset inspection, construction, rendering, verification, and repair rather than short interactive responses. One additional condition supplies GPT-6. Astra with 1,909 words of general editorial guidance covering brief interpretation, footage inspection, planning, construction, review, and delivery. The guidance is task-independent and does not disclose solutions.

The matched comparisons are informative about the relative contribution of harness and interaction mode. Switching between OpenCode and developer-specific harnesses does not produce statistically reliable differences in either human preference or task resolution. Curated guidance improves the point estimate for GPT-6. Astra from 21.4% to 26.8%, but the matched human win-or-tie difference is only $+0.6$ percentage points and is not significant. By contrast, computer use substantially degrades performance: GPT-6. Astra resolves 2 of 53 jointly evaluated tasks versus 12 for the coding condition, with a human win-or-tie difference of $-16.4$ percentage points and adjusted $p < 0.001$. This result does not establish that GUI interaction is intrinsically inferior; it establishes that this particular computer-use configuration, which lacked shell and code tools and could not directly access audio through the same workflow, was markedly less effective on these assignments.

Main performance results

The benchmark produces a low absolute resolution rate across all evaluated agents. The strongest condition, GPT-6. Astra in Codex CLI with curated guidance, resolves 15 of 56 tasks, or 26.8%. Claude Opus 5 in Claude Code follows at 23.2%, while GPT-6. Astra in either OpenCode or unguided Codex CLI resolves 21.4%. Across all 896 runs, only 125 tasks are resolved, for an aggregate resolution rate of 14.0% with a 95% confidence interval of 11.7–16.4%.

Agent condition	Tasks resolved	Resolution rate	Human win-or-tie rate
GPT-6. Astra, Codex CLI + guidance	15/56	26.8%	24.4%
Claude Opus 5, Claude Code	13/56	23.2%	23.2%
GPT-6. Astra, Codex CLI	12/56	21.4%	23.8%
GPT-6. Astra, OpenCode	12/56	21.4%	22.3%
Claude Fable 5.1, Claude Code	10/56	17.9%	23.5%
Claude Opus 5, OpenCode	10/56	17.9%	19.0%
GPT-5.6. Sol, OpenCode	10/56	17.9%	17.6%
All agents	125/896	14.0%	16.5%

The confidence intervals are wide because each agent receives only one run per task; for individual agents they are approximately $\pm 11$ percentage points. Consequently, the paper appropriately cautions that several top-ranked agents cannot be reliably distinguished. Runtime also does not explain success: median runtime ranges from 16 to 61 minutes, with Spearman correlation $\rho = 0.15$ between runtime and resolution. Estimated cost is more positively associated with performance, with $\rho = 0.79$ among 12 agents with cost records, although the best-performing guided condition costs approximately $9.76 per run rather than being the most expensive condition.

The human study establishes a stricter baseline for editorial quality. Across 2,582 judgments, editors prefer the professional reference edit in 83.5% of comparisons, prefer the agent edit in 11.1%, and report no meaningful preference in 5.5%. The aggregate human win-or-tie rate across agents is 16.5%. The low single-judgment reliability, $r = 0.93$0, reflects the subjectivity of pairwise editorial evaluation, but averaging three judgments per pair yields substantially more stable agent-level estimates, with estimated reliability of 0.84. The ranking remains highly stable when individual editors are removed.

Compliance is substantially stronger than editorial craft

The most consequential result concerns where agents fail. Of 771 unresolved runs, 562, or 73%, pass delivery, content, and brief tests but fail only the quality test. For 15 of the 16 agents, quality-test failure accounts for 61–90% of unresolved runs. Only six of the 863 delivered videos fail a delivery test, and 762 pass every brief test. This produces a strong and potentially contradictory claim: the evaluated agents are generally capable of producing technically valid videos that satisfy explicit requirements, but they are not reliably capable of producing edits that professional editors consider finished.

The distinction has direct implications for benchmark interpretation. A checklist-only evaluation would substantially overstate competence. The agents can render valid files, preserve required scene order, include specified dialogue or graphics, and avoid many forms of timeline corruption. Yet these capabilities do not imply adequate pacing, shot selection, sound design, visual finishing, or overall polish. The quality test is therefore not an optional aesthetic supplement; it determines whether the benchmark measures the intended professional deliverable rather than merely a syntactically valid video.

The quantitative editing differences support the human judgments. Relative to the reference edits, agents cut at 0.72 times the reference cut rate, use 0.76 times as many shots per edit, produce median shots 1.59 times longer, and hold their longest static shots 1.50 times longer. Human editors penalize slower pacing but do not show an analogous penalty for cutting faster than the reference. The paper estimates that an agent loses 6.6 percentage points of human win-or-tie rate for each halving of its cut rate below the reference rate, with a 95% confidence interval of 3.2–10.0 points.

Human comments identify overall polish, shot selection, graphics, transitions, story structure, sound design, and color as the principal strengths of the reference edits. In contrast, explicit technical defects are relatively infrequent among the strongest agents. For the six highest-rated agents, only 10% of rejected outputs contain a human-noted defect that a content test could plausibly detect, compared with 19% for the three weakest coding agents. The computer-use condition is a clear outlier: 28% of rejected edits contain such defects, including slates, crew dialogue, or retakes in 18.5% of losses.

The pattern of preferences varies by collection. Narrative quality drives 84–86% of tagged judgments in narrative collections, whereas graphics determine 91% of tagged UGC judgments. This result indicates that benchmark difficulty is partly collection-dependent and that a single aggregate score conceals different failure modes across production genres.

Process analysis and the missing audiovisual feedback loop

Trajectory analysis provides a mechanistic account of the quality gap. Coding agents devote 54% of their 111,636 native actions to source perception, 17% to verification, and only 9% to building. Eighty-three percent of source-perception reads involve frames or contact sheets. The first render occurs late, between 72% and 90% of the way through a run, and 98% of source-perception activity occurs before that first render. Agents therefore form plans from still images, transcripts, and measurements before observing the assembled audiovisual result.

This workflow is not merely inefficient; it limits the type of errors that agents can detect. Agents inspect their renders in 86–100% of runs for most coding conditions, and verification correlates with resolution across agents at $r = 0.93$1. However, only 5.5% of the problems they report after checking concern pacing, story, or shot choice, whereas 66% of human editors’ notes concern those dimensions. Agents predominantly verify form: file properties, repeated footage, synchronization, silence, and other mechanically observable defects. They rarely evaluate the emergent temporal and narrative properties that determine whether an edit communicates effectively.

The mismatch between self-assessment and actual performance is particularly severe. Agents claim full success in 93% of runs, including 95.5% of outputs that fail at least one benchmark test. This indicates that their verification routines do not provide calibrated uncertainty about either compliance or quality. The problem is not simply that agents lack an editing heuristic; they also lack a reliable evaluator capable of identifying when the constructed edit requires substantive revision.

The paper’s characterization is consequently precise: current agents are “clean assemblers” rather than finished editors. They are relatively reliable at delivery and explicit requirements, increasingly capable at basic narrative assembly, and weak at rhythm, shot selection, finishing, and self-judgment. The evidence supports a need for native audiovisual perception and iterative review of rendered sequences, but it does not establish which combination of continuous video encoders, audio-conditioned reasoning, editorial planning representations, or learned quality critics would solve the problem.

Task difficulty, model ability, and benchmark structure

A Rasch analysis shows that task difficulty varies more than agent ability. The estimated task standard deviation is 0.83 on the logit scale, compared with 0.51 for agents, yielding a variance ratio of 2.6. Thirty-five tasks are resolved by no agent, whereas only approximately five would be expected under random allocation of successes. Even the strongest agent is more likely to lose than to win or tie against the reference edit on 54 of the 56 tasks.

Collection membership explains approximately half of the variation in task difficulty. Human win-or-tie rates range from 28.7% for Cinestudy to 6.8% for UGC. Within collections, however, no individual task descriptor or required editing skill predicts difficulty after correction for multiple comparisons. Likewise, after task difficulty is removed, no agent demonstrates a reliable specialization for particular required skills; the joint permutation test yields $r = 0.93$2.

These findings constrain claims about model capability. The benchmark does not support a simple interpretation in which one model is consistently superior for, for example, interviews, graphics, or sound effects. Performance is strongly conditioned by the assignment itself, and the current sample does not identify stable agent-by-skill interactions. The benchmark is therefore better suited to measuring robust end-to-end reliability over heterogeneous tasks than to constructing a fine-grained skill profile for each model.

Limitations and open questions

The principal statistical limitation is that every agent runs only once per task. The reported confidence intervals quantify uncertainty across tasks but omit run-to-run variance caused by stochastic model behavior, tool-use trajectories, and rendering choices. Consequently, small differences in resolution rates should not be interpreted as stable model rankings.

The quality test has a further dependence on its calibration procedure. It is calibrated using the same pool of human judgments against which aggregate behavior is assessed, although the reported results use leave-one-agent-out margins to reduce circularity. The paper explicitly states that the test is validated per agent rather than per edit. In addition, professional reference edits are exemplars rather than certified ground truth; a reference may be less suitable than an alternative edit for a particular audience even when it serves as the benchmark comparison.

The human sample is experienced but heterogeneous: 43 freelance editors report an average of at least two years of experience, with most falling in the one-to-three-year band. Single judgments are noisy, and only aggregate rates are released to protect participants. The source media are also access-controlled, which supports licensing and privacy requirements but limits fully independent reproduction outside approved research teams. Finally, the benchmark samples 56 tasks from four collections, and collection effects are large. The absence of within-collection predictors should therefore be treated as a result about this task set, not as evidence that editorial skill requirements are generally unrelated to difficulty.

Several specific questions remain open. It is unclear whether agents would improve primarily through continuous audiovisual observation, earlier rendering and iterative revision, stronger editorial planning, specialized video-editing tools, or a calibrated critic trained on professional preferences. It is also unresolved whether repeated stochastic runs would materially increase resolution, whether agent ensembles could improve shot selection and pacing, and whether the current quality-test calibration would remain valid as agents produce edits substantially unlike the reference distribution.

Conclusion

TIMELINE-BENCH evaluates video editing as a complete professional workflow rather than as a collection of isolated media operations. Across 16 agents, the best condition resolves only 15 of 56 tasks, while the aggregate resolution rate is 14.0%. The dominant failure is not delivery or explicit brief compliance: 562 of 771 unresolved runs fail only the quality test. Human editors prefer the reference edits in 83.5% of judgments and identify deficiencies in polish, shot selection, pacing, graphics, sound, transitions, and finishing.

The benchmark consequently demonstrates a persistent separation between executable correctness and editorial quality. The evaluated agents can usually produce valid, requirement-compliant videos, but their perceptual and evaluative loops remain focused on detectable defects rather than narrative rhythm and viewing experience. By releasing the tasks, verifier, frozen evaluation constants, and per-run results, the paper provides an outcome-oriented test of whether future editing agents can close that specific gap.