MMTB: Evaluating Terminal Agents on Multimedia-File Tasks

Abstract: Terminals provide a powerful interface for AI agents by exposing diverse tools for automating complex workflows, yet existing terminal-agent benchmarks largely focus on tasks grounded in text, code, and structured files. However, many real-world workflows require practitioners to work directly with audio and video files. Working with such multimedia files calls for terminal agents not only to understand multimedia content, but also to convert auditory and visual evidence across related files into appropriate actions. To evaluate terminal agents on multimedia-file tasks, we introduce MultiMedia-TerminalBench (MMTB), a benchmark of 105 tasks across 5 meta-categories where terminal agents directly operate with audio and video files. Alongside MMTB, we propose Terminus-MM, a multimedia harness that extends Terminus-KIRA with audio and video perception for terminal agents. Together, MMTB and Terminus-MM support a controlled study of multimedia terminal agents, revealing how different forms of multimedia access shape task outcomes and determine which evidence agents rely on to construct executable terminal workflows. MMTB media and metadata are released at https://huggingface.co/datasets/mm-tbench/mmtb-media

• The paper introduces MMTB, a benchmark of 105 multimedia-file tasks to evaluate terminal agents using native audio-visual tools, demonstrating significant performance gains.
• It employs a suite of harnesses like Terminus-MM that dynamically route modality-specific tools to efficiently process audio, video, image, and text inputs.
• Results reveal that native audio-visual perception boosts success rates (e.g., Gemini-3.1-Pro improved from 12.4% to 37.1%), highlighting critical advantages over command-line proxies.

Evaluating Terminal Agents on Multimedia-File Tasks: An Expert Analysis of "MMTB: Evaluating Terminal Agents on Multimedia-File Tasks" (2605.10966)

Introduction

"MMTB: Evaluating Terminal Agents on Multimedia-File Tasks" addresses the notable shortfall in current terminal-agent benchmarks, namely the lack of systematic evaluation of agents' abilities to perform media-grounded workflows directly on audio and video files. Existing benchmarks like Terminal-Bench focus on text, code, and structured data, while AV question-answering benchmarks reduce multimodal operation to text-answering tasks, omitting full pipeline execution and artifact evaluation as required in realistic workflows. This work introduces MultiMedia-TerminalBench (MMTB), a benchmark with 105 rigorously curated multimedia-file tasks, accompanied by Terminus-MM—a terminal-agent harness that enables native audio and video perception in a persistent command-line environment.

Benchmark Design and Construction

MMTB is designed to evaluate agents in realistic multimedia processing and editing scenarios typically encountered by practitioners on platforms like Upwork or Fiverr. Each task comprises a persistent workspace of audio, image, and video assets, and specifies artifact-based evaluation—demanding the agent transform, extract, or align multimedia content using terminal commands to produce verifiable deliverables (e.g., structured records, edited media, or temporal alignments).

The benchmark construction pipeline begins with 163 candidate real-world multimedia workflows. Iterative automated validation, baseline reviews, and manual pruning yield a final suite of 105 tasks, spanning five meta-categories and 16 fine-grained workflow types, with empirical distributions annotated by perception and reasoning capability requirements.

Figure 1: MMTB construction pipeline (a), meta-category and fine-grained task-type statistics (b), and multi-label distribution of perception/reasoning capabilities (c).

Tasks employ the Harbor format to ensure agent-agnosticism and reproducibility, packaging each scenario as a self-contained directory with assets, instructions, tool interface constraints, required outputs, and deterministic evaluation logic.

Harnesses, Modalities, and Models

MMTB's evaluation protocol leverages a suite of harnesses derived from the Terminus family, each differing in the set of accessible native perception tools: from text-only (Terminus-2), to text+image (Terminus-KIRA), to combinations incorporating native audio (Terminus-A), video (Terminus-V), and the full T+I+A+V setting (Terminus-MM). Notably, Terminus-MM employs dynamic schema routing, exposing native perception tools only for modalities present in the workspace, thus minimizing unnecessary checks and tool chatter.

Evaluated agent configurations use prominent VLMs and omni-modal LLMs (Qwen3.5-122B, GPT-5.2, Gemini-2.5-Flash, Gemini-3.1-Pro), and include strong installable terminal-agent baselines: Codex CLI (OpenAI) and Claude Code (Anthropic), which offer their own tool interfaces and perception mediation.

Native Multimedia Access vs. Terminal Proxies

A core empirical finding is the pronounced improvement in success and efficiency when terminal agents are augmented with native audio and video perception. On Gemini-3.1-Pro, binary success rates increase from 12.4% (text-only) to 37.1% (full modality access), with partial success up to 46.9%. These gains are robust across backbones, and the gap over pure text or image-only access reflects the prevalence of tasks where perceptually grounded reasoning about sound events, timing, or AV alignment is irreplaceable.

Figure 2: Example MMTB task workflow—agents with native multimodal perception directly interpret raw files; text-only agents must reconstruct evidence via terminal tools, introducing inefficiency and loss.

Analyses of command-line proxy strategies (e.g., using ffmpeg, ASR, frame sampling) reveal substantial API and cost overheads—ratios from 1.6× to 7.7× compared to native access, with pathologies up to 30×–40×. While command-line conversions are sometimes successful, they systematically introduce additional failure points in tool setup, execution, and lossy intermediate representation processing, making them brittle and resource-inefficient in artifact-level benchmarks.

Agent Comparison, Harness Design, and Failure Signatures

MMTB establishes that neither native modality access nor terminal command execution in isolation is sufficient: the two solve overlapping but distinct task regimes, with 28 tasks only solved by Terminus-MM, 6 only by Codex CLI, and 11 by both.

Figure 3: Overlap of solved tasks between Terminus-MM and Codex CLI reveals significant non-nested regimes.

The exclusive success regions are highly diagnostic: Codex CLI-only successes hinge on workflows where command-line pipelines can deterministically transform media into tractable intermediate signals (e.g., OCR, audio energy, transcript-based alignment); Terminus-MM-only tasks require direct perceptual access, such as audio-visual event matching or per-interval artifact labeling.

Harness design further matters: exposing an unfiltered set of perception tools in full-modality agents induces redundant (and often costly) behavior, reducing performance—modality masking tailored to actual workspace content provides measurable benefit.

In-depth failure analysis shows that terminal-only agents predominantly fail on tool-setup, execution, or lossy proxy grounds, while full-modality agents' failures concentrate in evidence selection and reasoning steps, indicating the remaining bottleneck is robust cross-modal perceptual reasoning rather than tool manipulation.

Task Difficulty, Modality Dependency, and Domain Analysis

MMTB tasks are distributionally diverse in file counts, media durations, and workflow intent. Task-wise success inversely correlates (moderately) with media duration, reinforcing the importance of efficient perceptual pipelines for longer or more complex artifacts.

Performance gains from native modality access are not uniform across meta-categories: media production and compliance tasks benefit most from joint audio-visual grounding, whereas some personal or educational workflows are tractable via partial modality access or command-line proxies.

Figure 4: Domain-level modality dependency—binary success rates across meta-categories and harnesses highlighting concentrated modality gaps in media production and compliance workflows.

Fine-grained capability tag analyses reinforce these findings: tasks requiring audio-visual alignment, temporal localization, or music understanding see the largest differential in success rate between baseline and modem harnesses with native AV access.

Implications and Future Directions

MMTB operationalizes a rigorous, artifact-level benchmark for terminal agents in realistic multimedia scenarios, serving both as a stress-test for current agent architectures and an actionable blueprint for future modeling work. Results highlight several crucial points for future research:

• Agent architectures must jointly optimize for native multimedia access and robust terminal artifact construction. Tool-use can mitigate limited perception, but the cost and brittleness ceiling is explicit.
• Interface-level tool schema routing (modality masking) is a non-trivial design lever in interactive agent systems. Thoughtful tool exposure significantly boosts efficiency and success rates.
• Closing the task frontier requires advances in cross-modal evidence selection, stopping, and artifact formatting precision, beyond scaling current VLMs or improving proxy conversion pipelines.
• Benchmark hygiene should treat proxy workflows as regime indicators rather than shortcuts—command-line detours are evidence about agent-agent divergence, not contamination to suppress.

Practically, progress on MMTB is consequential for users automating media production, compliance, or annotation workflows using terminal agents—an increasingly common scenario with the proliferation of model-driven CLIs in both open and closed ecosystems.

Conclusion

"MMTB: Evaluating Terminal Agents on Multimedia-File Tasks" provides a validated, domain-grounded benchmark critical for diagnosing, quantifying, and advancing the state-of-the-art in agentic multimedia processing within persistent command-line environments. The deployment of workspace-aware multimodal harnesses like Terminus-MM demonstrates that full-stack, artifact-oriented evaluation is essential to measure true multimedia agent competency, setting a rigorous agenda for future work at the intersection of perception, agency, and reliable tool execution (2605.10966).