---
title: 'Terminus-MM: Multimedia Terminal Agent Harness'
url: https://www.emergentmind.com/topics/terminus-mm
type: topic
---

# Terminus-MM: Multimedia Terminal Agent Harness

Searching arXiv for the cited paper to ground the article.
Terminus-MM is a multimedia harness for terminal agents introduced alongside MultiMedia-TerminalBench (MMTB) in “MMTB: Evaluating Terminal Agents on Multimedia-File Tasks” [2605.10966]. It is defined as a drop-in extension of Terminus-KIRA, itself an extension of Terminus-2, that adds native audio and video “tools” on top of the existing terminal interface. In the reported formulation, Terminus-MM is designed to let a terminal agent directly operate over workspaces containing audio and video files, rather than relying exclusively on text, code, structured files, or proxy command-line extraction pipelines. Its role is both architectural and evaluative: it supplies multimedia perception primitives to the agent loop and serves as the harness used to study how modality access affects success on multimedia-file tasks.

## 1. Lineage and system definition

Terminus-MM is situated in a sequence of terminal-agent harnesses. Terminus-2 provides a CLI loop that issues Bash commands through `execute_commands`, observes `stdout/stderr`, and uses `task_complete` to signal the end of a task. Terminus-KIRA extends this setup with `view_image(path)`, which returns a caption or OCR/text dump of an image. Terminus-MM further adds `listen_audio(path[, start, end])`, which returns a transcript, speaker labels, or audio features, and `watch_video(path[, start, end])`, which returns a summary of a clip, motion cues, on-screen text, and related information [2605.10966].

The paper characterizes these additions as native audio and video “tools” layered on top of the existing terminal interface. This makes Terminus-MM a multimodal terminal-agent harness rather than a general multimedia application framework. Its operational emphasis is on enabling an LLM-driven agent to transform auditory and visual evidence across related files into executable terminal actions.

A central design feature is workspace awareness. At startup, Terminus-MM scans the workspace and dynamically routes only those tools whose modality is present. This means that image, audio, and video tools are exposed conditionally, rather than unconditionally. The paper presents this as a mechanism for controlling the action space seen by the model during inference.

## 2. Tool routing and agent loop

The paper gives two explicit algorithms for Terminus-MM: workspace-aware perception-tool routing and the main agent loop [2605.10966]. In the routing procedure, all files under `workspace_dir` are enumerated, file extensions are mapped to modality indicators, and a retained tool set is constructed from `execute_commands`, `task_complete`, and any relevant perception tools. If multimedia files are absent, the corresponding tools are not exposed.

This routing logic has several direct consequences. First, it ties the available API surface to the task workspace rather than to a static global configuration. Second, it removes irrelevant perception calls from the model’s tool schema. The reported ablation shows that this matters empirically: “Terminus-MM w/o masking” attains `32.4 %` binary and `42.6 %` partial success, whereas routed Terminus-MM attains `37.1 %` binary and `46.9 %` partial success. The paper attributes this to the prevention of wasted verification steps, such as audio calls on video-only tasks, and describes the gain as recovering `4–5 pp` of success.

The agent loop follows the standard pattern used by terminal-agent systems, but with multimedia tool calls included in the dispatch. A prompt is built from a system preamble, tool descriptions induced by the routed schema, prior interaction history, and the task instruction. The LLM then emits either a shell command, an image call, an audio call, a video call, or `task_complete`. Tool outputs are appended to history until termination. In the ASCII dataflow diagram, this process connects a Harbor task driver, a Terminus-MM agent, and the final workspace containing produced files.

A plausible implication is that Terminus-MM’s main novelty lies less in altering the high-level control loop than in modifying the observational interface available within that loop. The task-solving regime remains tool-mediated and iterative; what changes is the granularity and fidelity of perceptual access to multimedia files.

## 3. Audio and video perception stack

Terminus-MM relies on a single multimodal LLM backbone, exemplified in the paper by Google Gemini 3.1 Pro, for all three perception channels: image, audio, and video. The perception tools are implemented as API calls to this backbone rather than as purely local standalone decoders. The design therefore centralizes multimodal interpretation within one model family [2605.10966].

For audio, the paper specifies the internal sequence triggered by `listen_audio(file.wav,start,end)`. The harness reads the waveform $x[n]$ from `file.wav`, computes a short-time Fourier transform spectrogram,
$$
X(\tau, \omega) = \Sigma_n x[n]\cdot w[n-\tau]\cdot e^{-j\omega n},
$$
where $w$ is the analysis window, and feeds log-magnitude $|X(\tau,\omega)|$ frames, or a learned feature sequence, to the LLM’s audio encoder. The LLM then returns a transcript, speaker tags, or feature summaries.

For video, `watch_video(file.mp4,start,end)` samples frames $\{I_t\}$ at a fixed frame rate, given in the paper as examples such as `1 fps` or `2 fps`. Each frame is processed by a vision encoder, exemplified as a CNN or ViT patch embedding $\phi(I_t)`. An OCR layer may optionally extract on-screen text over $\phi(I_t)`. The resulting frame embeddings, together with an optional audio embedding for the interval, are concatenated and passed through a Transformer to produce a descriptive summary.

The paper states that both audio and video inputs are linear sequences, allowing the LLM’s cross-modal transformer to handle integration and alignment. This suggests that Terminus-MM is architecturally aligned with unified multimodal sequence modeling rather than with a collection of narrowly specialized perception modules. At the same time, the harness abstraction presented in the paper exposes only high-level tool outputs—transcripts, summaries, OCR, motion cues—rather than raw embeddings to the agent.

## 4. Role within MMTB

Terminus-MM is introduced together with MMTB, a benchmark of `105` self-contained Harbor tasks across `5 meta-categories` in which terminal agents directly operate with audio and video files [2605.10966]. Each task comprises an instruction, a workspace of multimedia files, and an evaluator $V_i$ that checks the final workspace state $y_i$ and returns $s_i \in [0,1]$.

The benchmark defines three reported metrics. Partial success is
$$
(1/N)\Sigma_i s_i.
$$
Binary success is
$$
(1/N)\Sigma_i 1[s_i \ge \tau_i],
$$
where $\tau_i$ is a task-specific threshold. The paper also reports API cost per task in USD and mean wall time per task in seconds.

In this evaluation setting, Terminus-MM is not merely an implementation detail; it is the mechanism by which full multimedia access is operationalized. The paper’s comparative setup distinguishes access regimes such as `T`, `T+I`, `T+I+A`, `T+I+V`, and `T+I+A+V`, where `T` denotes terminal access, `I` image, `A` audio, and `V` video. Terminus-MM corresponds to the full `T+I+A+V` regime.

A common misconception in terminal-agent evaluation is that image access substantially closes the gap between text-only operation and real-world multimedia workflows. The MMTB results argue against that interpretation: in the reported setup, text-only and text-plus-image agents remain below `16 %` success, which the paper presents as evidence that many tasks require audio or video grounding.

## 5. Empirical performance and cost profile

Using a Gemini-3.1-Pro backbone, the paper reports the following comparison across harnesses [2605.10966]:

| Harness | Access | Binary↑ / Partial↑ |
|---|---|---|
| Terminus-2 | T | `0.124 / 0.162` |
| Terminus-KIRA | T+I | `0.105 / 0.159` |
| Terminus-IA | T+I+A | `0.333 / 0.406` |
| Terminus-IV | T+I+V | `0.333 / 0.432` |
| Terminus-MM | T+I+A+V | `0.371 / 0.469` |

The corresponding cost and time figures are also reported. Terminus-2 has `0.772` USD cost and `538s` mean wall time; Terminus-KIRA `2.061` USD and `544s`; Terminus-IA `1.742` USD and `460s`; Terminus-IV `1.283` USD and `434s`; Terminus-MM `1.228` USD and `442s`. The paper summarizes these results with three observations: adding native audio or video yields large jumps over `T+I`; full `T+I+A+V` yields the best overall binary success (`37.1 %`) and partial success (`46.9 %`); and text-only or text-plus-image agents stall below `16 %` success.

The paper also quantifies proxy-conversion overhead when agents lack a modality and must invoke noisy CLI tools such as `ffmpeg` frames, ASR pipelines, or OCR. On tasks where both a partial agent and Terminus-MM succeed, the average API-cost ratio `(partial / MM)` ranges from `1.6×` to `7.7×`, with worst cases up to `42×`. The stated interpretation is that repeated transcript, frame, or feature extraction is substantially more expensive than a single native LLM perception tool call.

This result is significant because the improvement is not presented solely as a matter of absolute accuracy. It also concerns the computational and economic consequences of substituting native multimodal access for lossy proxy pipelines. A plausible implication is that multimodal terminal-agent evaluation should treat modality access as an efficiency variable as well as a capability variable.

## 6. Ablations, solver regimes, and failure structure

The paper includes several analyses intended to isolate what Terminus-MM contributes beyond generic terminal competence [2605.10966]. In the “modality ladder” ablation with Gemini-3.1, the reported binary success rates are `12.4 %` for `T` alone, `10.5 %` for `T+I`, `33.3 %` for `T+I+A`, `33.3 %` for `T+I+V`, and `37.1 %` for `T+I+A+V`. The lack of gain from `T+I` relative to `T` is one of the clearest findings in the paper, while the near parity of audio-only and video-only augmentations indicates that both modalities independently unlock substantial subsets of tasks.

The solver-regime Venn analysis compares Terminus-MM with Codex CLI. It reports `11` tasks solved by both systems, `28` tasks solved only by Terminus-MM, `6` tasks solved only by Codex CLI, and `60` tasks solved by neither. The paper labels the `28` Terminus-MM-only tasks as “grounding-limited” and the `6` Codex-only tasks as “pipeline-limited: simple OCR/ASR suffices.” This framing distinguishes failures caused by absent or lossy multimedia grounding from failures caused by deficiencies in reasoning, tool use, or artifact construction.

Failure signatures sharpen that distinction. Terminus-MM failures concentrate in model reasoning (`47 %`) and wrong or missed evidence (`23 %`). Codex CLI failures concentrate in tool timeouts, tool failures, and lossy intermediate extraction (`47 %`). The paper concludes from these analyses that a multimedia terminal agent needs both native audio and video grounding to avoid lossy CLI proxies and robust tool-use and artifact-construction abilities to avoid execution errors.

These findings also delimit the scope of Terminus-MM’s contribution. The harness addresses one bottleneck—multimedia grounding—more directly than another—general-purpose reasoning and reliable execution. The residual `60` unsolved tasks indicate substantial remaining headroom even when full modality access is available.

## 7. Interpretation and research significance

Within the framework of MMTB, Terminus-MM functions as an experimental instrument for studying how different forms of multimedia access shape task outcomes and determine which evidence agents rely on to construct executable terminal workflows [2605.10966]. Its importance therefore lies in both system design and measurement: it concretizes a multimodal terminal-agent interface and exposes the performance consequences of granting or withholding specific modalities.

Several broader implications follow from the reported results. First, the benchmark suggests that multimedia-file tasks are not well approximated by text-only or text-plus-image terminal benchmarks. Second, the routing mechanism shows that modality exposure itself is a controllable variable affecting success, not merely a passive feature of the environment. Third, the cost analysis indicates that native multimedia perception changes the economics of agent execution, especially when compared with repeated proxy extraction through external CLI tools.

At the same time, the empirical profile argues against an overly narrow reading of Terminus-MM as simply “an ASR and video-summary wrapper.” The failure analysis shows that even with native multimedia access, reasoning errors and missed evidence remain major failure modes. This suggests that the main challenge in multimedia terminal agents is not only perception fidelity, but also the coupling of perception with action selection, verification, and artifact construction in long-horizon terminal workflows.

In this sense, Terminus-MM marks a shift in terminal-agent evaluation from predominantly textual interaction toward multimedia-grounded execution. The paper presents it as the harness that makes such evaluation operational, while MMTB provides the task distribution on which its consequences can be measured.

Source: https://www.emergentmind.com/topics/terminus-mm