Terminus-KIRA: LLM Terminal Evaluation

• Terminus-KIRA is a minimal, extensible framework for evaluating LLM-driven agents in Unix-style environments by integrating native image perception into terminal workflows.
• It employs a sandboxed containerized workspace with a strict tool schema—supporting execute_commands, view_image, and task_complete—to ensure secure and controlled agent interactions.
• The system benchmarks agent performance on multimedia tasks using precise metrics, paving the way for multimodal extensions as seen in Terminus-MM.

Terminus-KIRA is a minimal, extensible harness for evaluating LLM-driven agents operating in a Unix-style terminal, with particular focus on integrating native image perception into terminal-based workflows. Designed to generalize the Harbor/Terminal-Bench paradigm for agent evaluation, Terminus-KIRA situates agents within a controlled, tool-mediated execution environment, enabling rigorous assessment of agentic planning, tool use, and artifact production across structured, code, text, and image-based tasks. The framework is a precursor to Terminus-MM, which extends these capabilities to audio and video modalities (Heo et al., 8 May 2026).

1. Architectural Principles and Execution Loop

Terminus-KIRA centers on a shell-driven evaluation harness built around three core elements: (i) a sandboxed Unix-like containerized workspace, (ii) a restricted and precisely defined tool schema, and (iii) a prompt template composed dynamically per task. Key architectural features include:

• Task Packaging (Harbor Format): Each evaluation task is an independent package with (a) a natural language user instruction, (b) a prepopulated sandboxed file directory, (c) a whitelist of allowed tools (typically core shell commands plus the view_image native primitive), (d) a specified output schema, and (e) an artifact-level task verifier.
• Agent-Harness Interaction: Operation proceeds via an agent-planner loop: a. The harness constructs a context-rich prompt fusing system instructions, tool signature descriptions, user instructions, and file listings. b. The LLM agent produces a tool call—either execute_commands(["…"]), view_image("…"), or task_complete(). c. The harness executes shell commands on behalf of the agent or, in the case of view_image, invokes a vision-encoder backend to generate a summary from image data. d. Tool outputs are fed back to the agent, iterating this process as the agent updates its plan. e. On task_complete(), the harness runs the artifact verifier to compute task-specific scores.

This kernel loop is highly general, compatible with a wide range of file-based workflow tasks and agent architectures.

2. Tool Schema and Perception Primitives

The harness enforces a constraining but extensible tool schema that isolates interactions to three atomic action types:

Tool Name	Functionality	Native/Commanded
execute_commands	Arbitrary shell commands	Commanded
view_image	Native image perception	Native
task_complete	Signals end-of-task verification	Commanded

The view_image primitive constitutes Terminus-KIRA’s only native perception tool, allowing direct image summary extraction via vision-LLM (VLM) backends. All perception beyond image inspection—such as audio or video understanding—must be accomplished through explicit shell toolchains (e.g., ffmpeg for frame extraction, whisper for ASR, tesseract for OCR, and Python scripts for signal analysis).

This design tightly couples direct native support for visual evidence with deliberate exclusion of audio and video as first-class primitives, forcing agents to synthesize non-visual perception via shell-based pipelines.

3. Data Processing Pipelines and Evaluation Metrics

Perception Pathways: For non-image media, agents must orchestrate multi-step shell workflows to convert audio/video artifacts into analyzable, text-based cues. Examples include:

• ffmpeg -i clip.mp4 -r 1/[FRAME_RATE] frame%03d.png for video frame sampling
• whisper --model small clip.wav > transcript.txt for speech-to-text (ASR)
• tesseract frame001.png ocr.txt for image-based OCR
• python compute_rms.py clip.wav > rms.csv for audio feature extraction

Native image perception through view_image leverages the model’s internal multimodal encoder to generate embeddings and textual summaries.

Scoring Protocol: Output artifacts are assessed by task verifiers using normalized scores $s_i \in [0, 1] = V_i(y_i; A)$. Two aggregate metrics are defined for agent $A$ over $N$ tasks:

• $$\text{BinarySuccess}(A) = \frac{1}{N} \sum_{i} \mathbb{1}[s_i \geq \tau_i]$$
• $$\text{PartialSuccess}(A) = \frac{1}{N} \sum_{i} s_i$$

where $\tau_i$ is the task-specific acceptance threshold.

Additionally, efficiency is quantified via mean API cost and mean wall-time per task.

4. System Integration and Module Interactions

The orchestrated execution environment comprises the following interacting modules:

• Agent LLM: Issues tool calls as dictated by the present workflow context.
• Terminus-KIRA Harness: Mediates tool calls, executes shell commands or native perception primitives, manages the filesystem, and aggregates outputs.
• Terminal Container: Provides a live bash environment with tool access and a persistent filesystem for the agent’s use.
• Artifact Verifier: Task-specific logic evaluates correctness and completeness of output artifacts.

The runtime information flow is:

[Agent LLM] ↕ (tool call) [Terminus-KIRA Harness] ←→ [Terminal container: bash + tools + filesystem] ↕ (tool output) [Agent LLM]

Only view_image is implemented as a direct harness-to-model call; all other analysis must be executed via execute_commands, reinforcing precise modular separation.

5. Empirical Performance and Limitations

In comparative studies on the MMTB (MultiMedia-TerminalBench) suite, Terminus-KIRA demonstrates the incremental value of adding native perception beyond text. For the Gemini-3.1-Pro backbone, Terminus-KIRA (supporting text and image, denoted T+I) achieves 10.5% binary and 15.9% partial success on multimedia workflow tasks. These results are significantly improved by incorporating subsequent modalities: adding audio (Terminus-IA: T+I+A) results in 33.3% binary/40.6% partial, and including video (Terminus-IV: T+I+V) attains 33.3%/43.2%, while the full multimedia baseline (Terminus-MM: T+I+A+V) obtains 37.1%/46.9% (Heo et al., 8 May 2026).

Terminus-KIRA’s architecture, with only image-native perception, leads to substantial redundancy and inefficiency when operating over audio/video files: agents expend up to 1.6–7.7 times higher API costs reconstructing missing modalities through shell toolchains relative to agents endowed with full native access. Failure analysis reveals that native-media agents (as in Terminus-MM) experience fewer tool-setup or execution timeouts (24% of failures), compared to terminal-only agents (47%), though both categories retain a material fraction of reasoning errors (47%).

6. Role in the Terminal Agent Ecosystem and Extensions

Terminus-KIRA constitutes the critical bridge between purely text/code-based terminal-agent evaluation and full multimodal (audio, visual, video) workflow automation. Its minimal but extensible harness formed the basis for developing Terminus-MM, which extends the modality tool schema with listen_audio and watch_video primitives. Terminus-MM also introduces a workspace-aware routing policy that activates only those perception tools relevant to the present file modalities, thus avoiding extraneous tool exposure and reducing agent distraction.

The contribution of Terminus-KIRA is twofold: establishing a shell-execution harness with native image inspection and providing a control for assessing the empirical benefit conferred by additional modality-native tools in agentic workflows. Its use in MMTB and related benchmarks underscores the ongoing transition from text-centric to multimedia-capable agentic evaluation frameworks (Heo et al., 8 May 2026).