---
title: 'Terminus-KIRA: LLM Terminal Evaluation'
url: https://www.emergentmind.com/topics/terminus-kira
type: topic
---

# Terminus-KIRA: LLM Terminal Evaluation

Terminus-KIRA is a minimal, extensible harness for evaluating large language model (LLM)-driven agents operating in a Unix-style terminal, with particular focus on integrating native image perception into terminal-based workflows. Designed to generalize the Harbor/Terminal-Bench paradigm for agent evaluation, Terminus-KIRA situates agents within a controlled, tool-mediated execution environment, enabling rigorous assessment of agentic planning, tool use, and artifact production across structured, code, text, and image-based tasks. The framework is a precursor to Terminus-MM, which extends these capabilities to audio and video modalities [2605.10966].

## 1. Architectural Principles and Execution Loop

Terminus-KIRA centers on a shell-driven evaluation harness built around three core elements: (i) a sandboxed Unix-like containerized workspace, (ii) a restricted and precisely defined tool schema, and (iii) a prompt template composed dynamically per task. Key architectural features include:

- **Task Packaging (Harbor Format):** Each evaluation task is an independent package with (a) a natural language user instruction, (b) a prepopulated sandboxed file directory, (c) a whitelist of allowed tools (typically core shell commands plus the `view_image` native primitive), (d) a specified output schema, and (e) an artifact-level task verifier.
- **Agent-Harness Interaction:** Operation proceeds via an agent-planner loop:
  a. The harness constructs a context-rich prompt fusing system instructions, tool signature descriptions, user instructions, and file listings.
  b. The LLM agent produces a tool call—either `execute_commands(["…"])`, `view_image("…")`, or `task_complete()`.
  c. The harness executes shell commands on behalf of the agent or, in the case of `view_image`, invokes a vision-encoder backend to generate a summary from image data.
  d. Tool outputs are fed back to the agent, iterating this process as the agent updates its plan.
  e. On `task_complete()`, the harness runs the artifact verifier to compute task-specific scores.

This kernel loop is highly general, compatible with a wide range of file-based workflow tasks and agent architectures.

## 2. Tool Schema and Perception Primitives

The harness enforces a constraining but extensible tool schema that isolates interactions to three atomic action types:

| Tool Name         | Functionality                   | Native/Commanded |
|-------------------|--------------------------------|------------------|
| execute_commands  | Arbitrary shell commands        | Commanded        |
| view_image        | Native image perception         | Native           |
| task_complete     | Signals end-of-task verification| Commanded        |

The **`view_image`** primitive constitutes Terminus-KIRA’s only native perception tool, allowing direct image summary extraction via vision-language model (VLM) backends. All perception beyond image inspection—such as audio or video understanding—must be accomplished through explicit shell toolchains (e.g., `ffmpeg` for frame extraction, `whisper` for ASR, `tesseract` for OCR, and Python scripts for signal analysis).

*This design tightly couples direct native support for visual evidence with deliberate exclusion of audio and video as first-class primitives, forcing agents to synthesize non-visual perception via shell-based pipelines.*

## 3. Data Processing Pipelines and Evaluation Metrics

**Perception Pathways:** For non-image media, agents must orchestrate multi-step shell workflows to convert audio/video artifacts into analyzable, text-based cues. Examples include:

- `ffmpeg -i clip.mp4 -r 1/[FRAME_RATE] frame%03d.png` for video frame sampling
- `whisper --model small clip.wav > transcript.txt` for speech-to-text (ASR)
- `tesseract frame001.png ocr.txt` for image-based OCR
- `python compute_rms.py clip.wav > rms.csv` for audio feature extraction

Native image perception through `view_image` leverages the model’s internal multimodal encoder to generate embeddings and textual summaries.

**Scoring Protocol:** Output artifacts are assessed by task verifiers using normalized scores $s_i \in [0, 1] = V_i(y_i; A)$. Two aggregate metrics are defined for agent $A$ over $N$ tasks:

- $$
  \text{BinarySuccess}(A) = \frac{1}{N} \sum_{i} \mathbb{1}[s_i \geq \tau_i]
  $$
- $$
  \text{PartialSuccess}(A) = \frac{1}{N} \sum_{i} s_i
  $$

where $\tau_i$ is the task-specific acceptance threshold.

Additionally, efficiency is quantified via mean API cost and mean wall-time per task.

## 4. System Integration and Module Interactions

The orchestrated execution environment comprises the following interacting modules:

- **Agent LLM:** Issues tool calls as dictated by the present workflow context.
- **Terminus-KIRA Harness:** Mediates tool calls, executes shell commands or native perception primitives, manages the filesystem, and aggregates outputs.
- **Terminal Container:** Provides a live bash environment with tool access and a persistent filesystem for the agent’s use.
- **Artifact Verifier:** Task-specific logic evaluates correctness and completeness of output artifacts.

The runtime information flow is:

```
[Agent LLM] 
    ↕ (tool call)
[Terminus-KIRA Harness] ←→ [Terminal container: bash + tools + filesystem]
    ↕ (tool output)
[Agent LLM]
```

Only `view_image` is implemented as a direct harness-to-model call; all other analysis must be executed via `execute_commands`, reinforcing precise modular separation.

## 5. Empirical Performance and Limitations

In comparative studies on the MMTB (MultiMedia-TerminalBench) suite, Terminus-KIRA demonstrates the incremental value of adding native perception beyond text. For the Gemini-3.1-Pro backbone, Terminus-KIRA (supporting text and image, denoted T+I) achieves 10.5% binary and 15.9% partial success on multimedia workflow tasks. These results are significantly improved by incorporating subsequent modalities: adding audio (Terminus-IA: T+I+A) results in 33.3% binary/40.6% partial, and including video (Terminus-IV: T+I+V) attains 33.3%/43.2%, while the full multimedia baseline (Terminus-MM: T+I+A+V) obtains 37.1%/46.9% [2605.10966].

Terminus-KIRA’s architecture, with only image-native perception, leads to substantial redundancy and inefficiency when operating over audio/video files: agents expend up to 1.6–7.7 times higher API costs reconstructing missing modalities through shell toolchains relative to agents endowed with full native access. Failure analysis reveals that native-media agents (as in Terminus-MM) experience fewer tool-setup or execution timeouts (24% of failures), compared to terminal-only agents (47%), though both categories retain a material fraction of reasoning errors (47%).

## 6. Role in the Terminal Agent Ecosystem and Extensions

Terminus-KIRA constitutes the critical bridge between purely text/code-based terminal-agent evaluation and full multimodal (audio, visual, video) workflow automation. Its minimal but extensible harness formed the basis for developing Terminus-MM, which extends the modality tool schema with `listen_audio` and `watch_video` primitives. Terminus-MM also introduces a workspace-aware routing policy that activates only those perception tools relevant to the present file modalities, thus avoiding extraneous tool exposure and reducing agent distraction.

*The contribution of Terminus-KIRA is twofold: establishing a shell-execution harness with native image inspection and providing a control for assessing the empirical benefit conferred by additional modality-native tools in agentic workflows.* Its use in MMTB and related benchmarks underscores the ongoing transition from text-centric to multimedia-capable agentic evaluation frameworks [2605.10966].

Source: https://www.emergentmind.com/topics/terminus-kira