---
title: Kamiwaza Agentic Merit Index (KAMI) Benchmark
url: https://www.emergentmind.com/topics/kamiwaza-agentic-merit-index-kami
type: topic
---

# Kamiwaza Agentic Merit Index (KAMI) Benchmark

The Kamiwaza Agentic Merit Index (KAMI) is an interactive, multi-step benchmark for evaluating the real-world agentic capabilities of large language models (LLMs) equipped with tool-use APIs. Unlike traditional static question answering or code generation benchmarks, KAMI assesses how reliably an LLM can ground its reasoning within its environment, sequence tool calls, recover from errors, and adapt under uncertainty. The benchmark is structured around filesystem, text extraction, CSV analysis, and SQL querying tasks, each instantiated as multiple randomized scenarios. Scoring is based on pooled accuracy across a large number of independent trials, yielding both aggregate measures and detailed behavioral traces for qualitative analysis. KAMI v0.1 aims to provide a rigorous foundation for analyzing interactive, environment-aware agentic behavior in LLM agents [2512.07497].

## 1. Formal Definition and Purpose

The KAMI benchmark is defined as a collection of interactive, multi-step scenarios designed to probe and quantify agentic robustness in LLMs that interface with tool-use APIs. Its evaluation targets include:

- Grounding: Does the model anchor its reasoning in live observations and environmental data?
- Sequencing: Does the model orchestrate the correct ordering of tool invocations?
- Recovery: Can the agent identify and correct errors within the allowed inference rounds?
- Adaptation: Is the model robust under varying uncertainty, environmental distractors, or workload?

KAMI distinguishes itself from static benchmarks by requiring actual state changes in an environment and by scoring based on the exactness of tool-mediated artifacts produced by the agent in each trial [2512.07497].

## 2. Mathematical Formulation

KAMI’s score is mathematically formulated as the trial-weighted pooled accuracy across all benchmark scenarios. Let $S = \{1, \dots, M\}$ be the set of agentic scenarios ($M=19$ in KAMI v0.1), $T_s$ the number of randomized trials per scenario ($T_s=240$), and $X_{s,t} \in \{0,1\}$ a per-trial success indicator (1 for correct, scoring-type-verified output; 0 otherwise). Then:

- Scenario-level accuracy: $A_s = \frac{1}{T_s} \sum_{t=1}^{T_s} X_{s,t}$
- Scenario weight: $w_s = \frac{T_s}{\sum_{k=1}^M T_k}$ (uniform in v0.1)
- Pooled KAMI score:
  \[
  \mathrm{KAMI} = \sum_{s=1}^M w_s A_s = \frac{1}{M}\sum_{s=1}^{M} A_s = \frac{\sum_{s=1}^M\sum_{t=1}^{T_s}X_{s,t}}{\sum_{s=1}^M T_s}
  \]

With fixed $T_s$ and $w_s$, each scenario contributes equally, and every trial is normalized to prevent overfitting to particular tasks or scenarios [2512.07497].

## 3. Benchmark Structure and Scenario Taxonomy

KAMI v0.1 comprises four principal task categories, each mapped to specific scenario ID series and defined operationally as follows:

| Category           | Scenario IDs   | Description                                                            |
|--------------------|---------------|------------------------------------------------------------------------|
| Filesystem         | Q201, Q202    | Create directories/files under implicit constraints                     |
| Text Extraction    | Q301, Q302    | Retrieve specific lines from large files using line-indexed reads       |
| CSV Analysis       | Q401, Q402, Q403| Compute counts, sums, averages over CSVs, requiring code execution     |
| SQL Querying       | Q501, Q502, Q503| Schema discovery, joins, filters, aggregation with distractors         |

Each scenario is instantiated as $T_s=240$ independent, randomized trials in the full benchmark. The qualitative study manually reviews a 12.5% sample—$30$ trials per scenario across selected models—surfacing both aggregate accuracy and fine-grained behavioral patterns. Non-core axes such as sanity checks (100-series), prompt engineering (600-series), and output format (700-series) are excluded from the scored set in the referenced work [2512.07497].

## 4. Scoring, Weighting, and Normalization

The aggregation of trial-level outcomes adheres to strict normalization:

- All scenarios employ identical trial counts and equal per-scenario weights ($w_s = 1/19$).
- KAMI’s trial-weighted pooling precludes selective overfitting; even the hardest scenarios contribute exactly $1/19$ to the final metric.
- Each trial’s evaluation is binary: $X_{s,t}=1$ only if the final artifact (file, JSON, etc.) exactly matches the scenario’s canonical solution; otherwise $0$.
- The scoring mechanism accommodates multi-step error recovery. A model is not penalized for initial mistakes if successful correction occurs within the maximum of $20$ inference rounds allotted per trial.

These criteria collectively ensure that KAMI reflects aggregate, scenario-balanced agentic reliability rather than idiosyncratic successes or degenerate strategies.

## 5. Example Evaluation and Interpretation

Analysis in [2512.07497] reports performance of DeepSeek V3.1 over ten benchmarked scenarios, each with 30 manually traced trials. The corresponding accuracies are tabulated as follows:

| Scenario | Correct Trials | Accuracy |
|----------|---------------|----------|
| Q201     | 30/30         | 1.000    |
| Q202     | 30/30         | 1.000    |
| Q301     | 27/30         | 0.900    |
| Q302     | 30/30         | 1.000    |
| Q401     | 30/30         | 1.000    |
| Q402     | 20/30         | 0.667    |
| Q403     | 30/30         | 1.000    |
| Q501     | 30/30         | 1.000    |
| Q502     | 14/30         | 0.467    |
| Q503     | 20/30         | 0.667    |

The subset KAMI score (pooled over 10 scenarios) is calculated as 
\[
\mathrm{KAMI}_{\text{subset}} = \frac{1.000+1.000+0.900+1.000+1.000+0.667+1.000+1.000+0.467+0.667}{10} \approx 0.870
\]
In the full benchmark (19 scenarios × 240 trials), the identical aggregation yields the reported “Pooled Accuracy” figure (e.g., DeepSeek V3.1 at 92.2%) [2512.07497].

## 6. Experimental Protocols and Design Assumptions

KAMI v0.1 enforces several design decisions to maintain interpretative clarity and reproducibility:

- A single-tool-per-round constraint, unadvertised in prompts, enforces stepwise agentic decomposition of tasks.
- Temperature is fixed at $0.4$ to balance deterministic execution and needed exploration.
- Each trial is capped at 20 inference rounds to avoid infinite loops or degenerate long-horizon behaviors.
- Core scenarios uniformly exercise multi-layered agent capabilities; auxiliary or meta scenarios are excluded from scoring and qualitative error analysis.
- The experimental sample in [2512.07497] comprises three models (Granite 4 Small, Llama 4 Maverick, DeepSeek V3.1) across a subset of scenarios and trials. This leaves comprehensive model coverage and tool-augmented qualitative assessment as future work.

A plausible implication is that the current version deliberately abstracts away from variance in prompt engineering or output format effects to focus on core agentic behaviors.

## 7. Significance and Comparison to Traditional Metrics

KAMI provides several conceptual and practical advantages over aggregate leaderboard metrics common in QA or code-completion evaluation:

- Fine-grained, per-trial interactive analysis: Each execution trace can be dissected for strategy, error recovery, and context handling.
- True multi-step agentic evaluation: Scenarios require ongoing schema inspection, adaptation to environmental distractors, and stateful tool-use sequences.
- Explicit measurement of recovery behavior: Success is not exclusively tied to first-attempt correctness, but to the agent’s overall capacity for diagnosis and correction within resource budgets.
- Robustness under context pollution: Scenario distractors probe the agent’s ability to discriminate between salient and irrelevant contextual signals, surfacing failure modes invisible to one-shot evaluation tasks.

The index thus supports both quantitative cross-model comparison and granular investigation of recurrent failure archetypes—premature action without grounding, over-helpfulness in response to missing inputs, vulnerability to context pollution, and execution fragility under load—facilitating principled benchmark evolution and informed system design [2512.07497].

Source: https://www.emergentmind.com/topics/kamiwaza-agentic-merit-index-kami