---
title: 'MTAgentBench: Benchmarks for Multi-Mission Agents'
url: https://www.emergentmind.com/topics/mtagentbench
type: topic
---

# MTAgentBench: Benchmarks for Multi-Mission Agents

MTAgentBench is the name used for two distinct benchmark concepts in the supplied research record. In “Multi-Mission Tool Bench,” it denotes MMTB, a benchmark for LLM-based tool-use agents operating across related, dynamically changing missions [2504.02623]. In “GitHarness: Git Init Your Harness Working Memory for Perpetual User Requirements,” it denotes a verifier-preserving benchmark for long-horizon collaboration in which user requirements evolve and agents must select, reuse, revise, or discard prior work [2609.36789]. The two benchmarks share an emphasis on dynamic interaction, but evaluate different capabilities: MMTB focuses on mission switching and dependency-aware tool invocation, whereas the GitHarness benchmark focuses on requirement tracking and versioned work-state management.

## 1. Terminology and research scope

The MMTB usage of MTAgentBench addresses the limitations of single-mission tool-use evaluation. Conventional tool-use benchmarks often approximate the workflow

$$
\text{one user request} \rightarrow \text{one tool-use episode} \rightarrow \text{one answer}.
$$

MTAgentBench instead models interactions as a sequence of missions,

$$
M_1 \rightarrow M_2 \rightarrow \cdots \rightarrow M_N,
$$

where later missions can depend on entities, parameters, results, or implicit context established earlier. The agent must retain relevant dialogue information, detect mission switches, select appropriate actions, avoid cross-mission interference, coordinate serial and parallel tool calls, and decide whether to answer directly, request clarification, or invoke tools [2504.02623].

The GitHarness usage addresses a related but different problem. An agent collaborates with a user over a long horizon while accumulating evidence, database queries, code, workspace changes, reasoning traces, and report drafts. New feedback may complete an underspecified requirement, introduce an additional dimension, or revise an existing requirement. The central problem is formulated as joint requirement tracking and selective work reuse: the agent must reconstruct the current requirement, identify a semantically compatible historical state, and update only the affected work [2609.36789].

These meanings should not be conflated. MMTB evaluates dynamic tool invocation within related missions. The GitHarness benchmark evaluates requirement-conditioned state transitions across mathematical reasoning, text-to-SQL, agentic search, software engineering, and research synthesis.

## 2. Multi-Mission Tool Bench

### Mission model

In MMTB, a mission is a user objective completed through one of four primary action types:

$$
\mathcal{A} =
\{A_{\mathrm{single}}, A_{\mathrm{multi}}, A_{\mathrm{chat}}, A_{\mathrm{clarify}}\}.
$$

The action types are:

1. **Single-tool invocation**: the mission can be completed with one tool call.
2. **Multiple-tool invocation**: the mission requires multiple tools or calls.
3. **Chat-only mission**: no suitable tool is required and the agent should answer from context or its own knowledge.
4. **Clarification mission**: a tool is appropriate, but required parameters are missing or cannot be safely inferred; the agent must ask the user rather than hallucinate parameters.

Multiple-tool invocation is operationally divided into serial execution, parallel execution, and mixed serial-parallel execution. Consequently, the benchmark has four primary action types but six operational mission categories.

A test case contains between one and four sequential missions. Relatedness is instantiated through implicit understanding, ellipsis, and long-term memory. A later mission can therefore be represented as

$$
M_i = f(U_i,D_{<i}),
$$

where $U_i$ is the current utterance and $D_{<i}$ is the preceding dialogue. The current request is not necessarily self-contained.

### Switching-space coverage

For $N$ missions, a switching sequence is

$$
\mathbf{A}_N=(A_1,A_2,\ldots,A_N), \qquad A_i \in \mathcal{A}.
$$

The complete primary-type switching space is

$$
\mathcal{S}_N=\mathcal{A}^N,
\qquad
|\mathcal{S}_N|=4^N.
$$

There are 16 primary-type sequences for two missions, 64 for three missions, and 256 for four missions. The four-mission partition contains one combination for each primary action-type sequence, providing complete coverage of the four-mission switching space. Multi-tool subtypes are used operationally, but the formal switching-space count remains $4^N$, rather than $6^N$.

The benchmark distinguishes related missions from merely consecutive missions. Later requests deliberately depend on earlier dialogue, so an agent must resolve context before selecting tools or filling arguments. This design targets weaknesses not captured by independent tool-calling episodes.

### Tools and simulated environments

The benchmark uses simulated external tools derived from the public-apis repository. The supplied description reports approximately 400 tool lists, approximately 1,600 tools, and 50 tool categories. Tool descriptions are refined with LLMs, explicit constraints and parameter requirements are added, complex parameter types such as enumerations, arrays, and objects are supported, and five LLM-based experts review the generated tools.

The tools are simulated rather than connected to live services. A tool agent produces API-like JSON responses, validates HTTP methods and parameters against tool specifications, generates realistic data, and handles edge cases. This improves control and reproducibility but does not reproduce live API latency, authentication, rate limits, changing external state, noisy service responses, or real user corrections [2504.02623].

## 3. Data generation and evaluation in MMTB

### Role-based generation

Although described as a multi-agent data-generation framework, the generator uses a single LLM with different role prompts. The five roles are user, planner, AI assistant, checker, and tool.

The user role generates missions and answers clarification questions. The planner interprets the mission, decides whether tools are needed, selects tools, identifies required and optional parameters, determines serial, parallel, or mixed execution, and produces structured action lists. The AI-assistant role communicates with the user, asks clarification questions, presents tool results, summarizes completed missions, and maintains the visible dialogue. The tool role simulates API execution. The checker verifies output format, action ordering, consistency between the planner’s reasoning, plan, and action list, tool dependencies, and realization of the prescribed mission type.

Single-mission generation samples a tool list, selects an action type, generates candidate missions, chooses a relationship type for related missions, conditions generation on earlier dialogue, manually selects and refines a candidate, executes planner, tool, AI, and checker roles, and retains the complete dialogue and execution trace. Multi-mission generation repeats this process sequentially for each target action-type sequence, conditioning each mission on prior dialogue, the selected relationship type, available tools, and the prescribed action type [2504.02623].

### Constraints and quality control

The generation process requires required parameters to appear in missions, permits optional parameters when appropriate, avoids explicit tool names in user requests, varies mission length and linguistic form, incorporates entities and constraints from tool descriptions, and requires clarification when parameters are missing. Tool dependencies must be respected. Parallel calls are allowed only where no parameter or logical dependency exists, and answer or clarification actions cannot execute concurrently with other tools. The planner’s natural-language plan must agree with its structured action list.

Quality control combines LLM-based role checking, checker verification, candidate generation and manual refinement, review of generated tools by five LLM experts, and validation of API-like responses and parameter schemas. The authors note that repeated generation and human intervention are required because LLMs do not always follow role instructions accurately.

### Dynamic decision-tree evaluation

MMTB evaluates multi-tool execution with a dynamic decision tree rather than a single reference sequence. For a tool set $V$ and dependency graph

$$
G=(V,E),
$$

an edge $(v_i,v_j)\in E$ indicates that $v_i$ must precede $v_j$ because of a parameter or logical dependency. The graph is treated as a directed acyclic graph for path generation.

At each stage, the evaluator identifies all zero-indegree tools. These may be selected individually as serial actions or grouped into permissible combinations as parallel actions. Depth-first topological traversal enumerates valid paths, including serial, parallel, and mixed serial-parallel paths. A singleton node represents one tool call; a grouped node such as $[0,1]$ represents a parallel call to tools 0 and 1.

During evaluation, an agent’s action is matched incrementally against the current candidate children. An invalid action terminates the path as incorrect. A valid action prunes incompatible subtrees, leaving only legal continuations. This accepts multiple dependency-valid execution orders while distinguishing correctness from efficiency.

The principal metrics are success rate and optimal path rate. Success rate measures the percentage of evaluated cases in which the agent completes a valid path. Optimal path rate measures the percentage completing a minimum-length valid path. The benchmark also reports accomplished progress, a softer measure of how far an agent proceeds through a mission. A valid but inefficient plan can therefore receive credit for correctness without being classified as optimal.

## 4. MMTB dataset and empirical findings

MMTB contains 1,024 test entries, evenly divided across single-, two-, three-, and four-mission partitions.

| Partition | Missions | Entries |
|---|---:|---:|
| Single-mission | 1 | 256 |
| Two-mission | 2 | 256 |
| Three-mission | 3 | 256 |
| Four-mission | 4 | 256 |

The study evaluates 24 models, with the main quantitative table reporting 15. These include closed-source general models, open-source general models, and specialized tool-use models. Reported model sizes range from approximately 0.5B to tens or hundreds of billions of parameters. Models are prompted through a common tool-using interface requiring mission status, tool-need analysis, required and optional parameter analysis, a natural-language plan, and an action list with tool names and JSON arguments [2504.02623].

The reported results show substantial variation by action type. Chat-only missions are generally easiest, while clarification missions are considerably harder. Parallel multi-tool execution is easier than serial or mixed execution for many models, whereas serial execution requires maintaining intermediate state and using earlier tool outputs as later arguments. Serial-plus-parallel missions are particularly challenging.

The best reported optimal-path rates are 41.08% for GPT-4o-2024-11-20 and 39.42% for o1-2024-12-17. GPT-4o records 51.59% on parallel multi-tool missions and 23.81% on mixed serial-parallel missions. o1 records 33.33% on serial multi-tool missions. Tool-use specialization does not guarantee robust planning: ToolACE-8B reaches 87.11% on chat-only missions but 0% on serial multi-tool cases.

Performance declines as the number of related missions increases. Specialized models are competitive with general models on isolated missions but decline rapidly as related missions accumulate, and even o1 shows a noticeable multi-mission decline. Long-term-memory relationships are consistently more difficult than implicit understanding or ellipsis. The reported findings support the interpretation that cross-mission context retention and information retrieval are major bottlenecks.

The error analysis distinguishes tool errors, parameter-name hallucinations, parameter-value hallucinations, and parameter-value errors. Stronger agents generally have a lower proportion of tool errors. Parameter-name hallucinations are relatively rare but serious, while parameter-value extraction errors are the most common parameter problem. The supplied paper does not fully specify the interpretation of the reported tool-error percentages, so they represent the authors’ reported error distributions rather than independently reconstructable probabilities.

## 5. Verifier-Preserving MTAgentBench

The second MTAgentBench is constructed within GitHarness as a benchmark for evolving user requirements and selective reuse of prior work [2609.36789]. It extends five public benchmarks into seven-turn interactions:

| Domain | Source benchmark | Native metric |
|---|---|---|
| Mathematical reasoning | GSM8K | Accuracy |
| Text-to-SQL | BIRD | Execution accuracy |
| Agentic search | BrowseComp-Plus | Answer accuracy |
| Software engineering | SWE-bench Verified | Resolved rate |
| Research synthesis | DeepResearch Bench | RACE |

The benchmark is called verifier-preserving because requirement changes modify the task specification while retaining a valid mechanism for evaluating the resulting artifact. Each instance has a final-task verifier and may have turn-level verifiers. Final correctness is evaluated against the revised requirement, rather than only the original task or the latest user utterance in isolation.

The benchmark uses three underlying agent harnesses: TCRAG, described as a stack-memory harness; StackPlanner, a hierarchical multi-agent planning harness; and OpenHands, an executable-workspace harness. It includes structured requirement and version-state annotations, final-task verifiers, optional turn-level verifiers, and execution-cost measurements.

### Requirement-change operators

The benchmark distinguishes three broad kinds of requirement change:

1. **Completion**: the user supplies missing details consistent with the existing direction.
2. **Elicitation**: the user introduces an additional task dimension.
3. **Shift**: the user revises, withdraws, or replaces a prior requirement.

The paper states that nine fine-grained operators instantiate these changes, but the supplied material does not enumerate them.

The benchmark evaluates whether an agent can remove obsolete information, preserve compatible work, update affected artifacts, and avoid unnecessary calls and token generation. Its motivating examples include changing the color of an umbrella in an existing visual artifact while preserving unrelated content, removing one section from a report without rewriting the remainder, and fixing one software bug without modifying unrelated files.

### Domain tasks

In mathematical reasoning, a multi-turn GSM8K problem is modified through later feedback. The expected artifact is a numerical or textual answer, and the native metric is accuracy.

In text-to-SQL, later feedback can change filters, requested fields, aggregation criteria, or other database constraints. The query is executed against the database, and the native metric is execution accuracy.

In agentic search, the agent accumulates search results, evidence, notes, and intermediate answers. Requirement changes can alter scope, constraints, entities, time periods, or comparisons. The native metric is answer accuracy.

In software engineering, a SWE-bench Verified repository and issue are transformed into a multi-turn interaction. The agent must preserve unrelated correct files and edits while responding to new reproduction details, behavioral requirements, or revised expectations. The native metric is resolved rate.

In research synthesis, the agent gathers evidence and produces a report while later feedback changes scope, requested sections, or evidentiary requirements. The native metric is DeepResearch Bench RACE.

The supplied text does not provide total benchmark size, per-domain counts, exact train/validation/test counts, difficulty levels, or the nine perturbation operators. It also does not provide direct standalone metrics for valid-work preservation, obsolete-work removal, changed-file locality, branch-selection accuracy, or state-selection precision and recall.

## 6. GitHarness and results on evolving requirements

GitHarness couples requirement states and work states in a branchable version history. Each successful turn creates a version containing the effective requirement, a recoverable work checkpoint, and parent or branch relationships. Historical checkpoints are immutable. When a user revisits an earlier state, the framework can restore that state and create a new branch.

The Git Agent uses three actions:

1. **Inspect**: read a prior version’s requirement.
2. **Commit**: resolve the current requirement and select a base version.
3. **Reuse**: select an existing version that already satisfies the requirement.

The framework supports exact reuse, fresh solving, and reuse-and-patch. Exact reuse returns an existing checkpoint that already satisfies the request. Fresh solving starts without a usable prior state. Reuse-and-patch forks a compatible state and modifies it locally. Failed executions leave the version graph unchanged, and only normally completed executions are committed.

The Git Agent is trained with interface-level black-box reinforcement learning while the downstream harnesses and task-execution models remain fixed. The learned policy resolves requirements and selects historical states; it does not directly generate the final SQL query, code patch, or research report. For GitHarness-RL, the Git Agent is a LoRA-GRPO-trained Qwen3-8B, while downstream execution uses the model specified by the experimental condition.

Training-free GitHarness beats Native in all 30 model–harness–domain settings and matches or exceeds the strongest competing baseline in 28 of 30 settings, including 27 strict wins and one tie. GitHarness-RL outperforms native cross-turn execution in 29 of 30 settings. Relative to Native, training-free GitHarness reduces average token usage by 35.8% on SQL, 54.5% on Search, 73.6% on Code, and 36.7% on Research.

Ablations show that historical version selection and execution-mode selection are both necessary. Under StackPlanner with DeepSeek-V4-Flash, removing historical version selection lowers performance from 94.0 to 84.0 on Math, from 78.0 to 62.0 on SQL, from 58.0 to 42.0 on Search, from 74.0 to 44.0 on Code, and from 49.96 to 45.59 on Research. Removing Inspect, exact-state reuse, ReuseAndPatch, or FreshSolve also reduces performance in multiple domains.

In 10–30-turn sessions, GitHarness is more robust than Native, Restart, and U-Fold. At 30 turns, it exceeds the strongest baseline by 6.0 points on Math and 12.79 RACE points on Research. Its Research performance decreases by 2.21 points from 10 to 30 turns, compared with at least 3.37 points for the baselines.

These results support the claim that explicit requirement-conditioned state management improves final performance and execution efficiency. They do not, however, fully quantify whether unaffected artifacts are preserved or whether obsolete artifacts are removed, because the supplied results emphasize final task success and token usage rather than artifact-level locality.

## 7. Evaluation reliability, limitations, and relation between the benchmarks

A separate study, “Benchmarking LLM Judges for Mobile Agent Evaluation,” provides methodological guidance for any MTAgentBench evaluator that relies on LLM-based judging [2608.11434]. It introduces MobileJudgeBench, which contains 931 human-annotated trajectories from six mobile-agent benchmarks, four agent models, and 68 Android apps. Its central premise is that a benchmark evaluates not only agents but also the judge used to determine success.

The study finds that a simple sampled-screenshot judge can be competitive with or exceed more elaborate judge pipelines. Across 30 judge variants, the best reported accuracy is 90.9% for the simple baseline with Gemini 3 Flash and 90.8% for the simple baseline with GPT-5-mini. F1 and balanced accuracy correlate more strongly with agent-ranking fidelity and success-rate estimation error than precision alone. The study also identifies conservative and permissive judge profiles: conservative judges produce more false negatives, while permissive judges produce more false positives.

These findings do not establish that the same judge, backbone, screenshot count, or sampling strategy is optimal for either MTAgentBench. They do support validating any automatic evaluator against redundant human or programmatic ground truth, reporting accuracy, precision, recall, F1, balanced accuracy, ranking fidelity, success-rate mean absolute error, uncertainty intervals, and conservative or permissive error profiles.

The two MTAgentBench meanings have distinct limitations. MMTB is limited to at most four missions, uses simulated tools, relies on human intervention in generation, and uses a single LLM with role-specific prompts rather than independently trained agents. Its switching space grows exponentially as $4^N$, and its optimality criterion is primarily minimum tool-path length rather than latency, cost, API-call count, or user-visible turns.

The GitHarness benchmark uses synthetic multi-turn interactions generated from public tasks with LLM assistance, author review, benchmark verification, and human inspection. Requirement tracking partly relies on an LLM judger. Absolute scores differ across TCRAG, StackPlanner, and OpenHands because their tools and execution procedures differ. The supplied material does not report confidence intervals, hypothesis tests, detailed calibration results, or a comprehensive contamination audit.

Taken together, the benchmarks define complementary dimensions of dynamic agent reliability. MMTB asks whether an agent can preserve conversational context while switching among related missions and coordinating dependency-constrained tools. The GitHarness benchmark asks whether an agent can connect evolving requirements to compatible historical work states and perform local updates without retaining stale requirements or restarting unnecessarily. Both indicate that isolated task success is insufficient for characterizing long-horizon agents: reliability also depends on context retention, state management, dependency-aware planning, clarification, selective reuse, and the validity of the evaluator itself.

Source: https://www.emergentmind.com/topics/mtagentbench