---
title: 'AgentFloor: LLM Tool-Use Benchmark'
url: https://www.emergentmind.com/topics/agentfloor
type: topic
---

# AgentFloor: LLM Tool-Use Benchmark

AgentFloor is a benchmark and evaluation harness for **LLM-based agents that use tools**. It consists of **30 deterministic tasks** built on **8 abstract tools** over an in-memory database and organized as a **six-tier capability ladder**, from instruction following without tools to long-horizon planning under persistent constraints. The benchmark was introduced to answer a deployment question that existing evaluations do not directly resolve: which parts of agent workflows genuinely require frontier models, and which can already be handled by smaller open-weight systems. In the reported evaluation, **16 open-weight models (0.27B–32B parameters)** and **GPT-5** are tested across **16,542 scored runs** [2605.00334].

## 1. Definition, motivation, and scope

AgentFloor was designed around a practical observation about production agentic systems: a single user request often triggers many model calls, and most of those calls are **short**, **structured**, and **operationally simple**. The benchmark therefore asks where the **“boundary of model necessity”** lies—specifically, whether small and mid-sized open-weight models are already sufficient for routine tool use, and where large frontier models still provide clear value [2605.00334].

This framing distinguishes AgentFloor from two neighboring benchmark traditions. First, single-turn function-calling suites such as BFCL, API-Bank, and Gorilla emphasize isolated tool calls rather than sequential workflows. Second, broader agentic environments such as ToolBench, WebArena, OSWorld, and SWE-Bench mix tool use with web interaction, GUI complexity, API drift, and contamination risk, making it difficult to separate failures of core tool-use cognition from failures induced by unstable environments. AgentFloor instead isolates the **cognitive demands of tool use** inside a deterministic abstract-tool environment [2605.00334].

The benchmark’s central research questions are explicitly tiered. It studies how model families behave as tasks progress from **pure instruction following** to **single-tool invocation**, **sequential chaining**, **branching**, **multi-source synthesis with conflict recovery**, and finally **long-horizon planning under persistent constraints**. It also studies cost, latency, and intervention sensitivity, rather than reporting only a single aggregate accuracy score [2605.00334].

## 2. Deterministic environment and six-tier capability ladder

All AgentFloor tasks execute in the same abstract environment: **8 deterministic tools** over an **in-memory fixture database**, with **no filesystem, no external APIs, no live web, and no time-varying state**. Every tool call is a pure function of its arguments and the fixture state, and the fixture state does not change unpredictably. This design is meant to ensure exact reproducibility and to prevent API drift from confounding results [2605.00334].

The tool surface is fixed:

- `search_records`
- `lookup_record`
- `get_attribute`
- `list_options`
- `check_constraint`
- `compare_records`
- `compute_value`
- `submit_decision`

AgentFloor contains **30 tasks**, arranged as **6 tiers** with **5 tasks per tier**. Each task has a **canonical prompt** and **4 paraphrase variants**. For four tasks—**A1, B1, C1, and E1**—the benchmark also includes **five instance variants** for robustness ablations. Step budgets increase with tier complexity [2605.00334].

| Tier | Added cognitive demand | Step budget |
|---|---|---:|
| A0 | Instruction following without tools | 1 |
| A | Single tool call | 2 |
| B | Sequential 2-tool chaining | 4 |
| C | Branching based on intermediate results | 6 |
| D | Multi-source synthesis with conflict recovery | 8 |
| E | Long-horizon planning under persistent constraints | 10 |

The progression is cumulative. Tier **A0** tests direct answer production under format and content constraints, with no tool use permitted. Tier **A** adds tool selection and argument formation. Tier **B** requires output from one tool to feed into another. Tier **C** introduces conditional branching over intermediate results. Tier **D** requires the model to reconcile conflicting or incomplete evidence from multiple tool calls. Tier **E** adds sustained planning, repeated constraint enforcement, and failure avoidance over long trajectories [2605.00334].

A notable design choice is that each tier introduces **exactly one new cognitive demand** beyond the previous one. This makes AgentFloor less a generic task collection than a calibrated ladder for diagnosing where capability drops occur [2605.00334].

## 3. Evaluation protocol, scoring, and statistical tests

AgentFloor is evaluated with a single Python harness at **temperature 0**, using a byte-identical system prompt across all main conditions. The benchmark relies only on each provider’s **native tool-calling interface**. If a model emits JSON in plain text rather than through the designated tool-calling field, those calls are ignored and the run counts as having used no tools. Models that probe as `text_leaked`, `no_tools`, or `error` are excluded [2605.00334].

The primary metric is **Task Completion Rate (TCR)**, defined as binary pass/fail per run:

$$
\mathrm{TCR} = \frac{\#\ \text{passed runs}}{\#\ \text{total runs}} \times 100\%.
$$

A run passes **if and only if** all four checker families pass:

1. **Final-answer match**
2. **Submission validator**
3. **Trajectory correctness**
4. **Forbidden behavior absence**

The forbidden-behavior category includes hallucinated tools, termination without answer, repeated identical tool calls without progress, and partial constraint checking when all constraints are required. There is **no partial credit** [2605.00334].

In a few tasks, the benchmark uses a deterministic cached **LLM judge** (`gpt-5-nano`) for semantic predicates: **A02/A03/A05** for hallucinated facts in summaries, and **E5** for inconsistent recovery. The cache is keyed by **SHA-256** and reported to achieve approximately **88% cache hit rate**, again in service of reproducibility [2605.00334].

Sampling is deliberately heavy. For each **model × tier** cell in the open-weight corpus, the evaluation uses **125 runs**: **5 tasks × 5 prompt variants × 5 repeated runs**. For **GPT-5**, the main comparison uses at least **45 runs per tier**, with **270 paired observations** overall. Variance is estimated with **10,000 bootstrap resamples** using **seed 42**, and the paper reports **95% confidence intervals** for most descriptive statistics [2605.00334].

For frontier equivalence, the authors use **TOST** and **non-inferiority testing** with a pre-registered margin of **±10 percentage points**. Letting

$$
\Delta = \mathrm{TCR}_{\text{gemma4:26b}} - \mathrm{TCR}_{\text{GPT-5}},
$$

equivalence holds at ±10 pp if the **90% CI** of $\Delta$ lies within $[-10, +10]$ pp, while non-inferiority at margin −10 pp requires the **95% CI lower bound** to exceed −10 pp [2605.00334].

## 4. Models evaluated and empirical findings

The evaluation covers **16 open-weight models**—from `functiongemma:270m` at **0.27B** to `qwen3:32b` at **32B**—plus **GPT-5** as the frontier reference. Open-weight systems are served via **Ollama**, while GPT-5 is accessed through the **OpenAI Responses API** [2605.00334].

The headline aggregate result is that the strongest open-weight model, **`gemma4:26b`**, is statistically equivalent to GPT-5 on the benchmark as a whole. Over the 30-task paired comparison, **`gemma4:26b` achieves 60.0% TCR**, **GPT-5 achieves 59.6%**, and the difference is **$\Delta = +0.4$ pp** with **90% CI \([-4.0, +5.1]\)**, which satisfies the pre-registered equivalence criterion [2605.00334].

Tier-by-tier, however, the picture is not uniform. On **A0**, `gemma4:26b` reaches **100%** while GPT-5 reaches **80%**, with **$\Delta = +20.0$ pp** and **95% CI \([+8.9, +33.3]\)**; this is a strict win for the open-weight model. On **A**, `gemma4:26b` reaches **96%** and GPT-5 **98%**, which is formally equivalent at ±10 pp. On **B**, `gemma4:26b` reaches **72%** and GPT-5 **82%**; the point estimates are close, but the confidence interval is too wide to prove equivalence. On **C**, `gemma4:26b` reaches **59%** and GPT-5 **51%**; again, the point estimates are close but the interval remains wide. On **D**, `gemma4:26b` reaches **32%** and GPT-5 **42%**, with neither model strong in absolute terms. On **E**, the distinction sharpens: `gemma4:26b` is reported at **0%**, while GPT-5 reaches **10.2%**, with **$\Delta = -10.2$ pp** and **95% CI \([-18.4, -2.0]\)**, making GPT-5 strictly superior on that tier [2605.00334].

The strongest broader conclusion is that **small and mid-sized open-weight models are already sufficient for much of the short-horizon, structured tool use work** represented by **A0, A, and much of B**, while **Tier E** exposes a persistent frontier advantage. Yet the frontier advantage is narrow rather than sweeping: even GPT-5 reaches only **10%** TCR on Tier E, and **no model reaches high reliability** there [2605.00334].

The “smallest model that’s reliable enough” analysis makes the deployment implication more concrete. Under a strict criterion requiring the **95% CI lower bound** to clear a threshold, **`qwen3:0.6b`** suffices for **60–70%** reliability on **A0**, **`ministral-3:3b`** clears **80%** on **A**, and **`qwen3.5:2b`** clears **70%** on **B**. No model clears even **60%** under this criterion on **C, D, or E** [2605.00334].

These results identify a sharp capability break between **B** and **C**. The benchmark authors emphasize a large cliff from sequential chaining to branching, suggesting that persistent agent difficulty begins not at tool invocation itself but at **conditional control flow and longer-horizon coordination** [2605.00334].

## 5. Failure modes and intervention sensitivity

AgentFloor includes an explicit failure taxonomy. The benchmark assigns each failed run a priority-ordered label: **F1 hallucinated tool**, **F2 malformed call**, **F4 step-budget exhausted**, **F5 early resignation**, **F5b plan-without-execute**, **F6 wrong tool**, **F7 partial completion**, and **F3 residual** [2605.00334].

The Tier-E comparison between GPT-5 and `gemma4:26b` is especially diagnostic. For **GPT-5** on E, the reported breakdown is **10% PASS**, **39% F5**, **24% F5b**, **18% F7**, **8% F4**, and **0% F1**. GPT-5 therefore tends to remain inside the tool schema, often forming plausible plans, but frequently stops early or fails to fully execute. By contrast, **`gemma4:26b`** on E is reported as **0% PASS**, **41% F4**, **36% F1**, **23% F5**, and **0% F5b**. It tends to exhaust the step budget, invent nonexistent tools when stuck, or resign after multiple attempts rather than merely planning in text [2605.00334].

This divergence matters because interventions are strongly **model-specific rather than universal**. An **explicit-submission prompt**—adding the instruction that the model **must call `submit_decision`**—raises **`ministral-3:8b`** on **E1** from **0/5 to 5/5**, a **+100 pp** lift, but produces no change for `gemma4:26b`, `qwen3:32b-nothink`, GPT-5, or GPT-5-mini on the same task. In the successful case, the model was already computing the right decision and simply failing to submit it through the tool interface; in the others, the failure occurred earlier in the reasoning chain [2605.00334].

A second intervention—**doubling the step budget** for GPT-5 on **D** and **E**—helps only marginally in aggregate, though it improves **E1** from **1/9 to 4/9**. The paper’s interpretation is that extra steps help only when the model is already exploring near the budget boundary; for most failures, the main problem is not insufficient budget but incomplete execution or resignation [2605.00334].

A third intervention concerns **reasoning mode** in the Qwen3 family. On **Tier B**, turning reasoning **off** improves `qwen3:32b` by **+12 pp**, but hurts `qwen3:8b` by **−9 pp** and `qwen3:14b` by **−12 pp**. This is presented as evidence that reasoning tokens are a tuning knob rather than a universally beneficial feature: at some scales they help chaining, while at others they consume output budget or exacerbate resignation behavior [2605.00334].

Most strikingly, a seemingly natural **plan/execute/submit** prompt harms performance across multiple models. The paper reports overall TCR drops of **−5 pp** for `gemma4:26b`, **−6 pp** for `qwen3:32b-nothink`, **−14 pp** for `nemotron-3-nano:4b`, and **−33 pp** for `ministral-3:8b`. Trace inspection indicates that models often comply with the instruction not to call tools during the planning phase, then never transition into actual execution. AgentFloor therefore treats prompt structuring itself as an empirical variable rather than a stable best practice [2605.00334].

## 6. Operational implications, cost structure, limitations, and reproducibility

The central deployment recommendation is explicit: use **small or mid-sized open-weight models** for the broad base of **routine, short-horizon, structured tool use**, and reserve **frontier models** for the narrower class of tasks that resemble **Tier E**. This is not presented as a universal recipe for all agents, but as a benchmark-backed routing principle [2605.00334].

Cost and latency estimates reinforce this routing argument. Under the paper’s assumptions, **`gemma4:26b`**, at matched aggregate reliability, is reported as approximately **15× cheaper per passed task on Mac**, about **3× cheaper per passed task on cloud GPU**, and roughly **2.5× faster per passed task** than GPT-5. The paper also highlights lower-cost operating points such as **`granite4:3b`**, which reaches **40% TCR** at **\$0.00046 per pass** on Mac, and **`ministral-3:8b`**, which reaches **55% TCR** at **\$0.00098 per pass** [2605.00334].

At the same time, the benchmark is explicit about its own limitations. It contains only **30 tasks**, uses a **deterministic abstract environment**, evaluates **native tool calling only**, includes **no human baseline**, and studies only **one frontier model** in depth. The higher-tier ceiling should therefore be read as a limit under this benchmark and protocol, not as a universal limit on all agent systems [2605.00334].

Reproducibility is unusually strong. The paper states that it releases the **benchmark tasks**, **evaluation harness**, **sweep configurations**, and the **full run corpus** of **16,542 scored runs**. The intended use is direct comparison under the same deterministic settings, fixed tool schema, and unchanged system prompt [2605.00334].

## 7. Broader research context and the wider “AgentFloor” idea

The name **AgentFloor** can be misleading if read literally. In the cited literature, the term primarily denotes the **tool-use benchmark** just described, but adjacent work uses “AgentFloor”-type language for systems that reason about **floor plans**, **spatial layout**, or **floor level** rather than tool-routing. This suggests two distinct research agendas sharing a label: one centered on **agentic tool use**, the other on **spatial intelligence**.

A particularly close neighboring benchmark is **Blueprint-Bench**, which evaluates whether models can convert apartment photographs into accurate 2D floor plans. Its main result is that current LLMs, image models, and agent systems perform at or below a random baseline on the composite similarity score, while human performance remains substantially superior. The paper is explicit that it is “directly relevant to any ‘AgentFloor’ concept: an agent that reconstructs floor plans from photos,” and it argues that merely adding agency and tools does not automatically confer the geometry-aware reasoning required for multi-view spatial reconstruction [2509.25229].

On the generative side, **ActFloor-GAN** formulates residential floorplan design as a **human-centric**, **activity-guided** two-stage process. It synthesizes a **human-activity map** and then uses that map to guide pixel-wise room-type prediction via a cycle-consistent conditional GAN. In the reported comparison, the method reaches **MSE 0.111**, **MAE 0.096**, and **vectorization success 83/100**, with a user study indicating stronger topological quality than the cited baseline by Wu et al. This line of work uses “agent” ideas in the sense of simulated human movement and controllable activity fields rather than tool-using LLM agents [2111.03545].

Another neighboring direction is **audio-visual floorplan reconstruction**. AV-Map predicts a 2D top-down floorplan from short input video sequences with synchronized ambisonic audio. On **85 large real-world environments**, the paper reports that with glimpses spanning **26% of an area**, the model estimates the whole area with **66% accuracy**, and that audio contributes both geometric and semantic evidence for unseen space. This is a different technical pathway to floor-oriented agent reasoning, grounded in embodied multimodal sensing rather than abstract tool trajectories [2012.15470].

A more explicitly topology-first synthesis framework appears in **GFLAN**, which factorizes floorplan generation into **Stage A** topological planning and **Stage B** geometric realization. Stage A sequentially allocates room centroids via probability maps, while Stage B constructs a heterogeneous room-boundary graph and applies a Transformer-augmented GNN to regress room boundaries. The paper’s framing is especially resonant with agentic decomposition: a planner-like module decides *where* rooms should go, and a refiner-like module decides *what precise shapes* they should take [2512.16275].

There is also work on vertical localization rather than floorplan generation. The 911-response paper “Predicting Floor-Level for 911 Calls with Neural Networks and Smartphone Sensor Data” describes a pipeline that detects building entry with a neural model and then uses barometric pressure differences to infer the caller’s floor. Across **63 experiments** in **five** NYC buildings, the system is reported to predict the correct floor level with **100% accuracy** under the building-conditioned setting. In the supplied synthesis, this is described as “almost exactly an ‘AgentFloor’ system,” but its object is floor index estimation rather than tool-use evaluation or floorplan synthesis [1710.11122].

Taken together, these neighboring works indicate that **AgentFloor** now sits at an intersection of at least three technical themes: **benchmarking tool-using agents**, **reconstructing or generating architectural layouts**, and **estimating vertical position in buildings**. A plausible implication is that future systems may combine these strands: small open-weight models could handle routine tool calls inside larger spatial pipelines, while specialized vision, geometry, or sensing modules handle the genuinely hard floor-oriented reasoning that current generalist agents still fail to solve [2605.00334][2509.25229][2512.16275].

Source: https://www.emergentmind.com/topics/agentfloor