AgentFloor: LLM Tool-Use Benchmark

• AgentFloor is a benchmark evaluating LLM-based agents using 8 deterministic tools across 30 tasks organized in a six-tier capability ladder.
• It isolates the cognitive demands of tool use, testing models from simple instruction following to long-horizon planning under persistent constraints.
• Empirical findings show that while small and mid-sized open-weight models perform on par with frontier models for routine tasks, challenges remain for complex decision-making.

AgentFloor is a benchmark and evaluation harness for LLM-based agents that use tools. It consists of 30 deterministic tasks built on 8 abstract tools over an in-memory database and organized as a six-tier capability ladder, from instruction following without tools to long-horizon planning under persistent constraints. The benchmark was introduced to answer a deployment question that existing evaluations do not directly resolve: which parts of agent workflows genuinely require frontier models, and which can already be handled by smaller open-weight systems. In the reported evaluation, 16 open-weight models (0.27B–32B parameters) and GPT-5 are tested across 16,542 scored runs (Karmakar et al., 1 May 2026).

1. Definition, motivation, and scope

AgentFloor was designed around a practical observation about production agentic systems: a single user request often triggers many model calls, and most of those calls are short, structured, and operationally simple. The benchmark therefore asks where the “boundary of model necessity” lies—specifically, whether small and mid-sized open-weight models are already sufficient for routine tool use, and where large frontier models still provide clear value (Karmakar et al., 1 May 2026).

This framing distinguishes AgentFloor from two neighboring benchmark traditions. First, single-turn function-calling suites such as BFCL, API-Bank, and Gorilla emphasize isolated tool calls rather than sequential workflows. Second, broader agentic environments such as ToolBench, WebArena, OSWorld, and SWE-Bench mix tool use with web interaction, GUI complexity, API drift, and contamination risk, making it difficult to separate failures of core tool-use cognition from failures induced by unstable environments. AgentFloor instead isolates the cognitive demands of tool use inside a deterministic abstract-tool environment (Karmakar et al., 1 May 2026).

The benchmark’s central research questions are explicitly tiered. It studies how model families behave as tasks progress from pure instruction following to single-tool invocation, sequential chaining, branching, multi-source synthesis with conflict recovery, and finally long-horizon planning under persistent constraints. It also studies cost, latency, and intervention sensitivity, rather than reporting only a single aggregate accuracy score (Karmakar et al., 1 May 2026).

2. Deterministic environment and six-tier capability ladder

All AgentFloor tasks execute in the same abstract environment: 8 deterministic tools over an in-memory fixture database, with no filesystem, no external APIs, no live web, and no time-varying state. Every tool call is a pure function of its arguments and the fixture state, and the fixture state does not change unpredictably. This design is meant to ensure exact reproducibility and to prevent API drift from confounding results (Karmakar et al., 1 May 2026).

The tool surface is fixed:

• search_records
• lookup_record
• get_attribute
• list_options
• check_constraint
• compare_records
• compute_value
• submit_decision

AgentFloor contains 30 tasks, arranged as 6 tiers with 5 tasks per tier. Each task has a canonical prompt and 4 paraphrase variants. For four tasks—A1, B1, C1, and E1—the benchmark also includes five instance variants for robustness ablations. Step budgets increase with tier complexity (Karmakar et al., 1 May 2026).

Tier	Added cognitive demand	Step budget
A0	Instruction following without tools	1
A	Single tool call	2
B	Sequential 2-tool chaining	4
C	Branching based on intermediate results	6
D	Multi-source synthesis with conflict recovery	8
E	Long-horizon planning under persistent constraints	10

The progression is cumulative. Tier A0 tests direct answer production under format and content constraints, with no tool use permitted. Tier A adds tool selection and argument formation. Tier B requires output from one tool to feed into another. Tier C introduces conditional branching over intermediate results. Tier D requires the model to reconcile conflicting or incomplete evidence from multiple tool calls. Tier E adds sustained planning, repeated constraint enforcement, and failure avoidance over long trajectories (Karmakar et al., 1 May 2026).

A notable design choice is that each tier introduces exactly one new cognitive demand beyond the previous one. This makes AgentFloor less a generic task collection than a calibrated ladder for diagnosing where capability drops occur (Karmakar et al., 1 May 2026).

3. Evaluation protocol, scoring, and statistical tests

AgentFloor is evaluated with a single Python harness at temperature 0, using a byte-identical system prompt across all main conditions. The benchmark relies only on each provider’s native tool-calling interface. If a model emits JSON in plain text rather than through the designated tool-calling field, those calls are ignored and the run counts as having used no tools. Models that probe as text_leaked, no_tools, or error are excluded (Karmakar et al., 1 May 2026).

The primary metric is Task Completion Rate (TCR), defined as binary pass/fail per run:

$$\mathrm{TCR} = \frac{\#\ \text{passed runs}}{\#\ \text{total runs}} \times 100\%.$$

A run passes if and only if all four checker families pass:

1. Final-answer match
2. Submission validator
3. Trajectory correctness
4. Forbidden behavior absence

The forbidden-behavior category includes hallucinated tools, termination without answer, repeated identical tool calls without progress, and partial constraint checking when all constraints are required. There is no partial credit (Karmakar et al., 1 May 2026).

In a few tasks, the benchmark uses a deterministic cached LLM judge (gpt-5-nano) for semantic predicates: A02/A03/A05 for hallucinated facts in summaries, and E5 for inconsistent recovery. The cache is keyed by SHA-256 and reported to achieve approximately 88% cache hit rate, again in service of reproducibility (Karmakar et al., 1 May 2026).

Sampling is deliberately heavy. For each model × tier cell in the open-weight corpus, the evaluation uses 125 runs: 5 tasks × 5 prompt variants × 5 repeated runs. For GPT-5, the main comparison uses at least 45 runs per tier, with 270 paired observations overall. Variance is estimated with 10,000 bootstrap resamples using seed 42, and the paper reports 95% confidence intervals for most descriptive statistics (Karmakar et al., 1 May 2026).

For frontier equivalence, the authors use TOST and non-inferiority testing with a pre-registered margin of ±10 percentage points. Letting

$$\Delta = \mathrm{TCR}_{\text{gemma4:26b}} - \mathrm{TCR}_{\text{GPT-5}},$$

equivalence holds at ±10 pp if the 90% CI of $\Delta$ lies within $[-10, +10]$ pp, while non-inferiority at margin −10 pp requires the 95% CI lower bound to exceed −10 pp (Karmakar et al., 1 May 2026).

4. Models evaluated and empirical findings

The evaluation covers 16 open-weight models—from functiongemma:270m at 0.27B to [qwen3](https://www.emergentmind.com/topics/qwen3):32b at 32B—plus GPT-5 as the frontier reference. Open-weight systems are served via Ollama, while GPT-5 is accessed through the OpenAI Responses API (Karmakar et al., 1 May 2026).

The headline aggregate result is that the strongest open-weight model, gemma4:26b, is statistically equivalent to GPT-5 on the benchmark as a whole. Over the 30-task paired comparison, gemma4:26b achieves 60.0% TCR, GPT-5 achieves 59.6%, and the difference is $\Delta = +0.4$ pp with 90% CI $[-4.0, +5.1]$, which satisfies the pre-registered equivalence criterion (Karmakar et al., 1 May 2026).

Tier-by-tier, however, the picture is not uniform. On A0, gemma4:26b reaches 100% while GPT-5 reaches 80%, with $\Delta = +20.0$ pp and 95% CI $[+8.9, +33.3]$; this is a strict win for the open-weight model. On A, gemma4:26b reaches 96% and GPT-5 98%, which is formally equivalent at ±10 pp. On B, gemma4:26b reaches 72% and GPT-5 82%; the point estimates are close, but the confidence interval is too wide to prove equivalence. On C, gemma4:26b reaches 59% and GPT-5 51%; again, the point estimates are close but the interval remains wide. On D, gemma4:26b reaches 32% and GPT-5 42%, with neither model strong in absolute terms. On E, the distinction sharpens: gemma4:26b is reported at 0%, while GPT-5 reaches 10.2%, with $\Delta = -10.2$ pp and 95% CI $[-18.4, -2.0]$, making GPT-5 strictly superior on that tier (Karmakar et al., 1 May 2026).

The strongest broader conclusion is that small and mid-sized open-weight models are already sufficient for much of the short-horizon, structured tool use work represented by A0, A, and much of B, while Tier E exposes a persistent frontier advantage. Yet the frontier advantage is narrow rather than sweeping: even GPT-5 reaches only 10% TCR on Tier E, and no model reaches high reliability there (Karmakar et al., 1 May 2026).

The “smallest model that’s reliable enough” analysis makes the deployment implication more concrete. Under a strict criterion requiring the 95% CI lower bound to clear a threshold, qwen3:0.6b suffices for 60–70% reliability on A0, ministral-3:3b clears 80% on A, and qwen3.5:2b clears 70% on B. No model clears even 60% under this criterion on C, D, or E (Karmakar et al., 1 May 2026).

These results identify a sharp capability break between B and C. The benchmark authors emphasize a large cliff from sequential chaining to branching, suggesting that persistent agent difficulty begins not at tool invocation itself but at conditional control flow and longer-horizon coordination (Karmakar et al., 1 May 2026).

5. Failure modes and intervention sensitivity

AgentFloor includes an explicit failure taxonomy. The benchmark assigns each failed run a priority-ordered label: F1 hallucinated tool, F2 malformed call, F4 step-budget exhausted, F5 early resignation, F5b plan-without-execute, F6 wrong tool, F7 partial completion, and F3 residual (Karmakar et al., 1 May 2026).

The Tier-E comparison between GPT-5 and gemma4:26b is especially diagnostic. For GPT-5 on E, the reported breakdown is 10% PASS, 39% F5, 24% F5b, 18% F7, 8% F4, and 0% F1. GPT-5 therefore tends to remain inside the tool schema, often forming plausible plans, but frequently stops early or fails to fully execute. By contrast, gemma4:26b on E is reported as 0% PASS, 41% F4, 36% F1, 23% F5, and 0% F5b. It tends to exhaust the step budget, invent nonexistent tools when stuck, or resign after multiple attempts rather than merely planning in text (Karmakar et al., 1 May 2026).

This divergence matters because interventions are strongly model-specific rather than universal. An explicit-submission prompt—adding the instruction that the model must call submit_decision—raises ministral-3:8b on E1 from 0/5 to 5/5, a +100 pp lift, but produces no change for gemma4:26b, qwen3:32b-nothink, GPT-5, or GPT-5-mini on the same task. In the successful case, the model was already computing the right decision and simply failing to submit it through the tool interface; in the others, the failure occurred earlier in the reasoning chain (Karmakar et al., 1 May 2026).

A second intervention—doubling the step budget for GPT-5 on D and E—helps only marginally in aggregate, though it improves E1 from 1/9 to 4/9. The paper’s interpretation is that extra steps help only when the model is already exploring near the budget boundary; for most failures, the main problem is not insufficient budget but incomplete execution or resignation (Karmakar et al., 1 May 2026).

A third intervention concerns reasoning mode in the Qwen3 family. On Tier B, turning reasoning off improves qwen3:32b by +12 pp, but hurts qwen3:8b by −9 pp and qwen3:14b by −12 pp. This is presented as evidence that reasoning tokens are a tuning knob rather than a universally beneficial feature: at some scales they help chaining, while at others they consume output budget or exacerbate resignation behavior (Karmakar et al., 1 May 2026).

Most strikingly, a seemingly natural plan/execute/submit prompt harms performance across multiple models. The paper reports overall TCR drops of −5 pp for gemma4:26b, −6 pp for qwen3:32b-nothink, −14 pp for nemotron-3-nano:4b, and −33 pp for ministral-3:8b. Trace inspection indicates that models often comply with the instruction not to call tools during the planning phase, then never transition into actual execution. AgentFloor therefore treats prompt structuring itself as an empirical variable rather than a stable best practice (Karmakar et al., 1 May 2026).

6. Operational implications, cost structure, limitations, and reproducibility

The central deployment recommendation is explicit: use small or mid-sized open-weight models for the broad base of routine, short-horizon, structured tool use, and reserve frontier models for the narrower class of tasks that resemble Tier E. This is not presented as a universal recipe for all agents, but as a benchmark-backed routing principle (Karmakar et al., 1 May 2026).

Cost and latency estimates reinforce this routing argument. Under the paper’s assumptions, gemma4:26b, at matched aggregate reliability, is reported as approximately 15× cheaper per passed task on Mac, about 3× cheaper per passed task on cloud GPU, and roughly 2.5× faster per passed task than GPT-5. The paper also highlights lower-cost operating points such as granite4:3b, which reaches 40% TCR at \$\Delta = \mathrm{TCR}_{\text{gemma4:26b}} - \mathrm{TCR}_{\text{GPT-5}},$00.00098 per pass (Karmakar et al., 1 May 2026).

At the same time, the benchmark is explicit about its own limitations. It contains only 30 tasks, uses a deterministic abstract environment, evaluates native tool calling only, includes no human baseline, and studies only one frontier model in depth. The higher-tier ceiling should therefore be read as a limit under this benchmark and protocol, not as a universal limit on all agent systems (Karmakar et al., 1 May 2026).

Reproducibility is unusually strong. The paper states that it releases the benchmark tasks, evaluation harness, sweep configurations, and the full run corpus of 16,542 scored runs. The intended use is direct comparison under the same deterministic settings, fixed tool schema, and unchanged system prompt (Karmakar et al., 1 May 2026).

7. Broader research context and the wider “AgentFloor” idea

The name AgentFloor can be misleading if read literally. In the cited literature, the term primarily denotes the tool-use benchmark just described, but adjacent work uses “AgentFloor”-type language for systems that reason about floor plans, spatial layout, or floor level rather than tool-routing. This suggests two distinct research agendas sharing a label: one centered on agentic tool use, the other on spatial intelligence.

A particularly close neighboring benchmark is Blueprint-Bench, which evaluates whether models can convert apartment photographs into accurate 2D floor plans. Its main result is that current LLMs, image models, and agent systems perform at or below a random baseline on the composite similarity score, while human performance remains substantially superior. The paper is explicit that it is “directly relevant to any ‘AgentFloor’ concept: an agent that reconstructs floor plans from photos,” and it argues that merely adding agency and tools does not automatically confer the geometry-aware reasoning required for multi-view spatial reconstruction (Petersson et al., 24 Sep 2025).

On the generative side, ActFloor-GAN formulates residential floorplan design as a human-centric, activity-guided two-stage process. It synthesizes a human-activity map and then uses that map to guide pixel-wise room-type prediction via a cycle-consistent conditional GAN. In the reported comparison, the method reaches MSE 0.111, MAE 0.096, and vectorization success 83/100, with a user study indicating stronger topological quality than the cited baseline by Wu et al. This line of work uses “agent” ideas in the sense of simulated human movement and controllable activity fields rather than tool-using LLM agents (Wang et al., 2021).

Another neighboring direction is audio-visual floorplan reconstruction. AV-Map predicts a 2D top-down floorplan from short input video sequences with synchronized ambisonic audio. On 85 large real-world environments, the paper reports that with glimpses spanning 26% of an area, the model estimates the whole area with 66% accuracy, and that audio contributes both geometric and semantic evidence for unseen space. This is a different technical pathway to floor-oriented agent reasoning, grounded in embodied multimodal sensing rather than abstract tool trajectories (Purushwalkam et al., 2020).

A more explicitly topology-first synthesis framework appears in GFLAN, which factorizes floorplan generation into Stage A topological planning and Stage B geometric realization. Stage A sequentially allocates room centroids via probability maps, while Stage B constructs a heterogeneous room-boundary graph and applies a Transformer-augmented GNN to regress room boundaries. The paper’s framing is especially resonant with agentic decomposition: a planner-like module decides where rooms should go, and a refiner-like module decides what precise shapes they should take (Abouagour et al., 18 Dec 2025).

There is also work on vertical localization rather than floorplan generation. The 911-response paper “Predicting Floor-Level for 911 Calls with Neural Networks and Smartphone Sensor Data” describes a pipeline that detects building entry with a neural model and then uses barometric pressure differences to infer the caller’s floor. Across 63 experiments in five NYC buildings, the system is reported to predict the correct floor level with 100% accuracy under the building-conditioned setting. In the supplied synthesis, this is described as “almost exactly an ‘AgentFloor’ system,” but its object is floor index estimation rather than tool-use evaluation or floorplan synthesis (Falcon et al., 2017).

Taken together, these neighboring works indicate that AgentFloor now sits at an intersection of at least three technical themes: benchmarking tool-using agents, reconstructing or generating architectural layouts, and estimating vertical position in buildings. A plausible implication is that future systems may combine these strands: small open-weight models could handle routine tool calls inside larger spatial pipelines, while specialized vision, geometry, or sensing modules handle the genuinely hard floor-oriented reasoning that current generalist agents still fail to solve (Karmakar et al., 1 May 2026, Petersson et al., 24 Sep 2025, Abouagour et al., 18 Dec 2025).