StateEval: Benchmark for Stateful API Workflows

Updated 6 July 2026

StateEval is a benchmark for assessing LLMs on executing stateful API calls and managing evolving program states in multi-step workflows.
It uses the StateGen framework to automatically generate executable tasks across diverse domains such as RESTful services, tensor operations, and voice call setups.
The evaluation focuses on valid sequencing, parameter reasoning, and semantic correctness through automated state validation and constraint checks.

StateEval is a benchmark for evaluating LLMs on realistic, stateful tool use that requires sequential API calls and correct management of evolving program state. Constructed automatically by the StateGen framework, it contains 120 manually verified test cases across three representative scenarios—Session Service, Tensor Operation, and ElevenLabs MCP—and is designed to measure whether an LLM can read API documentation, plan multi-step workflows, choose valid API calls and parameters, and maintain data and control dependencies throughout execution (Huang et al., 13 Jul 2025).

1. Definition, scope, and evaluation target

StateEval measures whether an LLM can understand natural language task descriptions and produce syntactically and semantically correct code that orchestrates a sequence of interdependent API calls to achieve a specified objective. Its scope is realistic, multi-API software tasks in three domains—web services, deep-learning tensor computation, and modern LLM-to-tool interaction via MCP—where correctness depends on both data and control flow. Each case is executable, and correctness is determined by automatic semantic oracles rather than static string matching (Huang et al., 13 Jul 2025).

The benchmark emphasizes four capabilities: valid sequencing under pre-/post-conditions and invariants, parameter reasoning and object state evolution, control-flow handling through if/else branches, and semantic correctness. Its design principles are automated generation from API documentation with a state-machine model and constraint validation, diversity and coverage through energy-based sampling over transition pairs, realistic program structure through a single if–else split per program, high-quality natural language instructions produced with a generator–evaluator LLM multi-agent protocol, and executable tests with recorded state transitions as ground-truth oracles (Huang et al., 13 Jul 2025).

A central misconception addressed by StateEval is that LLM tool-use evaluation can be reduced to isolated function selection or string-level answer checking. StateEval instead targets sequential and stateful constraints, in which correctness depends on preconditions, postconditions, invariants, and inter-call dependencies across an execution trace (Huang et al., 13 Jul 2025).

2. StateGen and reverse-generation of benchmark tasks

StateEval is produced by StateGen, a reverse-generation pipeline that first synthesizes valid API traces as executable programs and then translates them into natural language tasks. The generator maintains a program state space $S$ and parameter space $P$ , with each API call $o \in O$ acting as a transition with side effects on state:

$o: S \times P \to S$

For a state variable $s_j$ affected by transitions $o_j^1,\dots,o_j^t$ , the composed evolution is

$s_j^e = o_j^t(o_j^{t-1}(\dots o_j^1(s_j^0))).$

At generation step $i$ , StateGen computes the feasible action set

$A_i = \{o \in O \mid valid(o, S, O)\},$

using type and shape checks together with documented preconditions, so only feasible calls are considered (Huang et al., 13 Jul 2025).

To promote diversity, StateGen tracks transition pairs linked by shared state, specifically producer–consumer relationships of parameters. Less-frequent pairs are sampled preferentially using an energy-based scheme in which $E \propto 1/(\nu + \epsilon)$ for pair frequency $P$ 0, and sampling follows

$P$ 1

If a candidate introduces a previously unseen transition pair, it is selected for new coverage; otherwise, energy weights are used to avoid overfitting to common local patterns. An ablation reported in the paper shows that StateGen without pair-transition coverage yields lower coverage and slower convergence than the full method (Huang et al., 13 Jul 2025).

Program generation injects a single if–else split at a randomly chosen call site. The if branch continues the current generator, while the else branch uses a cloned generator seeded with current coverage bookkeeping to produce a competing path. The split condition uses a most-recent local variable on the left-hand side and a placeholder on the right-hand side, later replaced by a variable declared in an initialization block. Unreferenced return values are collected into a RESULT variable, which acts as the sink checked during evaluation (Huang et al., 13 Jul 2025).

Instruction translation uses a generator agent and an evaluator agent. The generator transforms initialization plus program into human-like instructions that are unambiguous, natural, and non-redundant; the evaluator iteratively returns either <OK>, suggestions for improvement, or <IMPOSSIBLE>, for up to three negotiation rounds, implemented with GPT-4.1. From 180 generated pairs, manual review removed 7 Session items, 12 Tensor items, and 8 MCP items, leaving 120 high-quality tasks (Huang et al., 13 Jul 2025).

3. Scenarios, APIs, and stateful workflows

StateEval comprises 120 test cases, with 40 per scenario. The three scenarios were chosen to cover stateful RESTful CRUD, tensor computation with shape-sensitive transformations, and LLM tool calling for speech workflows (Huang et al., 13 Jul 2025).

Scenario	Domain	Scale
Session Service	EvoMaster/cBioPortal sessions via MongoDB	40 cases; 5 APIs
Tensor Operation	PyTorch	40 cases; 6 APIs
ElevenLabs MCP	Text-to-speech, transcription, phone calls	40 cases; 6 APIs

In Session Service, the benchmark models RESTful CRUD over session documents stored in MongoDB. The five APIs are POST /sessions, GET /sessions/{id}, PUT /sessions/{id}, DELETE /sessions/{id}, and GET /sessions?userId=.... Pre-conditions include existence of sessionId, userId validity, and schema constraints. Post-conditions include accessibility of created IDs, update visibility in subsequent reads, and deletion effects on list cardinality. Invariants include schema adherence and unique IDs. The benchmark example sequence creates a session, reads it, updates an attribute, lists sessions for the user to verify the update, deletes the session, and confirms disappearance from the list (Huang et al., 13 Jul 2025).

In Tensor Operation, the benchmark uses six PyTorch APIs: four tensor-manipulation operations and two computation modules, torch.nn.Conv2d and torch.nn.Linear. The state constraints are shape algebra across calls, deterministic weight initialization, dtype compatibility, and device and batch semantics. The representative workflow creates a 4D tensor, permutes layout, extracts a centered crop, permutes back to N \times C \times H \times W, applies Conv2d, flattens, and feeds the result to Linear, with the oracle checking both shapes and numerical outputs (Huang et al., 13 Jul 2025).

In ElevenLabs MCP, the six typical tools are listVoices(), createOrUpdateVoice(name, styleParams), textToSpeech(text, voiceId, prosody), saveAudio(audioHandle, path), startOutboundCall(phoneNumber, voiceId, script), and either getCallStatus(callId) or transcribe(audioHandle). Constraints include the existence of voiceId, non-empty text, valid phone-number format, and persistence requirements before transcription if required. Post-conditions include voice/style persistence, successful audio creation, and call-status progression to "completed". Evaluation replaces remote calls with a local mock server that logs state transitions and produces consistent responses (Huang et al., 13 Jul 2025).

4. Formal model, validation, and benchmark metrics

StateEval formalizes sequential API interaction with states $P$ 2 representing both program-local variables and external elements, and actions $P$ 3 consisting of API calls. The transition function is written as

$P$ 4

with each action implementing $P$ 5. Constraints are organized into pre-conditions $P$ 6, parameter constraints $P$ 7, post-conditions $P$ 8, and invariants $P$ 9, such as schema adherence, ID uniqueness, shape and dtype consistency, and monotonic properties like call-status progression (Huang et al., 13 Jul 2025).

Semantic validation compares an execution’s final state $o \in O$ 0 against the recorded reference state $o \in O$ 1:

$o \in O$ 2

if all checked properties match, including IDs present or absent, tensor shapes and values, audio or call status, and RESULT sink contents; otherwise the score is $o \in O$ 3. For branching programs, the oracle records snapshots for both if and else outcomes by toggling the condition variable, so coverage is not confined to a single linear path (Huang et al., 13 Jul 2025).

The main reported success metric is pass@1. The paper states the general form of pass@k as

$o \in O$ 4

but StateEval reports pass@1. Coverage is also quantified through Adjacent Transition Coverage:

$o \in O$ 5

where $o \in O$ 6 is the number of unique APIs in the scenario. The benchmark additionally uses transition-pair coverage internally for StateGen’s sampler (Huang et al., 13 Jul 2025).

The failure taxonomy has three categories: syntax errors, where code fails to parse; execution errors, where runtime failure arises from wrong parameter types, shape mismatches, or invalid IDs; and result errors, where code executes but produces incorrect state transitions or outputs. Dataset-level realism statistics reported by the paper are 717.08 tokens for average instruction length, 442.11 tokens for average reference code length, 7.16 average function calls per sample, Path Depth 2.00, and Binding Count 5.03. Each execution trace contains 5 API calls; the injected if–else can increase total calls in the full program, but each branch’s execution remains at five calls (Huang et al., 13 Jul 2025).

5. Experimental results and diagnostic findings

The evaluation covers five models: Qwen2.5-Coder (32B), Llama-4-Scout (MoE; 17B/109B), Gemini-2.5-Flash, GPT-4.1-nano, and GPT-4.1. Overall pass@1 is 56% for GPT-4.1, 45% for Gemini-2.5-Flash, 35% for GPT-4.1-nano, 34% for Llama-4-Scout, and 25% for Qwen2.5-Coder (Huang et al., 13 Jul 2025).

Model	pass@1
GPT-4.1	56%
Gemini-2.5-Flash	45%
GPT-4.1-nano	35%
Llama-4-Scout	34%
Qwen2.5-Coder	25%

Performance varies sharply by scenario. GPT-4.1 reaches 78% on Tensor Operation, more than double its Session Service performance, and similar trends are reported for Gemini-2.5-Flash and GPT-4.1-nano. The paper states that this suggests tensor operations are better represented in training corpora and documentation. Session Service is the hardest scenario across models, likely because of less common API semantics and intricate data constraints (Huang et al., 13 Jul 2025).

Error distributions are scenario-specific. Execution errors dominate in Session Service and Tensor Operation, typically through invalid IDs or shape incompatibilities. Result errors dominate in ElevenLabs MCP, for example through wrong voice or call parameters and incorrect state transitions detected by the oracles. Some syntax errors persist, particularly around string formatting and f-strings (Huang et al., 13 Jul 2025).

The ablations support the generation methodology. StateGen without pair-transition coverage yields lower coverage and slower convergence than the full system. An LLM-only generation baseline, obtained by prompting GPT-4.1 to write programs directly, produced lower structural diversity and required substantial fixing to execute, consistent with known sampling biases (Huang et al., 13 Jul 2025).

StateEval is positioned against hand-curated or string-matching benchmarks by executing generated programs and verifying semantic correctness through recorded oracles over database states, tensor outputs and shapes, and MCP tool states. It requires reasoning over preconditions, postconditions, invariants, and control flow rather than selecting a single API. The paper further states that its tasks have higher Path Depth and Binding Count than HumanEval, DS-1000, BigCodeBench, and BFCL, and that they involve meaningful shared data across calls (Huang et al., 13 Jul 2025).

The benchmark’s main limitations are also explicit. Manual work is still required to model each scenario’s API documentation into state-machine transitions. Instruction translation quality is mediated by LLM agents, though this is mitigated by multi-agent negotiation and manual verification. External validity is limited to three scenarios, and expanding the domain set is presented as the next test of generality (Huang et al., 13 Jul 2025).

Reproducibility is supported by a Python codebase of approximately 7,800 lines. StateGen uses GPT-4.1 for generator and evaluator agents, with at most three negotiation rounds. The authors state that they will release the codebase and StateEval publicly. The evaluation environments are designed for repeatability: Session Service runs locally with MongoDB-backed state capture; Tensor Operation uses deterministic tensor initialization; and ElevenLabs MCP replaces remote calls with a sandboxed local mock server, thereby avoiding rate limits and nondeterminism (Huang et al., 13 Jul 2025).

The term StateEval also appears in distinct senses in adjacent work. StatEval, which is also referred to in its paper as StateEval, is a benchmark for statistical reasoning with 13,817 foundational problems and 2,374 research-level proof tasks (Lu et al., 10 Oct 2025). GroundEval presents a deterministic, judge-free framework for stateful agent evaluation and explicitly notes that its design aligns with “StateEval” needs where correctness is grounded in state and constraints (Flynt, 22 Jun 2026). SEval-Ex describes summarization consistency evaluation as statement-level StateEval (Herserant et al., 4 May 2025). The State Value Estimation Benchmark (SVEB) is presented as a StateEval instrument for assessing state-value estimation in LLM reinforcement learning (Chen et al., 28 May 2026). ASSESS gives a self-contained description of StateEval as a framework for formal statement similarity via Operator Trees and TransTED Similarity (Liu et al., 26 Sep 2025). This suggests that, beyond the specific benchmark introduced with StateGen, the label has become associated more broadly with evaluation settings in which correctness depends on explicit state, constraints, or structured verification.