EnvSimBench: LLM Environment Simulation Benchmark

• EnvSimBench is a benchmark that objectively evaluates LLM-based simulation by framing environment prediction as a fully observable, single-turn state prediction problem.
• It reformulates traditional simulation by providing the full before-state and code, enabling precise verification of state transitions and observation correctness.
• The benchmark stratifies samples by state-change complexity, revealing a 'state-change cliff' where models succeed in inert cases but falter with multiple simultaneous updates.

EnvSimBench is a benchmark for evaluating and improving LLM-based environment simulation, introduced to test the assumption that LLMs can accurately simulate environmental feedback for scalable agent training (Liu et al., 8 May 2026). The benchmark targets tool-interactive, stateful environments in which each agent action is implemented by deterministic code, and it reframes environment simulation into a fully observable single-turn state prediction problem so that evaluation is objective and LLM-free. In this formulation, the central capability is Environment Simulation Ability, defined as a model’s capacity to predict both the induced state transition and the observation resulting from an action given the current state and the action’s implementation logic (Liu et al., 8 May 2026).

1. Motivation and scope

Training scalable AI agents requires diverse, faithful, low-cost environments. Executable environments are described as expensive, brittle, and hard to scale, while manually crafted environments are also limited in diversity. EnvSimBench is motivated by the alternative paradigm of LLM-simulated environments, in which a model returns environment feedback for each action. The benchmark is built around the claim that this paradigm only works if LLMs can reliably simulate state transitions and observations; otherwise, agents optimize against corrupted rewards and the intended cost savings are offset by failure overheads (Liu et al., 8 May 2026).

The benchmark focuses on tool-interactive, stateful environments where each action is implemented by deterministic code. The examples explicitly listed include banking, logistics, authentication, reservations, and wildlife observations. Each environment maintains a persistent configuration as a structured JSON-like schema with nested objects, lists, and keys, and actions are tool calls with typed arguments. The implementation code encodes transition logic and constraints, including guards, side-effects, and observation formatting (Liu et al., 8 May 2026).

A central design choice is the reduction of environment simulation to a single-turn problem. Standard LLM simulators are described as operating under a POMDP in which the model receives only interaction history and must infer a belief state. EnvSimBench instead supplies the full before-state and the action code, converting evaluation into a fully observable MDP. This removes multi-turn state drift from the benchmarked prediction step and makes each sample independently verifiable. This suggests that EnvSimBench is intended less as a general-purpose agent benchmark than as a controlled diagnostic for transition fidelity and observation correctness (Liu et al., 8 May 2026).

2. Formalization of Environment Simulation Ability

EnvSimBench formalizes an environment as

$E = (S, A, T, O),$

with state space $S$, action space $A$, deterministic transition function $T: S \times A \to S$, and observation function $O: S \times A \to \Sigma^*$. Environment Simulation Ability is then defined as the ability to predict both the next state and the observation produced by an action, conditioned on the current state and the action’s implementation code. The simulator objective is written as

$$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$

and simulation is correct if and only if

$$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$

In practice, the model does not emit a full state directly; it outputs a set of typed state-change operations

$\widehat{\Delta} = \{\delta_1,\ldots,\delta_k\},$

which reconstruct the predicted after-state when applied to the before-state (Liu et al., 8 May 2026).

The benchmark also contrasts this MDP framing with the more typical POMDP regime for LLM simulators. Under the POMDP view, the model receives only the history

$h_t=(a_1,o_1,\ldots,a_{t-1},o_{t-1},a_t)$

and must jointly perform state estimation, transition reasoning, and observation generation:

$\hat{o}_t,\,\hat{s}'_t = f_{\theta}(h_t).$

EnvSimBench instead supplies the full state and code:

$S$0

This design isolates simulation ability from hidden-state inference and enables exact stepwise verification (Liu et al., 8 May 2026).

The operationalization emphasizes transition fidelity. EnvSim Ability can be reported as the tuple $S$1 by stratum, with Config Match treated as the primary reasoning metric. A plausible implication is that EnvSimBench distinguishes sharply between plausible textual feedback and actual state correctness, rather than treating response fluency as a sufficient indicator of simulator quality.

3. Benchmark composition and stratification

EnvSimBench contains 400 independently verifiable single-turn samples drawn from 167 distinct tool-interactive environments, seeded from 191 EnvScaler environments (Liu et al., 8 May 2026). Each sample is a quadruple $S$2 together with the action’s implementation code $S$3, and labels are obtained by executing the real environment. A diversity rule selects samples to maximize distinct environment coverage per subgroup.

The benchmark is stratified along three axes. The first is action outcome, separating Success from Failure. The Failure group is defined by $S$4 together with an error observation, and it tests whether models correctly withhold state changes when guards are not satisfied. The second is state-change complexity, measured by the cardinality of the change set $S$5. The third is input argument cardinality, which further partitions Failure and No-Change samples by whether the tool has 0 arguments or at least 1 argument (Liu et al., 8 May 2026).

The quantitative breakdown is as follows:

Subgroup	Definition	Count
Failure	$S$6 with an error observation	20
No-Change	$S$7	80
Simple	$S$8	50
Medium	$S$9	200
Difficult	$A$0	50

Within these groups, the benchmark further specifies that No-Change contains 40 samples each for 0-arg and $A$1-arg cases; Simple contains 25 samples per $A$2 and $A$3; Medium contains 50 samples per $A$4; and Difficult is distributed across $A$5 (Liu et al., 8 May 2026). The distribution is said to mirror empirical frequencies across source environments and to concentrate diagnostic resolution in the Medium tier where failure patterns first appear.

This stratification is important because it separates qualitatively different demands. No-change and failure cases test whether a simulator invents updates. Small-$A$6 success cases test local state tracking. Larger-$A$7 cases test simultaneous multi-field updates, auxiliary side-effects, and the preservation of consistency between feedback and state. The benchmark’s structure therefore supports failure analysis at the level of transition cardinality rather than only overall accuracy.

4. Labeling, validators, and scoring protocol

Ground-truth labels are produced by a deterministic external Python executor that runs $A$8 on $A$9. No LLM is used in labeling (Liu et al., 8 May 2026). This is the basis for the benchmark’s claim of independently verifiable supervision.

For each sample, a model must output both a predicted observation string $T: S \times A \to S$0 and a set of typed state-change operations $T: S \times A \to S$1. The predicted after-state is reconstructed as

$T: S \times A \to S$2

A prediction is counted as correct if and only if both exact conditions hold:

$T: S \times A \to S$3

The validation procedure parses the structured output, computes $T: S \times A \to S$4, reconstructs $T: S \times A \to S$5, compares $T: S \times A \to S$6 with $T: S \times A \to S$7, compares $T: S \times A \to S$8 with $T: S \times A \to S$9, records two booleans, and aggregates them by group (Liu et al., 8 May 2026).

The two core metrics are Feedback Match and Config Match:

$O: S \times A \to \Sigma^*$0

$O: S \times A \to \Sigma^*$1

Aggregate accuracy over a subset $O: S \times A \to \Sigma^*$2 is computed by averaging FM or CM over that subset. A joint correctness indicator is also defined:

$O: S \times A \to \Sigma^*$3

The scoring rules explicitly prioritize CM as the primary reasoning metric because it is robust to formatting conventions, whereas FM captures surface-response correctness but is vulnerable to format mismatches (Liu et al., 8 May 2026).

The prompting protocol includes the full before-state $O: S \times A \to \Sigma^*$4, the tool name and typed arguments $O: S \times A \to \Sigma^*$5, and the implementation code $O: S \times A \to \Sigma^*$6. No multi-turn history is provided, and frontier models are evaluated through provider APIs in non-thinking mode with identical prompts. The specialized simulator is trained with supervised fine-tuning, using internal reasoning traces during training in some settings, but it outputs only structured predictions at inference (Liu et al., 8 May 2026).

5. Empirical findings and the “state-change cliff”

EnvSimBench reports systematic evaluations of DeepSeek-V3.2, Qwen3.5-397B-A17B, GPT-5.4, Gemini-3.1-Pro-Preview, Claude-Sonnet-4.6, MiniMax-M2.7, and GLM-5 (Liu et al., 8 May 2026). The principal empirical finding is a universal “state-change cliff”: models perform near-perfectly when the state remains invariant and fail sharply when multiple state variables must be updated simultaneously.

On state-preserving operations, defined as Failure plus No-Change, all models achieve near-perfect CM of 97–100%. This is presented as evidence that none hallucinates changes when none are warranted. FM varies much more because of formatting conventions; Claude-Sonnet-4.6 and MiniMax-M2.7 are reported to “unwrap” structured responses, which produces FM as low as 10–15% despite 99–100% CM (Liu et al., 8 May 2026). This clarifies a common misreading of benchmark results: low textual exact-match can coexist with correct state prediction.

On state-changing operations, CM degrades rapidly as $O: S \times A \to \Sigma^*$7 increases. For Simple cases with $O: S \times A \to \Sigma^*$8, CM ranges from 22% for DeepSeek-V3.2 to 50% for MiniMax-M2.7, while FM often remains at least 80–96%. For Medium cases with $O: S \times A \to \Sigma^*$9, CM drops to 8–18% across models, whereas FM remains 65–90%. For Difficult cases with $$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$0, CM approaches 0–28% while FM often remains moderate to high; examples include GPT-5.4 with FM 74% and GLM-5 with FM 70% (Liu et al., 8 May 2026).

Overall, CM ranges from 32.5% for DeepSeek-V3.2 to 42.3% for Qwen3.5-397B-A17B, while FM ranges from 25.5% for Claude-Sonnet-4.6, attributed to formatting, to 80.5% for GLM-5 (Liu et al., 8 May 2026). The per-$$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$1 analysis makes the cliff more precise: once $$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$2, all models show a sharp drop; by $$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$3, CM is near zero for all evaluated systems, including 2.7% for DeepSeek-V3.2, 6.0% for Claude-Sonnet-4.6, 9.3% for Gemini-3.1-Pro-Preview, 10.7% for MiniMax-M2.7, 12.0% for GPT-5.4, and 11.3% for GLM-5 (Liu et al., 8 May 2026).

The benchmark further highlights FM–CM decoupling and silent corruption. On samples with $$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$4 where CM fails, 50–64% of failures for several models still have FM=1, and detailed per-$$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$5 breakdowns are described as often showing 60–100% FM among CM failures (Liu et al., 8 May 2026). In these cases, the agent receives plausible feedback while the state is wrong. The reported failure modes are hallucination, logical inconsistency, and state drift or statelessness. Hallucination includes fabricated constraints or invented entities not present in code. Logical inconsistency includes feedback claiming one outcome while the predicted state encodes another. State drift in the POMDP regime, or statelessness in the single-turn MDP regime, includes ignoring the provided before-state or missing runtime-dependent fields such as timestamps and auto-generated IDs.

6. Constraint-driven simulation, downstream use, and limitations

To address these failures, the paper introduces a constraint-driven simulation pipeline with four components: constraint exposure, schema enforcement, state tracking, and, in some settings, reasoning traces during training (Liu et al., 8 May 2026). Constraint exposure means providing the full before-state $$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$6 and $$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$7 in the prompt so that transition logic is explicit. Schema enforcement requires structured outputs with typed change operations $$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$8 and a schema-conforming observation, which validators then compare against the executor-produced ground truth. State tracking is handled by the single-turn MDP formulation, which eliminates multi-turn drift and allows each step to be independently verified.

Under this paradigm, a 4B specialized simulator called Full-Balance2 achieves 45.3% overall CM and 79.5% FM, surpassing all frontier LLMs on CM by 3.0 percentage points over Qwen3.5-397B-A17B and nearly matching the best reported FM of 80.5% from GLM-5 (Liu et al., 8 May 2026). It outperforms frontier LLMs by up to 10 percentage points CM for $$(\hat{o},\,\hat{s}') = f_{\theta}(s,\,a,\,code(a)),$$9, though all methods converge near zero beyond the cliff at $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$0. In downstream synthesis, replacing a large-model ensemble with Full-Balance2 in EnvScaler increases the number of passing environments from 191 to 204, corresponding to a 6.8% synthesis yield improvement, with approximately 59 times fewer parameters and costs reduced by over 90%. The cost reduction is expressed as

$$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$1

The reported ablations indicate that composition matters more than volume at 5–6K samples. Balance2, which mirrors the empirical $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$2 distribution, yields the best overall CM of 45.3% and FM of 79.5%. Full-parameter supervised fine-tuning outperforms LoRA on CM, with 35.0% versus 31.8% at a 5K data budget, and the largest advantage appears in the Simple tier at 34% versus 20%. Reasoning traces do not consistently improve CM at 5K samples, with 30.3% with traces versus 35.0% without, suggesting that traces may require more data or selective application to complex samples (Liu et al., 8 May 2026).

The benchmark release includes benchmark samples, executor-based labels, prompts, and evaluation scripts in a public repository. Each sample includes the JSON before-config $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$3, the tool call $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$4, the Python implementation $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$5, and the ground truth $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$6, $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$7, and $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$8. Reproduction consists of setting up the deterministic Python executor and verification scripts, querying a model with the MDP triple $$\hat{s}' = T(s,a)\quad\text{and}\quad \hat{o} = O(s,a).$$9, parsing the structured output to extract $\widehat{\Delta} = \{\delta_1,\ldots,\delta_k\},$0 and $\widehat{\Delta} = \{\delta_1,\ldots,\delta_k\},$1, computing FM and CM via exact equality, and aggregating results by group and by $\widehat{\Delta} = \{\delta_1,\ldots,\delta_k\},$2 (Liu et al., 8 May 2026). For integration into agent training, the simulator can be inserted as the environment backend in an overview pipeline such as EnvScaler, with CM used as the primary reliability signal and synthesized environments gated by pass accuracy thresholds such as at least 0.85.

The paper also identifies several limitations and future directions. Some strata have high sampling variance, notably Failure with $\widehat{\Delta} = \{\delta_1,\ldots,\delta_k\},$3 and Difficult sub-bins with approximately 4–6 samples, so future releases are said to require bootstrap error bounds. Overlap between training data and synthesis filters can bias downstream validation, motivating expanded holdouts and more hard samples with $\widehat{\Delta} = \{\delta_1,\ldots,\delta_k\},$4. The Difficult tier currently conflates bulk-uniform operations with heterogeneous multi-field updates, and future versions are proposed to separate these cases. Runtime values remain a structural challenge, because without actual execution models cannot infer clock-dependent fields or auto-generated keys; hybrid simulators that call minimal executors for such fields are suggested as a possible bridge (Liu et al., 8 May 2026).

EnvSimBench therefore serves as a fully observable, verifiable framework for measuring Environment Simulation Ability, diagnosing the discrepancy between plausible feedback and correct state transitions, and testing constraint-driven remedies for reliable LLM-based environment simulation (Liu et al., 8 May 2026). Its central result is that current systems can preserve state when no change is required but exhibit a universal failure mode once multiple simultaneous updates must be coordinated, with direct implications for reward integrity and the scalability of agent-training pipelines.