AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scale

Abstract: Agents learn to act through interaction with environments, yet the environments used for training are often manually constructed or synthesized around predefined tasks and benchmarks. This task-centric paradigm makes it difficult to scale environments that reflect realistic and evolving workflows where diverse tasks can naturally emerge from the underlying world. We introduce AgentMercury, a scalable framework for synthesizing executable environments from high-level business scenarios. Rather than constructing an environment for a specific task, AgentMercury first instantiates a persistent world with entities, services, tools, state, and executable cross-service invariants, from which diverse tasks and interaction trajectories can subsequently emerge. We construct 4,783 executable environments spanning 14 industries and 50 countries, and use them as training substrates for reinforcement learning. Despite being generated without targeting the evaluation benchmarks, policies trained on these business-oriented environments improve substantially on both enterprise workflows and out-of-domain benchmarks spanning reasoning, coding, scientific computing, and tool use. In our experiments, Qwen3.5-4B improves from 12.3 to 15.7 on EnterpriseOps-GYM and from 45.9 to 56.0 on AIME26 after training on AgentMercury environments. We further show that the construction process itself can be learned: fine-tuning Qwen3.5-35B-A3B on construction traces increases executable-world authoring success from 3.3% to 83.3% on held-out business scenarios. These results show that scenario-grounded environments can provide useful and generalizable learning signals beyond benchmark-specific training, while their construction can itself become a learnable capability.

• The paper presents the AGENTMERCURY framework, which reorganizes agent-environment interfaces for high-level business scenarios.
• AGENTMERCURY constructs persistent, stateful, multi-service environments, enabling agents to learn from multiple tasks and interactions sampled from coherent business worlds.
• Training on executable environments improves metrics in various benchmarks like ENTERPRISEOPS-GYM on 8 tasks, including AIME26 by over 100% and SCICode by over 12%, by 14 on average.

AGENTMERCURY proposes a reorganization of the agent-environment interface around three distinct roles: PLANET constructs the world, a task generator instantiates objectives within that world, and a policy acts through executable tools. The paper’s central claim is that scalable agent training should not be organized exclusively around benchmark-specific tasks. Instead, high-level business scenarios should be compiled into persistent, stateful, multi-service environments from which multiple tasks and interaction trajectories can be sampled. The framework is intended to make environment construction an explicit and potentially learnable component of the agent system (2608.20634).

Scenario-grounded environment construction

The paper identifies a structural limitation in conventional agent environments. Manually authored environments and task-conditioned synthesis pipelines generally begin with a predefined task, user instruction, or benchmark specification. This facilitates controlled evaluation, but it couples the underlying world to the objective used to define it. Increasing task count can therefore increase instance volume without substantially increasing diversity in services, state transitions, entities, or cross-system dependencies.

AGENTMERCURY reverses this construction order. Given a high-level business scenario $\sigma$, PLANET produces an executable world $w$ containing:

• entities and company identity;
• a service graph;
• persistent state schemas and seeded data;
• executable tools and observation interfaces;
• transition dynamics;
• world-level invariants represented as executable verification conditions.

Tasks are generated only after the world exists. A task generator modifies the initial state, produces a natural-language objective, and derives a task-specific rubric. Multiple task instances can consequently share the same services, state schema, transition logic, and invariants while differing in user intent, initial conditions, and required actions.

This factorization is technically important because it separates three types of semantics that are often conflated. The transition function determines how tool calls modify the software state. The task rubric determines whether a particular objective has been achieved. World-level invariants specify cross-service properties that should hold but are not automatically enforced by the simulator. For example, a workflow may require an agent to create a downstream record after an upstream business event. The environment does not perform that action on the agent’s behalf; it evaluates the resulting state against the invariant after interaction.

The distinction between visible and hidden invariant views further separates agent information from evaluation logic. An invariant may be discoverable through documents, policies, or service interactions, while its hidden executable form is retained by the grader. This arrangement allows the agent to reason about requirements without being given a direct evaluation oracle, while preserving deterministic post-episode verification.

Executable worlds and deterministic grading

The environments are implemented as executable software worlds rather than static text-generation tasks. Agents interact through MCP-style tools, receive observations through an environment interface, and modify persistent state through deterministic transitions. With a fixed initial seed and action sequence, trajectories can be replayed and re-evaluated. The resulting “golden reasoning trace” provides a reproducible basis for reward computation and failure analysis.

The grader evaluates the terminal state and trajectory against task-specific assertions and hidden world-level constraints. Whenever possible, these checks operate directly over database state and include record-existence, field-equality, and field-inequality assertions. This design is particularly suitable for long-horizon tool use because textual plausibility is insufficient: a successful trajectory must produce the required state changes across the relevant services.

The framework therefore avoids relying exclusively on LLM-based judging for the central training signal. The paper does use an LLM judge in selected interactive benchmark evaluations and includes a reward implementation with behavior penalties, but the synthesized environments themselves are grounded in programmatic execution and verification. This improves reproducibility and reduces ambiguity in reward assignment, although it does not eliminate possible specification errors in the generated worlds.

Scale and structural composition

The paper reports 4,783 executable environments spanning 14 industries and 50 countries. The RL corpus is described more specifically as containing 43,300 tasks instantiated from 4,326 company environments. These environments expose heterogeneous tools, state tables, and service combinations. The reported task corpus includes 2,287 industry descriptions, 842 tools, 222 state-table types, and 148 service combinations. Each environment exposes between 10 and 26 tools, with a mean of 16.1; each task touches between one and five services, with a mean of 2.9; and each task contains a mean of 5.4 programmatic assertions.

Corpus property	Reported value
Executable tasks	43,300
Company environments used for RL	4,326
Industry descriptions	2,287
Unique tools	842
State-table types	222
Service combinations	148
Tools per environment	10–26, mean 16.1
Services per task	1–5, mean 2.9
Assertions per task	2–10, mean 5.4

The corpus contains substantial cross-system structure: 25,991 of the 43,300 tasks are flagged as involving cross-system action risk. In the principal Qwen3.5-4B GRPO run, only 3,200 task instances received gradients over 200 training steps, representing 7.4% of the full task corpus. Nevertheless, those tasks covered 53.5% of environments, 62.9% of industries, 75.8% of tools, 61.7% of state tables, and 62.8% of service combinations. The authors use this result to argue that world-level coverage, rather than raw task count, is a more informative measure of training diversity.

The appendix qualifies this diversity claim. Textual descriptions are often generated from shared templates, and nearest-neighbor analysis yields a median TF-IDF similarity of 0.83, with 8.3% of tasks having a nearest neighbor above 0.99. Only 17% of these near-twin pairs originate from the same environment, suggesting that redundancy is driven substantially by shared templates across environments rather than merely by repeated seeds. More meaningful separation appears in tool-set space, where tasks cluster according to executable service structure. However, the number of exposed tools takes only five values—10, 14, 18, 22, and 26—because each additional resource contributes a fixed group of four tools. Thus, the task space has substantial operational variation but is not an unconstrained continuum.

Policy optimization results

AGENTMERCURY trains Qwen3.5 policies with GRPO and, at the larger model scale, SAO. The training environments are constructed independently of the evaluation benchmark tasks. This experimental separation is intended to test whether the environments provide transferable policy-learning signals rather than reproducing benchmark-specific task distributions.

On ENTERPRISEOPS-GYM, Qwen3.5-4B improves from an average score of 12.3 to 15.7 after GRPO training, a gain of 3.4 points or 27.6%. The strongest domain-level improvements occur in Drive and Email, each increasing by 9.4 points, from 6.2 to 15.6 and from 23.9 to 33.3, respectively. Hybrid increases from 11.7 to 17.0, Calendar from 10.4 to 13.1, Teams from 20.8 to 23.0, and HR from 9.5 to 10.8. CSM is the exception, declining from 9.2 to 5.6. The aggregate gain therefore coexists with a domain-specific regression, indicating that transfer is not uniformly distributed across enterprise workflows.

At the 35B-A3B scale, the base average is 24.8. GRPO raises it to 28.1, while SAO raises it to 28.3. These correspond to improvements of 13.3% and 14.1%, respectively. Both methods improve all eight enterprise domains in the reported results. The comparable performance under two optimization procedures supports the claim that the learning signal is not specific to GRPO, although the experiments do not establish equivalence between the algorithms or identify which environmental properties drive the gains.

Model and training	EnterpriseOps-GYM average
Qwen3.5-4B	12.3
Qwen3.5-4B + GRPO	15.7
Qwen3.5-35B-A3B	24.8
Qwen3.5-35B-A3B + GRPO	28.1
Qwen3.5-35B-A3B + SAO	28.3

The out-of-domain results are the paper’s strongest empirical support for scenario-grounded training. Qwen3.5-4B + GRPO improves on AIME26 from 45.9 to 56.0, HMMT from 28.5 to 35.4, LiveCodeBench from 36.6 to 44.0, SciCode from 22.6 to 25.7, BFCL from 30.3 to 31.7, GPQA-Diamond from 76.5 to 77.5, Tau-3 Airline from 48.8 to 58.7, and Tau-3 Retail from 70.4 to 73.6. Tau-3 Telecom slightly decreases from 92.5 to 91.9.

The transfer is particularly pronounced for the smaller model on mathematical reasoning and coding, despite those benchmarks not being represented as target tasks during environment construction. At 35B-A3B scale, the absolute improvements are smaller on already strong reasoning benchmarks but remain substantial for tool-oriented evaluation. SAO increases BFCL from 31.1 to 42.1, LiveCodeBench from 74.3 to 78.6, HMMT from 77.0 to 83.3, and GPQA-Diamond from 82.8 to 84.0. The BFCL result is consistent with the hypothesis that training on structured, multi-step tool interactions preferentially improves function selection and argument formation.

The authors also report reduced variance on interactive benchmarks. For example, Qwen3.5-35B-A3B’s base performance on Tau-3 Telecom is 49.1 ± 49.8, compared with 65.5 ± 23.8 after SAO training. On Retail, the standard deviation falls from 22.1 to 10.2, while Airline improves from 39.1 ± 10.6 to 50.9 ± 8.1. The implication is not merely higher expected reward but more consistent sequential behavior across sampled interactions. This interpretation is plausible for stateful tool-use tasks, although the number of independent evaluation runs is only three and the reported variance estimates should therefore be treated cautiously.

Training diagnostics indicate that reward increases while truncation declines and degenerate-response rates remain near zero. For the 4B GRPO run, reward rises toward approximately 0.55–0.60, while truncation falls from roughly 0.35 to nearly zero. For the 35B runs, GRPO reward increases from about 0.4 to 0.55–0.60, and SAO reward rises from approximately 0.35 to above 0.6. These observations argue against a simple reward-hacking explanation based on pathological output growth or response collapse. They do not, however, prove that the reward captures all aspects of competent business behavior; the environments’ own validators remain the operative definition of success.

Learning to author executable worlds

The second experiment treats environment construction as an agentic capability. Models receive held-out high-level business briefs and must produce executable world specifications. Success requires passing all 12 structural validators, rather than merely generating plausible text.

Strong API models achieve zero-shot oracle-pass rates ranging from 66.7% to 90.0%. The paper reports a mean of 80.7% in the detailed authoring analysis, while one displayed summary reports 78.3%; this inconsistency is not resolved in the text. Recipe conditioning, which supplies an invariant digest and a trimmed construction exemplar, does not reliably improve performance. Across the API models, the mean reportedly decreases from 80.7% to 78.0%, and exact McNemar tests yield $p \geq 0.375$ for every individual model. The result contradicts the straightforward expectation that more explicit procedural information should monotonically improve structured generation.

The dominant failure mode concerns cross-service invariants. Models frequently place both the trigger and target of an invariant within the same service, thereby preserving superficial syntactic validity while violating the intended inter-service dependency. GPT-5.4 exhibits this error on 10 of the 30 held-out briefs. Recipe conditioning can also increase formatting and parsing failures by imposing a longer output protocol. These results establish why executable validation is necessary: surface-level inspection cannot reliably detect semantically incorrect service graphs or invalid cross-service constraints.

Construction traces provide a substantially stronger intervention. The authors fine-tune Qwen3.5-35B-A3B on 29,823 samples covering brief-to-world generation, intermediate-stage completion, validator-guided repair, and intent-to-diff prediction. Full oracle success rises from 3.3% to 83.3% on the 30 held-out briefs. The model passes an average of 11.5 of 12 validators, with truncation and formatting failures nearly eliminated. A Fisher exact test reports $p = 1.2 \times 10^{-10}$, and the reported Wilson 95% confidence intervals are [0.6, 16.7] for the base model and [66.4, 92.7] after fine-tuning.

The recipe interaction produces an especially strong and contradictory result. For the base Qwen3.5-35B-A3B, recipe conditioning increases success from 3.3% to 20.0%. After fine-tuning, the same recipe reduces success from 83.3% to 10.0%; 27 of 30 recipe-conditioned generations fail cross-service validation. This asymmetry suggests that parameter-level learning and in-context procedural prompting are not interchangeable. Once the construction procedure is internalized, an external recipe may interfere with the learned serialization format or structural policy. The finding also argues against interpreting the fine-tuning gain as mere generic instruction following.

Limitations and open questions

The evidence is persuasive for feasibility but narrower than the framing of general environment scaling. The environment distribution is limited to synthetic business scenarios, and its realism is established primarily through structural properties—services, tools, state, and validators—rather than through comparison with operational enterprise traces or expert judgments of workflow fidelity. The scenarios, company identities, schemas, and invariants are generated within one construction pipeline, so correlated errors between authoring and grading remain possible.

The transfer experiments also lack several controls needed to isolate the source of improvement. The paper does not report matched training on an equal number of task-centric environments, ablations removing persistent state or cross-service invariants, or comparisons against synthetic tasks with equivalent tool and token budgets but reduced world structure. Consequently, the results demonstrate that AGENTMERCURY training is useful, but do not establish that scenario-grounded construction is superior to all alternative forms of executable task synthesis.

The out-of-domain evaluation is based on three independent runs, and some benchmark results show substantial variance, particularly for Tau-3. Several evaluations use repeated sampling per problem or task, so the effective statistical dependence and confidence intervals are not fully characterized. Moreover, the 4B SAO run was excluded from the principal comparison because its learning signal was judged insufficiently stable. This is a reasonable methodological decision, but it limits conclusions about optimizer–scale interactions.

The authoring evaluation uses only 30 held-out briefs. The large change from 3.3% to 83.3% is statistically strong under the reported test, but the interval remains broad because of the small sample. The discrepancy between the reported authoring means—78.3% in the displayed results and 80.7% in the accompanying analysis—also warrants clarification. Finally, PLANET is not yet coupled to policy failures or a learned world model. The generator does not select new scenarios adaptively based on capability gaps, and the system therefore does not yet demonstrate a closed environment-generation and policy-learning loop. The specific unresolved question is whether a policy can identify informative missing interaction structures and cause PLANET to synthesize environments that improve those capabilities without introducing exploitable artifacts.

Conclusion

AGENTMERCURY presents executable business worlds as the primary unit of scalable environment construction. Its separation of world synthesis, task instantiation, interaction, and deterministic grading supports persistent multi-service workflows and reusable training substrates. Training on 4,783 synthesized environments yields measurable gains on enterprise and unrelated reasoning, coding, scientific-computing, and tool-use benchmarks, while construction traces raise held-out world-authoring success from 3.3% to 83.3%. The results support treating environment construction as both a first-class systems role and a learnable capability, while leaving open whether the same framework can achieve empirically validated realism, stronger causal transfer controls, and adaptive generation driven by policy-level deficiencies.