AgentWorld Dataset: Dual-Modal Interactions

• AgentWorld Dataset is a research resource featuring two distinct modalities: embodied mobile manipulation for robotics and text-based interaction trajectories for language agents.
• The robotics variant supports household mobile manipulation with RGB images, proprioception, and dual-mode controls for both simple and multistage tasks.
• The language corpus comprises millions of text-based trajectories across seven domains, fostering next-observation prediction and advanced world model training.

“AgentWorld Dataset” denotes two distinct research artifacts that share a concern with environment interaction but differ fundamentally in modality, representation, and downstream purpose. In the mobile-robotics literature, AgentWorld is a dataset produced by an interactive simulation platform for household mobile manipulation, combining automated scene construction with dual-mode teleoperation and covering primitive and multistage tasks in living rooms, bedrooms, and kitchens (Zhang et al., 11 Aug 2025). In the language-agent literature, AgentWorld is a very large, unified corpus of environment interaction trajectories used to train Qwen-AgentWorld, a language world model that predicts next observations across seven agentic domains; in that setting, AgentWorldBench is a separate evaluation benchmark rather than the training corpus (Zuo et al., 23 Jun 2026).

1. Scope and naming

The shared name “AgentWorld” reflects two different notions of an environment-interaction dataset. The robotic AgentWorld Dataset is centered on embodied mobile manipulation in domestic scenes. Its observations are RGB images at $480\times640$ resolution plus robot proprioception, and its actions include arm commands, end-effector commands, base velocity commands, and a binary selector between locomotion and manipulation modes (Zhang et al., 11 Aug 2025).

By contrast, the AgentWorld corpus for Qwen-AgentWorld is text-native. All data follow a unified trajectory schema in which a system prompt encodes domain rules and constraints, and each turn is an $(\mathrm{action}_t,\mathrm{observation}_t)$ pair. The paper formalizes this as

$$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$

Its domains are MCP, Search, SWE, Terminal, Android, Web, and OS, with GUI environments represented by accessibility trees or view hierarchies rather than pixel frames (Zuo et al., 23 Jun 2026).

The two resources are therefore related at the level of interaction modeling, not at the level of data format or benchmark protocol. A plausible implication is that the name identifies a research theme—environment-grounded agents—rather than a single canonical dataset.

2. Mobile-manipulation AgentWorld: task taxonomy and scale

The robotic AgentWorld Dataset was created to enable training and evaluation of household mobile manipulation skills that combine navigation and dexterous interaction with everyday objects and articulated furniture. It targets both primitive operations—pick-and-place, open-and-close, push-and-pull—and multistage, long-horizon activities in realistic domestic settings, including serving drinks, organizing books, cleaning dishes, and stowing or heating food (Zhang et al., 11 Aug 2025).

Each task’s “Num of Trajectories” is reported as “number of assets × number of recorded sequences.” The dataset contains 5,050 trajectories across all categories.

Category	Representative tasks	Total trajectories
Pick & Place	Pick objects and place them in the bowl; pick objects and place them into the opened microwave/fridge/stove	800
Open & Close	Doors of rooms; doors of fridges/microwaves/stoves	800
Push & Pull	Pull out/push in drawers; push buttons on microwaves/stoves; push objects on tables away from the robot	500
Living Room	Organize books; pour the drink; throw away trash	1,150
Bedroom	Put the pillow on top; hang clothes; pick up alarm clock and push the button	1,150
Kitchen	Store dishes into the fridge; put food in the microwave; clean the dishes	650

The detailed counts reported for primitive tasks are as follows. Pick-and-place includes 100 trajectories for picking objects and placing them in the bowl, 300 for placing objects into the opened microwave/fridge/stove, 100 for picking objects out from the bowl, and 300 for picking objects out from the opened microwave/fridge/stove. Open-and-close includes 200 trajectories for room doors and 600 for appliance doors. Push-and-pull includes 200 for drawers, 200 for appliance buttons, and 100 for pushing objects on tables away from the robot (Zhang et al., 11 Aug 2025).

The multistage categories are organized by room type. In the living room, the tasks are “Pick up books on the table, walk to the shelf, organize,” “Pick up the pitcher and the cup and pour the drink,” and “Pick up trash on the table, walk to the trash bin, throw away,” for a total of 1,150 trajectories. In the bedroom, the tasks are “Slide objects on the bed away and put the pillow on top,” “Pick up clothes rack, open closets, hang,” and “Pick up alarm clock and push the button,” also totaling 1,150 trajectories. In the kitchen, the tasks are “Pick up dishes on the table and store into the fridge,” “Pick up food, turn around to open the microwave, put food in,” and “Pick up dishes on the rack and the sponge, clean the dishes,” totaling 650 trajectories (Zhang et al., 11 Aug 2025).

The object and asset taxonomy includes bowls, books, pitcher, cup, trash, pillow, clothes rack, alarm clock, dishes, sponge, and food items, together with articulated doors, drawers, and appliance buttons. This combination makes the dataset notable for coupling mobile base movement with articulated-object interaction inside room-scale tasks. The supplementary material, however, does not report the number of scenes or houses, exact counts of living rooms, bedrooms, and kitchens, frame rates, per-episode lengths, total hours, or train/val/test splits (Zhang et al., 11 Aug 2025).

3. Robotic platform, control parameterization, and scene construction

The AgentWorld simulation platform combines automated scene construction with teleoperation for data collection. The demonstrated scene-construction capabilities include automated layout generation with different room organizations, including stairs; semantic asset selection and placement; material configuration for walls, floors, and assets with varied materials; and interactive physics simulation with assets exhibiting collisions and joints (Zhang et al., 11 Aug 2025).

The platform supports Unitree G1 and H1 humanoids, each with 7-DOF arms; the Franka Emika Panda arm with 7 DOFs; the DOBOT X-Trainer dual-arm robot with 6 DOFs per arm; and the TRX-Hand5 anthropomorphic hand with integrated tactile feedback and 18 actuated DOFs. Dual-mode control is explicit: the action vector contains a binary flag that selects between locomotion and manipulation modes, enabling a single policy to switch between moving through rooms and interacting with objects (Zhang et al., 11 Aug 2025).

The reported observation space is

$s = [s_{qpos}, s_e, s_f],$

where $s_{qpos}$ denotes arm joint positions, $s_e$ denotes end-effector poses, and $s_f=(x,y,\theta)$ denotes mobile-base plane position and yaw. The action space is

$a = [a_{qpos}, a_e, a_f, 0/1],$

where $a_{qpos}$ are arm joint commands, $a_e$ is the end-effector command, $(\mathrm{action}_t,\mathrm{observation}_t)$0 are mobile-base velocity commands, and $(\mathrm{action}_t,\mathrm{observation}_t)$1 is the mode selector. For grippers, a single normalized value controls opening distance (Zhang et al., 11 Aug 2025).

For humanoid locomotion, the policy uses a foot trajectory generator from Lee et al. (2020) to produce a reference foot trajectory, and the policy outputs increments added to that reference. Random joystick commands are varied over time so that the objective is to follow diverse control inputs. Every 150 frames, a uniform-sampled offset is superimposed onto the default arm target joint positions to increase robustness to varied upper-limb poses. Termination conditions are base_contact, for fall detection when the body touches the ground, and time_out, for maximum episode time. The rewards encourage tracking commanded yaw or angular velocity and planar linear velocities, penalize $(\mathrm{action}_t,\mathrm{observation}_t)$2-axis velocity fluctuations, foot sliding, high joint accelerations or torques, joint limit violations, and joint deviations of arms, hip, and torso from defaults, and also encourage feet air time; the paper does not report reward weights (Zhang et al., 11 Aug 2025).

The supplementary material does not report the physics engine, friction and mass properties, collision or contact parameters, scene-generation algorithms, placement heuristics, camera intrinsics or extrinsics, the number or placement of cameras, depth or segmentation streams, semantic labels, affordances, task graphs, success flags, or contact-event annotations. This limits exact reconstruction of the platform while leaving the observation and action structures explicit (Zhang et al., 11 Aug 2025).

4. Robotic learning baselines, benchmark setup, and reported gaps

The robotic paper benchmarks imitation-learning methods including Behavior Cloning (BC), Action Chunking Transformer (ACT), Diffusion Policy (DP), and $(\mathrm{action}_t,\mathrm{observation}_t)$3, described as a vision-language-action flow model. BC uses a ResNet-18 image encoder; flattened visual features are concatenated with proprioceptive state; a 5-layer MLP head predicts actions; and layer normalization with $(\mathrm{action}_t,\mathrm{observation}_t)$4 is applied to the output. ACT, Diffusion Policy, and $(\mathrm{action}_t,\mathrm{observation}_t)$5 use LeRobot implementations. Action chunk sizes are 20 for basic tasks and 40 for multistage tasks, and temporal ensemble is applied for smoother actions (Zhang et al., 11 Aug 2025).

The reported training setup is uniform across methods except for $(\mathrm{action}_t,\mathrm{observation}_t)$6: batch size is 16 for BC, ACT, and Diffusion Policy, and 8 for $(\mathrm{action}_t,\mathrm{observation}_t)$7; training steps are 50K for basic tasks and 150K for multistage tasks; convergence is observed at 80K–100K steps; the hardware is an NVIDIA L20 GPU; and the approximate training durations are 6 hours for BC, 12 hours for ACT and Diffusion Policy, and 18 hours for $(\mathrm{action}_t,\mathrm{observation}_t)$8 (Zhang et al., 11 Aug 2025).

The paper also states the standard objectives used in these families. For behavior cloning, the loss is

$(\mathrm{action}_t,\mathrm{observation}_t)$9

for discrete actions when applicable, or

$$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$0

for continuous actions. For Diffusion Policy, the forward noising process is

$$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$1

with denoising objective

$$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$2

For ACT, the chunk-prediction objective is

$$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$3

For $$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$4, cross-entropy over actions conditioned on visual-language tokens is identified as standard: $$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$5 Attention-mechanism details for ACT and conditioning-architecture details for Diffusion Policy are not reported in the supplementary material (Zhang et al., 11 Aug 2025).

The principal limitation of the robotic benchmark is that quantitative evaluation is largely absent. Metrics such as success rate, path efficiency, time-to-success, and cumulative reward are not reported, and quantitative performance across tasks or environments for each baseline is not provided. The supplementary video shows imitation-learning performance in simulation for multistage tasks and a sim-to-real demonstration of a simple pick-and-place task, but no success rates or numeric comparisons are given. Sim-to-real transfer outcomes, ablations on domain randomization, locomotion mode toggling, and dataset-size scaling are likewise not reported (Zhang et al., 11 Aug 2025).

The paper explicitly notes reproducibility gaps: precise dataset splits, camera configurations, physics fidelity parameters, and quantitative evaluation are not provided. It also does not discuss ethical and safety considerations for real-world deployment near people or around heavy, hot, or sharp objects; instead, the recommended usage guidance advises safety checks, conservative velocity limits, collision monitoring, and fail-safe teleoperation overrides when transferring to real environments (Zhang et al., 11 Aug 2025).

5. AgentWorld for language world models: unified corpus, domains, and curation

In the Qwen-AgentWorld work, AgentWorld refers to a multi-source corpus of environment interaction trajectories collected to train a native language world model that predicts next environment observations across seven agentic domains. It is not a single monolithic public dump; it is a unified-format dataset used internally for continual pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL). The training data and AgentWorldBench are partitioned at the data-source level to avoid leakage, and AgentWorldBench is strictly out-of-distribution with respect to the training data (Zuo et al., 23 Jun 2026).

The world-model prediction target is formalized as

$$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$6

where $$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$7 is the system prompt and the conditioning includes full observation and action history. The observations are textual and structured. MCP uses JSON tool responses; Search includes result lists, snippets, and extracted pages; SWE includes tool outputs, diffs, and file contents; Terminal includes stdout and shell prompts; and Android, Web, and OS use textual accessibility trees or UI view hierarchies rather than pixel inputs (Zuo et al., 23 Jun 2026).

The paper reports “more than 10M environment interaction trajectories of 7 domains in real-world environments” at collection time. The explicitly quantified pools are 7,094 curated thinking trajectories for SFT, after rejection sampling with 69.2% retention from 10,250 candidates, and 92,308 RL training trajectories, with long-context prompts capped at 128k tokens (Zuo et al., 23 Jun 2026).

Domain	SFT	RL
MCP	179	4,156
Search	1,042	20,004
Terminal	1,580	34,125
SWE	249	8,181
Android	1,337	11,498
Web	1,605	8,716
OS	1,102	5,628
Total	7,094	92,308

The RL pool averages 19,443 tokens and 13.4 turns overall. The per-domain averages are reported as follows: MCP, 62,702 tokens and 28.9 turns; Search, 18,873 tokens and 6.2 turns; Terminal, 5,805 tokens and 12.0 turns; SWE, 36,734 tokens and 24.7 turns; Android, 30,064 tokens and 19.3 turns; Web, 19,417 tokens and 10.2 turns; and OS, 25,439 tokens and 12.4 turns (Zuo et al., 23 Jun 2026).

The data sources are complementary. They include dedicated agent infrastructure such as containerized code and tool backends, MCP servers, persistent terminals with shell-state tracking, and GUI environments outputting accessibility or view-hierarchy text; open interaction traces such as public terminal recordings and open-source agent tool-call logs, denoised via multi-agent cleaning; and in-house agentic trajectories converted into environment trajectories by stripping internal reasoning and retaining only action-observation pairs (Zuo et al., 23 Jun 2026).

Unified processing includes trajectory-to-turn expansion, retry-cycle skipping, no-change-turn filtering for GUI data, system-prompt construction with static and dynamic components, and prompt optimization via AutoResearch. The CPT stage additionally uses information-theoretic loss masking based on four surface-level statistics: $$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$8 Categories such as boilerplate and echo have low keep ratios, including 10% for boilerplate and 5% for echo, so that turns carrying genuine environment information are emphasized (Zuo et al., 23 Jun 2026).

6. Qwen-AgentWorld training stages, AgentWorldBench, and empirical findings

The AgentWorld corpus supports a three-stage pipeline. Stage 1, CPT, uses the standard autoregressive language-model objective

$$\begin{aligned} \mathrm{system\_prompt} &:= \mathrm{task\_description} \oplus \mathrm{action\_space} \oplus \mathrm{initial\_state} \oplus \mathrm{demonstrations} \oplus \mathrm{simulation\_instruction} \ \mathrm{turn}_t &:= (\mathrm{action}_t,\,\mathrm{observation}_t) \ \mathrm{trajectory} &:= \mathrm{system\_prompt} \oplus [\mathrm{turn}_1, \ldots, \mathrm{turn}_T]. \end{aligned}$$9

framed by the unified schema and supplemented with professional corpora from industry or manufacturing, cybersecurity, law, medicine, finance, and current events. Stage 2, SFT, uses next-token prediction to activate explicit next-state-prediction reasoning through “thinking trajectories,” with a 256k-token context window, prompt-template diversification across 10 variants, and multi-beam generation with rejection sampling by an independent judge. Stage 3, RL, uses GSPO, caps prompts at 128k tokens, expands each trajectory to exactly one turn to avoid the “Echo Trap,” and employs a hybrid rubric-and-rule reward in which a five-dimension rubric—Format, Factuality, Consistency, Realism, Quality—is combined with rule-based verification at a 9:1 ratio: $s = [s_{qpos}, s_e, s_f],$0 The rubric score is averaged and scaled to $s = [s_{qpos}, s_e, s_f],$1, the rule signal is scaled to $s = [s_{qpos}, s_e, s_f],$2, and strict tag extraction is used to prevent self-praise reward hacking (Zuo et al., 23 Jun 2026).

AgentWorldBench is the evaluation suite built from frontier-agent trajectories executed on real environments on established benchmarks. It contains 2,170 evaluation samples total. Its source interactions come from 5 frontier models across 9 established benchmarks, including Tool Decathlon, Terminal-Bench 1.0, Terminal-Bench 2.0, and OSWorld-Verified, with the remaining sets spanning SWE and GUI tasks from widely used suites. Text domains retain the first, last, and three uniformly sampled intermediate turns per trajectory, GUI domains prioritize challenging turns, and the final benchmark is uniformly downsampled to 50%. The benchmark composition is approximately 72.4% text domains, with SWE at 21.8% and Search at 21.1% as the largest components, and each GUI domain at approximately 9.2%. Judge prompts are calibrated via a double-blind Turing test; GPT-5.2 is selected for highest average accuracy; and rank correlations across Gemini 3 Flash, Claude Sonnet 4.5, and GPT-5.2 are high, with Spearman $s = [s_{qpos}, s_e, s_f],$3–$s = [s_{qpos}, s_e, s_f],$4 (Zuo et al., 23 Jun 2026).

The reported results tie the dataset directly to simulation fidelity. Qwen-AgentWorld-397B-A17B achieves the highest overall average on AgentWorldBench, 58.71, surpassing GPT-5.4 at 58.25. On text domains, its average is 58.07 versus 56.84 for GPT-5.4, including 57.73 versus 53.69 on Terminal and 68.49 versus 66.29 on SWE. Relative to base Qwen3.5 checkpoints, world-model training improves the 397B model from 54.74 to 58.71, a gain of $s = [s_{qpos}, s_e, s_f],$5, and the 35B model from 47.73 to 56.39, a gain of $s = [s_{qpos}, s_e, s_f],$6, with the latter surpassing Claude Sonnet 4.6 by $s = [s_{qpos}, s_e, s_f],$7 (Zuo et al., 23 Jun 2026).

The paper also reports cross-domain and downstream-transfer effects. RL trained only on Terminal improves held-out text domains without domain-specific supervision within 100 RL steps: SWE $s = [s_{qpos}, s_e, s_f],$8, Search $s = [s_{qpos}, s_e, s_f],$9, and MCP $s_{qpos}$0. Using Qwen-AgentWorld-397B-A17B as a simulator for OpenClaw Sim RL yields Claw-Eval $s_{qpos}$1 from 65.4 to 69.7 and QwenClawBench $s_{qpos}$2 from 47.9 to 55.0. Controllable simulation improves Tool Decathlon by $s_{qpos}$3 and MCPMark by $s_{qpos}$4 when targeted simulation instructions are added. In fictional-world Search Sim RL, the 35B model gains $s_{qpos}$5 on F1 Item and $s_{qpos}$6 on F1 Row, while the 397B model gains $s_{qpos}$7 and $s_{qpos}$8, respectively (Zuo et al., 23 Jun 2026).

The paper also emphasizes limits and risks. Search is described as particularly difficult, with the best score around 38 on AgentWorldBench, and factuality remains the lowest-scoring dimension even after RL. Domain imbalance in SFT and RL pools may bias capabilities. GUI fidelity is limited by text-only state representations. Open-trace data may retain rare artifacts despite filtering, and prompt-format dependencies may persist despite disjoint template usage across stages. The full corpus of more than 10 million trajectories is not released as a single public dataset, so reproducing the training setup requires engineering real or persistent simulated backends and collecting continuous logs at scale (Zuo et al., 23 Jun 2026).

Taken together, the two AgentWorld datasets exemplify two contemporary directions in environment-grounded AI research. One is an RGB-plus-proprioception dataset for household mobile manipulation with explicit dual-mode locomotion and manipulation control but limited quantitative reporting. The other is a large, text-native corpus for next-observation world modeling across heterogeneous agentic domains, paired with a reference-grounded benchmark and extensive empirical evaluation. The shared title masks substantial differences in ontology and use: one dataset trains embodied robot policies in simulated domestic scenes, while the other trains language world models to simulate tool-using and GUI-based environments.