StarDojo: Open Simulation Benchmark

• StarDojo is an open simulation benchmark that repurposes Stardew Valley to evaluate agentic multimodal behaviors, emphasizing production and social interactions.
• It features 1,000 curated tasks across five domains—Farming, Crafting, Exploration, Combat, and Social—with graded difficulty to assess long-horizon planning and decision-making.
• The framework supports varied evaluation regimes, from direct game-state APIs to screen-derived observations, enabling fine-tuned analysis of agent performance under realistic constraints.

StarDojo is an open, production–living simulation benchmark and environment built on the commercial game Stardew Valley to evaluate agentic multimodal LLMs on open-ended behaviors that couple productive activity, exploration, combat, social interaction, and economic decision-making (Tan et al., 10 Jul 2025). It comprises a full suite of 1,000 curated tasks and a compact StarDojo-Lite subset of 100 representative tasks, and has subsequently also been used under a more restrictive evaluation regime that preserves open-world dynamics while removing privileged game-state access and requiring control from screen-derived observations plus a shared executable action interface (Tan et al., 10 Jul 2025, Jiang et al., 18 May 2026).

1. Scope and design rationale

StarDojo was introduced to address a specific deficiency in existing interactive benchmarks: many evaluate either production-like behavior in nature-only worlds, dialog-centric social behavior, or short-horizon GUI control, but not the simultaneous and interdependent demands of production and social life under realistic constraints (Tan et al., 10 Jul 2025). Its motivating premise is that autonomous agents intended to operate in human society must balance resource generation, resource management, and social integration over extended horizons with dynamic environmental constraints.

The choice of Stardew Valley is central to this rationale. The game combines real-time days from 6 AM to 2 AM, stamina and energy budgets of 270 energy per day initially, weather, four 28-day seasons, production chains spanning farming, mining, foraging, animal husbandry, and more than 100 crafting recipes, together with a town containing approximately 45 NPCs, quests, gifts, festivals, relationship and heart mechanics, marriage and family, and a functioning economy with shops, upgrades, and community development projects (Tan et al., 10 Jul 2025). This combination makes it possible to evaluate whether a multimodal foundation agent can ground perception in action, plan across days and seasons, conduct transactions, and build relationships while managing limited time and energy.

In comparative terms, the benchmark is positioned against several neighboring evaluation regimes. Atari and Crafter emphasize low-level control or small games with narrow semantics and no social or economic systems; CivRealm emphasizes macro-level, turn-based strategy rather than individual-level embodiment and fine-grained control; MineDojo and Minecraft provide open-ended production environments but with limited human-like social mechanisms and arduous 3D control; OSWorld and WebArena focus on short-horizon software tasks; and dialog-oriented social benchmarks such as Smallville do not integrate production and economic demands (Tan et al., 10 Jul 2025). StarDojo’s design therefore targets a distinct research niche: open-ended embodied behavior in a persistent world with both livelihood and community structure.

2. Task suite and benchmark construction

The benchmark contains 1,000 curated tasks spanning five domains: Farming, Crafting, Exploration, Combat, and Social (Tan et al., 10 Jul 2025). Each task is labeled Easy, Medium, or Hard, with heuristic per-episode step caps of Easy $\le 30$, Medium $\le 50$, and Hard $\le 150$. For efficient evaluation, the StarDojo-Lite subset contains 100 tasks with a total difficulty distribution of 56 Easy, 23 Medium, and 21 Hard tasks (Tan et al., 10 Jul 2025).

Domain	StarDojo-Lite tasks	Representative examples
Farming	21	clear_10_weeds_with_scythe, harvest_5_parsnip
Crafting	14	craft_1_furnace, produce_1_copper_bar_with_furnace
Exploration	28	go_to_pierre’s_general_store, mine_1_amethyst_with_pickaxe
Combat	12	kill_1_green_slime_with_rusty_sword, kill_1_rock_crab_with_rusty_sword
Social	25	sell_5_parsnip_to_pierre, earn_200_friendship_with_harvey

The Lite subset’s per-domain composition is explicitly enumerated: Farming 21 tasks, Crafting 14, Exploration 28, Combat 12, and Social 25 (Tan et al., 10 Jul 2025). The full set of representative task forms includes simple one-shot objectives such as tilling tiles or crafting an item, navigation-heavy objectives such as reaching a store or mine floor, multi-stage production objectives such as mine $\rightarrow$ smelt $\rightarrow$ craft chains, and social objectives such as purchasing animals, upgrading tools, or accumulating friendship points.

Each task is specified in a YAML schema keyed by name and description and may include an id, object, quantity, tool requirements, initial save, init_commands, evaluator type, and difficulty (Tan et al., 10 Jul 2025). At task start, the benchmark loads a designated game save and applies init_commands to standardize prerequisites such as items, date, weather, skill levels, and unlocked areas. Evaluators then track incremental progress toward the target, such as the number of items harvested or friendship points gained. This design aims to reduce variance arising from world randomness and fragile UI scripting while keeping task semantics explicit and reproducible.

A separate extended objective, the Playthrough challenge, asks an agent starting from scratch to accumulate 1,000,000 in-game currency over hundreds of in-game days (Tan et al., 10 Jul 2025). That objective is not merely a larger instance of the modular task suite; it integrates seasonal planning, unlocking areas via quests, energy and time management, crafting and skill progression, and social–economic strategy over a much longer horizon.

3. Environment architecture and interaction regimes

The original StarDojo environment is built around StarDojoMod, a C# mod on SMAPI, Stardew Valley’s official modding framework, which connects agents to the live game engine through a socket server (Tan et al., 10 Jul 2025). In that configuration, the environment provides direct access to rendered frames, internal game state such as player position, energy, time, inventory, and local or global map information, and callable game functions as high-level skills. A Python wrapper exposes these capabilities through a unified observation–action API that eliminates keyboard and mouse emulation and supports Windows, macOS, Linux, and headless Linux through Xvfb.

The environment also implements a pause–resume mechanism in which the game is paused while the model reasons and unpaused to execute actions (Tan et al., 10 Jul 2025). This removes the need to keep the game window active and decouples inference latency from in-game time, a feature that the original paper uses as part of its ablation analysis. Parallel execution is supported via multiple independent instances bound to unique ports, with communication accelerated through shared memory and per-observation retrieval as low as approximately 30 ms.

In the original benchmark protocol, observations are multimodal. They include a gameplay screenshot at configurable resolution from 360p to 4K, with 720p $(1280 \times 720)$ used in the main experiments; structured textual state containing character information, game-level globals, and a local $N \times N$ agent-centered grid with $N=7$ in the main experiments; and previous-step visual and textual observations for temporal context (Tan et al., 10 Jul 2025). The textual state can include health, energy, money, position, inventory, selected item, time, day, season, weather, current menu, and local tile attributes such as terrain, crops, debris, exits, buildings, furniture, and NPCs. Map-level global information exists in the API but is excluded from the main experiments to fairly test exploration.

The original action space is a high-level abstraction of human-play semantics: move(x, y), use(direction), interact(direction), choose_item(slot_index), attach_item(slot_index), detach_item(), craft(item), choose_option(option_index, quantity, direction), menu(option, menu_name), and navigate(name) (Tan et al., 10 Jul 2025). In the main experiments, navigate(name) is excluded to focus on realistic exploration, and agents may output up to two skills per decision.

Later work employed StarDojo in a more restrictive setting. In the SPIKE evaluation, the benchmark “keeps open-world dynamics but requires control without a privileged game-state API, using screen-derived observations and a shared executable action interface,” and each step provides multimodal observations

$o_t=(v_t,u_t,f_t),$

where $v_t$ is a visual screenshot, $\le 50$0 is textual or UI context derived from on-screen text or state summarization, and $\le 50$1 comprises runner feedback fields such as progress flags, position or facing changes, inventory deltas, menu or dialogue transitions, and execution messages (Jiang et al., 18 May 2026). The same work states that these signals are benchmark runner signals rather than privileged internal game-state APIs. This establishes an important distinction: StarDojo is an environment with rich instrumentation, but it can also be instantiated under stricter screen-grounded evaluation protocols.

4. Formalization and evaluation methodology

StarDojo formalizes the environment as a partially observable Markov decision process. The underlying MDP is

$\le 50$2

and at timestep $\le 50$3 the agent receives

$\le 50$4

where $\le 50$5 is the screenshot and $\le 50$6 is textual state, together with $\le 50$7 for temporal context (Tan et al., 10 Jul 2025). A policy $\le 50$8 selects actions over the abstract action space, and actions are executed atomically or as short sequences of at most two skills. Episodes terminate on task success or when the step cap $\le 50$9 is reached, with $\le 150$0 for Easy, $\le 150$1 for Medium, and $\le 150$2 for Hard tasks.

The benchmark’s core metric is task success rate. For $\le 150$3 tasks, with success indicator $\le 150$4, the overall success rate is

$\le 150$5

Per-domain scores are defined by averaging over tasks in each domain and combining them as a micro-average weighted by domain size (Tan et al., 10 Jul 2025). Across repeated runs, the benchmark reports mean success and standard deviation, with three runs per task used by default.

Evaluation is based on unified automatic instrumentation. Let $\le 150$6 denote evaluator-accumulated progress toward quantity $\le 150$7, and let

$\le 150$8

extract incremental gains from consecutive textual states. Then progress obeys

$\le 150$9

and success occurs when $\rightarrow$0 (Tan et al., 10 Jul 2025). The benchmark uses a generic incremental progress signal from the evaluator in addition to the binary completion flag; there is no dense reward shaping beyond evaluator-derived increments. Instrumentation is grounded in StarDojoMod APIs, enabling stateful detection of inventory changes, tile updates, menu actions, friendship points, kill counts, and related events.

A later evaluation protocol introduced Budgeted SR to make large-model call efficiency visible on StarDojo-Lite (Jiang et al., 18 May 2026). Under the benchmark-native assumption of 4 LLM calls per step, the original step caps are translated into difficulty-dependent LLM-call budgets of 120, 200, and 600 calls for easy, medium, and hard tasks respectively. In that protocol, runs terminate on success, budget exhaustion, or terminal environment failure; all large-model completion requests are counted, including transport retries, parser or format retries, reflection, reasoning, action proposals, reactive action selection, memory queries and summarization, retry, and replanning, whereas visual encoding, deterministic index lookups, Event Trigger rules, and environment steps are not counted (Jiang et al., 18 May 2026). A related recovery-oriented metric defines

$\rightarrow$1

with Recovery/Stuck Ratio given by $\rightarrow$2 (Jiang et al., 18 May 2026).

5. Empirical performance and benchmark usage

The original StarDojo paper reports extensive evaluation of seven state-of-the-art multimodal LLM agents on StarDojo-Lite, with GPT-4.1 as the best-performing model at $\rightarrow$3 overall success rate (Tan et al., 10 Jul 2025). Gemini 2.5 Pro reaches $\rightarrow$4, Claude 3.7 Sonnet $\rightarrow$5, Llama 4 Maverick $\rightarrow$6, GPT-4.1 mini $\rightarrow$7, Qwen2.5-VL-72B $\rightarrow$8, and Gemma 3 27B $\rightarrow$9. For GPT-4.1, the per-domain results are Farming $\rightarrow$0, Crafting $\rightarrow$1, Exploration $\rightarrow$2, Combat $\rightarrow$3, and Social $\rightarrow$4. The difficulty breakdown is consistent across models: some easy tasks are solved, whereas medium and hard tasks are near-zero success across the board.

The failure analysis in the benchmark paper identifies three recurring bottlenecks. First, visual understanding is weak for small or intricate UI elements and environmental affordances, including entrances and exits, tile states such as tilled or watered ground, and whether an object or NPC is within interaction range (Tan et al., 10 Jul 2025). Second, multimodal reasoning is brittle: textual local surroundings help substantially, and advanced models often use vision to approach targets and text to refine final positioning, but image–text inconsistencies frequently lead to repetitive ineffective actions. Third, low-level control and long-term planning remain poor: movement is often excessively cautious and tile-by-tile, craft calls are sometimes misused, and agents do not reliably navigate across maps, manage day transitions, or execute multi-day routines.

StarDojo has also been used as a testbed for cost-aware hierarchical control. On the Lite-100 split, SPIKE reports Success Rate $\rightarrow$5, Budgeted SR $\rightarrow$6, Tokens per Task $\rightarrow$7, Latency per Step $\rightarrow$8, and Recovery/Stuck Ratio $\rightarrow$9 under a default Qwen3.5-397B-A17B backend (Jiang et al., 18 May 2026). In the same evaluation, the strongest baseline for raw success rate is CRADLE at $(1280 \times 720)$0 SR, $(1280 \times 720)$1 tokens per task, $(1280 \times 720)$2 latency per step, and Recovery/Stuck Ratio $(1280 \times 720)$3; SPIKE therefore improves SR by $(1280 \times 720)$4 percentage points, a $(1280 \times 720)$5 relative gain, while reducing token consumption by $(1280 \times 720)$6 and latency by $(1280 \times 720)$7. For Budgeted SR, the strongest budgeted baseline is a baseline labeled StarDojo at $(1280 \times 720)$8, and SPIKE improves over it by $(1280 \times 720)$9 points, a $N \times N$0 relative gain (Jiang et al., 18 May 2026).

These results show that the benchmark is used in at least two distinct but compatible ways: as a direct measure of open-ended multimodal competence and as a platform for studying the success–cost trade-off of long-horizon control under token and latency budgets. A plausible implication is that StarDojo’s utility extends beyond simple leaderboard comparison to the analysis of controller scheduling, memory design, and reasoning allocation.

6. Limitations, contested interpretations, and research significance

StarDojo’s reported limitations are substantive rather than incidental. Version 1 focuses primarily on early- and mid-game content; advanced areas such as the Desert and Ginger Island and the fishing mini-game are not included (Tan et al., 10 Jul 2025). The benchmark requires a purchased copy of Stardew Valley, which constrains out-of-the-box accessibility. Reported evaluations in the initial release are concentrated on the 100-task Lite subset and seven models, leaving broader sweeps for future work.

The benchmark also surfaces characteristic failure modes that are themselves research targets. Typical failures include missing building entrances and exits, misreading tile states, failing to detect NPCs or discrete resources, misjudging interaction distance, poor map-scale navigation, failure to return home and sleep, weak budgeting of time and energy, inability to schedule multi-day crop routines, difficulty with gift preferences and multi-step shop interactions, missing intermediary components in crafting trees, and inefficient small-step movement that exhausts step budgets before reaching goals (Tan et al., 10 Jul 2025). Combat is especially weak because agents struggle to adapt to moving enemies.

Later work adds a second layer of limitations specific to restricted-access, budgeted evaluation. In the SPIKE setting, misrouting can occur when progress is visually subtle or failure signals are noisy; stale memory retrieval can bias both controllers; combat and social tasks remain challenging because of moving enemies and long inter-house routes; and recovery still depends on explicit large-model invocations, which preserves cost and latency pressure (Jiang et al., 18 May 2026). A representative failure case described there involves repeated short local moves around obstacles without reliable target localization or search-and-engage behavior, eventually exhausting the budget.

One recurring misconception is to treat StarDojo as either a fully instrumented privileged-API environment or a purely screen-grounded benchmark with no internal access. The published record supports a more precise interpretation. The original environment exposes direct game-state and callable skill APIs through StarDojoMod (Tan et al., 10 Jul 2025), whereas later work deliberately evaluates on StarDojo under a constrained protocol using screen-derived observations and shared runner feedback rather than privileged internal state (Jiang et al., 18 May 2026). This suggests that StarDojo functions both as an environment and as a benchmark family whose observation and control interfaces can be varied to test different forms of agentic competence.

The broader significance of StarDojo lies in the gap it exposes. Despite supportive abstractions such as pause–resume execution, structured text, and clean action APIs, the best initial StarDojo-Lite performance is only $N \times N$1 (Tan et al., 10 Jul 2025), and even more specialized control architectures remain far from robust performance on long-horizon tasks (Jiang et al., 18 May 2026). The benchmark therefore serves as a concrete stress test for grounded perception, multimodal fusion, hierarchical control, memory and mapping, social and economic reasoning, and real-time or budget-aware decision-making in persistent worlds.