Simia-RL: LLM-Driven RL Framework

Updated 8 November 2025

Simia-RL is a machine learning framework that uses LLMs to simulate environments, eliminating the need for custom-built simulators.
It combines an agent model with an LLM-based environment simulator that generates state transitions, tool feedback, and structured reward signals.
Empirical results show that Simia-RL outperforms traditional simulators on key agent benchmarks, accelerating research and scaling deployment.

Simia-RL is a machine learning framework designed to enable reinforcement learning (RL) for agents in the absence of real environment implementations by leveraging LLMs as environment simulators. It replaces bespoke, task-specific simulation code with an LLM capable of generating state transitions, tool/API feedback, and programmatic reward signals. The approach aims to address the bottleneck of environment engineering in large-scale, agentic RL, thereby enabling scalable, flexible, multi-domain training for language-based agents and multi-agent systems, and has demonstrated improved performance over both open and closed-source baselines on major agent benchmarks (Li et al., 3 Nov 2025, Fujii et al., 12 Oct 2025).

1. Motivation and Context

Traditional RL for agentic LLMs requires engineering bespoke simulators or real environments with complex APIs and reward logic. Such implementations are costly, brittle, and do not scale easily to new domains. Simia-RL circumvents this by utilizing powerful LLMs (e.g., o4-mini, GPT-5) to simulate environment transitions, feedback, and rewards based on structured prompts encoding the task, tool schema, and interaction history. This allows RL policy optimization without the need for domain-specific simulation code, greatly accelerating research and deployment cycles (Li et al., 3 Nov 2025).

2. Framework Architecture

Simia-RL consists of two main components:

Agent Model: Typically a large LLM (such as Qwen, Llama, or other open models), trained to produce actions in a given environment context.
LLM Environment Simulator: A separate LLM (e.g., o4-mini), prompted to simulate the environment’s reaction to agent actions. It generates next-state observations, tool/API outputs or error messages, and evaluates task success for reward computation.

Prompt construction provides detailed instructions, tool/API schemas, sample trajectories, and the full interaction history. The LLM simulator produces environment feedback by interpreting the agent's action in context and delivers both feedback and reward (e.g., a binary success/failure flag or more nuanced criteria codified in structured output).

Iterations of the RL training loop proceed entirely through agent–LLM environment interaction. The environment LLM is typically invoked with high context window and explicit validation constraints (e.g., tool schemas, output formats, stepwise history) (Li et al., 3 Nov 2025).

3. RL Training Procedure

The Simia-RL optimization procedure replicates classical RL, with the distinction that agent rollouts are not sampled from a software or physical environment but from the LLM simulator. The key steps are:

Action Selection: The agent chooses an action from its policy given the current (simulated) observation.
Environment Simulation: Action and context are sent to the environment LLM, which returns the next observation, any tool/API outputs, error messages, and a programmatically derived reward.
Reward Assignment: The LLM computes the reward, most commonly as a binary indicator of task completion, but optionally with structured confidence or error explanations (see JSON schema example in (Li et al., 3 Nov 2025)).
Policy Update: The agent policy is updated using RL algorithms such as Generalized Reward Policy Optimization (GRPO) or Proximal Policy Optimization (PPO), using simulated rollouts of (state, action, reward, next state).
Iteration: Steps 1-4 are repeated with new samples or longer trajectories. All samples are generated entirely by LLM simulation, and batch sizes, sequence lengths, and temperature parameters are set as appropriate for memory and diversity.

Prompt engineering is critical, with highly structured, environment-constraining prompts to the LLM simulator, including action schemas, tool specs, policy rules, and example trajectories (see paper’s appendix for complete prompt patterns).

4. Empirical Results and Benchmarks

Simia-RL-trained models have been validated on several widely cited agentic RL benchmarks:

Model	Airline (τ²-Bench)	Retail (τ²-Bench)	OfficeBench (Avg)	AgentBench (Avg)
GPT-4o	48.0	60.4	31.1	44.2
o4-mini	57.0	69.3	-	-
Simia-Tau (Qwen2.5-32B)	56.0	61.7	-	-
xLAM-2-70B	49.3	63.2	-	-
Simia-OB (Qwen3-8B)	-	-	44.0	-
Simia-AB (Qwen3-8B)	-	-	-	42.6

Models fine-tuned and RL-trained via Simia-RL on simulated data match or outperform state-of-the-art closed models, such as GPT-4o, with smaller parameter counts.
Data ablation studies show that scaling up simulated data further improves performance, and RL conducted with LLM-simulated environments yields higher agent performance—especially on complex tasks (e.g., OfficeBench 3-app: 34.5 vs. 28.6 for the real environment).
LLM-simulated feedback provides adaptive, context-rich guidance compared to real environments that yield fixed error messages, resulting in richer policy learning signals and improved robustness (Li et al., 3 Nov 2025).
Side-by-side comparison with simulators such as VirtualTaobao, RL4RS, RecSim, and KuaiSim demonstrates that replacing engineered simulation with LLM-driven simulation offers scalability and higher empirical fidelity when prompt constraints and reference trajectories are well designed (Zhao et al., 2023).

5. Methodological Foundations and Implementation

The Simia-RL training regime integrates established RL paradigms but adapts the data interface:

State, action, reward, and transition tuples are sampled through prompt-driven, context-aware LLM calls.

Reward computation is embedded in the LLM output, often with explicit structure, e.g.:

{
  "reasoning": "...",
  "evidence": "...",
  "task_success": true/false,
  "confidence": <float>
}

Policy optimization follows standard actor–critic or policy-gradient methods as implemented in frameworks such as RAGEN built atop VeRL.
Rollout and batch parameters are set for large-scale simulation: context lengths up to 12,000 tokens (agent), 60,000 tokens (simulator), rollout batches of 16, and 64 RL steps typical in training runs.
Prompt design requires providing action schemas, tool and API specification tables, result formatting constraints, and example reference trajectories to ensure the environment LLM’s responses remain valid and non-degenerative.
By relying on LLM simulation, no environment-specific code/engineering is necessary beyond the prompt and schema configuration.

6. Advantages, Limitations, and Implications

Advantages

Domain independence: Simia-RL generalizes to entirely new domains and tool suites by simply modifying prompt structure or schema without rewriting simulation code.
Rapid scalability: Large-scale diverse trajectory data can be synthesized, supporting robust SFT and RL fine-tuning.
Research acceleration: Removes the traditional bottleneck of environment reimplementation, vastly speeding up experiments and deployment for agentic RL.
Supervised and RL synergy: RL fine-tuning on simulated environments further enhances performance over SFT alone, leveraging richer LLM-based feedback mechanisms.
Empirical parity with closed models: Demonstrated performance rivaling or exceeding state-of-the-art proprietary models of much larger size.

Limitations

Simulation fidelity is bounded by the LLM’s reasoning capacity and adherence to prompt constraints.
Prompt brittleness: Incorrect specification or inadequate reference trajectories can yield invalid environment transitions or reward leakage.
May not scale without loss of precision in extremely complex, high-frequency, or safety-critical environments where nuanced real-world stochasticity is essential.

A plausible implication is that as LLM simulators improve in reasoning and in handling complex schemas, the Simia-RL approach could fully supplant traditional engineered simulators for many classes of agentic tasks.

7. Future Directions

The Simia-RL paradigm promotes a new research and engineering model: "environments as language," wherein any new agent training domain is instantiated as a set of prompt and schema instructions for an LLM simulator. This suggests rapid domain transfer, on-the-fly adaptation to unseen tasks, and persistent improvement of open models even in the absence of proprietary APIs or code.

Anticipated research directions include:

Extending prompt strategies to cover hierarchical, multi-stage, or adversarial multi-agent environments.
Incorporating more structured reward signals (e.g., nuanced gradations, counterfactual evaluation) via enhanced simulator prompting.
Merging Simia-RL methodology with frameworks for counterfactual simulation and multi-agent/counterfactual regularization, as explored in related work on RL for animal behavior modeling and cross-session recommendation systems (Fujii et al., 12 Oct 2025, Zhao et al., 2023).
Automated prompt and schema induction for new domains to further reduce configuration overhead.
Integrating with platforms for ongoing deployment-time RL, where real and simulated (LLM-based) environments coexist for continual agent adaptation and robustness.

Simia-RL thus defines a generalizable, scalable foundation for RL-based LLM agent training, removing environment engineering as a barrier to progress and supporting multi-domain, high-fidelity agent development at scale.