---
title: 'Simia-RL: LLM-Simulated RL Framework'
url: https://www.emergentmind.com/topics/simia-rl-framework
type: topic
---

# Simia-RL: LLM-Simulated RL Framework

The Simia-RL framework is a reinforcement learning methodology in which both environment transitions and reward feedback are simulated by a large language model (LLM) rather than a fixed-coded environment or real-world API. Conceived as part of a broader effort to eliminate brittle and labor-intensive environment engineering for language agents, Simia-RL enables scalable, environment-agnostic agent training. By leveraging LLMs to simulate both environment feedback and reward computations, Simia-RL permits training and evaluation of agents entirely in synthetic contexts, supporting both supervised and reinforcement learning modalities. The framework is positioned as a unified solution for robust language agent training, with demonstrated empirical gains on multiple complex tool-use and dialogue benchmarks.

## 1. Markov Decision Process Formulation

Simia-RL is defined as a standard reinforcement learning problem over a Markov Decision Process (MDP) $(\mathcal S, \mathcal A, \gamma)$, where:
- $\mathcal S$ is the state space, comprising full agent–environment dialogue histories, tool-use contexts, and schema information.
- $\mathcal A$ is the action space, including free-form text replies and structured tool calls.
- $\gamma \in [0, 1]$ is the discount factor.

The key distinction is that both the transition function $T(s'|s, a)$ and reward function $r(s, a)$ are implemented by LLM-based simulators, not by hand-coded or real environments. Formally:
\[
\bigl(s', r_t\bigr) \sim \left( \widehat T(s'|s, a), \widehat r(s, a) \right)
\]
where $\widehat T$ and $\widehat r$ are outputs of the LLM environment simulator and reward model, respectively. The episodic reward is binary, with successful task completion (as defined by oracle criteria) yielding $r_t = 1$, and failure yielding $r_t = 0$.

The agent objective is to maximize the expected discounted return:
\[
J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t = 0}^T \gamma^t r_t \right]
\]
where trajectories $\tau$ are simulated entirely with LLM-based feedback.

## 2. Algorithmic Structure

Training proceeds in interleaved policy rollouts and updates, closely akin to PPO or GRPO, with all feedback provided by the LLM-based simulator.

### High-Level Algorithm

1. **Initialize** policy $\pi_\theta$ (typically an instruction-tuned transformer such as Qwen or Llama).
2. **For** $N$ RL iterations (e.g., 64):
    - **Collect $K$ simulated episodes**: for each,
        - Start from an initial state $s_0$.
        - For up to max-turns:
            - Sample action $a_t \sim \pi_\theta(\cdot | s_t)$.
            - Query the LLM simulator with $(s_t, a_t)$ to obtain $(s_{t+1}, r_t)$.
            - Store $(s_t, a_t, r_t, s_{t+1})$.
3. **Compute** advantage estimates $\hat A_t$ (e.g., GAE).
4. **Update** $\theta$ by maximizing the clipped surrogate PPO or GRPO objective:
    \[
    L^{\mathrm{CLIP}}(\theta) = \mathbb{E} \left[ \min \left( r_t(\theta) \hat A_t, ~ \mathrm{clip}\left( r_t(\theta), 1 - \epsilon, 1 + \epsilon \right) \hat A_t \right) \right]
    \]
    where $r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$.
5. **Optional** fine-tuning with supervised learning on high-reward trajectories.

#### Pseudocode

```python
initialize θ
for iteration in range(N):
    trajectories = []
    for rollout in range(K):
        s = s0
        for t in range(T):
            a = pi_theta(s)
            s_prime = LLM_env_simulator(prompt=(s, a))
            r = LLM_reward_computer(prompt=(s, a, s_prime))
            store(s, a, r, s_prime)
            s = s_prime
    compute_advantages()
    update_theta()  # PPO/GRPO with L_CLIP objective
```

## 3. LLM-Based Simulation Details

Both the transition simulator and reward function are implemented as prompt-based queries to a large instruction-tuned LLM, typically o4-mini or GPT-5.

- **Simulator/Reward Models**: Designed atop transformer-based LLMs with context windows up to 60k tokens.
- **Environment Feedback Prompts**: Bundle tool schemas, available functions and signatures, interaction formatting instructions, and at least one exemplar trajectory.
- **Reward Prompts**: Include natural-language task success criteria, full history, and explicit instructions to return JSON-formatted success/failure with justification and confidence.
- **Prompt Diversity and Amplification**: Trajectories are generated with stochastic sampling (temperature=1.0), supporting the construction of tens of thousands of unique episodes. The same environment prompt is reused, ensuring API consistency but supporting stochasticity in simulator output.

## 4. Training Regime and Hyperparameters

Simia-RL employs RL training hyperparameters as follows:
- **Learning rate**: $1 \times 10^{-6}$
- **Rollouts per iteration**: 16 episodes
- **Gradient steps**: 64 per training epoch
- **Mini-batch size**: 32
- **Clip parameter**: $\epsilon = 0.28$
- **KL penalty coefficient**: 0.001
- **Sampling temperature**: 0.7; Top-P: 1.0
- **Max rounds per episode**: 25
- **Max context length**: $12,000$ tokens

Exploration incentives are implemented via entropy bonuses from GRPO. A cosine annealing scheduler with $10\%$ warm-up is used. No explicit data augmentation is needed, as the LLM-simulated environment naturally injects trajectory diversity due to its stochastic outputs.

## 5. Empirical Evaluation and Performance

Experiments target benchmarks requiring realistic multi-tool, multi-turn agent interaction:

**Benchmarks evaluated:**
- $\tau^2$-Bench (Airline, Retail) using GPT-4.1 as a user simulator.
- OfficeBench (2-apps, 3-apps workflows).
- AgentBench (Operating System, WebShop, Mind2Web).

**Performance metrics (selected results):**

| Benchmark          | Closed-source Baseline | Simia (SFT) | Simia-RL (SFT+RL) |
|--------------------|-----------------------|-------------|-------------------|
| $\tau^2$-Bench     | GPT-4o: 54.2          | 58.9 (Qwen2.5-32B) | 51.0 (Qwen3-8B)   |
| OfficeBench        | GPT-4: 31.1           | 44.0 (Qwen3-8B)    | 49.6 (Qwen2.5-7B) |
| AgentBench         | GPT-4o: 38.1          | 42.6 (Qwen3-8B)    | N/A               |

Ablation analyses demonstrate that synthetic–simulated trajectories yield similar or better downstream performance versus real-environment data, with greater scalability. Using different LLM simulators (o4-mini, GPT-5) produces nearly indistinguishable downstream results, subject to domain effects. RL training on LLM-simulated environments converges more quickly and robustly than training against real test-bed APIs, plausibly due to richer and more consistent feedback.

## 6. Strengths, Limitations, and Prospective Extensions

**Strengths**
- Eliminates dependency on handcrafted environments or APIs for agent training.
- Supports data amplification and diversity, scaling SFT and RL to tens of thousands of trajectories with minimal engineering effort.
- Open-source models fine-tuned with Simia-RL achieve or surpass the performance of larger closed models on standard tool-use and workflow benchmarks.

**Limitations**
- Experimental evaluation is limited in scope (Airline, Retail, Office, WebShop, OS, Mind2Web).
- LLM simulators may introduce distributional shifts or biases relative to deployment environments.
- The binary reward function constrains the granularity of credit assignment; structured reward shaping remains open.

**Potential Extensions**
- Application to robotics, science education/tutoring, multi-agent coordination.
- Integration of hierarchical or curriculum-based RL leveraging subtask decomposition guided by LLM simulators.
- Ensembles of LLMs for more robust or diversified simulation feedback.
- Human-in-the-loop corrections to improve simulator accuracy during online fine-tuning.

In sum, Simia-RL establishes that LLM-based environment and reward simulation constitute a viable and scalable alternative to traditional environment engineering for both supervised and reinforcement learning of complex language agents, enabling flexible and robust training pipelines independent of real-world APIs or testbeds.

Source: https://www.emergentmind.com/topics/simia-rl-framework