Online LLM Simulation Overview

• Online LLM simulation is a method that dynamically queries large language models to emulate system behaviors in real time, maintaining session state.
• It utilizes runtime invocation and contextualized prompts to achieve high response fidelity and efficient error handling across dynamic test scenarios.
• Benchmarks reveal exceptional status-code and response-shape fidelity compared to static mocks, establishing its effectiveness in dynamic system emulation.

Online LLM Simulation

Online LLM simulation refers to the use of LLMs, accessed and executed at runtime, to generate high-fidelity, stateful, and context-sensitive responses that emulate the behavior of complex systems or agents in dynamic scenarios. In contrast to traditional pre-generated mock artifacts or static stubs, online LLM simulation leverages the model’s dynamic capabilities to reason over live inputs, track state across a session, and produce outputs that are sensitive to prior exchanges and evolving system context. This paradigm has recently enabled breakthroughs in software testing, social simulation, agent-based modeling, and realistic system orchestration, often matching or exceeding the behavioral fidelity of hand-crafted mocks, pattern-mined models, or specification-driven stubs (Zhang, 6 Apr 2026).

1. Core Principles and Architectural Features

Online LLM simulation is defined by several architectural and methodological principles, distinguishing it from conventional simulation or mocking approaches:

• Runtime Invocation: The LLM is queried synchronously in response to each incoming interaction or event, rather than operating via a static replay or pre-fabricated artifact. This enables the model to integrate immediate context, scenario state, and prior interactions at inference time.
• State Maintenance Across Requests: The system maintains a sliding window of interaction history (typically request–response pairs) within the LLM’s context, allowing for implicit emulation of cross-request state and long-range behavioral dependencies. This supports rich, stateful workflows such as transactional operations, multi-phase interaction, and context-sensitive protocol handling.
• Direct Contextualization from Source Artifacts: Prompts for the LLM are constructed at session initialization by combining dependency source code, caller code, summarized production traces, and high-level instructions. These signals encode the functional and behavioral specification that the model is asked to emulate, allowing for rapid adaptation to new APIs or system versions without offline artifact regeneration (Zhang, 6 Apr 2026).
• On-the-fly Error Handling and Fidelity Correction: The simulation pipeline includes programmatic validation of the LLM’s outputs (typically JSON format), with retry strategies for ill-formed responses and explicit guidance for reformatting. This design ensures robust emulation under the constraints of LLM generative variability.

2. Prompt Engineering and State Representation

Online LLM simulation systems employ carefully engineered system prompts to ground the model’s behavior in relevant code, environment details, and empirical trace data:

• Prompt Composition: The system prompt is assembled from truncated dependency source code (∼8 kB), caller source code (∼5 kB), summarized production traces, and explicit instructions to simulate cross-request state and return strictly typed outputs (JSON with expected status/body/headers). This approach leverages the LLM’s ability to infer functional requirements and error-handling logic directly from source-level signals.
• History Window Management: For each incoming request, the pipeline serializes the HTTP method, path, headers, query, and body into a user message, appends it to the cumulative history of the scenario, and passes the sequence to the LLM for inference. Up to 20 recent request–response exchanges are typically retained to fit context length constraints, with oldest pairs dropped as necessary.
• Stateful Scenario Context: Advanced designs expose explicit “reset” or “scenario” endpoints that allow the test harness to inform or reset the LLM about scenario boundaries, improving repeatability in integration test campaigns (Zhang, 6 Apr 2026).

3. Fidelity Metrics and Empirical Findings

Performance in online LLM simulation is quantified by its ability to emulate the behavioral interface of the emulated system (e.g., microservices), under live or replayed traffic. The following two metrics are core:

Metric	Definition
Status-Code Fidelity	Fraction of test cases with matching status codes: $$\mathrm{status\_code\_fidelity} = \frac{\left|\{\,r : \hat s_r = s_r\,\}\right|}{N}$$
Response-Shape Fidelity	Fraction with matching response structure (top-level JSON keys): $$\mathrm{response\_shape\_fidelity} = \frac{\left|\{\,r : \mathrm{shape}(\hat y_r) = \mathrm{shape}(y_r)\,\}\right|}{N}$$

Empirical evaluation across benchmarks (Google Online Boutique, Sock Shop, and custom demos) demonstrates:

• White-box mode (dependency source included in prompt): 99% status-code fidelity, 99% response-shape fidelity (109/110 scenarios).
• Black-box mode (only caller code + traces): 94% status, 75% shape.
• By contrast, record-replay mocks achieve only 62% status, 16% shape fidelity; pattern-mining and transition system methods perform even lower on complex, stateful APIs.
• Critical insight: dependency source is the primary enabler for response-structure fidelity. Without it, the LLM can often infer basic error logic, but complex and stateful structures degrade in accuracy (Zhang, 6 Apr 2026).

4. Cost, Latency, and Model Diversity

Online LLM simulation systems run at cost and latency regimes suitable for interactive or batch testing:

• Per-dependency invocation cost ranges by model: \$0.16–\$0.82 (Claude Opus, Sonnet, Kimi, MiniMax), each achieving $\ge97\%$ status/shape fidelity in white-box mode.
• Average LLM call latency is ∼3 seconds per dependency request, with full-scenario suites (110 test scenarios, 177 calls) executed in ∼9.4 minutes.
• Outcome stability is observed across three LLM families (within 3% variance), indicating robustness of this simulation paradigm to underlying model architecture.

5. Insights, Limitations, and Generalization

Key insights from recent large-scale studies:

• Online LLM simulation with cross-request context consistently matches or exceeds behavioral fidelity achieved by hand-crafted stubs, record-replay artifacts, or pattern-extracted transition systems.
• Stateful history maintenance (i.e., non-Markovian context) is essential—typed intermediate representations or static models degrade on complex workflows (down from 99% to 55% fidelity).
• The approach is resilient to incomplete signals: even with only production traces or caller code, a substantial fraction of status behaviors are captured, but response structure requires code-level input.
• Limitations include context window truncation for very long-lived scenarios, restriction to HTTP/JSON protocols (no gRPC or binary bus protocols tested), and validation metrics currently focused on shape/status rather than nested value or property invariants.
• Future directions identified include value-level or property-based response validation, hybrid LLM + IR enforcement, protocol extension, context compression/summarization, and challenges in long-term run-to-run reproducibility under model versioning (Zhang, 6 Apr 2026).

6. Applications Beyond Microservices

While the most rigorously documented instantiation of online LLM simulation is in microservice dependency mocking (Zhang, 6 Apr 2026), conceptual and practical extensions are evident or explicit in adjacent literature:

• Agent-based social simulation (e.g., legal societies, legislative bodies): LLMs embedded in agent roles, maintaining local memory, and executing context-conditioned behavior in continuous online environments (Wang et al., 28 Oct 2025, Baker et al., 2024).
• Judgeless or counterfactual OSN moderation: online, parallel feed simulation where model outputs under alternative histories enable causal attribution and intervention evaluation (Fidone et al., 10 Nov 2025).
• Human–agent interaction in education, medicine, or economic games: LLMs acting as real-time simulators, maintaining causal state (e.g., case history, patient symptoms, memory banks) and responding with high instructional or clinical fidelity (Voigt et al., 19 Aug 2025, Zhang et al., 2024).
• Routing and orchestration in LLM endpoint selection: online model selection policies that route based on query features and immediate reward feedback, with context maintained throughout the batch (Tsai et al., 31 Mar 2026).

Adoption of online LLM simulation is accelerating in any domain where dynamic, stateful emulation of complex, opaque, or brittle dependencies is preferable to static or precomputed artifacts. This marks a fundamental shift toward integrating LLMs "in the loop" for automated testing, system integration, and policy evaluation (Zhang, 6 Apr 2026).