Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents

Abstract: LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. To address this limitation, we turn to a known equivalence between RL and Bayesian inference. We propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is a posterior over trajectories whose prior we define as the fixed black-box LLM agent. We employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged. We validate AMC on three diverse environments from the AgentGym benchmark, demonstrating significant improvements over prompting baselines and even outperforming Group Relative Policy Optimization (GRPO) as we scale the test-time compute of our method. AMC demonstrates the feasibility of performing principled RL-style optimization of black-box LLM agents. Code is available at https://github.com/layer6ai-labs/Agentic-Monte-Carlo

• The paper introduces Agentic Monte Carlo as a novel RL paradigm for black-box agents that circumvents gradient-based updates.
• It leverages Sequential Monte Carlo methods with a learned value function to approximate Bayesian policy inference for optimized action trajectories.
• Empirical evaluations demonstrate AMC's strong performance and cost efficiency compared to traditional prompting and white-box RL approaches.

Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents

Introduction and Motivation

Scalable, agentic use of LLMs is trending toward interaction with proprietary, black-box foundation models made available only via inference APIs. Standard reinforcement learning pipelines—such as PPO and GRPO—depend on parameter access for gradient-based policy updates, excluding their application to closed-source LLM policies. As a result, black-box inference-time alignment and optimization has become a critical bottleneck, with most prior art limited to engineered prompting or exploration of open-weight surrogates. "Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents" (2606.05296) introduces a new paradigm: formulating RL for black-box LLM agents as Bayesian policy inference and leveraging Sequential Monte Carlo (SMC) methods for approximate inference-time optimization of action trajectories, steered by a learned value function.

Figure 1: AMC enables RL-style optimization of black-box LLM policies by learning a value function for SMC-based trajectory search, in contrast to gradient-based RL, which is restricted to open-weight policies.

Methodological Framework

The paper formalizes KL-regularized RL as variational inference, showing that maximizing expected reward with a KL prior is equivalent to sampling trajectories from a Bayesian posterior that modulates the fixed pre-trained LLM prior $\pi$ by a reward-based likelihood, i.e.,

$(\tau) \propto \pi(\tau) \exp(r(\tau) / \beta),$

where $\tau$ denotes a trajectory, $\pi$ the base LLM policy, and $r$ the cumulative reward. The optimal Bayesian posterior is intractable to sample from directly, especially without access to black-box policy probabilities. The authors instead employ SMC, simulating agent rollouts in parallel, weighting and resampling trajectories according to approximated importance weights based on a learned value function $V_\theta$.

The importance weight recursion used in AMC, derived from theoretical foundations, is

$$w_t = w_{t-1} \exp\left(V(\mathbf{s}_t) - V(\mathbf{s}_{t-1}) + r(\mathbf{s}_{t-1})/\beta\right),$$

where $V(\mathbf{s}_t)$ is a soft value function, and all weights and rewards are computed at the trajectory state level. The value function is trained via regression on Monte Carlo rollouts from the fixed prior policy.

Positioning with Respect to Prior Art

AMC distinguishes itself from previous SMC applications for language agents in key ways. Prior works (e.g., "Fleet of Agents" [klein2025fleetagentscoordinatedproblem]) employed built-in heuristics or prompt-level verifiers for trajectory weighting, while "Rollout Roulette" [puri2025rollout] operated in process-reward model (PRM) domains. In contrast, AMC leverages a parametric value function learned from interaction data to produce RL-grounded optimality estimates, and is explicitly constructed for agentic, multi-step interaction in black-box LLM settings, where only API-level stepwise access is available and policy logits are inaccessible.

Experimental Validation

AMC was evaluated on the AgentGym benchmark, focusing on WebShop (e-commerce), SciWorld (multi-step science reasoning), and TextCraft (Minecraft-style crafting). Each environment presents unique challenges: open-ended search with semantic constraints (WebShop), combinatorially complex procedural tasks (SciWorld), and deep compositional planning (TextCraft).

AMC was benchmarked against (1) prompting-based Best-of-$N$ sampling, (2) Fleet-of-Agents-style SMC with in-context prompting for state evaluation, and (3) state-of-the-art GRPO for white-box RL. Across all environments and both open-weight and black-box policies (GPT-4.1, GPT-5.1), AMC yielded significant improvement over parallel sampling and SMC-prompted agents. Importantly, AMC's improvements come without modifying or fine-tuning the base agent.

Figure 2: On SciWorld, AMC consistently outperforms Best-of-N and GRPO, both for black-box GPT-5.1 and open-weight Qwen-2.5 LLMs.

Notably, AMC's RL-style trajectory search often exceeds the performance of GRPO as test-time compute (number of SMC particles) is scaled, despite the fact that GRPO relies on full parameter access and expensive online rollouts. On WebShop and SciWorld, AMC achieves strong policy improvements for both large and small LLM priors, and closes the performance–cost gap between smaller black-box models and their more expensive counterparts.

Figure 3: Across WebShop, SciWorld, and TextCraft, AMC dominates Best-of-N in terms of sample efficiency (trajectories needed for target reward).

Ablations and Analysis

Ablation analyses probe AMC's cost and efficiency trade-offs, value function training methods, value model architectures, and trajectory selection policies. Key insights include:

• Value Function Necessity: Training a domain-specific value function is necessary; zero-shot prompting for value estimation does not consistently outperform Best-of-N.
• Data Efficiency: AMC reaches target performance with fewer trajectories than Best-of-N, minimizing API inference cost.
• Surrogate Training for Value Model: Training the value function on open-weight surrogate policy data still brings robust performance gains when applied to black-box policies, offering cost advantages in high-API-cost regimes.
• Cost and Scalability: AMC enables smaller black-box models (e.g., GPT-4.1-mini) to match or exceed the performance of larger black-box models at a fraction of the cost.
• Failure Modes: For nearly deterministic or extremely strong black-box priors, AMC can saturate or slightly underperform Best-of-N due to reduced diversity in sampled states.

Trajectory Analysis and Resampling Strategies

The optimality of AMC is predicated not only on value function learning but also on when and how often to resample. Experiments indicate that infrequent, well-placed SMC resampling steps (rather than dense, per-step resampling or dynamic ESS-based triggers) deliver maximal gains while controlling value function evaluation cost.

Figure 4: On WebShop, task-specific fixed-step resampling (e.g., at step 6) yields higher performance than frequent or ESS-based resampling.

Theoretical and Practical Implications

AMC advances the interface between RL and Bayesian inference for agentic LLMs, demonstrating that RL-style, reward-driven behavior can be instilled in black-box policies without gradients or parameter updates. This positions AMC as a practical alignment and optimization tool in scenarios where most advanced LLMs are API-only and cost-sensitive. It implies that, as model complexity and proprietary control increases, algorithmic advances in inference-time search will become necessary to extract policy improvements, and that amortized value function learning can close some of the gap traditionally dominated by direct RL methods.

Future Directions

Limitations include reliance on the quality of the learned value model, computational burden of sampling and SMC, and possible suboptimality when priors are near-deterministic. Future work may leverage adaptive resampling, value function distillation, temporal-difference learning updates for value models, and hybrid integration with open-weight off-policy RL. The theoretical underpinnings of SMC in high-dimensional, structured agentic state spaces, and the interaction between reward/value function mis-specification and search-induced sample bias, are outstanding research directions.

Conclusion

Agentic Monte Carlo (AMC) represents a theoretically principled and empirically validated paradigm for reward-driven optimization of black-box LLM agents. By reframing RL as Bayesian inference and applying SMC guided by a learned value function, AMC enables inference-time performance scaling of agentic LLMs without direct policy mutation. Empirical results across diverse, high-complexity domains substantiate AMC's efficacy, specifically in closing performance and cost gaps for black-box agents, and in matching or surpassing the best white-box RL test-time agents under realistic compute regimes. This work provides a foundation for future advances in practical black-box LLM agent alignment in an era where policy gradients are increasingly inaccessible.