Continuous Monte Carlo Graph Search (CMCGS)

• Continuous Monte Carlo Graph Search (CMCGS) is a framework that extends MCTS to continuous and mixed state-action spaces, enabling scalable planning and optimization.
• It leverages graph-based search with clustering and importance-sampling policies to counteract combinatorial explosion in high-dimensional domains.
• Empirical benchmarks in quantum circuit synthesis and continuous control highlight CMCGS’s sample efficiency and superior solution quality.

Continuous Monte Carlo Graph Search (CMCGS) is a framework extending Monte Carlo Tree Search (MCTS) to efficiently address planning and combinatorial optimization in settings characterized by continuous or mixed discrete-continuous state and action spaces. Unlike conventional MCTS, which suffers from combinatorial explosion in high-dimensional or continuous domains due to its branching-tree structure and per-action node allocation, CMCGS leverages graph-based search, clustering, and importance-sampling-based policies to achieve tractable and sample-efficient exploration. It has been developed and evaluated independently in both quantum circuit synthesis (Rosenhahn et al., 2023) and continuous-control planning (Kujanpää et al., 2022) contexts, providing a unifying methodology for scalable exploration, solution construction, and parameter optimization.

1. Foundations and Problem Formalization

CMCGS is motivated by limitations of classical MCTS when applied to domains featuring uncountably infinite action/state spaces or environments in which many paths lead to equivalent or similar results. In such problems, naive tree-based expansion leads to tree-size explosion, severely limiting the practical horizon and throughput for real-world planning or synthesis tasks (Kujanpää et al., 2022).

For quantum circuit optimization, the state space is the set of partial quantum circuits, each corresponding to a unitary $U \in \mathrm{U}(2^N)$ built as a left-ordered product of elementary gates. Each unique unitary is a vertex $v$ in a (potentially cyclic) directed graph $G = (V, E)$, with the root $v_0$ as the identity operator $\mathbb{I}$. The action space at a node consists of choosing a discrete gate $O \in \mathcal{OP}$ and, if the gate is parameterized, selecting real parameters $\phi \in \mathbb{R}^d$ (e.g., rotations $R_Z(\phi)$). The search objective is to maximize a task-dependent reward function $s(U) \geq 0$ for a (partial or complete) circuit $U$, e.g., $s(U) = 1/(1 + L(U))$ where $L(U) = \|U_{\mathrm{target}} - U\|_F$ for synthesis tasks (Rosenhahn et al., 2023).

In continuous-control settings, the environment is modeled as an MDP with continuous state and action spaces. Standard MCTS cannot efficiently represent all states and actions, so CMCGS instead builds layered graphs, where nodes in each layer represent clusters of states sharing similar local dynamics or policies. Each node $q$ maintains parameterized distributions over both the states it represents and the local policy over actions (Kujanpää et al., 2022).

2. Core Framework and Algorithmic Structure

The canonical CMCGS algorithm follows a four-phase iteration, analogous to MCTS but with modifications to fit the continuous and/or graph-based context:

1. Selection

In each iteration, a node $v$ (quantum synthesis) or $q$ (continuous planning) is selected with probability proportional to its current estimated quality. For quantum circuits, Poisson selection is used: $\pi_v = s_v / \sum_{w \in V} s_w$. In continuous planning, selection can be $\epsilon$-greedy, balancing stochastic sampling from the node's action distribution with exploitation of elite actions in its buffer (Rosenhahn et al., 2023, Kujanpää et al., 2022).

2. Expansion

At the chosen node, an action is sampled:

• For quantum circuits: (a) pick a discrete gate $O$ (uniformly or via a learned proposal $\rho(O)$), (b) if parameterized, sample $\phi$ from a per-node proposal density $q_v(\phi)$, e.g., a Gaussian (centered at the empirical mean or importance-weighted towards low-loss regions). The resulting new unitary is constructed as $\tilde{U} = O(\phi) U_v$. If an equivalent unitary exists in $V$ (to tolerance $\epsilon$), an edge is added; otherwise, a new node is inserted (Rosenhahn et al., 2023).
• For continuous planning: sample an action $a$ from $\pi_q(a)$, simulate the environment step, and use the resulting next state $s'$ to assign execution to the cluster $q'$ that maximizes the state density $p_{q'}(s')$ (Kujanpää et al., 2022).

3. Simulation/Rollout

From the newly created or matched node, a rollout (simulation episode) is performed, either to a fixed depth or until a quality threshold is met. In quantum synthesis, this involves further random or policy-driven application of gates to complete a candidate circuit, after which the reward $s_{\text{rollout}}$ is computed. In continuous planning, random or policy-driven rollouts add further steps and accumulate reward (Rosenhahn et al., 2023, Kujanpää et al., 2022).

4. Backpropagation

The path traversed in the selection and expansion phases is updated by propagating the rollout reward:

• Quantum synthesis: For each visited $v$, increment visit count $N_v \to N_v+1$ and update $s_v$ either via running average or max-update.
• Continuous planning: For each transition $(s_t, a_t, s_{t+1}, q_t)$, the sample is added to the node's buffer. If buffer conditions are met, clustering (via Ward's linkage) may split or refine clusters; updated distributions $p_q(s)$ and $\pi_q(a)$ are fitted on the elite subset as described below (Kujanpää et al., 2022).

Key innovation: For continuous parameters, proposal distributions are continuously refined by importance-weighting according to observed rollout rewards. Gradient-based updates or Gaussian mixture models may be used to accelerate concentration in high-reward areas (Rosenhahn et al., 2023).

3. Representation of State, Action, and Proposal Distributions

Quantum Circuit Synthesis

Nodes represent unique partial circuits with associated unitary matrices. The expansion phase uses a proposal density $q_v(\phi)$ for parameterized gates, which may be updated in response to observed outcomes, concentrating sampling in promising parameter regions. Alternative update rules include maintaining a histogram, fitting a bounded-variance Gaussian, or sampling from a Gibbs distribution as $q_v(\phi) \propto \exp(-L(O(\phi)U_v)/T)$ (Rosenhahn et al., 2023). Resampling and shrinking variance are used to avoid wasting simulation budget on unproductive regions.

Continuous Control Planning

Each node in layer $t$ is a cluster of similar states with distributions:

• $p_q(s_t) = \mathcal{N}(s_t; \mu_q^s, \Sigma_q^s)$: where the node 'lives' in state space.
• $\pi_q(a_t) = \mathcal{N}(a_t; \mu_q^a, \Sigma_q^a)$: its stochastic action policy.

Action distributions are maintained by refitting to the top $r_\text{elite}$ quantile of buffer experiences; variance updates use conjugate-prior Bayesian inference: $\alpha' = \alpha + n/2$, $\beta' = \beta + (1/2)\sum_i\|a_i - \mu_q^a\|^2$, yielding $\sigma^2 = \beta' / (\alpha' - 1)$. Clustering methods (agglomerative, Ward's linkage) manage per-layer expansion and avoid uncontrolled graph growth (Kujanpää et al., 2022).

4. Theoretical Properties and Convergence Remarks

For quantum synthesis, provided that (i) rewards are bounded, (ii) all gate actions have nonzero proposal probability at each iteration, and (iii) backpropagation uses unbiased updates, every finite circuit prefix is visited infinitely often due to Poisson node selection. This ensures that node reward estimates $s_v$ converge to their true expected rewards, and the best-discovered circuit approaches an (approximate) optimum as the number of iterations grows. Embedding the parametric expansions into a continuous-armed bandit setting (e.g., HOO algorithms) allows sharper analysis if proposal variance is controlled and ergodicity is ensured (Rosenhahn et al., 2023).

No formal convergence proofs are provided for the general CMCGS planning variant; regret and consistency analysis is left open, but empirical results indicate that the mechanism is robust in practice in a variety of domains (Kujanpää et al., 2022).

5. Practical Implementation and Complexity Considerations

Computational Complexity

• Selection and simulation are $O(\text{depth} \times d_\text{model})$ per trajectory.
• Buffer management and clustering cost $O(n_t^2)$ per triggered clustering at time step $t$ (mitigated by keeping $n_t / m$ small).
• Distributional updates are $O(|D_q| \times \mathrm{dim})$ per visited node (Kujanpää et al., 2022).

Graph Construction and Parallelization

CMCGS constructs layered directed acyclic graphs (for continuous MDP planning) or generic directed graphs with functionally equivalent nodes merged (quantum synthesis), greatly reducing redundant computation versus trees. Batch-parallel rollouts and vectorized model calls are recommended for efficiency and allow natural scaling to multi-core or GPU environments (Kujanpää et al., 2022).

Hyperparameters and Integration

Key hyperparameters include: depth-expansion threshold $m$, exploration probability $\epsilon$, number of top actions $N_\text{top}$, replay buffer size per node $|D_q|_\text{max}$, initial/max graph depth $d_\text{init}/d_\text{max}$, rollout length $N_r$, maximum clusters per layer $n_\text{max}$, and action-distribution Bayesian priors $(\alpha, \beta)$. Practical implementation leverages PyTorch/NumPy and scikit-learn for clustering; high-dimensional spaces may require latent-state projection prior to clustering (Kujanpää et al., 2022).

6. Empirical Results and Benchmarks

Quantum Circuit Synthesis Benchmarks (Rosenhahn et al., 2023)

Task	CMCGS Performance	Baseline Comparison
3-qubit QFT	Order-of-magnitude fewer circuit samples;	Outperforms random sampling (RS), GA, PF, SA in sample and code efficiency
Cellular automata	Discovers universal compact circuits for all 256 rules	Consistent, rapid synthesis via graph expansion
QML classifiers	95% (Iris), 90% (Wine), 92% (Zoo) accuracy	Matches/exceeds decision trees, shallow NNs, unhandcrafted

Continuous Control Benchmarks (Kujanpää et al., 2022)

Environment	CMCGS	CEM, Random Shooting (RS), Others
Toy multimodal bandit	99% full-reward rate	CEM avg 65%
2D navigation/expl.	Solves with higher sample efficiency	All baselines slower/less reliable
DMC suite	Higher mean episode return (5/7, 6/7 envs)	CMCGS: 856; CEM: 767 (mean, PlaNet img)

This suggests that CMCGS consistently discovers higher-quality solutions with fewer samples, particularly in tasks with sparse rewards, complex state-action spaces, and where redundant explorations would otherwise dominate the computational cost.

7. Distinctive Features and Limitations

Key features of CMCGS are (i) asymmetrical graph-based expansion, (ii) importance-sampling for both discrete and continuous action choices, (iii) local buffer-based clustering for scalable width and depth control, and (iv) proposal refinement via reward feedback and, optionally, gradient-driven parameter updates (Rosenhahn et al., 2023, Kujanpää et al., 2022). Unlike tree-based approaches, graph merging avoids redundant expansion of functionally equivalent states.

The main current limitation is the absence of proved convergence or regret bounds in general domains, and the clustering cost may be non-negligible in very large or high-dimensional state spaces, requiring pragmatic design of hyperparameters and possible latent-space reduction (Kujanpää et al., 2022). Future work is suggested to address theoretical analyses and to improve the integration of surrogate models and scalable clustering algorithms.

---

References:

• "Continuous Monte Carlo Graph Search" (Kujanpää et al., 2022)
• "Monte Carlo Graph Search for Quantum Circuit Optimization" (Rosenhahn et al., 2023)