Equivariant Reinforcement Learning for Clifford Quantum Circuit Synthesis

Abstract: We consider the problem of synthesizing Clifford quantum circuits for devices with all-to-all qubit connectivity. We approach this task as a reinforcement learning problem in which an agent learns to discover a sequence of elementary Clifford gates that reduces a given symplectic matrix representation of a Clifford circuit to the identity. This formulation permits a simple learning curriculum based on random walks from the identity. We introduce a novel neural network architecture that is equivariant to qubit relabelings of the symplectic matrix representation, and which is size-agnostic, allowing a single learned policy to be applied across different qubit counts without circuit splicing or network reparameterization. On six-qubit Clifford circuits, the largest regime for which optimal references are available, our agent finds circuits within one two-qubit gate of optimality in milliseconds per instance, and finds optimal circuits in 99.2% of instances within seconds per instance. After continued training on ten-qubit instances, the agent scales to unseen Clifford tableaus with up to thirty qubits, including targets generated from circuits with over a thousand Clifford gates, where it achieves lower average two-qubit gate counts than Qiskit's Aaronson-Gottesman and greedy Clifford synthesizers.

• The paper introduces an RL framework leveraging qubit permutation equivariance and message passing to synthesize Clifford circuits with near-optimal two-qubit gate counts.
• It employs a reverse-reduction MDP formulation and a curriculum-based training strategy to minimize costly CZ gate usage while significantly reducing computational time compared to previous methods.
• The size-agnostic, symmetry-aware architecture demonstrates strong generalization to larger circuits, offering scalable quantum compilation for fault-tolerant protocols.

Equivariant Reinforcement Learning for Clifford Quantum Circuit Synthesis

Problem Overview

The paper addresses Clifford circuit synthesis for devices with all-to-all qubit connectivity, where the synthesis target is to implement a given Clifford unitary (represented as a binary symplectic matrix or "tableau") using a sequence of native gates (H, S, CZ), with a strong incentive to minimize the number of two-qubit gates—here, CZs—since they dominate compilation cost and physical error rates. The Clifford synthesis problem admits an exact group-theoretic representation, but the search space for optimal circuits grows super-exponentially. Previous methods face a trade-off: polynomial-time algorithms (e.g., Aaronson-Gottesman) produce suboptimal circuits, while template or search-heavy optimal solvers are only tractable for few qubits and are computationally expensive.

Methodology

The paper proposes a reinforcement learning (RL) framework, accompanied by a size-agnostic neural architecture explicitly equivariant to qubit relabelings (permutations), to produce a general-purpose policy for Clifford synthesis. The reverse-reduction MDP formulation is employed: for a given Clifford tableau, the agent sequentially chooses gates to reduce it to the identity. The episode reward includes penalties for single- and two-qubit gates, a sparse reward for successful synthesis, and a progress term reflecting the Hamming distance from the identity.

The network architecture leverages the symmetries of Clifford synthesis. Input tableaus are reshaped into four $n\times n$ quadrants, mapping pairwise qubit interactions. These are embedded and aggregated into qubit tokens, updated via message passing (permitting communication between qubits), and ultimately produce policy logits for each action (single- or two-qubit gate) as well as a value estimate. By construction, the architecture is equivariant under permutations of qubits, ensuring the RL policy respects physical indistinguishability.

A curriculum is used for learning: episodes correspond to random walks from the identity of increasing lengths, progressively increasing problem difficulty while ensuring a sufficiently dense reward regime during early training phases.

Experimental Evaluation

Six-Qubit Regime. Evaluation on the Bravyi et al. 1003-tableau six-qubit benchmark demonstrates that the learned policy, with no access to optimal references during training, matches the optimal two-qubit gate count in 99.2% of cases (995/1003), with the maximal gap being one CZ gate in all remaining instances. The agent reaches within one CZ gate of optimality on the complete suite in 21 seconds total, and finds optimal circuits in all but eight cases within three hours. In contrast, the previous state-of-the-art (template-based methods) matches 97.9% (982/1003) after 217 hours, and remains unable to close the last gaps after over 576 cumulative hours.

Notably, greedy decoding of the trained policy is highly efficient, delivering over 500/1003 optimal circuits in less than one second for the complete suite.

Scalability Beyond Training Size. The size-agnostic, symmetry-aware architecture enables generalization to much larger circuits. After initial training on six-qubit instances, fine-tuning on ten-qubit Clifford circuits permits generalization to synthesizing circuits up to 30 qubits. On these, the RL policy consistently yields lower average CZ counts than Qiskit’s polynomial-time synthesizers (Aaronson-Gottesman and the Bravyi et al. greedy approach), sometimes by hundreds of gates.

Ablation Studies. Architectural ablations demonstrate that the combination of qubit-level equivariance and inter-qubit message passing is critical: the message-passing and relation-aware transformer models are tied at best, conventional attention is weaker, and flat or non-communicating MLPs underperform by 2–3 CZ gates per circuit.

Limitations. Although network weights are size-agnostic, inference becomes increasingly costly as $n$ grows due to the $O(n^2)$ two-qubit action space. For large $n$, searching for optimal actions per step remains a bottleneck, suggesting possible future work in factoring or hierarchically compressing the action space, or employing policy-guided search planners.

Theoretical and Practical Implications

The construction of a policy class respecting qubit permutation symmetry is grounded in finite MDP symmetry theory, guaranteeing that invariant rewards and equivariant transitions yield an equivariant optimal policy, hence justifying the adopted architectural constraints. This enables a single neural policy to generalize across qubit numbers, eliminating the need for network reparameterization or separate models per quantum device size.

Practically, this method provides a scalable and computationally efficient solution to Clifford synthesis, with immediate utility for quantum compilers seeking to minimize error-prone entangling gates. As Clifford synthesis is a core subproblem in surface code architectures and fault-tolerant quantum protocols, this method can facilitate improved logical qubit throughput and reduced logical error rates.

The results suggest that reinforcement learning, when combined with symmetry-aware neural architectures and appropriate curricula, can outperform both traditional heuristic and search-based approaches on combinatorial synthesis tasks previously considered infeasible for learning-based techniques.

Future Directions

Potential improvements include further reduction of inference costs via action space factoring, introduction of structured macro-actions, or integration with search methods (e.g., Monte Carlo tree search) during rollouts. Extensions of the equivariant RL framework to other compilation tasks (Pauli network synthesis, T-count optimization, stabilizer state preparation) are promising. The size-agnostic policy mechanism may generalize to other group-structured quantum or classical optimization tasks.

Conclusion

This work advances the state-of-the-art in Clifford circuit synthesis by introducing an RL-based, size-agnostic architecture that leverages problem symmetries for efficient, high-quality circuit synthesis. Numerical results show that it achieves near-optimal two-qubit gate counts in polynomial time across previously intractable regimes and exhibits strong generalization properties. The techniques presented here have significant implications for quantum compilation and form a foundation for future research into symmetry-aware, scalable neural synthesis on combinatorial quantum control problems (2605.10910).