Quantum Asynchronous Advantage Actor-Critic

• QA3C is a hybrid quantum–classical reinforcement learning method that integrates variational quantum circuits as both actor and critic within an asynchronous training framework.
• It employs distinct quantum models, including dressed VQCs and ANO-VQCs with adaptive measurement techniques, to achieve performance comparable to or exceeding classical A3C baselines.
• The approach leverages asynchronous workers and policy-gradient optimization while revealing trade-offs between circuit depth and measurement design, guiding future quantum RL research.

Quantum Asynchronous Advantage Actor-Critic (QA3C) denotes a family of hybrid quantum–classical reinforcement-learning methods that embed parameterized quantum models into the asynchronous advantage actor-critic paradigm. In the published formulations currently available, both the policy (“actor”) and the value estimator (“critic”) are realized by variational quantum circuits (VQCs) or related quantum modules, while asynchronous workers interact with separate environment instances and update shared global parameters. The 2023 formulation studies asynchronous training of advantage actor-critic variational quantum policies with “dressed” VQCs, and the 2025 formulation explicitly presents QA3C with adaptive non-local observables (ANO) as the measurement layer for both actor and critic (Chen, 2023, Lin et al., 25 Jul 2025).

1. Research emergence and scope

The 2023 work situates QA3C in the broader development of quantum machine learning and quantum reinforcement learning, emphasizing that training QRL algorithms with VQCs requires significant amount of computational resources and proposing asynchronous training as a remedy. Specifically, it chooses the asynchronous training of advantage actor-critic variational quantum policies and reports numerical simulations in which, within the tasks considered, asynchronous training of QRL agents can reach performance comparable to or superior than classical agents with similar model sizes and architectures (Chen, 2023). The 2025 work extends the design space by introducing an adaptive non-local observable paradigm within VQCs for QRL, and instantiates that paradigm inside both Deep Q-Network and Asynchronous Advantage Actor-Critic algorithms; in its A3C instantiation, the method is explicitly described as QA3C (Lin et al., 25 Jul 2025).

A central point in the literature is that QA3C is defined more by its training protocol and actor–critic decomposition than by a single canonical circuit ansatz. Both formulations share asynchronous worker-based optimization, separate actor and critic parameterizations, and policy-gradient learning with value baselines. They differ materially in state encoding, readout design, and which parts of the quantum model are trainable. This suggests that QA3C is best understood as a methodological class rather than a fixed architecture.

Aspect	2023 formulation	2025 formulation
Quantum backbone	“dressed” VQC	ANO-VQC
Readout	measured expectation values $\langle Z \rangle$ plus classical post-processing	adaptive non-local observable $H(\phi)$
Actor/critic parameters	separate $\theta$ and $\theta_v$	separate $(\theta_{\text{actor}}, \phi_{\text{actor}})$ and $(\theta_{\text{critic}}, \phi_{\text{critic}})$
Asynchrony	master process with $N=80$ workers, $S=5$	spawn $N$ workers, segment length $t_{\max}$
Baseline comparison	classical A3C with nearly matched parameter counts	classical A3C with local Pauli measurements and standard VQC-A3C

2. Hybrid quantum–classical architectures

In the 2023 formulation, both actor and critic are parameterized by a “dressed” VQC placed between two small classical linear layers. A pre-processing layer maps the raw observation vector $H(\phi)$0 into an 8-dimensional real vector $H(\phi)$1, one entry per qubit. The VQC core is an 8-qubit parameterized circuit with two variational layers, giving $H(\phi)$2 free parameters per VQC. A post-processing layer maps the 8 measured expectation values $H(\phi)$3 into a logit vector of length $H(\phi)$4 for the actor and into a single scalar $H(\phi)$5 for the critic. The core circuit initializes each qubit in $H(\phi)$6, applies a Hadamard gate, encodes each $H(\phi)$7 using $H(\phi)$8 followed by $H(\phi)$9, applies a ring of CNOTs, applies a three-parameter single-qubit rotation $\theta$0 on each qubit, repeats the entangling-and-rotation block for the second variational layer, and finally measures each qubit in the Pauli-$\theta$1 basis (Chen, 2023).

In the 2025 formulation, each of the actor and critic is a hybrid quantum–classical module built from an ANO-VQC. The architecture uses $\theta$2 qubits; in CartPole experiments $\theta$3, while in MiniGrid the high-dimensional raw observation is linearly projected down to 4 features and then assigned to 4 qubits. The encoding layer $\theta$4 has depth 1 and applies Hadamard on every qubit followed by single-qubit rotations $\theta$5 or $\theta$6. The variational block $\theta$7 consists of $\theta$8 layers of nearest-neighbour CNOT entanglers and single-qubit parameterized rotations $\theta$9, with typical choices $\theta_v$0 for CartPole and $\theta_v$1 for MiniGrid. The key innovation is the adaptive non-local observable $\theta_v$2 with $\theta_v$3-local support acting as measurement; a sliding overlapping $\theta_v$4-qubit window is used to generate $\theta_v$5 outputs, and in all reported QA3C experiments $\theta_v$6 (Lin et al., 25 Jul 2025).

The architectural contrast is consequential. In the 2023 design, measurement is fixed to local Pauli-$\theta_v$7 expectations and expressivity is concentrated in the variational circuit plus surrounding classical layers. In the 2025 design, the measurement operator itself becomes trainable. This suggests that published QA3C variants occupy different points in the trade space between circuit expressivity, classical post-processing, and measurement design.

3. Policy, value, and objective functions

For the 2023 dressed-VQC agent, the stochastic policy over a discrete action set $\theta_v$8 is

$\theta_v$9

where $(\theta_{\text{actor}}, \phi_{\text{actor}})$0 is the circuit output and $(\theta_{\text{actor}}, \phi_{\text{actor}})$1 are included in $(\theta_{\text{actor}}, \phi_{\text{actor}})$2. The advantage is defined as

$(\theta_{\text{actor}}, \phi_{\text{actor}})$3

with $(\theta_{\text{actor}}, \phi_{\text{actor}})$4 as the $(\theta_{\text{actor}}, \phi_{\text{actor}})$5-step return. The actor update uses the policy-gradient surrogate

$(\theta_{\text{actor}}, \phi_{\text{actor}})$6

while the critic minimizes

$(\theta_{\text{actor}}, \phi_{\text{actor}})$7

The actor objective may equivalently be written as

$(\theta_{\text{actor}}, \phi_{\text{actor}})$8

or, in minimization form,

$(\theta_{\text{actor}}, \phi_{\text{actor}})$9

Gradients of circuit observables are computed via the parameter-shift rule

$(\theta_{\text{critic}}, \phi_{\text{critic}})$0

and these derivatives are auto-differentiated through the classical pre- and post-networks (Chen, 2023).

For the 2025 ANO-VQC formulation, the quantum state is

$(\theta_{\text{critic}}, \phi_{\text{critic}})$1

For each action $(\theta_{\text{critic}}, \phi_{\text{critic}})$2, a distinct adaptive observable $(\theta_{\text{critic}}, \phi_{\text{critic}})$3 is assigned, and the action score is

$(\theta_{\text{critic}}, \phi_{\text{critic}})$4

The actor policy is

$(\theta_{\text{critic}}, \phi_{\text{critic}})$5

and the critic is

$(\theta_{\text{critic}}, \phi_{\text{critic}})$6

The per-time-step losses are

$(\theta_{\text{critic}}, \phi_{\text{critic}})$7

$(\theta_{\text{critic}}, \phi_{\text{critic}})$8

and

$(\theta_{\text{critic}}, \phi_{\text{critic}})$9

The overall per-worker loss is

$N=80$0

A distinctive feature is that both $N=80$1 and the observable parameters $N=80$2 are jointly optimized, with a shift rule available for $N=80$3 because $N=80$4 depends linearly on the matrix entries (Lin et al., 25 Jul 2025).

4. Asynchronous optimization mechanics

The 2023 method follows a canonical asynchronous setup with a master process holding global parameters $N=80$5 and spawning $N=80$6 independent workers; in the reported experiments, $N=80$7 on an 80-core CPU. Each worker pulls a local copy $N=80$8, interacts with its own environment instance for up to $N=80$9 steps or until terminal, accumulates gradients while working backward from the final step to the start of the segment, applies an atomic asynchronous optimizer step using Adam with $S=5$0, $S=5$1, $S=5$2, and then immediately refreshes its local parameters from the global state. No replay buffer is needed. The paper states that the parallelism both decorrelates training samples and greatly speeds up wall-clock training (Chen, 2023).

The 2025 formulation preserves the same asynchronous actor-critic logic while expanding the parameter state to four sets:

$S=5$3

Each worker interacts with its own copy of the environment for up to $S=5$4 steps, collects tuples $S=5$5, computes the $S=5$6-step return

$S=5$7

and forms the advantage estimate $S=5$8. Local gradients are then computed for actor and critic, including both circuit and observable parameters, and applied asynchronously to the shared global parameters in Hogwild! style. The reported optimizer is RMSProp with learning rate $S=5$9, discount $N$0, entropy weight $N$1, and value weight $N$2 (Lin et al., 25 Jul 2025).

Across both versions, the defining operational feature of QA3C is therefore not merely the presence of a quantum policy network, but the coupling of that network to many asynchronously updating environment workers. In both papers, the actor–critic decomposition is used to reduce variance relative to pure return-based updates, while asynchronous parallelism is used to improve sample decorrelation and training throughput.

5. Benchmark configurations and reported performance

In the 2023 study, the environments are Acrobot, Cart-Pole, and MiniGrid-SimpleCrossing. Acrobot uses a 6-dimensional observation, 3 discrete actions, reward $N$3 per step, and success threshold $N$4. Cart-Pole uses a 4-dimensional observation, 2 actions, and reward $N$5 per step until failure. MiniGrid-SimpleCrossing uses a 147-dimensional compact observation, 6 actions, and a sparse reward of 1 at goal minus step penalty, with three variants having 1–3 valid wall crossings. Shared hyperparameters are $N$6, update horizon $N$7, discount factor $N$8, and 8-qubit two-layer VQCs, yielding 48 parameters per circuit and total quantum parameters of 96 for actor plus critic. The classical baseline uses the identical network layout except that the 8-qubit VQC is replaced by a single classical linear layer $N$9, so total parameter counts are nearly matched. All runs use 100 000 training episodes and report the moving average over the past 100 episodes. On Acrobot, the QA3C agent steadily improves and converges to the goal threshold faster and more stably than the classical baseline, which struggles to climb out of local minima. On Cart-Pole, the classical baseline learns slightly faster in the first $t_{\max}$0 episodes, but the QA3C agent overtakes and reaches higher final average reward with fewer fluctuations. On SimpleCrossing, for the $t_{\max}$1 and $t_{\max}$2 variants the QA3C agent converges significantly faster and to higher returns than the classical A3C, while in the easiest $t_{\max}$3 case both methods perform similarly (Chen, 2023).

In the 2025 study, the reported QA3C tasks are CartPole, MiniGrid-$t_{\max}$4, and MiniGrid-SimpleCrossing S9N1. CartPole uses $t_{\max}$5 qubits and $t_{\max}$6; the MiniGrid tasks reduce observations to 4 features and use $t_{\max}$7. Each experiment is averaged over 5 seeds, and the reported metrics are episode return for CartPole and success rate for MiniGrid. On CartPole, QA3C with ANO and $t_{\max}$8 reaches an average return of 400 within $t_{\max}$9 episodes, versus $H(\phi)$00 for classical A3C with local Pauli measurements, and fails entirely when only $H(\phi)$01 or only measurement is used. On MiniGrid-$H(\phi)$02, QA3C attains $H(\phi)$03 success in $H(\phi)$04 episodes, whereas standard VQC-A3C plateaus at $H(\phi)$05. On SimpleCrossing, QA3C reaches $H(\phi)$06 success, VQC-A3C reaches $H(\phi)$07, and measurement-only reaches $H(\phi)$08 (Lin et al., 25 Jul 2025).

Taken together, the published evidence supports a narrower but technically specific conclusion: under the reported simulated benchmark conditions, QA3C matches or surpasses classical A3C baselines and local-measurement VQC baselines with comparable or explicitly stated architectures. The stronger claim of a general quantum advantage is not established in these papers; rather, the reported improvements are benchmark- and implementation-specific.

6. Measurement design, limitations, and research directions

A major methodological divergence between the two formulations concerns measurement. The 2023 method uses fixed local Pauli-$H(\phi)$09 measurements and relies on the VQC plus surrounding classical layers for representational capacity. The 2025 method promotes measurement to a trainable object by introducing a $H(\phi)$10-local Hermitian observable

$H(\phi)$11

with

$H(\phi)$12

and total parameter count approximately $H(\phi)$13. The paper’s ablation results state that measurement locality $H(\phi)$14 controls expressive power: on MountainCar in DQN, performance improves markedly from $H(\phi)$15, with diminishing returns beyond $H(\phi)$16, and once $H(\phi)$17 is large enough, measurement-only matches the performance of full ANO. The same study explicitly describes a “depth $H(\phi)$18 measurement” trade-off, according to which one may increase $H(\phi)$19 and the $H(\phi)$20-parameter count instead of increasing circuit depth $H(\phi)$21 (Lin et al., 25 Jul 2025).

The main limitations are also explicit. In the 2023 study, all experiments are simulated and no real QPU hardware was used; the authors note that noise, limited qubit connectivity, and measurement error on current NISQ devices may degrade performance. The circuits remain shallow, with two variational layers, to stay within NISQ constraints, and deeper circuits are presented as a possible route to greater expressivity but one that would require error mitigation (Chen, 2023). The 2025 study similarly frames shallow circuits as desirable because deeper circuits exacerbate noise on NISQ hardware, and positions ANO as a way to improve expressivity without adding circuit depth (Lin et al., 25 Jul 2025).

Several future directions are stated directly in the literature. The 2023 paper identifies deployment on multi-QPU or cloud-accessed hardware with true quantum parallelism among workers, more advanced advantage estimation such as generalized advantage estimation or entropy regularization, hybrid encodings such as amplitude or basis encoding for higher-dimensional inputs, and extensions to continuous action spaces such as quantum soft-actor-critic (Chen, 2023). The 2025 paper points instead toward scaling expressive measurements and exploiting the depth-versus-measurement trade-off (Lin et al., 25 Jul 2025). A plausible implication is that future QA3C research will increasingly treat readout design, not only ansatz design, as a primary control knob for reinforcement-learning performance.