CleanQRL: Standardized Quantum RL Library

• CleanQRL is a Python library for quantum reinforcement learning that standardizes and simplifies hybrid quantum-classical algorithm implementations.
• It employs a single-file design philosophy to enhance readability, debuggability, and benchmark consistency across various RL algorithms and environments.
• The framework integrates PennyLane, PyTorch, and Ray Tune for efficient PQC-based policy approximation, hyperparameter tuning, and reproducible experiments.

CleanQRL is a Python library for quantum reinforcement learning (QRL) that provides lightweight, single-file implementations of QRL algorithms, explicitly modeled after CleanRL. It is presented as a standardized, readable, and reproducible foundation for variational QRL, specifically in the hybrid quantum-classical setting where a parameterized quantum circuit (PQC) replaces the neural-network function approximator used in classical reinforcement learning. In this design, the agent follows the logic of a classical RL algorithm, while policy or value approximation is performed by a trainable PQC optimized through classical control flow and optimization routines (Kruse et al., 10 Jul 2025).

1. Definition, scope, and motivating problem

CleanQRL is situated at the intersection of quantum computing and machine learning, within the subfield of quantum reinforcement learning. Its stated motivation is infrastructural rather than algorithmic novelty: QRL is described as a young field in which researchers typically combine a classical RL software stack with a separate quantum computing framework, producing implementations that are fragmented, inconsistent, and difficult to compare. CleanQRL is introduced to address this gap by offering a common implementation substrate for variational QRL (Kruse et al., 10 Jul 2025).

The library targets variational, hybrid quantum-classical RL. In this paradigm, the overall learning loop remains recognizably classical: environments, optimization logic, rollout collection, and most engineering machinery follow standard RL practice. The quantum component enters through the function approximator, where a PQC stands in for the neural network used in conventional agents. This positioning is important because CleanQRL does not propose a distinct environment API or a wholly new training formalism; instead, it localizes the quantum modification within otherwise familiar RL workflows.

A central claim of the project is that QRL research has lacked a standardized and comprehensive collection of implementations. CleanQRL therefore presents itself as a reproducible implementation framework for many QRL algorithms, with an emphasis on benchmarkability, inspectability, and rapid adaptation. This suggests an infrastructure role analogous to that played by CleanRL in classical reinforcement learning, but specialized for the additional constraints of hybrid training and quantum circuit simulation.

2. Single-file design philosophy and implementation architecture

The defining design principle of CleanQRL is its single-file philosophy. Each algorithm is implemented as a self-contained script rather than distributed across a large modular codebase. The rationale is that modular RL libraries such as Stable-Baselines, RLlib, TorchRL, Pearl, Tianshou, and Dopamine are powerful, but their critical logic is often spread across many files and abstractions, making implementations harder to inspect and modify. CleanQRL instead aims to keep “all moving wheels in plain sight,” accepting some code duplication in exchange for clarity, debuggability, lower engineering overhead, and educational value (Kruse et al., 10 Jul 2025).

The library is closely aligned with CleanRL in both structure and implementation logic. For the classical RL portion, CleanQRL largely follows CleanRL’s implementation decisions. The principal difference is localized to the agent definition. Both classical and quantum agents are implemented as torch.nn.Modules and expose the same input/output behavior. In the classical case, the function approximator is a standard PyTorch neural network; in the quantum case, it is a PQC implemented with PennyLane and wrapped so that it behaves like a PyTorch module. Because of this interface alignment, the outer training loop can remain almost unchanged across classical and quantum variants.

This architectural choice is central to the library’s benchmarking agenda. CleanQRL includes both classical and quantum versions of each algorithm precisely so that comparisons can be made under nearly identical training code. The paper presents this as a corrective to a recurring problem in QRL, namely that baseline implementations often differ in multiple uncontrolled ways, obscuring whether observed differences arise from the learning method or from surrounding engineering choices.

The script layout is deliberately repetitive. Each script generally contains three principal components: def make_env, which initializes the environment; class Agent(nn.Module), which defines either the classical model or the quantum PQC-based model; and def training_function, which contains the training loop, loads hyperparameters, and executes learning. Quantum scripts add two further elements: class ObservationWrapper, used for observation normalization, and def parametrized_quantum_circuit, which defines the PQC ansatz itself. The paper emphasizes that observation normalization is especially important in QRL because of qubit count limitations and the periodicity of Pauli rotation gates.

The single-file rule is not absolute. Replay buffers and interfaces to Jumanji environments are handled outside the strict single-file paradigm, described as a pragmatic compromise required to preserve readability.

3. Algorithms, environments, and supported problem classes

CleanQRL provides single-script implementations of many QRL algorithms and explicitly positions itself relative to quantum variants of major RL families including REINFORCE, DQN, PPO, and DDPG (Kruse et al., 10 Jul 2025). The paper’s figures particularly emphasize DQN and REINFORCE, but the stated objective is broader algorithmic coverage across different environment and action-space regimes.

The environment coverage is organized by state and action type. The library includes scripts for continuous state spaces with discrete actions, exemplified by CartPole; continuous state spaces with continuous actions, exemplified by Pendulum; and discrete state spaces with discrete actions, exemplified by FrozenLake. In addition, it supports selected Jumanji environments and more specialized QRL setups. These include graph-based approaches such as TSP and Hamiltonian-based approaches such as Knapsack.

This breadth is significant because the paper does not frame CleanQRL as narrowly tied to one specific RL family or one benchmark suite. Instead, it is presented as a reusable foundation across multiple action/state-space combinations and across more specialized structured-input settings. The library supports any environment following Gymnasium conventions, as well as selected Jumanji environments, and notes that custom environments can be adapted so long as they conform to those interfaces. This reinforces the project’s design choice to integrate with existing classical RL practice rather than establish a separate quantum-specific ecosystem.

The inclusion of both classical and quantum versions of the implemented algorithms is methodologically important. CleanQRL’s stated aim is not simply to provide quantum scripts, but to standardize side-by-side benchmarking. A plausible implication is that the library functions simultaneously as an experimentation platform and as a baseline-definition mechanism for future QRL work.

4. Quantum modeling and circuit-aware benchmarking

On the quantum side, CleanQRL uses PennyLane together with PyTorch. The PQCs are typically hardware-efficient ansatzes based on prior QRL literature, especially the DQN-style ansatz from Skolik et al. The example circuit described in the paper consists of an encoding block in which state features $s$ are encoded and scaled by trainable parameters $\lambda$, a variational block with trainable parameters $\theta$, and an entangling block, repeated for $n$ layers and followed by measurement. The notation explicitly highlighted is $s$ for state, $\lambda$ for trainable scaling parameters in the encoding block, $\theta$ for trainable variational parameters, and $n$ for the number of repeated layers (Kruse et al., 10 Jul 2025).

The paper does not provide full RL objective equations, Bellman targets, or policy-gradient derivations. Its mathematical emphasis is instead on implementation-level description and on PQC parameterization. The main explicitly mentioned gradient-estimation mechanism is the parameter-shift rule. This matters because it changes what constitutes computational cost in QRL. Classical RL commonly evaluates sample efficiency in terms of environment steps to solution, but in variational QRL there is an additional cost dimension: quantum circuit executions.

Because parameter-shift gradient estimation requires a number of circuit evaluations that grows linearly with the number of circuit parameters, CleanQRL’s benchmarking records not only standard learning curves but also efficiency with respect to circuit executions. This is one of the library’s most distinctive methodological commitments. It treats circuit executions as a first-class benchmarking quantity, rather than assuming that environment-step efficiency alone is an adequate metric for hybrid quantum-classical methods.

The observation wrapper used in quantum scripts is also tied to this hardware-aware viewpoint. The paper stresses that normalization is especially important because of qubit count limitations and the periodicity of Pauli rotation gates. CleanQRL therefore embeds preprocessing choices directly into the script structure rather than leaving them implicit.

5. Experiment management, reproducibility, and prototyping workflow

A major component of the CleanQRL ecosystem is integration with Ray Tune. The paper gives two reasons for this emphasis: quantum models are computationally expensive to simulate in runtime and memory, and QRL algorithms are highly sensitive to hyperparameters, often more so than their classical counterparts. Ray Tune is therefore used for distributed computing across multiple nodes, parallel training runs, and hyperparameter optimization (Kruse et al., 10 Jul 2025).

The library exposes four run modes. main.py loads one config file and trains one agent without Ray Tune. main_batch.py loads a list of config files and runs them sequentially. tune.py performs Ray Tune hyperparameter optimization for one config. tune_batch.py runs sequential Ray Tune optimizations over multiple configs. This arrangement is intended to support both small local experiments and larger distributed searches.

For logging and benchmarking, CleanQRL integrates Weights & Biases. It also supports custom JSON logging and Ray Tune logs for CPU/GPU and memory utilization. The paper links this tooling directly to reproducibility. For each environment, the authors provide a Weights & Biases report comparing classical and quantum variants of the applicable algorithms, along with configuration files intended to reproduce the results. These reports are designed to be compatible with the Open RL Benchmark.

The framework is also presented as a rapid-prototyping platform. The tutorial material extends beyond standard RL tasks and includes graph-based QRL inspired by graph neural network ideas and custom ansatzes for problems such as TSP; Hamiltonian-based QRL, where problem Hamiltonians in the style of QAOA are used as ansatz structure; maze games as benchmark environments shared across multiple QRL paradigms, including amplitude amplification-based and free-energy-based approaches; and noise models intended to bridge ideal statevector simulation and real quantum hardware. The paper presents these tutorials as evidence that new QRL ideas can be integrated with little overhead and that the framework is intended to evolve with the field.

6. Empirical position, limitations, and relation to adjacent work

The paper presents example comparisons between classical and quantum REINFORCE on Acrobot-v1 and classical and quantum DQN on CartPole-v1, shown through Weights & Biases reports (Kruse et al., 10 Jul 2025). However, the stated emphasis is not on claiming universal quantum superiority. Instead, the contribution is framed as a consistent benchmarking and experimentation platform built around aligned code structure, reproducible configuration files, logging, and circuit-aware efficiency metrics.

Several limitations are explicitly acknowledged. The provided PQCs are primarily example ansatzes and are not claimed to be optimal. The authors warn that hardware-efficient ansatzes can suffer from the barren plateau phenomenon, making scalability task-dependent and potentially requiring redesign of the ansatz for more complex problems. The observation wrapper may also need adjustment for sophisticated environments. In addition, the authors state that they did not perform extensive hyperparameter searches for all benchmarked environments, because doing so would be computationally expensive; they instead encourage the community to contribute improved configuration files.

In the broader research landscape, CleanQRL should be distinguished from several similarly named but substantively different lines of work. “Quarl: A Learning-Based Quantum Circuit Optimizer” is an RL system for sequential circuit rewriting and optimization, not a library for variational quantum reinforcement learning algorithms (Li et al., 2023). “Qurts: Automatic Quantum Uncomputation by Affine Types with Lifetime” addresses type-theoretic automatic uncomputation and clean disposal of temporary quantum values, rather than RL experimentation infrastructure (Hirata et al., 2024). “Clean quantum and classical communication protocols” studies clean communication complexity and the restoration of non-output registers to their initial states (Buhrman et al., 2016), while “Quantum Relational Hoare Logic” develops a relational verification logic for pairs of interactive quantum programs (Unruh, 2018). These works share the vocabulary of cleanliness, cleanup, or quantum control, but they address distinct problems.

Within QRL itself, CleanQRL’s main claimed advantages are standardization, readability, reproducibility, ease of modification, benchmarking consistency, educational value, and reduced engineering overhead. It does not introduce a single new QRL algorithm so much as provide a transparent implementation framework for many of them. In that sense, its contribution is infrastructural: it aims to make QRL research easier to inspect, compare, tune, benchmark, and extend (Kruse et al., 10 Jul 2025).