Hamiltonian Q-Learning

• Hamiltonian Q-Learning is a reinforcement learning framework that embeds value optimization within Hamiltonian or HJB structures for continuous and high-dimensional control problems.
• It leverages techniques such as Hamilton–Jacobi deep Q-networks, Hamiltonian Monte Carlo sampling, and quantum annealing to improve sample efficiency and convergence.
• The approach unifies classical and quantum methods with robust theoretical guarantees and practical applications in deterministic, stochastic, and quantum optimization tasks.

Hamiltonian Q-Learning is a class of methods and theoretical frameworks that reformulate or extend classical Q-learning by embedding its value-optimization and policy-synthesis steps within Hamiltonian or Hamilton–Jacobi–Bellman (HJB) structures. These methods span multiple domains—including deterministic continuous-time control, high-dimensional stochastic reinforcement learning, quantum hardware optimization, and continuous-time diffusions—with the common feature of leveraging Hamiltonian dynamics or HJB partial differential equations (PDEs) to derive update laws, sampling mechanisms, or quantum encodings for the Q-function and associated policy.

1. Hamilton–Jacobi Q-Learning: Continuous-Time Control and HJB Theory

Hamilton–Jacobi Q-Learning arises from applying the dynamic programming principle directly to continuous-time deterministic systems with Lipschitz-constrained controls. For the system

$$\dot x(t) = f(x(t), a(t)), \quad x \in \mathbb{R}^n,\ a \in \mathbb{R}^m,$$

with infinite-horizon discounted cost

$$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$

the continuous-time Q-function is defined as

$$Q(x_0, a_0) = \sup_{\substack{a(\cdot): a(0) = a_0,\, \|\dot a\| \le L}} \int_{0}^{\infty} e^{-\gamma t} r(x(t), a(t))\,dt.$$

The dynamic programming (DP) principle yields, in the viscosity solution sense, the Hamilton–Jacobi–Bellman PDE: $$\gamma Q(x,a) - \nabla_x Q(x,a) \cdot f(x,a) - L \bigl|\nabla_a Q(x,a)\bigr| - r(x,a) = 0.$$ A semi-discrete HJB equation is introduced to allow updates from discrete samples without discretizing or approximating the underlying continuous dynamics: $$Q^{h,\star}(x,a) = h\,r(x,a) + (1-\gamma h)\sup_{|b| \le L} Q^{h,\star}(\xi(x,a;h), a + h b).$$ This framework leads to a rigorous Q-learning update compatible with deep neural architectures, establishing convergence in the limit of vanishing time discretization and guaranteed contraction of the Bellman operator under bounded-Lipschitz system and reward assumptions (Kim et al., 2020).

2. Hamiltonian Q-Learning via Hamiltonian Monte Carlo

In high-dimensional stochastic MDPs where the transition kernel is only known up to an unnormalized density, Hamiltonian Q-Learning leverages Hamiltonian Monte Carlo (HMC) to draw next-state samples for Q-function updates. The key steps are as follows:

• For each sampled state-action pair $(s,a)$, HMC is used to sample $N_{\mathrm{HMC}}$ successor states $s'_i$ from a density $\pi(s'|s,a) \propto \mathcal{P}(s'|s,a)\,c(s')$.
• The partial Bellman update is

$$\widehat Q^{t+1}(s,a) = r(s,a) + \frac{\gamma}{N_{\mathrm{HMC}}} \sum_{s' \in \mathcal{H}_t} \max_{a'} Q^t(s', a').$$

• Only a subset $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$0 is updated at each iteration, and the full Q-matrix is reconstructed by nuclear-norm minimization based low-rank matrix completion:

$$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$1

This approach achieves near-optimal sample efficiency in high-dimensions (complexity matching minimax rates), provided the true Q-matrix is approximately low-rank, the transition density is smooth, and nuclear-norm matrix completion is feasible (Madhushani et al., 2020).

3. Quantum Hamiltonian Q-Learning: K-Spin Mapping and Quantum Optimization

A further generalization encodes the Bellman optimality equations for an infinite-horizon discounted MDP as a pseudo-Boolean polynomial cost function $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$2, which is equivalent to a generalized K-spin Hamiltonian: $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$3 Optimal policies correspond to the ground state of this Hamiltonian. Quantum heuristics—such as adiabatic quantum annealing (AQA) and the quantum approximate optimization algorithm (QAOA)—can be applied to minimize $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$4 and thus solve for the optimal policy directly, bypassing iterative Bellman updates. Simulation and quantum annealing on small MDPs confirm that the ground state encodes the classical optimal policy. This formulation reveals intrinsic analogies with classical field theory and suggests hardware-efficient approaches for near-term quantum devices (Jones et al., 2020).

4. Continuous-Time Hamiltonian Q-Learning in Stochastic Diffusions

Extensions to continuous-time, entropy-regularized stochastic control require re-interpreting the conventional Q-function, which naturally collapses in the $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$5 limit. The "little $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$6-function"—the $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$7-rate of value improvement for immediate deviation from the default policy—plays the role of a Hamiltonian-based advantage rate: $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$8 where $$J_{x(0)=x_0,\,a(0)=a_0}[a(\cdot)] = \int_0^\infty e^{-\gamma t} r(x(t), a(t)) \, dt,$$9 is the standard Hamiltonian of the controlled diffusion (including drift, diffusion, reward, and entropy regularization). Martingale characterizations provide on- and off-policy update rules, and actor-critic algorithms emerge naturally from the structure. The relationship to SARSA and policy-gradient is made precise, highlighting discretization-invariant continuous-time learning (Jia et al., 2022).

5. Algorithmic Realizations and Empirical Evaluation

Algorithmic realization of Hamiltonian Q-learning principles includes, for deterministic continuous-time systems, the Hamilton–Jacobi Deep Q-Network (HJDQN):

• $$Q(x_0, a_0) = \sup_{\substack{a(\cdot): a(0) = a_0,\, \|\dot a\| \le L}} \int_{0}^{\infty} e^{-\gamma t} r(x(t), a(t))\,dt.$$0 is parameterized by a neural network.
• Replay-buffered transitions are used to form targets via the semi-discrete HJB equation.
• Policy extraction is direct: the optimal control increment is given by the ODE $$Q(x_0, a_0) = \sup_{\substack{a(\cdot): a(0) = a_0,\, \|\dot a\| \le L}} \int_{0}^{\infty} e^{-\gamma t} r(x(t), a(t))\,dt.$$1, eliminating the need for a separate actor network.
• Empirically, HJDQN is competitive or superior to actor-critic baselines such as DDPG on standard continuous-control (MuJoCo) and high-dimensional LQR tasks. For large-dimension LQR, HJDQN achieves stable learning where DDPG fails (Kim et al., 2020). For stochastic high-dimensional MDPs, the HMC-based Hamiltonian Q-learning demonstrates reduced sample complexity and comparable or better Q-learning accuracy than classical or deep RL baselines on controlled pendulum systems and further scales via low-rank completion (Madhushani et al., 2020). Quantum Hamiltonian approaches match classical Q-learning optima, with resource scaling discussed in terms of logical qubits and gate depth (Jones et al., 2020). Continuous-time q-learning eliminates discretization sensitivity and supports both on- and off-policy learning in entropy-regularized diffusions (Jia et al., 2022).

6. Theoretical Guarantees and Limitations

All major Hamiltonian Q-learning frameworks provide contraction properties for their respective Bellman-type operators, convergence to near-optimal solutions under standard regularity assumptions, and provable consistency in the continuous- and discrete-time limits (Kim et al., 2020, Madhushani et al., 2020, Kim et al., 2019, Jia et al., 2022). Key technical assumptions include:

• Smoothness and Lipschitz continuity of dynamics, rewards, and (for low-rank methods) incoherence of the Q-matrix.
• Existence of viscosity solutions for HJB PDEs.
• Feasibility of parameter selection for HMC or quantum algorithms. Noted limitations include HMC parameter tuning, convex optimization overhead in matrix completion, sensitivity to poorly mixing dynamics in HMC, and exponential scaling for quantum hardware with problem size (Madhushani et al., 2020, Jones et al., 2020).

7. Significance and Future Directions

Hamiltonian Q-learning generalizes the scope of reinforcement learning algorithms by:

• Enabling principled treatment of continuous-time, continuous-control settings under model-free and data-driven conditions, naturally bridging to stochastic optimal control.
• Delivering sample and computational efficiency in high-dimensional, structure-rich problems (via HMC and low-rank exploitation).
• Building a direct pathway from RL problem statements to quantum-implementable Hamiltonians, foreshadowing scalable quantum-enhanced decision-making.
• Unifying policy gradient and temporal-difference ideas in a continuous-time martingale-or Hamiltonian-based calculus. Future research will address robustness under more general noise models, improved numerical stability for deep architectures, scalability of quantum approaches, and theoretical guarantees beyond idealized assumptions (Kim et al., 2020, Madhushani et al., 2020, Jones et al., 2020, Jia et al., 2022, Kim et al., 2019).