Continuous-Time Reinforcement Learning

• Continuous-Time Reinforcement Learning is a framework that models sequential decision-making in dynamic systems using continuous dynamics (ODEs, SDEs, CTMDPs).
• It employs advanced methodologies such as model-based learning, actor-critic updates, and physics-informed networks to tackle exploration, sample efficiency, and distributional objectives.
• CTRL has broad applications in robotics, finance, and neuroscience, addressing challenges like high-dimensional nonlinearity and adaptive measurement scheduling.

Continuous-time reinforcement learning (CTRL) is the study and design of sequential decision-making algorithms for dynamical systems whose evolution is naturally and fundamentally continuous in time. Unlike standard discrete-time reinforcement learning, where interactions and policies are typically constructed on a fixed temporal grid, CTRL frameworks model agent–environment interactions as either deterministic ordinary differential equations (ODEs), stochastic differential equations (SDEs), or continuous-time Markov decision processes (CTMDPs). Core paradigms in CTRL address both the theoretical and computational challenges of learning optimal (or near-optimal) policies under general function approximation, sample-efficient exploration, distributional objectives, and adaptive measurement protocols. This area is motivated by physical, biological, and engineered systems that evolve continuously, such as robotics, finance, neuroscience, and controlled stochastic processes.

1. Formal Models and Objective Criteria

CTRL proceeds from analytic foundations established by continuous-time dynamical system theory and stochastic control. The canonical models are:

• Ordinary Differential Equation (ODE) Control: Systems with state $x(t) \in \mathbb{R}^d$ evolving under

$\dot{x}(t) = f(x(t), u(t)),$

where $u(t)$ is the (possibly stochastic) control action.

• Stochastic Differential Equation (SDE) Control: The state process $x(t)$ evolves under

$dx(t) = f(x(t), u(t))\,dt + g(x(t), u(t))\,dW(t),$

with $W(t)$ a (multi-dimensional) Wiener process and possibly state- and action-dependent diffusion terms. Policies $u(t) = \pi(x(t), t)$ may be deterministic or randomized.

• Continuous-Time Markov Decision Process (CTMDP): Described by state space $\mathcal{X}$, action set $\mathcal{U}$, transition rates, and reward functions $r(x, u)$. State transitions occur over continuous time according to a rate matrix $R$.

The agent seeks to optimize a performance criterion over a finite or infinite time horizon, such as cumulative rewards

$$J(\pi) = E_\pi\left[\int_{0}^{T} r(x(t), u(t))\,dt \right],$$

or discounted infinite-horizon returns. Distributional objectives, risk measures, or constraints may also be considered (Wiltzer et al., 2022).

The central value function $V(x)$ and action-value (Q-function) $Q(x,u)$ satisfy the continuous-time Hamilton–Jacobi–Bellman (HJB) equation, for example: $$0 = \max_{u \in \mathcal{U}} \left\{ r(x, u) + \nabla V(x)^\top f(x, u) + \tfrac{1}{2} \text{Tr}[g(x, u)g(x, u)^\top \nabla^2 V(x)] \right\}.$$

2. Algorithmic Paradigms and Function Approximation

Key algorithmic approaches in CTRL include:

• Model-Based CTRL: Dynamics $f^*$ are unknown and are learned using probabilistic function approximators. Bayesian neural ODEs or Gaussian processes (GPs) are frequently used to represent $f^*$ and to capture epistemic uncertainty. Model-based methods leverage optimistic planning principles under epistemic confidence sets to drive sample-efficient exploration and learning (Yıldız et al., 2021, Treven et al., 2023, Iten et al., 28 Oct 2025). For CTMDPs, this generalizes to construction of confidence sets over drift and reward functions utilizing the distributional Eluder dimension to control approximation error (Zhao et al., 20 May 2025, Zhao et al., 4 Aug 2025).
• Actor–Critic and Policy Gradient Methods: Policy parameterizations are updated using temporal-difference (TD) errors and gradients computed with respect to continuous-time occupation measures and instantaneous advantage-rate functions. The continuous-time policy gradient theorem is established as

$$\nabla_\theta J(\theta) = \mathbb{E}_{\rho_\pi} [\nabla_\theta \log \pi_\theta(a|x) \, A_\pi(x, a)],$$

where $A_\pi$ is the instantaneous advantage computed via the generator of the SDE and the current value function (Zhao et al., 2023, Jia et al., 2022, Hua et al., 20 Oct 2025).

• Value Iteration and Physics-Informed Networks: Direct neural solution of the continuous-time HJB equations using physics-informed neural networks (PINNs) and value-gradient iteration modules ensures accurate propagation of value gradients and robust value estimation, overcoming the curse of dimensionality in multi-agent continuous-time RL (Wang et al., 11 Sep 2025).
• Adaptive Sensing and Event-Driven RL: Addressing the cost of interaction, time-adaptive RL frameworks treat both action selection and holding duration as joint decisions, recovering a discrete MDP over (action, duration) pairs that is solved with standard RL algorithms (Treven et al., 2024).

3. Measurement Strategies, Sample Efficiency, and Regret Bounds

A distinctive challenge in continuous time is the selection and scheduling of measurements for efficient learning:

• Measurement Selection Strategies (MSS): The regret and sample complexity of CTRL algorithms crucially depend on how often and where to sample the system (state and/or derivatives). Strategies include equidistant, adaptive receding-horizon, and oracle sampling. Adaptive MSSs focus samples in regions of high epistemic uncertainty, leading to sublinear regret with significantly fewer samples than naive equidistant approaches (Treven et al., 2023, Treven et al., 2024, Iten et al., 28 Oct 2025).
• Instance-Dependent Guarantees: Regret bounds in CTRL are tightly coupled to problem-dependent quantities such as total reward variance and measurement resolution. Recent work shows that instance-dependent bounds, scaling with reward variance and eluder dimension of transition marginal densities, can outperform worst-case, horizon-based bounds and become insensitive to the measurement schedule if observation frequency adapts to environment complexity (Zhao et al., 4 Aug 2025).
• Computational Efficiency: Structured policy updates, batch rollouts, and switching thresholds yield substantial reductions in policy-update and roll-out counts without loss of sample efficiency (Zhao et al., 20 May 2025).

4. Exploration–Exploitation Trade-offs and Distributional CTRL

CTRL must resolve continuous-time exploration–exploitation trade-offs:

• Entropy and Epistemic Regularization: Methods incorporate entropy regularization terms or explicit uncertainty bonuses in the objective, yielding policies that randomize actions over time and regimes. This enables principled exploration and smooths deterministic switching into probabilistic regime selection, as established rigorously in generator-randomized switching problems (Huang et al., 4 Dec 2025, Wang et al., 2019).
• Distributional RL in Continuous Time: Beyond mean value optimization, CTRL extends to distributional objectives using distributional HJB equations, quantile-based distributional approximations, and JKO gradient flow algorithms with entropic regularization. Quantile parameterizations eliminate extra statistical diffusivity terms and enable online, unbiased distributional learning, outperforming discrete-time DRL baselines on return distribution objectives (Wiltzer et al., 2022).

5. Advanced Topics: Multi-Agent Systems, Hybrid Controls, and Theoretical Foundations

• Multi-Agent Reinforcement Learning: Centralized critics and decentralized actors with PINN and VGI modules scale CTRL to multi-agent and high-dimensional settings, circumventing the curse of dimensionality with sample-based learning (Wang et al., 11 Sep 2025).
• Random Measure Theory and Relaxed Controls: Recent advances provide rigorous mathematical foundations for measure-valued control execution, establishing grid-sampling limit theorems that unify the "exploratory SDE" and "sample SDE" formulations, supporting both theoretical analysis and TD/actor-critic algorithm derivation (Bender et al., 2024).
• Linear–Quadratic and Affine Systems: For linear systems, randomized parameterization, stabilizing projections, and Riccati feedback yield sharp instance-dependent regret bounds scaling as $\mathcal{O}(\sqrt{T\,p}\log T)$, and adaptive dynamic programming via decentralized excitable integral RL achieves provable convergence and closed-loop stability in large-scale, nonlinear systems (Faradonbeh et al., 2021, Wallace et al., 2023).

6. Applications and Empirical Studies

CTRL techniques have demonstrated empirical success in domains including:

• Robotics and Classic Control: Pendulum, CartPole, Acrobot, and MuJoCo benchmarks are addressed using continuous-time model-based RL, with GP/Bayesian neural ODE dynamics models attaining higher sample efficiency and robustness to variable observation intervals and noise than discrete-time baselines (Yıldız et al., 2021, Treven et al., 2023, Iten et al., 28 Oct 2025).
• Finance: Mean-variance portfolio selection and asset-liability management are solved via entropy-regularized, exploratory control formulations with continuous-time policy gradient and actor-critic updates, outperforming both adaptive control and high-capacity neural RL methods in stability and Sharpe ratio (Wang et al., 2019, Huang, 27 Sep 2025).
• Diffusion Model Fine-Tuning: CTRL aligns generative diffusion models to human feedback by controlling score functions as actions within a continuous-time SDE optimization framework, outperforming discrete RL baselines on downstream image quality and alignment metrics (Zhao et al., 2024, Zhao et al., 20 May 2025).
• Omega-Regular Specifications and Logic-Guided RL: CTRL algorithms can address infinite-horizon logic specifications, such as those expressed in Büchi automata and LTL, by translating automaton objectives to reward structures and employing continuous-time Q-learning (Falah et al., 2023).

7. Limitations, Challenges, and Open Directions

Open challenges in CTRL include high-dimensional nonlinearity, real-world determinism versus full observability, computational complexity of optimistic planning in function approximation regimes, online adaptation of measurement frequency, handling partial observability (continuous-time POMDPs), and scalable batch planning for real-time deployment. Ongoing research seeks tighter instance-dependent performance bounds, adaptive information-theoretic exploration strategies, and extensions to event-triggered safety constraints.

---

Continuous-time reinforcement learning unifies stochastic control, function approximation, probabilistic modeling, and optimization theory to permit genuinely time-continuous, sample-efficient, and theoretically grounded learning in complex dynamic environments. Significant advances in the past five years have addressed key issues in sample efficiency, computational scaling, exploration, and distributional objectives, positioning CTRL as an essential paradigm for control-theoretic and learning-based applications in science and engineering. Recent efforts toward rigorous grid-sampling, martingale-based TD learning, and distributional control further expand the theoretical and practical toolkit available for continuous-time RL research (Yıldız et al., 2021, Zhao et al., 20 May 2025, Iten et al., 28 Oct 2025, Bender et al., 2024, Wiltzer et al., 2022).