RT-HCP: Real-Time Hybrid Control

• RT-HCP is a real-time reinforcement learning framework that fuses sample-efficient planning with low inference time for direct on-robot control.
• It employs a delay-aware d-step MPC approach that buffers action sequences to overcome slow planning and meet strict control frequency requirements.
• RT-HCP integrates a physics-informed dynamics model with a learned actor-critic policy, achieving superior performance on platforms like the Furuta pendulum.

RT-HCP is a real-time reinforcement-learning framework for learning controllers directly on physical robots under simultaneous sample-efficiency and inference-time constraints. It is introduced as “Real-Time Hybrid Control with Physics-informed model” and is motivated by the observation that model-based reinforcement learning can be highly sample efficient yet still fail on hardware when its inference time exceeds the robot’s control period. The method addresses this by combining a physics-informed dynamics model, $d$-step MPC, and a learned actor-critic policy within a delay-aware execution framework, and it is validated on a real FURUTA pendulum platform with a $\Delta t = 20\ \text{ms}$ sampling period (Asri et al., 8 Sep 2025).

1. Concept and problem setting

RT-HCP is formulated for the regime in which a controller must be trained directly on the robot, making real data scarce, expensive, and operationally constrained. The central problem is not only sample efficiency but also the requirement that a controller produce actions fast enough to satisfy a fixed control frequency. In the RT-HCP formulation, this becomes critical for model-based RL (MBRL) methods because they often require trajectory optimization at each control step, which can make inference too slow for deployment on the physical system (Asri et al., 8 Sep 2025).

The paper distinguishes this setting from standard simulation-centric RL by treating inference delay as a first-class systems problem. If the inference time for a single-step decision is $T_i^1$, then delay appears whenever

$T_i^1 > \Delta_t,$

where $\Delta_t$ is the robot sampling period. In that case, the controller does not produce a new action when the plant requires one, creating an execution gap. A core premise of RT-HCP is therefore that sample-efficient planning must be redesigned so that the robot remains continuously actuated while planning is still in progress (Asri et al., 8 Sep 2025).

This suggests a broader interpretation of real-time RL on hardware: practical performance is governed by a three-way trade-off among control quality, data efficiency, and decision latency rather than by reward or sample complexity alone.

2. Delay-aware control framework

RT-HCP is built around a general framework for delayed inference in which the controller does not compute a single action but rather a sequence of actions. If inference spans $d$ control steps, the planner produces

$[a_t, a_{t+1}, \dots, a_{t+d-1}],$

and stores this sequence in an action buffer. The robot then executes buffered actions at every control step while the next sequence is being computed. The paper describes this as $d$-step MPC, with the explicit purpose of avoiding execution gaps (Asri et al., 8 Sep 2025).

To recover a reinforcement-learning formulation compatible with this delayed actuation pattern, the paper defines a delay-MDP with augmented spaces:

$\mathcal{A}' = \mathcal{A}^d,$

$\mathcal{S}' = \mathcal{S}^d \cdot \mathcal{A}^d,$

$\Delta t = 20\ \text{ms}$0

$\Delta t = 20\ \text{ms}$1

The augmented state contains the current observed state, the missed intermediate states, and the buffered actions. The paper writes an augmented state in the form

$\Delta t = 20\ \text{ms}$2

Its stated purpose is to restore the Markov property despite delayed inference (Asri et al., 8 Sep 2025).

The framework also provides a concrete rule for selecting the execution horizon. One chooses a sampling period $\Delta t = 20\ \text{ms}$3 and planning horizon $\Delta t = 20\ \text{ms}$4 such that

$\Delta t = 20\ \text{ms}$5

measures the actual inference time $\Delta t = 20\ \text{ms}$6, and then selects an execution horizon $\Delta t = 20\ \text{ms}$7 satisfying

$\Delta t = 20\ \text{ms}$8

with

$\Delta t = 20\ \text{ms}$9

The controller then uses $T_i^1$0-step MPC with $T_i^1$1 (Asri et al., 8 Sep 2025).

3. Algorithmic composition

RT-HCP combines three components: a physics-informed dynamics model, $T_i^1$2-step MPC, and a learned actor-critic policy. Its learning loop proceeds in three stages. First, the agent performs online planning and data collection on the real robot using hybrid planning that combines MPC with a learned policy $T_i^1$3, acting for $T_i^1$4 steps from the precomputed action buffer and storing transitions in a real replay buffer $T_i^1$5. Second, every $T_i^1$6 steps, it updates both the physics-informed model and the policy $T_i^1$7 using real data. Third, the updated model generates synthetic rollouts stored in $T_i^1$8, which are then used to further train $T_i^1$9. The paper explicitly characterizes this as essentially a Dyna-style learning loop adapted to real-time robotics with delayed inference (Asri et al., 8 Sep 2025).

The planning objective is given as an MPC-style optimization:

$T_i^1 > \Delta_t,$0

subject to

$T_i^1 > \Delta_t,$1

In the paper’s interpretation, this objective uses a learned transition model $T_i^1 > \Delta_t,$2 and a terminal value term $T_i^1 > \Delta_t,$3, allowing shorter planning horizons than a purely long-horizon optimizer would require (Asri et al., 8 Sep 2025).

A key design claim is that RT-HCP improves the trade-off among performance, sample efficiency, and inference time by using shorter planning horizons, a small CEM population, policy-initialized CEM samples, and a terminal Q-function. The learned policy thus serves not only as an execution policy but also as a planning prior that accelerates the optimizer (Asri et al., 8 Sep 2025).

4. Physics-informed model and optimization details

The paper instantiates RT-HCP on the real Furuta pendulum, with state

$T_i^1 > \Delta_t,$4

and action

$T_i^1 > \Delta_t,$5

The dynamics prior is a frictionless two-link rotary inverted pendulum model derived from Euler–Lagrange equations. The accelerations are written as

$T_i^1 > \Delta_t,$6

with explicit expressions for $T_i^1 > \Delta_t,$7, $T_i^1 > \Delta_t,$8, $T_i^1 > \Delta_t,$9, and $\Delta_t$0 given in the paper. The analytical prior is stated to omit friction, actuator delay, cable effects, and parameter mismatch, so RT-HCP adds a residual neural network consisting of a 4-layer MLP with 16 neurons per layer to correct the physics model (Asri et al., 8 Sep 2025).

For online trajectory optimization, RT-HCP uses CEM with $\Delta_t$1 iterations, $\Delta_t$2 samples, and 50 samples initialized from the policy $\Delta_t$3. These choices are explicitly motivated by inference-time constraints: CEM is the expensive part of MPC, and reducing its computational burden is central to keeping control real-time on the platform (Asri et al., 8 Sep 2025).

This architecture makes RT-HCP hybrid in two senses. It is hybrid between model-based and model-free learning, and it is hybrid between analytical dynamics and learned residual correction. A plausible implication is that the method uses structure where it is reliable and learning where that structure is incomplete.

5. Empirical evaluation on the Furuta pendulum

The evaluation is conducted on a simple but high frequency FURUTA pendulum platform with $\Delta_t$4. RT-HCP is compared against TD3, RT-TDMPC, and RT-PETS, with the model-based baselines adapted to the same delay-handling framework for fairness (Asri et al., 8 Sep 2025).

The reported results emphasize both learning speed and closed-loop quality. RT-HCP reaches a reward of 300 after about 60k steps (~20 minutes), whereas TD3 needs about 100k steps and PETS needs about 160k steps. In final control performance, RT-HCP achieves the fastest swing-up, the most stable stabilization, and the smallest rotor deviation, with a reported swing-up time about 1.67 s and rotor deviation about 0.53 rad (Asri et al., 8 Sep 2025).

The paper also studies how performance changes with planning horizon. RT-HCP is reported to degrade less than pure data-driven methods as horizon increases, which the authors attribute to the robustness of the physics-informed model over longer rollouts. Likewise, the learned RT-HCP model is reported to predict trajectories more accurately than the baselines, which matters because delayed-inference control requires reliable open-loop predictions to populate the action buffer (Asri et al., 8 Sep 2025).

The empirical interpretation given in the paper is that RT-HCP succeeds because it jointly addresses the two principal bottlenecks of direct real-robot learning: limited real experience and slow planning. The results are therefore presented not merely as an improvement in reward, but as evidence that delay-aware planning and physics-informed modeling can be coupled without sacrificing real-time execution.

6. Relation to adjacent robotics methods and nomenclature

RT-HCP occupies a distinct position among contemporary robot-learning systems. In the paper’s own comparison, TD3 offers very fast inference but poor sample efficiency; PETS is sample efficient but can be too slow because of planning overhead; and hybrid methods such as RT-TDMPC reduce some of that burden but still rely on purely data-driven models and do not explicitly solve delayed inference in the same way (Asri et al., 8 Sep 2025).

It is also distinct from the robotics method RT-H, which stands for Robot Transformer with Action Hierarchies and introduces a language-motion intermediate layer between task instructions and low-level actions. RT-H is a vision-language imitation-learning framework based on action hierarchies, whereas RT-HCP is a real-time control framework built around delayed-inference MPC, a physics-informed model, and actor-critic learning (Belkhale et al., 2024). The similarity of abbreviations can obscure that these methods address different problems: RT-H targets multi-task policy structure and language interventions, while RT-HCP targets direct on-robot learning under strict timing constraints.

Within this landscape, RT-HCP can be understood as a robotics-specific answer to a systems-level question: how to preserve the sample-efficiency advantages of MBRL when the hardware platform is control-hungry and the planner is computationally slow. The paper’s answer is to convert single-step decision making into buffered sequence execution, restore the Markov structure through state-action augmentation, and reduce planning cost by coupling MPC to a physics-informed model and a learned policy prior (Asri et al., 8 Sep 2025).