---
title: 'RT-HCP: Real-Time Hybrid Control'
url: https://www.emergentmind.com/topics/rt-hcp
type: topic
---

# RT-HCP: Real-Time Hybrid Control

RT-HCP is a real-time reinforcement-learning framework for learning controllers directly on physical robots under simultaneous sample-efficiency and inference-time constraints. It is introduced as **“Real-Time Hybrid Control with Physics-informed model”** and is motivated by the observation that model-based reinforcement learning can be highly sample efficient yet still fail on hardware when its inference time exceeds the robot’s control period. The method addresses this by combining a physics-informed dynamics model, **\(d\)-step MPC**, and a learned actor-critic policy within a delay-aware execution framework, and it is validated on a real **FURUTA pendulum** platform with a **\(\Delta t = 20\ \text{ms}\)** sampling period [2509.06714].

## 1. Concept and problem setting

RT-HCP is formulated for the regime in which a controller must be trained **directly on the robot**, making real data scarce, expensive, and operationally constrained. The central problem is not only sample efficiency but also the requirement that a controller produce actions fast enough to satisfy a fixed control frequency. In the RT-HCP formulation, this becomes critical for **model-based RL (MBRL)** methods because they often require trajectory optimization at each control step, which can make inference too slow for deployment on the physical system [2509.06714].

The paper distinguishes this setting from standard simulation-centric RL by treating **inference delay** as a first-class systems problem. If the inference time for a single-step decision is \(T_i^1\), then delay appears whenever

$$
T_i^1 > \Delta_t,
$$

where \(\Delta_t\) is the robot sampling period. In that case, the controller does not produce a new action when the plant requires one, creating an **execution gap**. A core premise of RT-HCP is therefore that sample-efficient planning must be redesigned so that the robot remains continuously actuated while planning is still in progress [2509.06714].

This suggests a broader interpretation of real-time RL on hardware: practical performance is governed by a three-way trade-off among control quality, data efficiency, and decision latency rather than by reward or sample complexity alone.

## 2. Delay-aware control framework

RT-HCP is built around a general framework for delayed inference in which the controller does not compute a single action but rather a **sequence of actions**. If inference spans \(d\) control steps, the planner produces

$$
[a_t, a_{t+1}, \dots, a_{t+d-1}],
$$

and stores this sequence in an action buffer. The robot then executes buffered actions at every control step while the next sequence is being computed. The paper describes this as **\(d\)-step MPC**, with the explicit purpose of avoiding execution gaps [2509.06714].

To recover a reinforcement-learning formulation compatible with this delayed actuation pattern, the paper defines a **delay-MDP** with augmented spaces:

$$
\mathcal{A}' = \mathcal{A}^d,
$$

$$
\mathcal{S}' = \mathcal{S}^d \cdot \mathcal{A}^d,
$$

$$
\mathcal{T}' = \mathcal{T}^d,
$$

$$
\mathcal{R}' = \mathcal{R}^d.
$$

The augmented state contains the current observed state, the missed intermediate states, and the buffered actions. The paper writes an augmented state in the form

$$
s'_t = \{s_{d\cdot t}, s_{d\cdot t-(d-1)}, \ldots, s_{d\cdot t-1}, a_{d\cdot t}, \ldots, a_{d\cdot t + (d-1)}\}.
$$

Its stated purpose is to restore the **Markov property** despite delayed inference [2509.06714].

The framework also provides a concrete rule for selecting the execution horizon. One chooses a sampling period \(\Delta_t\) and planning horizon \(H^p\) such that

$$
T_i^{H^p} < H^p \Delta_t,
$$

measures the actual inference time \(T_i^{H^p}\), and then selects an execution horizon \(H^e\) satisfying

$$
H^e_{\min} \le H^e \le H^p,
$$

with

$$
H^e_{\min} = \mathrm{int}\left(\frac{T_i^{H^p}}{\Delta_t}\right) + 1.
$$

The controller then uses **\(d\)-step MPC with \(d = H^e\)** [2509.06714].

## 3. Algorithmic composition

RT-HCP combines three components: **a physics-informed dynamics model**, **\(d\)-step MPC**, and **a learned actor-critic policy**. Its learning loop proceeds in three stages. First, the agent performs online planning and data collection on the real robot using hybrid planning that combines **MPC** with a learned policy \(\pi\), acting for \(d\) steps from the precomputed action buffer and storing transitions in a real replay buffer \(D_{\text{real}}\). Second, every \(N\) steps, it updates both the physics-informed model and the policy \(\pi\) using real data. Third, the updated model generates synthetic rollouts stored in \(D_{\text{im}}\), which are then used to further train \(\pi\). The paper explicitly characterizes this as essentially a **Dyna-style** learning loop adapted to real-time robotics with delayed inference [2509.06714].

The planning objective is given as an MPC-style optimization:

$$
A = \arg\max_{a_{t_0:t_0+H}} \left( \sum_{t=t_0}^{H} \gamma^{t-t_0} R(s_t, a_t) + \gamma^{H-t_0} Q(s_{t_0+H}) \right)
$$

subject to

$$
s_{t+1} = \hat{\mathcal{T}}_\theta(s_t, a_t).
$$

In the paper’s interpretation, this objective uses a learned transition model \(\hat{\mathcal{T}}_\theta\) and a terminal value term \(Q\), allowing shorter planning horizons than a purely long-horizon optimizer would require [2509.06714].

A key design claim is that RT-HCP improves the trade-off among performance, sample efficiency, and inference time by using **shorter planning horizons**, a **small CEM population**, **policy-initialized CEM samples**, and a **terminal Q-function**. The learned policy thus serves not only as an execution policy but also as a planning prior that accelerates the optimizer [2509.06714].

## 4. Physics-informed model and optimization details

The paper instantiates RT-HCP on the **real Furuta pendulum**, with state

$$
s_t = (\alpha_t, \beta_t, \dot{\alpha}_t, \dot{\beta}_t)
$$

and action

$$
a_t = V_t.
$$

The dynamics prior is a **frictionless two-link rotary inverted pendulum model derived from Euler–Lagrange equations**. The accelerations are written as

$$
\begin{pmatrix}\ddot{\alpha} \\ \ddot{\beta}\end{pmatrix}
=
M^{-1}(\beta)
\left(
\begin{pmatrix}\tau(a) \\ 0\end{pmatrix}
-
N(\beta,\dot{\alpha},\dot{\beta})
-
G(\beta)
\right),
$$

with explicit expressions for \(M(\beta)\), \(N(\beta,\dot{\alpha},\dot{\beta})\), \(G(\beta)\), and \(\tau(a)\) given in the paper. The analytical prior is stated to omit friction, actuator delay, cable effects, and parameter mismatch, so RT-HCP adds a **residual neural network** consisting of a **4-layer MLP** with **16 neurons** per layer to correct the physics model [2509.06714].

For online trajectory optimization, RT-HCP uses **CEM** with **\(I = 3\) iterations**, **\(P = 500\) samples**, and **50 samples initialized from the policy \(\pi\)**. These choices are explicitly motivated by inference-time constraints: CEM is the expensive part of MPC, and reducing its computational burden is central to keeping control real-time on the platform [2509.06714].

This architecture makes RT-HCP hybrid in two senses. It is hybrid between model-based and model-free learning, and it is hybrid between analytical dynamics and learned residual correction. A plausible implication is that the method uses structure where it is reliable and learning where that structure is incomplete.

## 5. Empirical evaluation on the Furuta pendulum

The evaluation is conducted on a **simple but high frequency FURUTA pendulum platform** with **\(\Delta t = 20\ \text{ms}\)**. RT-HCP is compared against **TD3**, **RT-TDMPC**, and **RT-PETS**, with the model-based baselines adapted to the same delay-handling framework for fairness [2509.06714].

The reported results emphasize both learning speed and closed-loop quality. RT-HCP reaches a reward of **300 after about 60k steps (~20 minutes)**, whereas **TD3 needs about 100k steps** and **PETS needs about 160k steps**. In final control performance, RT-HCP achieves the **fastest swing-up**, the **most stable stabilization**, and the **smallest rotor deviation**, with a reported **swing-up time about 1.67 s** and **rotor deviation about 0.53 rad** [2509.06714].

The paper also studies how performance changes with planning horizon. RT-HCP is reported to degrade **less** than pure data-driven methods as horizon increases, which the authors attribute to the robustness of the physics-informed model over longer rollouts. Likewise, the learned RT-HCP model is reported to predict trajectories more accurately than the baselines, which matters because delayed-inference control requires reliable open-loop predictions to populate the action buffer [2509.06714].

The empirical interpretation given in the paper is that RT-HCP succeeds because it jointly addresses the two principal bottlenecks of direct real-robot learning: limited real experience and slow planning. The results are therefore presented not merely as an improvement in reward, but as evidence that delay-aware planning and physics-informed modeling can be coupled without sacrificing real-time execution.

## 6. Relation to adjacent robotics methods and nomenclature

RT-HCP occupies a distinct position among contemporary robot-learning systems. In the paper’s own comparison, **TD3** offers very fast inference but poor sample efficiency; **PETS** is sample efficient but can be too slow because of planning overhead; and hybrid methods such as **RT-TDMPC** reduce some of that burden but still rely on purely data-driven models and do not explicitly solve delayed inference in the same way [2509.06714].

It is also distinct from the robotics method **RT-H**, which stands for **Robot Transformer with Action Hierarchies** and introduces a language-motion intermediate layer between task instructions and low-level actions. RT-H is a vision-language imitation-learning framework based on action hierarchies, whereas RT-HCP is a real-time control framework built around delayed-inference MPC, a physics-informed model, and actor-critic learning [2403.01823]. The similarity of abbreviations can obscure that these methods address different problems: RT-H targets multi-task policy structure and language interventions, while RT-HCP targets direct on-robot learning under strict timing constraints.

Within this landscape, RT-HCP can be understood as a robotics-specific answer to a systems-level question: how to preserve the sample-efficiency advantages of MBRL when the hardware platform is control-hungry and the planner is computationally slow. The paper’s answer is to convert single-step decision making into buffered sequence execution, restore the Markov structure through state-action augmentation, and reduce planning cost by coupling MPC to a physics-informed model and a learned policy prior [2509.06714].

Source: https://www.emergentmind.com/topics/rt-hcp