State History Augmented LQR

• State History Augmented LQR is a control strategy that extends classical LQR by integrating past state trajectories via time-delay embedding for enhanced handling of nonlinear and hybrid dynamics.
• It employs lifting techniques to create globally linear representations, allowing the use of standard LQR design while managing periodic behaviors and switching events.
• Validated on systems like the bouncing pendulum and bipedal walker, this method improves stabilization performance, though it introduces high-dimensional challenges requiring careful model reduction.

A State History Augmented Linear Quadratic Regulator (LQR) extends the classical LQR framework by incorporating past state trajectories—rather than relying solely on the current state—for feedback control. This augmentation is motivated by the need to manage complex, possibly nonlinear or hybrid systems (such as those with switching dynamics or periodicity) where Markovian models may be insufficient or where richer temporal context improves control accuracy. The use of time-delay embeddings and related techniques allows for the construction of globally linear representations of such systems, enabling the application of LQR theory and synthesis in domains previously intractable for classical methods.

1. Time-Delay Embedding and System Lifting

State history augmentation is primarily enabled through time-delay embedding. In this approach, a sequence of past states (and possibly inputs) is stacked into a higher-dimensional “delay-embedded” state vector. For a given discrete-time sequence $\{x_k, u_k\}$, the embedding forms blocks such as

$$X_i = [x_i, x_{i+1}, ..., x_{i+N}] \ U_i = [u_i, u_{i+1}, ..., u_{i+N}]$$

These vectors are used to construct Hankel matrices containing overlapping windows of the system’s past trajectory.

A finite-dimensional linear operator $L$ is then identified such that

$H_{k+1} = L H_k, \qquad L = H_{k+1} H_k^+$

where $H_k$ and $H_{k+1}$ are Hankel matrices constructed from time-shifted delay-embedded data and $^+$ denotes the pseudoinverse. This linear model is a data-driven approximation of the Koopman operator, which acts on the space of observables formed by the delay blocks. The crucial insight is that even for nonlinear or hybrid systems, if their periodic or switching behavior is consistent in timing and mode sequence, time-delay embedding can “lift” the dynamics into a linear form in this expanded space (Yang et al., 19 Jul 2025).

2. State History Augmented LQR Formulation

Once a lifted linear model is obtained, feedback synthesis proceeds via a retrospective (as opposed to purely reactive) control architecture. The augmented state $X_i$ incorporates the current and $N$ previous states; the augmented input $U_i$ is similarly stacked.

The global linear model can be expressed as:

$$\begin{bmatrix} X_{i+1} \ U_{i+1} \end{bmatrix} = \begin{bmatrix} L_{11} & L_{12} \ L_{21} & L_{22} \end{bmatrix} \begin{bmatrix} X_i \ U_i \end{bmatrix}$$

Defining the error dynamics about a reference periodic trajectory $(X^d, U^d)$ yields:

$\hat{X}_{i+1} = A \hat{X}_i + B \hat{U}_i$

where $\hat{X}_i = X_i - X_i^d$ and similarly for $\hat{U}_i$. The control law is constructed as:

$U_i = U_i^d - K_{\text{LQR}} \hat{X}_i$

with gain $K_{\text{LQR}}$ determined by solving the standard discrete-time algebraic Riccati equation for the linearized, delay-embedded system. This law utilizes a block of past states and inputs, effectively incorporating temporal context into each decision (Yang et al., 19 Jul 2025).

3. Application to Periodic Hybrid Systems

The methodology has been demonstrated on periodic hybrid systems such as the bouncing pendulum and the simplest planar bipedal walker. For these systems, the dynamics involve both continuous phases and discrete event-driven transitions. The embedding window is chosen to span at least one full cycle of the periodic evolution, ensuring that every relevant mode and switching event is represented within the data.

For each system, a delay embedding (e.g., $N=110$, $M=90$, $dt=0.01s$ for the hybrid pendulum) is constructed and used to fit a matrix Koopman operator $L$. The accuracy of this linear model is quantified using error metrics such as root-mean-square error (RMSE) on repeated open-loop rollouts. When external disturbances (e.g., impulsive kicks) are applied, the state history augmented LQR successfully stabilizes the system’s periodic orbit by deploying feedback based on the entire recent trajectory, not just the instantaneous state (Yang et al., 19 Jul 2025).

4. Design Advantages and Implementation Considerations

Key advantages of the state history augmented LQR include:

• Suitable for Hybrid/Nonlinear/Periodic Dynamics: Time-delay embedding transforms complex and possibly non-smooth periodic hybrid systems into globally linear models, to which LQR theory applies.
• Retrospective Feedback: The controller has a memory effect, leveraging the sequence of past states to produce less reactive and potentially smoother control actions.
• Global Linearization: As the lifted model is linear, standard design, analysis, and robustness tools from LQR carry over directly.

However, several technical and practical challenges must be addressed:

• Dimensionality: The embedding can result in high-dimensional lifted states, with direct implications for computational cost in control synthesis and execution.
• Embedding Consistency: The approach is contingent on the periodicity and invariance of the hybrid switching logic. Perturbations that shift switching times can undermine the quality of the Koopman-based linear approximation.
• Reference Trajectory Tracking: The practical method involves tracking a rich reference (limit cycle) and computing error dynamics using the augmented state.

Model reduction techniques may be necessary to render the approach tractable in systems with very long cycles or high embedded dimension.

5. Relation to Broader LQR Theory and Future Directions

State history augmentation generalizes the feedback model by synthesizing controllers that process windows of recent trajectory data, rather than assuming perfect Markovianity. This aligns with, but also extends, themes in robust and output feedback LQR, where augmented system representations are deployed to address uncertainty or partial observability.

Open directions for future research include relaxing the assumptions of strict periodicity and synchronization in hybrid systems, developing automatic model reduction for delay-embedded lifted models, and integrating data-driven Koopman modeling with other control strategies for general non-smooth or non-periodic hybrid dynamics (Yang et al., 19 Jul 2025).

6. Conclusion

State History Augmented LQR leverages time-delay embedding and Koopman operator theory to build linear models from nonlinear, periodic, and hybrid dynamical systems. Controllers synthesized in this lifted space use retrospective information over past temporal windows, affording precise and robust stabilization even in the presence of hybrid events. The methodology’s success in complex benchmark systems underscores its potential as a general tool in modern control, provided high-dimensionality and embedding assumptions are managed appropriately.