Inverse Optimal Control for Linear Quadratic Problem with Poisson Jumps: Model-Free Inverse Reinforcement Learning Approaches

Abstract: This paper addresses the inverse optimal control (IOC) problem for stochastic linear systems subject to both Brownian motion and Poisson jumps, using an inverse reinforcement learning (IRL) framework. Given a target feedback gain from an expert, the objective is to identify an equivalent cost functional-specifically, the set of all cost weights-that yields this same gain. To solve this problem when system dynamics are unknown, we propose two model-free, off-policy IRL algorithms that operate entirely from data, circumventing the need to solve the generalized algebraic Riccati equation or compute the cost weights analytically. The first is an inverse Q-learning algorithm that constructs data-driven equations from expert demonstrations to compute the Q-function matrix, with equivalent cost weights updated algebraically and without requiring additional trajectory data. The second is a model-free off-policy inverse policy iteration algorithm that leverages data collected under an initial stabilizing policy, offering a complementary approach suited to different data availability scenarios. Crucially, by decoupling the data-collection behavior policies from the policies being iteratively updated, both algorithms can learn equivalent cost weights from sufficiently excited trajectories without identifying the system dynamics or jump intensity. Numerical simulations validate the effectiveness of the proposed methods.

• The paper proposes two novel model-free IRL methods to infer cost weight matrices from expert demonstrations and trajectory data.
• The algorithms guarantee convergence and stability by leveraging persistent excitation and rigorous rank conditions in system identification.
• Empirical results validate that learned feedback gains closely match expert policies, despite unobservable system dynamics and non-unique cost functionals.

Inverse Optimal Control for Stochastic LQ Systems with Poisson Jumps: Model-Free IRL Approaches

Introduction

The paper "Inverse Optimal Control for Linear Quadratic Problem with Poisson Jumps: Model-Free Inverse Reinforcement Learning Approaches" (2607.03169) addresses the inverse optimal control (IOC) problem for stochastic linear systems influenced by both continuous noise (Brownian motion) and discontinuities (Poisson jumps). The core challenge lies in deducing equivalent cost functionals that generate a prescribed feedback gain, typically derived from expert demonstrations, without access to the underlying system dynamics or jump intensity parameters. To this end, the authors propose two novel, model-free, off-policy inverse reinforcement learning (IRL) algorithms: an inverse Q-learning framework leveraging expert demonstrations and a model-free off-policy inverse policy iteration method utilizing learner trajectories collected under an arbitrary stabilizing policy.

This work extends classical IOC and IRL theory, which largely focused on deterministic or continuously-driven systems, toward practical, data-driven approaches compatible with stochastic systems exhibiting jump behavior. The methods circumvent the necessity for solving stochastic Riccati equations with jumps, instead relying on direct data-driven learning of cost weights, marking a significant advancement in the operationalization of IOC for systems with realistic, irregular uncertainty profiles.

Problem Formulation

The system dynamics are cast as a multivariate stochastic differential equation driven by both Brownian motion $W(t)$ and a Poisson random measure $N(ds,de)$. The objective is to discover cost weight matrices $(N, R)$ so that the optimal feedback control law generated from solving the associated stochastic LQ problem yields the same gain as an expert-provided $K_\mathcal{T}$. The mathematical formulation is:

$$\begin{aligned} dX(s) &= [AX(s) + Bu(s)]ds + [CX(s) + Du(s)]dW(s) \ & + \int_{\mathcal{E}} [E(e)X(s-) + F(e)u(s)]\widetilde{N}(ds,de) \end{aligned}$$

Given access only to expert gain $K_\mathcal{T}$ and trajectory data, but no system parameters, the IOC problem is recast as a constrained IRL challenge: find $(N, R)$ so that the induced optimal feedback gain matches $K_\mathcal{T}$.

The solution set is formally characterized using stochastic Riccati equations with Poisson jumps, but the algorithms eschew explicit Riccati-based computation in favor of sample-based learning.

Model-Free Inverse Q-Learning Algorithm

The first approach utilizes expert demonstrations to construct a data-driven Q-function matrix. The iterative algorithm constructs an algebraic equation connecting the Q-function evaluated on observed state-control pairs to the cost weights, exploiting the temporal difference between successive states and the known expert policy.

Figure 1: All simulations using Algorithm 1 (inverse Q-learning): (a) Convergence of $\mathbf{Q}$, $N$, and $N(ds,de)$0; (b) State trajectories of expert and learner under $N(ds,de)$1; (c) Control performance under $N(ds,de)$2.

Key technical features:

• Algebraic Update of Cost Weights: The cost matrix $N(ds,de)$3 is refined iteratively using the discrepancy between the learned policy and the expert gain, without further trajectory sampling.
• Persistent Excitation: A probing noise is injected into the expert inputs to ensure the excitation of all system modes and guarantee the uniqueness of the Q-function estimation.
• Stability and Convergence Guarantees: The algorithm provably generates a sequence of stabilizing feedback gains converging to $N(ds,de)$4.

Empirical results show that $N(ds,de)$5 aligns closely with the expert gain (error $N(ds,de)$6), even though $N(ds,de)$7 may differ significantly from the original $N(ds,de)$8, confirming the non-uniqueness and equivalence classes of optimal cost functionals in stochastic IOC with jumps.

Model-Free Off-Policy Inverse Policy Iteration Algorithm

The second algorithm operates when expert trajectories are unavailable. The learner collects its own state and input trajectories by adopting an arbitrary stabilizing policy $N(ds,de)$9, augmented with probing noise to ensure data richness. The algorithm builds vectorized matrix equations involving Kronecker products to relate trajectory statistics to unknown cost weights, then solves these via batch least-squares.

Figure 2: All simulations using Algorithm 2 (inverse policy iteration): (a) Convergence of $(N, R)$0, $(N, R)$1, and $(N, R)$2; (b) State trajectories of expert and learner under $(N, R)$3; (c) Control performance under $(N, R)$4.

Technical points:

• Off-Policy Data Utilization: By decoupling the data collection policy from the iteration of feedback gains, the method is agnostic to the underlying stabilizer and does not require access to expert data.
• Algebraic Uniqueness via Rank Conditions: Rigorous rank conditions on the constructed data matrices ensure that the least-squares solution is unique and statistically robust.
• Equivalence of Algorithmic Iterations: The sequence of learned cost weights and gains is proven to be equivalent to the inverse Q-learning approach; both methods converge to feedback gains arbitrarily close to $(N, R)$5.

Numerically, the learned gain $(N, R)$6 achieves near-identical performance to the expert policy, with rapid system stabilization and output tracking, despite discrepancies in the recovered cost weights.

Figure 3: State trajectory $(N, R)$7 under an initial stabilizer $(N, R)$8, demonstrating excitation and system stabilization.

Numerical Results

Both algorithms are benchmarked on a 2D stochastic LQ system with specified matrices for $(N, R)$9, $K_\mathcal{T}$0, $K_\mathcal{T}$1, $K_\mathcal{T}$2, and jump coefficients. Simulations validate convergence of learned feedback gains to the expert's $K_\mathcal{T}$3, with strong tracking performance and control. The cost weights $K_\mathcal{T}$4 and $K_\mathcal{T}$5 differ from $K_\mathcal{T}$6, reinforcing the theoretical assertion that multiple distinct cost functionals can underlie the same optimal policy in the jump-diffusion LQ setting.

Implications and Future Directions

The introduced model-free, off-policy IRL algorithms represent significant progress for practical IOC in systems with jump disturbances. They facilitate data-driven learning of interpretable cost functionals under minimal information requirements—no model identification, no jump intensity estimation, no Riccati solving.

Theoretically, these results clarify the structure of equivalence classes for cost functionals in stochastic LQ problems with jumps, extending the reach of classical IOC and IRL. Practically, there is direct applicability to finance, power systems, and cyber-physical domains where unmodeled shocks and unknown system dynamics prevail.

Future lines of inquiry may target:

• Extension to nonlinear stochastic systems with jumps.
• Online adaptive algorithms with nonstationary environments.
• Incorporation of partial observability or output-feedback constraints.
• Real-world deployment in high-dimensional, multi-agent settings.

Conclusion

This paper establishes two rigorously proven, model-free IRL algorithms for stochastic LQ systems with Poisson jumps, enabling the identification of equivalent cost functionals solely from trajectory data and desired feedback gains. The convergence, stability, and consistency properties are analytically and empirically confirmed. The results advance IOC and IRL theory for complex stochastic systems, paving the way for robust, interpretable data-driven control design in challenging, uncertain environments.