Inverse Stochastic Control: Theory & Algorithms

Updated 21 January 2026

Inverse stochastic control is a framework that infers underlying cost structures from observed data in environments with stochastic, nonlinear, and partially observed dynamics.
The discipline leverages convex optimization and variational principles to recast complex control problems into tractable optimization programs with global guarantees.
Algorithmic implementations include backward recursion, maximum likelihood estimation, and sum-of-squares relaxations to ensure robust and efficient cost recovery.

Inverse stochastic control is the discipline concerned with inferring underlying objective functions or cost structures in stochastic control problems from observed data, typically agent demonstrations or empirical trajectories. It extends the framework of inverse optimal control to environments with stochastic, potentially nonlinear and partially observed dynamics, rigorously integrating probabilistic modeling, convex optimization, and variational principles. The field encompasses foundational results for both model-based and model-free settings, including consistent estimators, optimality conditions, and computational algorithms for a broad range of stochastic systems.

1. Problem Formulations and Theoretical Foundations

Inverse stochastic control (ISC) seeks to determine the underlying cost function or objective (and sometimes system noise parameters) that make observed agent behavior optimal, or nearly optimal, relative to a forward stochastic control problem. This involves settings where agents are observed in environments governed by stochastic (possibly nonlinear or non-stationary) dynamics, with the demonstrator’s policy optimal for some latent cost. The standard formalism consists of:

Controlled Markov or diffusion dynamics, either discrete or continuous time, often with general nonlinearities and additive or multiplicative noise (Garrabe et al., 2023, Wang et al., 27 Nov 2025, Nakano, 2020).
Expert demonstrations: datasets composed of observed state-action or trajectory sequences generated by an agent assumed to act optimally (or approximately so) with respect to their latent cost function.
Cost parameterization: Usually, the instantaneous or cumulative cost is modeled as a linear combination of known feature functions with unknown weights, but generalizations to nonconvex and nonstationary forms appear (Garrabe et al., 2023, Wang et al., 27 Nov 2025).

Fundamental theoretical results include well-posedness proofs, uniqueness and identifiability conditions (usually up to positive scaling), and convexity of the inverse estimation objectives under suitable parameterizations. For example, it is established that for linear quadratic diffusions with quadratic control penalties and sufficient regularity, the inverse problem can be reduced to solving a root-finding problem for a monotonic functional of the cost parameter (Nakano, 2020), and that, more generally, inverse estimation is well-posed provided the observed controls are non-degenerate over a set of positive measure.

2. Convex and Variational Approaches

A central advance in recent work is the development of convex optimization formulations for the inverse problem, even in the presence of highly nonlinear, non-stationary, and stochastic environments. Key features include:

Convexity: Forward and inverse problems are recast as strictly convex programs over occupancy measures (distribution over state-action pairs) or over the weights of parameterized costs, making global optima accessible by off-the-shelf solvers (Garrabe et al., 2023, Wang et al., 27 Nov 2025, Garrabé et al., 2023).
Finite- and infinite-dimensional LPs: The forward stochastic optimal control problem is lifted to an infinite-dimensional linear program (LP) over occupancy measures, with the dual associated to the Bellman value function. The inverse problem then becomes an LP or conic program conditioned to ensure optimality (KKT conditions) (Wang et al., 27 Nov 2025).
Sum-of-squares (SOS) relaxations: For nonlinear systems, the nonnegativity constraints arising from Bellman slack or complementary slackness can be enforced using SOS polynomials, leading to tractable semidefinite programming relaxations (Wang et al., 27 Nov 2025).
Variational principles: The minimization of suboptimality gaps (difference between observed and optimal costs for candidate latent objectives) leads to variational inverse formulations. The dual problem is shown to coincide with a generalized dynamic Schrödinger bridge (stochastic optimal transport with marginal constraints), establishing a strong mathematical link between ISC and stochastic optimal transport (Nakano, 14 Jan 2026).

3. Algorithmic Implementation and Computational Methods

Inverse stochastic control has motivated a variety of algorithmic strategies, typically tailored to the specific structure (finite vs. infinite horizon, linear vs. nonlinear, full vs. partial observability). Prominent approaches include:

Backward recursion and convex cost recovery: For finite-horizon systems, forward problems are solved via backward induction to compute value-like quantities, and the optimal randomized policy has an explicit exponential form ("Boltzmann policy") (Garrabe et al., 2023, Garrabé et al., 2023).
Convex maximum likelihood estimation: Under an exponential-family policy model, the negative log-likelihood for observed data reduces to a sum of log-sum-exp and linear terms in the cost parameters, yielding strict convexity (Garrabe et al., 2023); closed-form expressions for the gradient exist and global convergence is attainable.
Sum-of-squares polynomial approximation: Occupancy-measure LPs are approximated by polynomials and SOS after discretization, providing finite-dimensional convex programs for nonlinear systems with guaranteed statistical consistency (Wang et al., 27 Nov 2025).
Bi-level and alternating optimization: In special settings like linear-quadratic Gaussian (LQG) or linear-quadratic sensorimotor (LQS) models, the inverse problem is formulated as a bi-level program. The outer optimization matches mean and covariance of trajectories, while the inner solves Riccati-type recursions for optimal control, with moment-propagation and variance explained (VAF) criteria in the outer loss (Karg et al., 2023, Karg et al., 2022).
Kernel-based collocation for HJB equations: When explicit value functions are unavailable, kernelized collocation methods allow for flexible numerical solution of forward HJB equations, enabling plug-and-play cost-parameter estimation from data (Nakano, 2020).
Variational parameter estimation: For continuous-time diffusions, the suboptimality gap over observed marginal laws is minimized directly with respect to latent cost parameters (no nested forward/backward sweeps), with stochastic gradient descent when closed-form solutions are available (Nakano, 14 Jan 2026).

4. Extensions: Partial Observability, Nonlinearities, and Model-Free Regimes

Modern ISC frameworks accommodate realistic complexities such as partial observability, nonlinearities, and weak modeling assumptions:

Partial observability: Probabilistic approaches unify maximum causal entropy RL and explicit noise models for sensory and motor systems, employing belief-state tracking (e.g., with extended Kalman filters) and local linearization for efficient pseudo-likelihood computation. This allows disentanglement of perceptual noise from behavioral costs and efficient parameter recovery even with missing actions (Straub et al., 2023).
Model-free or off-policy IRL: Least-squares approaches estimate cost parameters directly from batches of observed (possibly exploratory) data, without knowledge of full system dynamics. This is accomplished by recasting the inverse problem as a regression over integral equations derived from stochastic maximum principles and Lyapunov conditions (Sun et al., 2024).
Nonconvex and nonstationary cost estimation: Convex optimization methods for cost recovery remain valid for nonconvex, time-varying, and stochastic settings, as demonstrated in nonlinear and high-dimensional environments (Garrabe et al., 2023).

5. Numerical Experiments, Empirical Results, and Performance

Numerical validation spans canonical control problems, robotic tasks, and applications in sensorimotor neuroscience. Across methods, the following empirical outcomes are reported:

Consistent, statistically convergent estimators: As sample size or approximation fidelity increases, cost recovery error converges to zero with high probability (Wang et al., 27 Nov 2025).
Superior speed and scalability: Convex inverse algorithms based on direct cost-maximization or gradient-based optimization display runtime orders of magnitude faster than classic approaches based on nested value iteration or policy evaluation (Garrabe et al., 2023, Karg et al., 2023).
Accurate cost-structure and noise recovery: Methods match expert cost weights and noise parameters in LQG, LQS, and nonlinear settings with low relative errors and variance-accounted-for rates exceeding 99% in simulated and real-robot benchmarks (Karg et al., 2022, Karg et al., 2023).
Robustness and generalizability: Inverse solutions remain stably performant under moderate model mismatch, stochastic perturbations, or variations in observed data fidelity. IOC-based controllers exhibit greater robustness than direct behavioral cloning when tested in mismatched environments (Wang et al., 27 Nov 2025).
Disentanglement of noise and behavioral cost: Specialized probabilistic approaches are empirically able to distinguish increased epistemic noise from explicit behavioral cost, even under intertwined perceptual and pragmatic objectives (Straub et al., 2023).

6. Mathematical and Conceptual Connections

The structure of inverse stochastic control reveals numerous conceptual and mathematical connections:

Relationship to optimal transport: The dual equivalence between variational ISC and generalized dynamic Schrödinger bridge problems links ISC with the stochastic optimal transport framework, fundamentally unifying inference under stochastic control with transport-theoretic path coupling (Nakano, 14 Jan 2026).
Quantum inverse scattering: Formulations in terms of linear Markov decision processes (LMDPs) enable an equivalence between Bellman–HJ equations and Schrödinger-type PDEs, with potential recovery via one-sided inverse scattering (Marčenko integrals) (Schneider et al., 2022).
Maximum entropy and linearly-solvable MDPs: The exponential-twisting (Boltzmann) form for optimal policies in convex inverse frameworks generalizes MaxEnt IRL from discrete MDPs to broad stochastic systems (Garrabe et al., 2023, Garrabé et al., 2023).

7. Open Challenges and Future Directions

Areas for ongoing research include:

Explicit sample complexity bounds for finite data regimes (Garrabe et al., 2023).
Extendibility to constrained or safety-critical policies, multi-agent interactions, and real-time (online) inverse control.
Kernelized or nonparametric cost representations for infinite-dimensional or highly structured feature spaces.
Efficient algorithms for inverse inference in high-dimensional continuous-time diffusions without closed-form Riccati or HJB solutions (Nakano, 14 Jan 2026).
Further integration of ISC with stochastic optimal transport and Schrödinger bridge theory to enable scalable, distribution-constrained inference.
Theory and practice for structured regularization (sparsity, group norms) to handle large feature sets or physiological priors (Garrabe et al., 2023).

Inverse stochastic control has thus emerged as a foundational domain in contemporary control theory, bridging stochastic optimal control, probabilistic inference, learning from demonstration, and variational analysis to enable principled identification of costs and behavioral objectives directly from data (Garrabe et al., 2023, Wang et al., 27 Nov 2025, Nakano, 14 Jan 2026, Karg et al., 2023, Straub et al., 2023, Garrabé et al., 2023, Schneider et al., 2022, Sun et al., 2024, Karg et al., 2022, Nakano, 2020).