LQG solution for POMDP without estimating states: A minimum variance approach

Abstract: This paper investigates the control of discrete-time linear time-invariant (LTI) systems subject to incomplete and corrupted measurements. Specifically, we focus on designing a Linear Quadratic Gaussian (LQG) controller without relying on explicit state estimation. By leveraging minimum variance duality, our approach allows the current control input to be represented as a linear function of available measurements and previously applied inputs, successfully reducing the task to a tractable deterministic optimization problem. We provide theoretical justification for this framework and demonstrate its practical effectiveness through numerical experiments.

• The paper presents a novel deterministic LQG control strategy for POMDPs that bypasses explicit state estimation via a minimum variance approach.
• It introduces a duality framework linking observable measurement histories to control synthesis, ensuring quadratic loss minimization under noise and constraints.
• Empirical validations on systems like the cart-pole and MIT Cheetah 3 demonstrate performance comparable to traditional methods while simplifying constraint integration.

Minimum Variance LQG Control for POMDPs Without State Estimation

Introduction and Context

The standard approach for LQG control in partially observable systems relies on the Separation Principle, in which optimal control is achieved by sequentially combining Kalman filtering for state estimation with linear feedback based on the estimated state. However, a growing body of literature underscores that this separation is neither necessary nor universally optimal, especially in scenarios with non-classical information structures, constraints, or non-Gaussianities in the observation channels, thereby challenging the universal applicability and structural optimality of certainty equivalence. This work addresses the synthesis and computation of LQG controls under partial observability without explicit state estimation, reframing the problem through the lens of minimum-variance duality and direct measurement feedback.

Formulation and Methodology

The paper considers discrete-time LTI systems with process and measurement noise—modeled by standard additive Gaussian processes—where the state is not directly accessible and observations are limited and noisy. The system is formalized via standard state-space equations, with an LQG quadratic cost in terms of state and input penalties. With only access to finite observation histories and inputs, the central problem is the synthesis of a causal feedback controller minimizing the expected quadratic loss relative to the ideal full-state LQG solution, all without performing state estimation as a computational intermediate.

To this end, the authors adopt and generalize minimum-variance duality theory, constructing a deterministic surrogate optimization in place of classical stochastic estimation. Through a dual process and its associated sequence $\{z_i, \alpha_i\}$, they represent the control as a weighted sum of measurements and previous inputs, derived not from estimated state trajectories but from an explicit minimum-variance solution.

Figure 2: The optimal cost in the minimum variance optimization is convergent and tracks the empirical MSE achieved by the control policy over batch simulations.

The main technical result, codified in Theorem 1, proves that the stochastic LQG tracking objective—with partial observability—admits an equivalent deterministic formulation in dual variables. The optimal control is given analytically as:

$$\hat{u}_t = z_t^\top \hat{x}_0^- + \sum_{i=0}^{t-1} z_i^\top B u_{t-i-1} - \sum_{i=0}^t \alpha_i^\top y_{t-i}$$

where $z_t, \alpha_t$ are recursively computed via the dual dynamics and the minimum variance criterion. This provides a direct input synthesis law dependent only on observable signals and input history.

Figure 1: Averaged control signals over multiple simulations for the three approaches indicate the empirical mean remains near zero, with minimal deviation and similar dispersion.

Algorithmic Implications and Constrained Extensions

A key advantage of this duality-based approach is the seamless incorporation of constraints. Because the control law arises from a deterministic, finite-dimensional optimization, convex input (or output) constraints can be imposed directly as feasibility regions in the policy synthesis, circumventing the nonconvexities typically encountered in stochastic separation-based frameworks. The paper formalizes this extension for convex $\mathcal{U} \subset \mathbb{R}^p$ and demonstrates that the constrained optimization remains convex and efficiently solvable using modern numerical optimization packages.

Empirical Validation

The efficacy of the proposed approach is demonstrated via two canonical systems:

1. Cart-Pole (Unconstrained): The minimum-variance controller is deployed on a linearized cart-pole system. Statistical analysis over 200 simulation batches shows that both the classical estimation-then-control and the direct minimum variance approaches yield nearly identical control and state processes, with empirical means and variances indistinguishable for practical purposes.

Figure 3: Simulation results for each state dimension reveal the empirical means of trajectories coincide for the standard and proposed methods.

2. MIT Cheetah 3 Quadruped (Constrained): The methodology is scaled to the 13-state, 12-input rigid body model for the MIT Cheetah 3 quadruped, employing realistic force and measurement constraints. Evaluation of ground reaction forces and pose tracking under stochastic disturbance again demonstrates negligible performance gap relative to the classical constructor and ideal full-state feedback.

Figure 4: MIT Cheetah 3 quadruped model used for the constrained control experiment.

Figure 5: Leg ground reaction force profiles show all approaches yield closely matched efforts converging to the reference value.

Figure 6: The reference pose and orientation of the MIT Cheetah 3 is accurately tracked under disturbances by all three control strategies.

Theoretical and Practical Implications

This minimum variance approach refines and extends duality frameworks for stochastic control. By circumventing the state estimation stage, it aligns policy synthesis more directly with observable quantities, supporting a variety of non-standard architectures (e.g., event-triggered, communication-limited, or nonclassical feedback). Furthermore, the formulation facilitates constrained LQG synthesis in real-world applications—robotics, aerospace, and networked systems—where input limits, non-Gaussian noise, or bilinear observations disrupt standard separation-based pipelines.

The strong empirical evidence, especially in high-dimensional and physical systems, indicates that for LTI-Gaussian systems, direct minimum variance approaches do not sacrifice performance relative to the classical separated designs, even under constraints. This challenges the necessity of the Kalman filter in practical LQG feedback synthesis when only quadratic performance is targeted and supports a paradigm shift towards direct measurement-centric policy parameterization.

Future Directions

The minimum variance duality framework is a promising foundation for further advances in stochastic optimal control. Potential avenues include:

• New theory and algorithms for LQG problems with non-Gaussian noise, non-quadratic costs, or nonlinear dynamics, where duality principles could be exploited for tractable surrogates.
• Application to distributed and networked control settings, where constraints or partial information violate the assumptions required for separation.
• Integration with robust, adaptive, or learning-based controllers to handle model misspecification or real-time adaptation.
• Generalization to partially observable nonlinear MDPs and formulation of minimum variance policies utilizing modern optimization and (possibly) learning paradigms.

Conclusion

This work provides a formal, minimum variance LQG solution methodology for POMDPs that avoids explicit state estimation, leveraging duality to formulate a deterministic optimal control synthesis dependent only on observable histories and inputs. Empirical results confirm competitive performance with the classical architecture, with added advantages in incorporating constraints and simplifying implementation. This paradigm invites reconsideration of foundational stochastic control design and has significant implications for high-assurance and resource-constrained applications.