Partial Process Observability

• Partial process observability is the condition where a system’s complete internal state is inaccessible, requiring inference from limited or noisy observations.
• It is fundamental across fields like control theory, robotics, and reinforcement learning, where estimating true states from indirect data is essential.
• Researchers use quantitative measures, empirical estimation techniques, and supervisory control frameworks to address the challenges posed by limited observability in both discrete and continuous systems.

Partial process observability refers to the condition in a dynamic system where the complete internal state is not directly or fully accessible to an observer or control agent. Instead, inference about the state must be made from limited, potentially noisy, or indirect measurements, events, or observations. This stands in contrast to fully observable systems, where the entire state is known at all times. Partial observability is a prevalent characteristic of real-world systems, arising from limitations in sensor technology, communication constraints, privacy requirements, or inherent system complexity. Effectively managing partial process observability is crucial across diverse fields, including control systems, artificial intelligence, robotics, monitoring, and verification.

Definition and Core Concepts of Partial Observability

At its core, partial observability implies a divergence between the true system state and the information available to an entity interacting with or monitoring the system. In systems described by state $s \in S$, an agent receives observations $o \in O$, where $o$ is a function of $s$, and possibly actions $a \in A$, and previous states. The observation function $\mathcal{H}(s)$ may be many-to-one, noisy, or provide information about only a subset of state variables.

In discrete-event systems (DES), partial observability typically manifests as unobservable events. The set of all possible events $\Sigma$ is partitioned into observable events $\Sigma_o$ and unobservable events $\Sigma_{uo}$. An observer or supervisor perceives only a projection $P: \Sigma^* \to \Sigma_o^*$ of the actual sequence of events (a trace) that occurs. A key challenge in DES supervisory control under partial observation is ensuring that the supervisor can make correct decisions and maintain system properties (like safety) without ambiguity arising from unobservable events.

For continuous systems, such as those described by Partial Differential Equations (PDEs), the state space is often infinite-dimensional. Sensors typically provide measurements at discrete spatial locations or times, or of integrated quantities, yielding only a finite-dimensional, partial view of the system state. The relationship between the infinite-dimensional state $u(x, t)$ and the finite-dimensional observation $y(t)$ is given by an observation operator, $y(t) = \mathcal{H}(u(\cdot, t))$. The goal is often to estimate the current or future state, or specific properties of the state, from the history of observations.

In reinforcement learning (RL) and planning problems, partial observability leads to Partially Observable Markov Decision Processes (POMDPs), where the agent's optimal policy depends on its history of observations and actions, or a derived belief state, rather than the true environment state. The agent must infer the current state distribution to act optimally.

Quantitative Measures and Consistency for Continuous Systems

Quantifying the degree of partial observability is essential for system analysis and sensor placement. For infinite-dimensional systems like PDEs, traditional binary observability criteria are insufficient given partial sensor coverage. A quantitative measure of partial observability can be defined relative to a finite-dimensional subspace of the state space targeted for estimation.

One such quantitative measure is the unobservability index, defined as the ratio $\rho / \epsilon$ (1111.5846, 1401.0235). Given a finite-dimensional subspace $W$ of the state space, a reference trajectory $u(t)$ with initial condition $u(0) \in W$, and a perturbation size $\rho$, $\epsilon$ is the minimum norm of the difference in output trajectories over a time interval $[t_0, t_1]$ for any perturbed initial condition $\hat{u}(0) \in W$ such that $\| \hat{u}(0) - u(0) \|_X = \rho$.

$$\epsilon = \inf \|\mathcal{H}(\hat u) - \mathcal{H}(u)\|_Y$$

$$\mathrm{Unobservability \; index} = \frac{\rho}{\epsilon}$$

A smaller index indicates stronger partial observability over the subspace $W$, meaning a smaller change in initial state within $W$ leads to a detectable change in output.

Calculating this index directly can be challenging, especially for nonlinear systems. For linear systems or finite-dimensional approximations, the quantity $\epsilon^2 / \rho^2$ relates to the smallest eigenvalue $\sigma_{\min}$ of a Gramian matrix restricted to the estimation subspace $W$. For linear systems with full state observability, this Gramian coincides with the classical observability Gramian from control theory. For nonlinear systems, an empirical Gramian can be constructed by perturbing the system's initial state in the directions of the basis vectors of $W$, simulating the system, and measuring the corresponding output deviations (1111.5846, 1401.0235). The smallest eigenvalue of this empirical Gramian provides a first-order approximation of the unobservability, with the unobservability index approximated by $1/\sqrt{\sigma_{\min}}$.

A critical concern when analyzing PDEs is the consistency of measures computed on finite-dimensional approximations. For the defined unobservability index, it has been proven that if the PDE is approximated using well-posed numerical schemes, the unobservability index computed on the discretized system converges to the index of the continuous PDE as the discretization is refined (1111.5846, 1401.0235). This ensures that numerical analysis on finite-dimensional models provides a meaningful estimate of the partial observability of the original infinite-dimensional system. Examples like Burgers' equation and the shallow water equation demonstrate this consistency (1401.0235).

Partial Observation in Discrete-Event Systems Control

Partial observation is a central difficulty in synthesizing supervisors for DES that enforce desired behaviors while only perceiving a subset of events. Standard observability definitions suffer from the issue that the union of observable sublanguages is not necessarily observable, which complicates the computation of the maximally permissive supervisor.

Relative observability is a concept introduced to address this challenge. A language $K$ is relatively observable with respect to an "ambient language" $C$ if any two traces in $C$ that are indistinguishable under the observation projection $P$ remain indistinguishable when considering their continuations within $K$ or their marked status (1306.2422). Formally, for $s, s' \in \Sigma^*$ with $P(s) = P(s')$:

1. $\forall \sigma \in \Sigma$, $$s\sigma \in \overline{K}, s' \in \overline{C}, s'\sigma \in L(G) \implies s'\sigma \in \overline{K}$$
2. $$s \in K, s' \in \overline{C} \cap L_m(G) \implies s' \in K$$

Relative observability is stronger than traditional observability but weaker than normality (which requires indistinguishability within the entire plant language). Crucially, relative observability is preserved under set union, guaranteeing the existence of a unique supremal relatively observable sublanguage, which can be computed using algorithmic procedures (1306.2422). This allows for the synthesis of supervisors that are less conservative than those based on normality, potentially permitting more system behavior while adhering to observation constraints. Applications include automated guided vehicle (AGV) systems, where disabling unobservable but controllable events can be done safely if relative observability is maintained (1306.2422).

In hierarchical supervisory control, where a complex low-level system is abstracted into a simpler high-level system, ensuring that control decisions made at the high level are valid and optimal for the low level under partial observation requires specific conditions. Conditions like observation consistency (OC) and local observation consistency (LOC) relate the indistinguishability of behaviors at the low and high levels given their respective observations (1912.07309). While OC and LOC are decidable for regular systems (finite automata models), they do not guarantee that the supremal observable or relatively observable sublanguages computed at the high level correspond exactly to those at the low level. To address this, modified observation consistency (MOC) is proposed, a stronger condition that does guarantee the preservation of the supremal normal sublanguage and ensures the supremal relatively observable sublanguage at the high level is at least as large as that at the low level (1912.07309). Decidability of MOC for regular systems allows for algorithmic verification of this property, supporting the design of sound hierarchical controllers for partially observed DES.

State and Belief Estimation in Partially Observable Systems

A fundamental necessity in partially observable systems is the ability to infer the unobserved state or aspects of it from available observations. This often involves maintaining and updating a belief state, which is a probability distribution over the possible true states of the system.

In the context of computer programs interacting with partially observable environments, traditional approaches require manual implementation of state estimators and environment models for formal verification. Belief Programming offers a methodology where the environment model is embedded directly in the program, and a runtime system automatically manages the belief state (2101.04742). Specialized programming constructs like choose (to represent environmental non-determinism or unobserved values), observe (to incorporate new observations and prune the belief space), and infer (to query properties of the belief state using modal operators like $\Box$ for certainty and $\Diamond$ for possibility) allow developers to write code that explicitly reasons about uncertainty. An accompanying Epistemic Hoare Logic (EHL) enables formal verification of such programs by reasoning about assertions on the belief state using these modal operators. The CBLIMP implementation demonstrates the feasibility of this approach for tasks like controlling the Mars Polar Lander with uncertain sensor data (2101.04742).

For predictive monitoring of hybrid systems, where predicting future safety violations is critical but only partial and noisy observations are available, state estimation is a key step (2108.07134). A learning-based approach uses a neural state estimator (NSE) to reconstruct the sequence of true states from observation histories. This two-step method (estimation followed by a neural state classifier for reachability prediction) is shown to be more accurate than direct end-to-end prediction from observations. Combining this with Conformal Prediction (CP) allows the system to provide probabilistic prediction regions with guarantees on coverage and to quantify uncertainty, which in turn enables high error detection rates and supports active learning by focusing on retraining on uncertain inputs (2108.07134).

In reinforcement learning, especially in complex environments with continuous state and action spaces, explicit state or belief tracking can be challenging. Methods leveraging learned models can provide an alternative to explicit memory structures like RNNs. By learning an abstract MDP model of the environment from observation-action traces, an RL agent can use the learned model's internal states as additional features representing implicitly tracked history (2206.11708). This approach allows tabular Q-learning to perform effectively in certain partially observable domains without requiring explicit recurrent neural networks, achieving better performance than memory-augmented deep RL baselines in some challenging scenarios (2206.11708). More recently, integrating Kalman filter layers into deep RL architectures allows the network to explicitly maintain a Gaussian distribution (mean and covariance) representing the belief over a latent state, providing a principled mechanism for filtering noise and reasoning about uncertainty directly within the policy learning process (2409.16824). This demonstrates superior performance in tasks where uncertainty is critical for decision-making, such as best arm identification and noisy continuous control.

Addressing Partial Observability in Reinforcement Learning

Partial observability presents significant challenges for standard RL algorithms, which are typically designed for fully observable Markov Decision Processes (MDPs). Agents in Partially Observable Markov Decision Processes (POMDPs) must learn policies that map observation histories (or belief states) to actions.

Standard DRL algorithms like TD3 and SAC, which use one-step bootstrapping, tend to perform poorly and get stuck in local optima under partial observability (2209.04999). PPO, which uses multi-step bootstrapping (like $\lambda$-returns), is more robust as it aggregates information over several timesteps, partially mitigating the lack of full state information. Introducing multi-step bootstrapping into TD3 and SAC (resulting in MTD3 and MSAC) significantly improves their robustness under partial observability (2209.04999). This highlights that providing more temporal context to value estimation is crucial when observations are incomplete.

Training RL agents for POMDPs can be aided by leveraging privileged information (like the true state) available during offline training or in simulation. Traditional methods often suffer from an "imitation gap" when a privileged teacher's policy is too complex for the partial-observation agent to imitate effectively. Guided Policy Optimization (GPO) proposes a co-training framework where a guider (using privileged information) and a learner (using partial observations) are mutually constrained to stay aligned (2505.15418). The learner is primarily trained by imitating the guider, but the guider is regularized to stay close to the learner's policy, ensuring the guidance is imitable. This theoretical framework demonstrates that the learner's updates can be equivalent to valid RL policy updates, allowing it to achieve RL-style optimality despite primarily using imitation, and shows strong empirical performance on tasks requiring memory and robustness to noise (2505.15418).

Another approach leveraging privileged information is Partially Observable Guided Reinforcement Learning (PO-GRL), which uses a curriculum that smoothly transitions from full observability to partial observability during training (2104.10986). By starting training with full state information, the agent learns the task structure efficiently, then gradually adapts to partial observations. This method avoids the sharp optimization difficulty of pure POMDP training and has been shown to improve performance and stability in continuous control and real-robot tasks with partial observability (2104.10986).

Theoretical analysis of the bias-variance trade-off in RL shows that under partial observability, the effective planning horizon can be shorter (2407.15820). As observation loss increases, the Blackwell-optimal discount factor decreases, suggesting that shallow planning (using a smaller discount factor) may be sufficient or even preferable in POMDPs. This is because the ability to influence and distinguish long-term returns is limited under uncertainty, reducing the bias penalty of shorter horizons while gaining robustness against compounding estimation errors.

For navigation tasks under partial observability, generating long-horizon plans from limited local observations is challenging. Value-guided Diffusion Policy (VDP) uses a diffusion model to generate multi-step action trajectories, guided by a learned value function derived from state estimations using Bayesian filtering (2404.02176). The value function, based on an approximate POMDP solution (QMDP), helps the agent evaluate candidate plans and select the one with the highest expected value, steering exploration towards goals even when not directly observed. This approach allows for versatile navigation and zero-shot transfer from 2D-trained policies to 3D environments using semantic segmentation and projection of point clouds from RGB-D inputs (2404.02176).

Even in adversarial RL settings, partial observability plays a significant role. For instance, in autonomous cyber defense, an attacker with only partial visibility of the network can still train a surrogate model to craft effective causative (poisoning) attacks against a defender's RL agent by perturbing its limited observations (1902.09062). This underscores the vulnerability introduced by partial observability. Countermeasures like the "inversion defense" attempt to proactively counteract such perturbations during training without explicit knowledge of the attack (1902.09062).

Modeling Process Networks with Partial Observations

Many real-world systems can be modeled as networks of interconnected processes, where the outputs of intermediate subprocesses are not always fully observable. Traditional Gaussian Processes (GPs) are limited by the curse of dimensionality and typically assume full observation of inputs and outputs. Gaussian Process Networks (GPNs) decompose a complex GP into a network of coupled subprocesses.

The Partially Observable Gaussian Process Network (POGPN) framework extends GPNs to explicitly handle partial, indirect, noisy, or missing observations at any node in the network (2502.13905). Each node represents a subprocess modeled by a GP, but the observed output is linked to the subprocess's latent function via an observation lens (a likelihood function). This allows for flexible modeling of various sensor types (e.g., Gaussian for regression, categorical for classification) and noise characteristics.

POGPN performs joint variational inference over the latent functions of all nodes in the network, leveraging doubly stochastic variational inference and inducing point approximations for scalability. This enables incorporating information from partial observations throughout the network structure during training and inference. The framework supports different training strategies, such as ancestor-wise training (jointly updating all ancestors of observed nodes) or node-wise training (iteratively updating nodes) (2502.13905). Empirical evaluations on datasets like the Jura geo-chemical data and EEG sensor data demonstrate that incorporating these partial, heterogeneous internal observations significantly improves the predictive performance and flexibility of the network compared to models that only use full or final outputs, highlighting the value of partial observability in improving model accuracy and robustness in complex process networks.

Multi-Agent Coordination and Monitoring under Partial Observability

Partial observability introduces unique challenges and opportunities in multi-agent systems (MAS), affecting coordination, monitoring, and interaction dynamics.

In the context of norm monitoring, where an entity (the norm monitor) checks if agents comply with rules but can only observe a subset of their actions, detecting violations or fulfiLLMents is difficult (1505.03996). The monitor must infer what unobserved actions might have occurred based on observed actions and state changes. This problem is formalized as reconstructing unobserved actions. Both full reconstruction (enumerating all possible consistent unobserved action sets) and approximate reconstruction algorithms are proposed. Approximate reconstruction, though not guaranteeing discovery of all violations, significantly increases the detection rate compared to monitors that only use observed actions, with lower computational cost, making it practical for large MAS (1505.03996).

In ad hoc teamwork, an agent must collaborate with unknown teammates on unknown tasks without prior coordination, based only on its own partial observations (2201.03538). The ATPO (Ad Hoc Teamwork with Partial Observability) algorithm addresses this by maintaining a Bayesian belief over a library of possible tasks/teammates, using the agent's history of actions and partial observations to update this belief and infer the teammate's likely behavior. The agent's policy is a mixture of optimal policies for each hypothesized task, weighted by its belief. This allows the agent to effectively identify the active task and coordinate with teammates based solely on how its limited perceptions evolve, demonstrating robustness and scalability in tasks like simulated robot navigation and cooperative cooking (2201.03538).

Finally, understanding the implications of an observer having partial observability is crucial for designing agents that can effectively communicate or conceal information through their actions. The PO-OAMDP (Partial Observer-Aware Markov Decision Process) framework models settings where an agent knows the observer perceives only a partial view of the system state and actions (2502.10568). This allows formalizing concepts like legibility (making goals clear), explicability (making behavior conformant with observer expectations), and predictability (making future actions/states guessable) under partial observation. The framework accounts for dynamic hidden variables (like changing goals or immediate next actions) that the observer tries to infer. Planning in PO-OAMDPs involves the agent reasoning about and influencing the observer's belief state, which is updated based on the observer's limited perceptions. This requires adapting planning algorithms like HSVI and highlights that optimal strategies in PO-OAMDPs involve intricate manipulation of shared, partially observable information spaces (2502.10568).