---
title: Finite-State Controllers in POMDPs
url: https://www.emergentmind.com/topics/finite-state-controllers-fscs
type: topic
---

# Finite-State Controllers in POMDPs

A finite-state controller (FSC) is a compact policy representation that encodes a mapping from observation and internal memory to action in a sequential decision process. FSCs are central to planning and control under partial information, especially in partially observable Markov decision processes (POMDPs) and their generalizations. Their adoption is motivated by the undecidability and intractability of computing history-dependent or fully belief-dependent policies in such domains. FSCs provide a finite-memory approximation, supporting both theoretical guarantees and scalable synthesis or learning in a variety of stochastic and deterministic settings.

## 1. Formal Definition and Classes of Finite-State Controllers

Let a POMDP be described by a tuple $(S, A, Z, T, O, R, \gamma)$, where $S$ is a finite state space, $A$ a finite action set, $Z$ a finite observation set, $T:S \times A \to \mathrm{Dist}(S)$ the transition kernel, $O:S \times A \times Z \to [0,1]$ the observation kernel, $R: S \times A \to \mathbb{R}$ the reward, and $\gamma \in (0,1]$ the discount factor.

A general stochastic finite-state controller for a POMDP is the tuple
\[
(Q,\, \pi(\cdot, \cdot),\, \eta(\cdot, \cdot, \cdot, \cdot),\, q_0)
\]
where
- $Q = \{q_1, \ldots, q_{|Q|}\}$ is the (finite) set of controller nodes (“memory” states),
- $\pi : Q \times A \to [0,1]$ specifies the probability $\pi(q, a)$ of picking action $a$ at node $q$, with $\sum_{a \in A} \pi(q, a) = 1$ for all $q$,
- $\eta : Q \times A \times Z \times Q \to [0,1]$ gives the probability $\eta(q, a, z, q')$ of moving from $q$ to $q'$ after taking $a$ and observing $z$, with $\sum_{q' \in Q} \eta(q, a, z, q') = 1$,
- $q_0 \in Q$ is the initial node.

Variants are common:
- Deterministic FSCs: $\pi(q, a)$ and $\eta(q, a, z, q')$ take values in $\{0,1\}$, specifying unique action-selection and memory-update mappings.
- Mealy machine models (memory-update depends only on current node and observation, independent of action).
- Memoryless policies are a special case with $|Q| = 1$.

For Decentralized POMDPs (Dec-POMDPs), each agent can have its own FSC, optionally stochastic and with distinct local observations and actions [2109.08755].

## 2. Value Evaluation and Policy Improvement in POMDPs

Given an FSC, its policy-induced value function is tractable to evaluate. For each $q \in Q$, let $V_q \in \mathbb{R}^{|S|}$ denote the value vector at node $q$. The Bellman evaluation equations are:
\[
V_q(s)\ =\ \sum_{a \in A} \pi(q,a) R(s,a)
\ +\ \gamma \sum_{a \in A} \sum_{s' \in S} \sum_{z \in Z} \sum_{q' \in Q}
\pi(q, a) \eta(q,a,z,q') T(s,a,s') O(s', a, z) V_{q'}(s')
\]
The value of a belief $b$ is $V(b) = \max_{q \in Q} \sum_{s} b(s) V_q(s)$ [1206.3263].

Policy improvement is typically implemented by a linear program (LP) that, for each node $q$, finds $\pi(q, \cdot)$ and $\eta(q, \cdot, \cdot, \cdot)$ maximizing the uniform lift $\epsilon$ s.t. the improved $V_q + \epsilon$ remains dominated by the new one-step backup under the current controller parameters. If $\epsilon^* > 0,$ this update strictly improves the policy on the tangent belief set [1206.3263].

## 3. Synthesis and Learning Algorithms for FSCs

### 3.1 Bounded Policy Iteration and Sparsity
The bounded policy iteration (BPI) algorithm improves FSCs via node-wise LPs. Empirically, most node parameter vectors are extremely sparse: in typical benchmarks, the number of nonzero elements per node grows only as $O(|A| + |Z|)$ even as $|Q|$ increases. The sparse BPI algorithm exploits this by solving a rapidly converging sequence of reduced LPs limited to the nonzero parameters. This enables improvements in per-iteration scalability by up to $80\times$ in observed benchmarks, restoring near-linear scaling in $|Q|$ without loss of policy improvement [1206.3263].

### 3.2 Inductive Synthesis, CEGAR, and Parameter Synthesis Methods
Recent frameworks model the entire $k$-FSC design space as either (a) a symbolic family with parameters for action and transition functions, or (b) a parametric Markov chain (pMC) with respect to these parameters [2203.10803, 1710.10294]. Inductive synthesis leverages counterexample-guided abstraction refinement (CEGAR) and oracles (e.g., abstraction MDPs for tight value bounds, counterexamples to block suboptimal candidates). This approach enables efficient search for small, correct-by-construction FSCs (often just 1–5 nodes) in both discounted and indefinite-horizon settings, and naturally incorporates multi-objective constraints [2203.10803].

Parameter synthesis allows computation of not just single optimal FSCs but entire permissive regions of parameters that yield correct policies, leveraging existing toolchains (Storm, PRISM, PARAM). Formally, for a desired threshold $\lambda$, one searches for parameter assignments $u$ such that $f(u) > \lambda$ for a rational (reachability, expected reward) function $f$ arising from the pMC [1710.10294].

### 3.3 Hybrid and Symbiotic Synthesis Frameworks
A further development is the “search and explore” symbiotic framework, integrating belief-based and inductive methods in parallel. Belief-based search synthesizes FSCs from finite fragments of the belief MDP, while inductive search handles large families via abstraction, yielding faster synthesis, higher controller value, and lower memory usage. Alternating these modalities, each phase benefits from tighter via-cuts or action restrictions informed by the other [2305.14149].

### 3.4 Learning Variable-Size FSCs and Data-Driven Settings
Expectation-maximization algorithms with nonparametric priors (e.g., stick-breaking) allow learning variable-size FSCs from data in Dec-POMDPs, adjusting the effective memory size by posteriors. This is achieved using variational Bayesian EM over an implicit infinite family, automatically concentrating the posterior on a small, performant controller [1505.00274]. For safe offline RL, finite-memory SPI with FSCs is feasible: a data-driven history MDP is estimated from behavior policy traces, policy improvement is then constrained on state-action pairs with insufficient data, and the solution is mapped directly back into an improved FSC with performance guarantees [2301.04939].

## 4. Structure, Hierarchies, and Expressiveness

FSCs can be flat or hierarchical. In generalized planning, hierarchical FSCs allow one controller to call subcontrollers (including recursively), dramatically increasing expressiveness and modularity. The existence of parameterized, recursive hierarchical FSCs allows solutions to families of problems (e.g., traversals in trees of depth $d$) with $O(d)$ controller states, versus $\Omega(2^d)$ for any flat FSC. Compilation techniques map controller synthesis—modular or hierarchical—into classical planning problems, with correctness guarantees for the resulting controller [1911.02887].

## 5. Complexity, Scalability, and Empirical Performance

Synthesizing or optimizing stochastic FSCs is computationally hard: deterministic $k$-FSC search is NP-complete; randomized versions (with real-valued probabilities) lie in the existential theory of the reals (ETR-complete) [2203.10803, 1710.10294]. The raw parameter space scales as $(|A| \cdot k)^{k \cdot |Z|}$, but practical methods exploit:
- Sparsity: limiting search or LP size to the parameter vectors’ support [1206.3263].
- Memory-model reductions: limiting the number of nodes relevant per observation [2203.10803].
- Symbolic and symmetry breaking: pruning isomorphic or unreachable configurations [2203.10803].
- Heuristics: modular controller seeding, action restriction by reference policies, value-driven memory injection [2305.14149].

Empirical results demonstrate that small FSCs—sometimes as compact as 1–5 states—can match or outperform state-of-the-art solvers on canonical benchmarks. Hierarchical and variable-size FSCs yield further advantages in expressiveness and scalability. The sparse BPI and symbiotic methods, in particular, achieve high performance with orders-of-magnitude reduction in synthesis time and controller size [1206.3263, 2305.14149, 2203.10803].

## 6. Correctness Guarantees and Specification Languages

Correctness of FSCs is studied with respect to satisfaction probabilities and reward thresholds. Formal specification languages often include:
- Reachability: $P_{\geq \lambda}$ constraints (probability to reach a goal),
- Expected cost/reward: $R_{\leq \mu}$ or $R_{\geq \mu}$,
- Termination and goal satisfaction: total probability of terminating (LTER), probability of goal-termination (LGT) [1905.07028].

Synthesis algorithms such as Pandor for stochastic domains provide soundness and completeness theorems for the returned FSCs, backed by α/λ (value/contribution) bound propagation and analysis of history likelihoods and looping contributions. For FSC plus parameter synthesis (pMC-based), solution regions can be certified by SMT solving or parameter lifting [1710.10294, 1905.07028].

In deterministic or hybrid settings, certified approximations and discrete optimization (Bellman inequalities, small-gain theorems) can yield controllers with provable bounds on performance, convergence rates, or stability [0903.5535].

## 7. Applications and Extensions

FSCs are used in robotics, AI planning, network protocols, verification, and safe offline RL. They support extensions to decentralized control, hierarchical and modular planning, infinite- and indefinite-horizon tasks, multi-objective specifications, and settings with partially known or data-sampled models.

Notably, FSCs enable:
- Nash equilibrium search and best-response synthesis in Dec-POMDPs by reducing the best-response computation to a (potentially high-dimensional) POMDP and compressing the resulting value function into bounded-size controllers [2109.08755],
- Safe policy improvement constrained by data and finite memory in offline RL, with high-probability performance bounds [2301.04939],
- Efficient and scalable planning in deterministic POMDPs with FSC-based solvers, producing compact policies applicable in large real-world robotic planning scenarios [2505.00596].

The general trend is toward exploiting structure—sparsity, hierarchy, modularity, and parameterization—for synthesis and learning of practical FSCs with theoretical guarantees in large-scale domains.

Source: https://www.emergentmind.com/topics/finite-state-controllers-fscs