Bounded Policy Synthesis in Stochastic Systems

• Bounded Policy Synthesis (BPS) is a method for constructing policies within finite time horizons or memory limits to meet complex objectives in stochastic environments.
• It utilizes techniques like SMT solving, tableau encodings, and mixed-integer programming to enforce temporal, probabilistic, and risk-aware constraints in models such as MDPs and POMDPs.
• BPS frameworks guarantee soundness and completeness under bounded conditions while enabling practical applications in robotics, autonomous driving, and multi-agent control.

Bounded Policy Synthesis (BPS) refers to a family of algorithmic methods for synthesizing policies for stochastic systems under a priori resource or structural bounds, such as finite execution horizon or finite policy memory, with correctness or performance guarantees relative to complex, often temporal or risk-aware, objectives. BPS formalizes the process of constructing, via constraint solving or sampling-based search, a policy that meets strict qualitative or quantitative requirements within explicitly bounded settings, such as Partially Observable Markov Decision Processes (POMDPs), Markov Decision Processes (MDPs), and systems governed by temporal logic or risk specifications (Wang et al., 2018, Akella et al., 2022, Frick et al., 2017, Baumgartner et al., 2017).

1. Formal Problem Definitions and Models

BPS frameworks operate across a variety of underlying models, unified by the decomposition of policy search space subject to explicit bounds.

• POMDP Setting: A POMDP is specified as $(S, A, T, \Omega, O, b_0)$, with $S$ finite states, $A$ actions, $T$ the transition kernel, $\Omega$ observations, $O$ the observation kernel, and $b_0$ an initial belief (probability distribution over $S$) (Wang et al., 2018).
• MDP Setting: For policy synthesis with temporal logic constraints, an MDP is $(S, s_{\mathrm{init}}, A, P, L)$, with $P$ a probabilistic transition kernel and $S$0 state labels, combined with a memory bound $S$1 for finite-memory policies (Baumgartner et al., 2017).
• Control Systems: In robust policy synthesis for linear/nonlinear systems affected by disturbances, bounded policy synthesis is applied to parameterized feedback laws, subject to system dynamics, and temporal or safety specifications (Frick et al., 2017, Akella et al., 2022).

BPS always posits a finite structure—horizon length, memory, sample budget, or policy complexity—that restricts the otherwise intractable synthesis space.

2. Expressive Objective Specifications: Temporal and Risk Constraints

BPS supports a wide spectrum of formal specification languages.

• Safe-Reachability Objectives: Under POMDPs, the specification demands that with probability at least $S$2, the process reaches $S$3 within horizon $S$4 without visiting $S$5 with probability exceeding $S$6 (Wang et al., 2018).
• Temporal Logic (BLTL, PCTL*): BPS handles bounded linear temporal logic (BLTL) and probabilistic computation tree logic* (PCTL*) specifications, translating temporal requirements (e.g., “eventually always safe”) into constraints over trajectories or policies (Frick et al., 2017, Baumgartner et al., 2017).
• Risk-Aware Objectives: Policies can be synthesized to minimize coherent risk measures, commonly Conditional Value-at-Risk (CVaR) or Entropic Value-at-Risk (EVaR), over robustness metrics (e.g., trajectory deviations), using statistical guarantees from sampled trajectory data (Akella et al., 2022).

This flexible specification layering enables BPS to meet safety, reachability, and performance requirements simultaneously, even under partial observability or environmental uncertainty.

3. Symbolic and Tableaux-Based Constraint Encodings

Policy synthesis is operationalized via compact symbolic or analytic encodings of the bounded policy search space.

• SMT-Based Encoding: In the POMDP context, BPS constructs symbolic constraints on time-indexed belief variables, enforcing initial state conditions, belief updates, stepwise safety, and terminal reachability within a horizon $S$7. All constraints are encoded in a Satisfiability Modulo Theories (SMT) formula $S$8 (Wang et al., 2018).
• Tableaux Construction: For MDPs with PCTL* constraints, analytic tableaux are used to build goal-directed constraint systems. Branching rules systematically decompose complex temporal logic objectives into nonlinear equality and inequality constraints over policy probabilities (Baumgartner et al., 2017).
• Mixed-Integer Programming: BPS with BLTL specifications utilizes inner-approximate encodings as mixed-integer quadratic programs (MIQPs), supporting affine feedback policies and robust constraint satisfaction over all bounded disturbance realizations (Frick et al., 2017).

This symbolic characterization leverages the structure of the underlying bounded system and objective to reduce the policy synthesis task to a finite constraint satisfaction or optimization problem.

4. Algorithmic Realization and Solvers

BPS solutions are typically achieved through a combination of iterative constraint solving, sampling, and scenario-based selection.

• Incremental SMT Solving: The core BPS algorithm for safe-reachability POMDPs proceeds by incrementally solving SMT instances for increasing horizons, pruning policy trees by backtracking and pushing/popping incremental scopes, and assembling full policies only if all observation branches succeed (Wang et al., 2018).
• Scenario-Based BPS: When risk measures are addressed, BPS uses a two-stage sample-based process: inner loops construct confidence-bounded risk evaluations for candidate policies, while an outer scenario program selects “good decisions” that statistically guarantee top-tail performance relative to all policies (Akella et al., 2022). Sample complexity is dictated by desired confidence and quantile level.
• Tableau Constraint Solving: In the PCTL* tableau approach, the constraint system is built via sequent expansion and resolved via linear or nonlinear programming, depending on whether deterministic or stochastic policies are permitted. Loop-checking ensures termination, and BSCC forcing provides qualitative temporal property satisfaction (Baumgartner et al., 2017).
• Mixed-Integer Quadratic Programming: MIQP solvers such as Gurobi or CPLEX are employed to solve inner-approximated robust synthesis problems for linear systems with BLTL constraints under uncertain disturbances (Frick et al., 2017).

A central principle is re-use of learned constraints or sampled data, substantially mitigating worst-case complexity induced by the exponential size of the reachable policy or belief space.

5. Theoretical Guarantees and Complexity Analysis

BPS frameworks provide the following fundamental guarantees when executed to completion under their respective assumptions:

• Soundness and Completeness: If there exists a policy within the bounded structure (horizon, memory, or policy class) satisfying the objective specification, the BPS procedures are guaranteed to find it (completeness), and any returned policy is correct by construction (soundness) (Wang et al., 2018, Baumgartner et al., 2017, Akella et al., 2022, Frick et al., 2017).
• Complexity: While BPS constrains synthesis to a finite domain, the worst-case computational complexity remains high—typically exponential in horizon $S$9 and number of observations/actions for POMDPs, and nonelementary for general PCTL* formulae (due to tableau branching and nonlinearity). Practical runtime is significantly improved by search-space restriction techniques, symbolic reuse, incremental solving, and inner-approximate or scenario-based relaxations (Wang et al., 2018, Baumgartner et al., 2017).
• Statistical Guarantees: In sample-based risk-aware BPS, empirical risk bounds hold with explicitly controlled confidence guarantees, and scenario selection theorems ensure the selected policy achieves desired quantile performance relative to the entire decision class (Akella et al., 2022).

Guaranteed performance is always relative to the imposed structural bounds (e.g., the policy class or optimization horizon).

6. Applications and Empirical Evaluation

BPS has been practically instantiated and empirically validated in diverse applications:

• Robotic Planning with Partial Observability: Safe-reachability BPS synthesized policies for a kitchen robot navigating a grid environment with uncertain obstacles, achieving feasible solutions in a “low hundreds” of SMT calls despite $A$0 possible belief-space leaves—policy and solver resource requirements remained practical up to moderately high problem sizes (Wang et al., 2018).
• Autonomous Driving Lane Change: MIQP-based BPS with affine feedback was applied to a lane-changing scenario, robustly satisfying BLTL specifications over all admissible disturbances and minimizing control effort relative to open-loop policies (Frick et al., 2017).
• Risk-Aware Multi-Agent Control: In the Robotarium testbed, BPS yielded control gain selections for unicycle robots that provably minimized tail risk (CVaR, EVaR) with confidence guarantees, outperforming baseline controllers in safety-critical robustness tails (Akella et al., 2022).
• Policy Synthesis Under General Temporal Logic: Tableaux-based BPS enabled the synthesis of finite-memory policies for MDPs under rich PCTL* constraints, supporting arbitrarily nested probabilistic temporal operators (Baumgartner et al., 2017).

Reported wall-clock times for realistic BPS instances are on the order of seconds to minutes when focusing search on relevant constrained slices, validating the practical tractability of the approach.

7. Limitations, Extensions, and Related Directions

BPS is subject to the following foundational limitations and areas for future refinement:

• Memory/Structure Bounds: All guarantees are relative to the imposed bounds (horizon $A$1, memory $A$2, policy structure). Unbounded policies yield undecidable synthesis problems (e.g., for PCTL*), making BPS intrinsically incomplete for the unrestricted setting (Baumgartner et al., 2017).
• Nonlinearity and Solver Scalability: For general stochastic policies and rich temporal logic objectives, constraint systems become highly nonlinear, limiting direct application of off-the-shelf solvers and motivating further abstraction or heuristic search techniques.
• Conservatism of Approximations: Practical BPS settings employing inner approximations (e.g., MIQP relaxations) may yield conservative solutions, potentially excluding feasible policies and impacting optimality (Frick et al., 2017).
• No Global Optimality for Nonconvex Classes: BPS in nonconvex feasible regions may only deliver percentile guarantees; absence of cost reward optimization in some BPS variants restricts them to feasibility rather than optimality (Baumgartner et al., 2017, Akella et al., 2022).

Active research explores extending BPS to larger fragments of temporal logic, integrating cost/reward measures, improving solver efficiency, and developing domain-specific algorithmic enhancements.

---

Key References:

• (Wang et al., 2018) "Bounded Policy Synthesis for POMDPs with Safe-Reachability Objectives"
• (Baumgartner et al., 2017) "Tableaux for Policy Synthesis for MDPs with PCTL* Constraints"
• (Akella et al., 2022) "Sample-Based Bounds for Coherent Risk Measures: Applications to Policy Synthesis and Verification"
• (Frick et al., 2017) "Robust Control Policies given Formal Specifications in Uncertain Environments"