Combinatorial Policy Synthesis

• Combinatorial Policy Synthesis is a systematic approach to building decision-making policies in environments with combinatorial state, action, and constraint spaces, often modeled as families of MDPs.
• It employs advanced algorithmic techniques such as game-based abstractions, SMT-based probabilistic model checking, tableau methods, and decision-tree learning to satisfy complex temporal and quantitative constraints.
• The methodology guarantees soundness, completeness, and scalability, enabling robust applications in multi-agent control, planning under uncertainty, and explainable policy representation.

Combinatorial Policy Synthesis refers to the systematic construction or optimization of policies for decision-making systems—often modeled as Markov Decision Processes (MDPs) or related frameworks—where the underlying state, action, environment, or constraint spaces are combinatorial in nature. This encompasses settings with structured constraints, functional specifications, parametric uncertainty, and large or infinite families of problem instances, commonly seen in combinatorial optimization, control synthesis, planning under uncertainty, and program synthesis.

1. Formal and Algorithmic Foundations

A combinatorial policy synthesis problem typically arises when the set of possible policies, environments, or both is combinatorially large or parameterized. Common setups include:

• Families of MDPs: The task is to synthesize a (possibly small) set of policies that collectively achieve desired properties (e.g., safety, reachability, quantitative objectives) across all members of a family parameterized by environment variables. This family may be indexed by discrete parameters I, leading to collections like $\mathcal{M} = \{M_i \mid i \in I\}$ (Andriushchenko et al., 2024).
• Structural Constraints: The synthesis is subject to representational restrictions, such as limiting the policy to a decision tree of bounded depth, finite-state controller size, or stratification properties (Demirović et al., 2024, Heck et al., 11 Nov 2025).
• Complex Specifications: Specifications can be given in rich logics such as linear temporal logic (LTL), graph temporal logic (GTL), or probabilistic extensions (PCTL*, ω-regular), requiring satisfaction of behavioral, temporal, probabilistic, and steady-state constraints (Křetínský, 2021, Baumgartner et al., 2017, Cubuktepe et al., 2020).

The synthesis objective is typically to find, for all $i \in I$, a memoryless or finite-memory policy $\pi$ such that the resulting closed-loop system satisfies hard constraints, meets probabilistic thresholds, or optimizes a reward under all parameter instances, possibly in the face of adversarial uncertainty (Heck et al., 11 Nov 2025).

2. Model Classes and Problem Formulations

The combinatorial nature of the synthesis problem can arise through several channels:

Model Class	Combinatorial Source	Synthesis Objective
Parameterized MDP families	Environment indexed by finite/infinite parameters	Find minimal set of policies covering all satisfiable indices
Robust/Constrained synthesis	Policy parameterization and adversarial environment	Satisfy constraints for all environments and policy instances
Finite-state controllers (FSCs)	Exponential number of parameterized Mealy machines	Maximize expected reward or guarantee specification
Decision-tree policies	All tree shapes × predicate assignments	Find minimum-depth/size tree policy, optimal steps-to-goal
Symbolic policies from features	Rule-based combinations over large feature pools	Generalize sample behaviors with guaranteed termination
Distributed multi-agent factored MDP	Local agent policies, neighbor consistency constraints	Jointly synthesize distributed controllers

Notable concrete problem formulations include:

• Existential-universal quantification: $$\exists\,\text{policy}\ \forall\,\text{environment}:$$ specification satisfied (Heck et al., 11 Nov 2025)
• Optimal coverage over families: Minimal number of policies such that every $M_i$ is covered (Andriushchenko et al., 2024)
• Rule-based generalization: Find stratified rule-sets covering all good traces and none of the bad (Bonet et al., 2 Sep 2025)
• Functional synthesis under logical constraints: Find a decision-tree or automaton-based policy that is feasible for all input parameters (Azeem et al., 2024, Demirović et al., 2024)

3. Core Algorithmic Techniques

A diverse set of combinatorial and algorithmic approaches underpins state-of-the-art policy synthesis:

Game-based Abstractions and Symbolic Search

Recursive game-based abstractions and refinement lead to strong generalization over MDP families. The key insight is to build a turn-based stochastic game abstraction $G_I$ encoding maximal and minimal performance across parameter indices, supporting divide-and-conquer construction of policy trees. Each node of the tree corresponds to a subfamily of parameter instances labeled by a policy or infeasibility (Andriushchenko et al., 2024).

Satisfiability-Modulo-Probabilistic-Model-Checking (SMPMC)

This paradigm couples SAT/SMT solvers with efficient probabilistic model checking. Here, robust feasibility (for all environment parameters) is encoded as an ∃–∀ first-order formula, with policy parameters as existential and environment parameters as universal variables. A custom “theory solver” performs tight coupling between logical constraints and probabilistic property checking for candidate assignments (Heck et al., 11 Nov 2025).

Tableau-Based and Linear-Programming Methods

Complex logical constraints (e.g., PCTL*, LTL with steady-state and quantitative reward thresholds) are encoded as (possibly nonlinear) constraint systems or linear programs:

• Tableau-based synthesis constructs a non-deterministic, combinatorial tableau whose expansion encodes policy decisions as real-valued variables, capturing all satisfaction requirements in an analytic proof tree. Branching corresponds to policy choices and formula structure (Baumgartner et al., 2017).
• Linear-programming synthesis maps satisfaction of temporal, steady-state, and reward constraints to LP variables representing transitions, flows, and recurrent state-action visitation frequencies. Feasible solutions can be interpreted as randomized or finite-memory policies; synthesis reduces to solving a high-dimensional but structured optimization problem (Křetínský, 2021).

Policy Learning and Generalization

• Decision-tree learning from small-instance optimal policies (DTL): Exact solutions from small problem instances are used to learn parameterized policies via decision-tree classifiers, enabling robust, explainable policies deployable to arbitrarily large instances without explicit exploration (Azeem et al., 2024).
• Symbolic feature-based synthesis: Policies are represented as sets of rules over features generated from a grammar or concept pool. A greedy hitting-set or SAT-based algorithm selects minimal feature sets that generalize positive sample traces and avoid negatives, enforcing structural acyclicity and safe termination (Bonet et al., 2 Sep 2025).
• Finite-state controller synthesis via symbiotic search: Integrates belief-MDP fragment exploration and inductive abstraction-refinement in the large, combinatorial space of memory-based policies, leveraging tight pruning by seeding each search with the other's incumbents (Andriushchenko et al., 2023).

Combinatorial Enumeration with Pruning

• Exhaustive enumeration with pruning, as in optimal decision-tree synthesis for black-box dynamical systems, imposes discretization of predicates and tree size, applying trace-based pruning to cut the search space, providing optimality within prescribed bounds (Demirović et al., 2024).
• Distributed optimization in factored multi-agent MDPs: Large agent systems are decomposed by neighborhood structure, producing per-agent LP subproblems that are solved in parallel with neighbor-consistency constraints coordinated via ADMM (Cubuktepe et al., 2020).

4. Scalability, Complexity, and Practical Performance

Combinatorial policy synthesis typically faces exponential blowup in policy class size, parameter dimensions, or logical formula size. However, advanced algorithmic techniques (game abstraction, symbolic model checking, distributed optimization, feature-pool pruning, and strategic search ordering) yield practical tractability in large problems for the following reasons:

• Game-based and symbolic abstractions avoid explicit enumeration of all parameter combinations, often reducing the number of needed policy candidates by several orders of magnitude for large families ($10^7$–$10^8$ instances) (Andriushchenko et al., 2024).
• Satisfiability-modulo model checking with clause learning and model-based quantifier instantiation achieves completeness and soundness and outperforms monolithic MILPs or conversion to SMT(LRA), scaling up to 20,000-state MDPs and moderate parameter spaces (Heck et al., 11 Nov 2025).
• Feature-driven and rule-based generalization handles state spaces in the millions and feature pools of up to 260,000, with greedy hitting-set routines operating in polynomial time in the size of the compressed feature space (Bonet et al., 2 Sep 2025).
• Distributed ADMM decomposition in multi-agent MDPs enables linear scalability in agent count ($O(M)$), given bounded neighbor interactions (Cubuktepe et al., 2020).
• Tableau, LP, and automata-based constructions support complete, correct synthesis for unrestricted logical constraints, though with worst-case EXPTIME dependence on formula size and policy memory (Křetínský, 2021, Baumgartner et al., 2017).
• Empirically, trace-based or structure-based pruning reduces explicit enumerate-and-test search spaces by one to two orders of magnitude, helping keep low-depth decision-tree synthesis practical for low-dimensional systems (Demirović et al., 2024).

5. Policy Representation and Generalization

Combinatorial policy synthesis methods support a range of policy representations:

Policy Class	Typical Representation	Synthesis Approach
Memoryless policies	Functions $S\rightarrow A$ or tabular decision rules	Explicit assignment, decision-tree
Finite-state controllers (FSC)	Mealy machines with memory and observation mapping	Inductive and belief-MDP search
Decision-tree policies	Axis-aligned predicate trees with bounded depth/size	Exhaustive search + pruning, or learning
Rule-based symbolic policies	Conjunctions/disjunctions of features with effect rules	Hitting-set, SAT solving
Population or distributional	Latent-conditioned policy families, e.g., COMPASS	Population-based RL + latent space search

Recent advances emphasize explainability and generalization capability, notably decision-tree learning from optimal policies on small instances, which yields parameter-independent policies for arbitrarily large parameter values in MDPs, with empirical near-optimality even well outside the training distribution (Azeem et al., 2024). Symbolic rule-based policies learned over expressive feature grammars enable coverage of complex planning domains with strong acyclicity guarantees (Bonet et al., 2 Sep 2025).

6. Applications and Empirical Results

Applications of combinatorial policy synthesis span a wide range:

• Model checking over parameterized families: Efficiently synthesizing a minimal set of controllers that provide specification coverage for millions of system variants (Andriushchenko et al., 2024).
• Robust and constrained policy optimization: SAT+PMC techniques tackle structural constraints (e.g., small trees, minimal complexity) and environmental uncertainty, outperforming classical abstraction-refinement approaches in both performance and uniqueness of solution (Heck et al., 11 Nov 2025).
• Multi-agent systems with spatial-temporal constraints: Distributed LP+ADMM schemes synthesize local controllers for hundreds of agents under global graph temporal logic specifications, scaling linearly with agent count (Cubuktepe et al., 2020).
• Symbolic and explainable planning: Hitting-set and feature-based approaches synthesize guaranteed acyclic, closed symbolic policies from examples in large state-and-feature settings, providing interpretable rules for generalized planning problems (Bonet et al., 2 Sep 2025).
• Learning-based policy adaptation and generalization: Methods such as COMPASS and DT learning enable strong generalization and zero-shot transfer in classic combinatorial optimization benchmarks (TSP, CVRP, JSSP) via latent-conditioned policy manifolds or tree-structured classifiers (Chalumeau et al., 2023, Azeem et al., 2024).

Empirical evaluations consistently demonstrate that combinatorial methods, when leveraging problem structure and advanced abstraction, drastically improve scalability, policy compactness, and coverage over naive enumeration or purely data-driven RL baselines (Andriushchenko et al., 2024, Chalumeau et al., 2023, Azeem et al., 2024).

7. Theoretical Properties, Guarantees, and Open Challenges

Combinatorial synthesis pipelines are characterized by strong theoretical guarantees in multiple regards:

• Soundness and completeness: Tableau and LP approaches are provably sound and complete for expressive logic specifications, within bounded policy-memory classes (Křetínský, 2021, Baumgartner et al., 2017).
• Termination and coverage: Policy tree, decision-tree, and stratified rule-based policy synthesis algorithms guarantee termination and optimality/correctness within defined bounds and under explicit search constraints (Andriushchenko et al., 2024, Demirović et al., 2024, Bonet et al., 2 Sep 2025).
• Complexity bounds: While combinatorial in worst-case, most frameworks exploit structure (symmetry, parameter invariance, locality, sparsity) to avoid state explosion.
• Limitations: Some approaches lack a priori generalization guarantees when policy structure depends sensitively on parameter values or if expressivity of feature/predicate pools is insufficient (Azeem et al., 2024, Bonet et al., 2 Sep 2025).

Open questions and ongoing research include robust and efficient synthesis for high-dimensional decision-tree or automaton policies, finer-grained abstraction-refinement over parameter spaces, and integration of symbolic and neural approaches for scalable, explainable combinatorial synthesis.

---

References

• "Policies Grow on Trees: Model Checking Families of MDPs" (Andriushchenko et al., 2024)
• "Constrained and Robust Policy Synthesis with Satisfiability-Modulo-Probabilistic-Model-Checking" (Heck et al., 11 Nov 2025)
• "In Search of Trees: Decision-Tree Policy Synthesis for Black-Box Systems via Search" (Demirović et al., 2024)
• "Learning General Policies From Examples" (Bonet et al., 2 Sep 2025)
• "1-2-3-Go! Policy Synthesis for Parameterized Markov Decision Processes via Decision-Tree Learning and Generalization" (Azeem et al., 2024)
• "LTL-Constrained Steady-State Policy Synthesis" (Křetínský, 2021)
• "Tableaux for Policy Synthesis for MDPs with PCTL* Constraints" (Baumgartner et al., 2017)
• "Policy Synthesis for Factored MDPs with Graph Temporal Logic Specifications" (Cubuktepe et al., 2020)
• "Combinatorial Optimization with Policy Adaptation using Latent Space Search" (Chalumeau et al., 2023)
• "Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing" (Liang et al., 2018)
• "Search and Explore: Symbiotic Policy Synthesis in POMDPs" (Andriushchenko et al., 2023)
• "Understanding Curriculum Learning in Policy Optimization for Online Combinatorial Optimization" (Zhou et al., 2022)
• "UNSAT Solver Synthesis via Monte Carlo Forest Search" (Cameron et al., 2022)