Bellman Value Decomposition for Task Logic in Safe Optimal Control

Abstract: Real-world tasks involve nuanced combinations of goal and safety specifications. In high dimensions, the challenge is exacerbated: formal automata become cumbersome, and the combination of sparse rewards tends to require laborious tuning. In this work, we consider the innate structure of the Bellman Value as a means to naturally organize the problem for improved automatic performance. Namely, we prove the Bellman Value for a complex task defined in temporal logic can be decomposed into a graph of Bellman Values, connected by a set of well-known Bellman equations (BEs): the Reach-Avoid BE, the Avoid BE, and a novel type, the Reach-Avoid-Loop BE. To solve the Value and optimal policy, we propose VDPPO, which embeds the decomposed Value graph into a two-layer neural net, bootstrapping the implicit dependencies. We conduct a variety of simulated and hardware experiments to test our method on complex, high-dimensional tasks involving heterogeneous teams and nonlinear dynamics. Ultimately, we find this approach greatly improves performance over existing baselines, balancing safety and liveness automatically.

• The paper develops an exact decomposition algebra that reduces broad temporal-logic specifications to Avoid, Reach-Avoid, and novel Reach-Avoid-Loop Bellman equations.
• The authors introduce VDPPO, which embeds the resulting Value graph in shared actor-critic networks and outperforms automaton-based PPO and trajectory optimization across simulated tasks and hardware experiments.
• The method improves scalability for nested goals, multiple agents, and recurring constraints, while remaining limited by exponential graph growth, deterministic dynamics, and convergence guarantees that require the discount factor to approach one.

Overview

This paper, by Sharpless, So, Hirsch, Herbert, and Fan (UCSD and MIT), addresses the problem of synthesizing optimal policies for tasks specified in temporal logic (TL) over deterministic discrete-time systems. The central observation is that the Bellman Value function for a complex TL specification can be exactly decomposed into a directed graph of simpler Value functions, each governed by one of three "atomic" Bellman equations: the Avoid ($\mathcal{A}$) BE, the Reach-Avoid ($\mathcal{RA}$) BE, and a novel Reach-Avoid-Loop ($\mathcal{RA}_\ell$) BE introduced for always-eventually specifications. Building on this decomposition algebra, the authors propose VDPPO (Value-Decomposition PPO), which embeds the decomposed Value graph (DVG) into shared actor-critic networks conditioned on node embeddings. The method is validated in simulation on four domains and on hardware with a heterogeneous team of Crazyflie drones and a Unitree Go2 quadruped.

The work generalizes the dual-objective decompositions of (Sharpless et al., 19 Jun 2025) to a substantially broader class of predicates, culminating in a master theorem covering conjunctions of Until operators composed with $\mathsf{G}(\cdot)$ recurrences and persistent safety constraints.

Problem formulation

Given dynamics $x_{t+1} = f(x_t, a_t)$ and atomic predicates defined by bounded functions $r_i$, the paper adopts quantitative semantics via robustness scores $\rho[\mathsf{p}]$ (Donzé–Maler style), where satisfaction corresponds to $\rho \ge 0$. The Safe Optimal Control Problem is

$$V^*[\mathsf{p}](x) := \max_\alpha \rho[\mathsf{p}](\xi_x^\alpha),$$

an infinite-horizon objective defined by temporal extrema rather than discounted sums. This distinction is consequential: extremum-based objectives select for outlying performance (worst-case safety, best-case achievement), whereas discounted-sum objectives select for average behavior. The known contractive Bellman operators for $V^*[\mathsf{Gq}]$ (Avoid) and $\mathcal{RA}$0 (Reach-Avoid) from [fisac2019bridging] serve as fixed points converging to $\mathcal{RA}$1 as $\mathcal{RA}$2.

A key motivating remark establishes that the algebra of TL does not transfer to Values: while $\mathcal{RA}$3 holds exactly, only the inequality $\mathcal{RA}$4 holds, and it is strict whenever no single action sequence achieves both subgoals. Concrete counterexamples are provided in the appendix. This motivates developing an algebra specific to Value functions rather than importing logical identities wholesale.

Decomposition results

Agreeable algebra. Lemma 1 identifies when Value algebra coincides with logic: because $\mathcal{RA}$5 commutes with maxima, $\mathcal{RA}$6 and $\mathcal{RA}$7, where $\mathcal{RA}$8 is the predicate induced by $\mathcal{RA}$9. These equivalences underpin all subsequent decompositions; corollaries recover chains of $\mathcal{RA}_\ell$0 Values for nested Untils and the reach-stay decomposition $\mathcal{RA}_\ell$1.

N-Until-Conjunction. Theorem 2 shows that for $\mathcal{RA}_\ell$2,

$\mathcal{RA}_\ell$3

Because the result is recursive, iterative application yields $\mathcal{RA}_\ell$4 sub-Values, each solvable by the discounted $\mathcal{RA}_\ell$5-BE. An extension handles an appended $\mathcal{RA}_\ell$6 constraint, reducing to the Reach-Always-Avoid case of (Sharpless et al., 19 Jun 2025). The implication is that multi-goal sequential tasks with shared obstacles reduce to a single abstract reach-avoid problem whose "reach" predicate encodes feasibility of the remaining logic — a dense, aligned learning signal unavailable to flat reward designs.

Recursive ($\mathcal{RA}_\ell$7-composed) decompositions. For $\mathcal{RA}_\ell$8 — the liveness-flavored alternative to the stricter $\mathcal{RA}_\ell$9 — Theorem 3 gives the self-referential characterization $\mathsf{G}(\cdot)$0. Theorem 4 extends this to a loop of $\mathsf{G}(\cdot)$1 coupled Values for $\mathsf{G}(\cdot)$2. The authors are explicit that these implicit characterizations do not guarantee uniqueness or existence of the Value, and may be ill-defined in continuous state spaces; existence is certified separately via finite-recurrence limits under finite states, finite actions, or compact actions with continuous dynamics (using König's lemma and Cantor's intersection theorem).

To make these Values computable, Lemma 5 introduces the $\mathsf{G}(\cdot)$3-BE,

$\mathsf{G}(\cdot)$4

and proves it is a contraction (via a sup-norm argument exploiting Lipschitz continuity) whose unique fixed point converges to the desired Value as $\mathsf{G}(\cdot)$5. A naive application of the standard $\mathsf{G}(\cdot)$6-BE here would fail to be contractive because the Value appears in both the $\mathsf{G}(\cdot)$7 and $\mathsf{G}(\cdot)$8 terms; this is the technical crux enabling learning-based solution of recurrence specifications such as $\mathsf{G}(\cdot)$9.

Master theorem. Theorem 6 combines the above: for predicates of the form $x_{t+1} = f(x_t, a_t)$0, the Value reduces to a single $x_{t+1} = f(x_t, a_t)$1 problem whose residual recursively decomposes via the loop structure. Appendix proofs proceed by symbolic rearrangement of the logic (verified with Spot) followed by application of the agreeable-algebra lemmas.

The appendix also develops policy extraction: an $x_{t+1} = f(x_t, a_t)$2-step Q-function recursion over the DVG yields optimal actions, with a pre-computable tree of comparisons minimizing required history — addressing the fact that optimal TL policies are inherently history-dependent without full state augmentation.

Algorithm: VDPPO

Two components operationalize the theory:

• valtr: a compiler that parses a TL string into an AST, applies logical equivalences to reach the canonical form of Theorem 6, and emits the DVG as a DAG whose nodes are predicates, negations, min/max operations, and Value functions. Topological sorting enables exact dynamic-programming solutions in low dimensions.
• VDPPO: a PPO variant in which a single shared trunk serves all DVG nodes, conditioned on one-hot node embeddings. Each node's advantage estimate is computed with its appropriate atomic BE ($x_{t+1} = f(x_t, a_t)$3, $x_{t+1} = f(x_t, a_t)$4, or $x_{t+1} = f(x_t, a_t)$5), bootstrapping current Value estimates across nodes rather than solving sequentially. At deployment, satisfying a trigger condition switches the active node embedding.

The shared-trunk design is justified by ablations: sharing critic parameters alone erodes success rate, sharing actor parameters improves it, and combining both matches unshared performance while training roughly $x_{t+1} = f(x_t, a_t)$6-times faster than the per-decomposition architecture of (Sharpless et al., 19 Jun 2025), which scales combinatorially with specification complexity.

Simulation results

Experiments address three questions against LCRL hasanbeig2022lcrl and TL-MPPI [halder2025trajectory], with three seeds and evaluation on 256 initial conditions.

Question	Setting	Finding
TL complexity (breadth/depth)	DoubleInt, increasing goals or nesting	VDPPO consistently best; baselines achieve ≤40% success at depth $x_{t+1} = f(x_t, a_t)$7
Agent scaling	Up to 5 agents + specs jointly	VDPPO is the only method solving the 5-agent task
Difficult dynamics	Herding, Delivery, Manipulator	Highest success rate by significant margin in all three

The depth result is attributed to the exponential decay of lucky satisfaction probability for nested specifications, which penalizes non-decompositional methods. The breadth/agent scaling results indicate that decomposition into smaller subproblems mitigates both sparse-reward and exploration-dimensionality pathologies simultaneously.

Hardware results

Hardware deployment uses Crazyflie drones tracked by HTC Vive Lighthouse base stations (with a propeller-less CF mounted on the Go2 for localization), with policies inferred on a laptop CPU broadcasting velocity setpoints.

Herding: one CF and the Go2 herd three autonomous sheep CFs through a narrow gap to a target region, under $x_{t+1} = f(x_t, a_t)$8 with asymmetric herder dynamics. Emergent division of labor is observed: the fast CF gathers the herd through the gap while the slower Go2 positions itself to block dispersal — behavior not explicitly programmed.

Delivery: two CFs and the Go2 perform recurrent delivery and resupply under a specification dominated by $x_{t+1} = f(x_t, a_t)$9 terms (solved largely via the $r_i$0-BE), with collision, obstacle, and no-fly-zone constraints. Targets relocate upon visitation, requiring goal-conditioned policies. The Go2 anticipates CF trajectories to minimize resupply detours, and no crashes were observed during live operation.

These demonstrations carry practical weight because the specifications involve heterogeneous teams, nonlinear relative dynamics, and recurrent liveness constraints — combinations that automaton-augmented RL and sampling-based trajectory optimization handle poorly at this scale.

Limitations and open questions

Several caveats are stated plainly in the paper. First, the recursive $r_i$1 Value characterizations do not guarantee uniqueness or existence in general; certification requires finite-state, finite-action, or compact-action assumptions, and the finite-recurrence procedure used for certification is acknowledged as impractical for large state spaces. Second, the convergence guarantees for the atomic Bellman equations hold in the limit $r_i$2; the behavior of VDPPO at practical discount factors, and whether the learned fixed point approximates $r_i$3 well on specific tasks, is empirical rather than theoretically bounded. Third, the decomposition class, while broad, excludes arbitrary nestings of TL operators outside the canonical form of Theorem 6, and the value-graph size grows exponentially ($r_i$4) in the number of conjuncts, so scalability to very large specifications remains open. Fourth, the shared-embedding hypothesis — that distinct DVG Values admit a common representation — is validated empirically but not characterized theoretically; identifying when representation sharing degrades approximation quality is unresolved. Finally, the framework assumes deterministic dynamics; extension to stochastic systems, where extremum-based Values require risk-sensitive reformulations, is not addressed.

Conclusion

The paper establishes an exact decomposition algebra for Bellman Values of a substantial class of temporal logic predicates, introduces a contractive Bellman equation for always-eventually (recurrence) specifications, and demonstrates that embedding the resulting Value graph into a shared-network PPO variant yields large performance gains over automaton-based and sampling-based baselines, including successful hardware deployment on heterogeneous drone-quadruped teams. The principal theoretical contribution — separating Value algebra from robustness-semantics algebra and supplying correct atomic rules — provides a principled alternative to reward shaping and Lagrangian tuning for safe, temporally structured control.