Library-based Neuro-symbolic Motion Planning

• Library-based neuro-symbolic motion planning is a framework that combines modular libraries of skills, operators, and predicates to enable interpretable and scalable robotic decision-making.
• It employs a bilevel planning algorithm where high-level symbolic planning guides neural subgoal sampling and control, ensuring efficient task decomposition and trajectory refinement.
• Empirical results demonstrate notable data efficiency, strong zero-shot generalization, and formal correctness guarantees across diverse robotic domains.

Library-based neuro-symbolic motion planning is an approach that integrates symbolic high-level reasoning and neural low-level control—packaged as modular, reusable libraries of skills, operators, samplers, or predicates—to achieve scalable, generalizable decision-making in robotics and task-and-motion planning (TAMP) domains. These frameworks offer abstractions that support data-efficient learning, long-horizon planning, task transfer, and formal correctness guarantees, with empirical validation across diverse robotic environments (Silver et al., 2022, Chitnis et al., 2021, Khodeir et al., 2021, Sun et al., 2022, Liang et al., 2024).

1. Neuro-Symbolic Skill and Operator Abstractions

Library-based approaches formulate neuro-symbolic modules as multi-part entities, typically consisting of:

• Typed arguments: E.g., a parameter vector $v = (v_1,\dots,v_n)$ specifying object types or roles.
• Symbolic operator: A lifted STRIPS-style action template $o = \langle v, P, E^+, E^- \rangle$ with preconditions $P$, add effects $E^+$, and delete effects $E^-$, all defined over abstract predicates.
• Neural components: Typically, a subgoal-conditioned policy $\pi(o; x, x') \rightarrow u$ mapping a current continuous state $x$ and subgoal $x'$ to low-level actions $u$, and a subgoal sampler $\mathcal{S}(o; x) \Rightarrow x' \sim \mathcal{D}$ producing feasible next states consistent with the operator’s effects (Silver et al., 2022).
• Transition models and samplers: For NSRTs, a neural transition model $o = \langle v, P, E^+, E^- \rangle$0 approximates object-specific state changes, and an action sampler $o = \langle v, P, E^+, E^- \rangle$1 produces candidate continuous actions (Chitnis et al., 2021).

These "library" modules can be indexed by their symbolic signatures, parameterized over arbitrary objects, and retrieved or composed as needed in new planning contexts. The library may also include reusable procedural generators for perception and control (e.g., in PDDLStream, families of streams for IK, grasping, or collision checking (Khodeir et al., 2021); in VisualPredicator, both primitive and recursively defined neuro-symbolic predicates (Liang et al., 2024)).

2. Bilevel and Hierarchical Planning Algorithms

The essential planning pipeline in library-based neuro-symbolic systems follows a bilevel, search-then-sample paradigm:

• Outer loop (symbolic planning): A high-level planner (e.g., A* over abstract states) sequences lifted operators or HLAs, assembling candidate plan skeletons (Silver et al., 2022, Chitnis et al., 2021).
• Inner loop (continuous refinement): For each symbolic step, neural samplers and/or policies search for feasible subgoals and control sequences in continuous space, typically by repeated sampling and rollout; failures trigger backtracking or fallback to other abstract plans.
• Library integration: Libraries provide modularity, with each operator/predicate or stream acting as a building block that can be instantiated with new objects, objects sets, or parameterized on task-specific constraints (Liang et al., 2024, Khodeir et al., 2021).

A representative PLAN–REFINE pseudocode (from (Silver et al., 2022)): the planner enumerates top-$o = \langle v, P, E^+, E^- \rangle$2 abstract plans, and for each tries up to $o = \langle v, P, E^+, E^- \rangle$3 subgoal samplers/refinements per action; successful sequences yield full low-level plans.

3. Automated Learning of Library Components

Most frameworks construct their neuro-symbolic libraries from demonstration or exploration data, proceeding by:

• Temporal segmentation: Cutting demonstration trajectories into meaningful action-execution segments whenever high-level contact or task predicates change.
• Partitioning and lifting: Clustering action segments into equivalence classes by matching effect signatures up to argument renaming; lifting from concrete to variable-parameterized operators (Silver et al., 2022, Chitnis et al., 2021).
• Symbolic operator/predicate learning: Intersection of precondition abstractions; effect lifting; optionally, online predicate invention and selection using operator learning objectives or VLM prompting (Liang et al., 2024).
• Neural module learning: Behavioral cloning for policies (supervised regression from state/subgoal to action), Gaussian maximum-likelihood for subgoal generators, and discriminative regularization to ensure subgoal quality (Silver et al., 2022). For NSRTs, separate networks are trained for transition modeling, action sampling, and applicability classification (Chitnis et al., 2021).
• Formal training: For temporal-logic planning, neural nets are trained to represent exactly the symbolic transitions encoded by a discrete memory or automaton, using projection-based constraints to guarantee symbolic consistency (Sun et al., 2022).

The result is a skill/operator library or predicate+operator set, generalizing to novel objects, configurations, or goal specifications.

4. Search Control via Neural Ranking and Predicate Invention

Advanced frameworks enhance library-based planning by using learned neural heuristics or performing online predicate/operator invention:

• Neural search control: In PDDLStream, a GNN-based scoring function prioritizes which procedural generator (stream) to expand next, focusing planning effort on likely-relevant objects or subgoals (Khodeir et al., 2021). For each stream, a per-stream MLP scores instantiations for their predicted plan relevance.
• Predicate invention: VisualPredicator proposes new neuro-symbolic predicate candidates by VLM prompting, transition discrimination, or unconditional generation, then selects or prunes predicate sets based on best-first search and task-solving performance (Liang et al., 2024).
• Hierarchical refinement: Some frameworks also allow recursive/derived predicates and operators (e.g., concept predicates in VisualPredicator), further enhancing modularity and abstraction.

This suggests an ongoing trend toward coupling symbolic expressiveness with neural perceptual and organizational mechanisms for deep task abstraction and sample-efficient learning.

5. Empirical Evaluation and Performance Metrics

Evaluation protocols typically span:

• Domains: Simulated robotics environments with combinatorial object arrangements, 3D manipulation, tool use, doors, stacking, or pouring (e.g., “Cover”, “Blocks”, “Coffee”, “Painting” (Silver et al., 2022, Liang et al., 2024, Chitnis et al., 2021)).
• Metrics: Test set solve rate (fraction of tasks solved), planning time (per solved instance), data/sample complexity, planning budget (number of abstract plans sampled), and OOD (out-of-distribution) generalization gap (Liang et al., 2024).
• Baselines: Monolithic neural policies (GNN BC), fixed abstraction planners, vanilla symbolic planning, model-free RL, or HRL methods.
• Results: Library-based neuro-symbolic planners—especially those with subgoal samplers and top-$o = \langle v, P, E^+, E^- \rangle$4 plan enumeration—typically achieve 80–100% success on held-out tasks with modest demonstration counts (100–1000), outperforming monolithic or no-sampler ablations, with notable data efficiency and zero-shot generalization. VisualPredicator, in particular, matches or nearly matches an "Oracle" human-engineered abstraction, showing strong gains over pure HRL or VLM-planning approaches, especially in complex domains (Silver et al., 2022, Liang et al., 2024).

A selection of results:

Domain	BPNS Solve % (Silver et al., 2022)	NSRT Solve % (Chitnis et al., 2021)	VisualPredicator Solve % (Liang et al., 2024)
Cover	80–99	98.4 (easy)	100
Blocks	—	—	96
Coffee	—	—	65
Painting/Kitchen	90+	85.0	—

Confidence intervals and OOD gaps are reported in individual papers. BPNS and VisualPredicator show strong zero-shot robustness, as in the generalization to unseen numbers or arrangements of objects.

6. Interpretability and Theoretical Guarantees

The modularity and transparency of the library approach yield several advantages:

• Interpretability: Each skill, operator, or predicate is human-readable—implemented as short Python classes or explicit STRIPS operators—enabling direct inspection and debugging. The full symbolic abstraction (PDDL-style) is accessible for expert audit (Liang et al., 2024).
• Correctness and guarantees: In temporal-logic settings, the symbolic “discrete memory” (abstract MDP × automaton) ensures that the composed neural planner provably retains the correctness properties of the symbolic plan, with quantifiable bounds on generalization and suboptimality; see Theorems 5.1–5.2 in (Sun et al., 2022).
• Failure handling and explanation: When learning from data, failed transitions are used to augment symbolic preconditions and prune infeasible plans, further strengthening the robustness of the planning loop (Chitnis et al., 2021).

A plausible implication is that library-based neuro-symbolic frameworks will continue to be preferred for settings where transparency, data efficiency, and generalization across variable domain instantiations are critical.

7. Connections to Related Work and Future Directions

These frameworks unify and extend previous efforts in TAMP, HRL, and relational abstraction, addressing several open challenges:

• Extensibility: Modular libraries allow lifelong accretion of skills, operators, and predicates, supporting continuous refinement and transfer (Silver et al., 2022, Liang et al., 2024).
• Planning under partial observability: Visual or language-grounded neuro-symbolic predicates can incorporate perception and reasoning directly over sensor data (Liang et al., 2024).
• Integration with existing solvers: Symbolic planners (e.g., Fast Downward), motion-planning backends (OMPL, MoveIt!), or PDDLStream orchestration can all directly consume learned neuro-symbolic libraries (Chitnis et al., 2021, Khodeir et al., 2021).
• Generalization to temporal logic: Extended frameworks handle temporal goals via symbolic automata, with libraries of locally optimal neural controllers per abstract transition (Sun et al., 2022).

As empirical studies expand, future research may further sharpen neural-symbolic synergy for hierarchical, explainable, data-driven lifelong robot learning and deployment.