Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition

Abstract: Learning from Demonstration (LfD) enables robots to learn complex behaviors from expert examples, yet existing approaches often fail to generalize to new compositions of known skills without retraining. Modern generative policies model distributions over action trajectories alone, thus are unable to reason about the symbolic outcomes required for robust composition. We propose that skills should jointly model action trajectories and the symbolic outcomes they induce. To address this gap, we introduce Predicate Action Skills (PACTS), a class of closed-loop visuomotor policies that model skills as a joint generative process over action and predicate belief trajectories, producing coherent action-outcome rollouts within a single model. Jointly generating actions and predicates enables PACTS to learn internal representations that improve both action generation and predicate classification. Furthermore, we demonstrate zero-shot composition of learned skills via planning by leveraging online predicate predictions from PACTS as a symbolic interface for sequencing and monitoring execution. Project website: https://planpacts.github.io/

• The paper demonstrates the Pacts framework that jointly models action trajectories and predicate belief traces to enable coherent zero-shot skill composition.
• The framework employs denoising diffusion probabilistic modeling and conditional flow matching with segmentation tools to enhance action–predicate coherence and predictive accuracy.
• The joint modeling strategy bridges sensorimotor control with symbolic reasoning, facilitating real-time monitoring, planning, and recovery in diverse robotic tasks.

Joint Generative Predicate-Action Policies for Zero-Shot Skill Composition

Motivation and Problem Formulation

The paper "Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition" (2605.20648) introduces Predicate-Action Skills (Pacts), a class of closed-loop visuomotor policies that jointly model action trajectories and symbolic predicate-belief trajectories. The traditional approach in Learning from Demonstration (LfD) focuses solely on distributions over action sequences, which results in limited generalization to novel task compositions due to the decoupling of action generation from symbolic state predictions. This work advocates that manipulation requires the tight coupling of actions and their symbolic outcomes for robust long-horizon composition and zero-shot recomposition.

Pacts models skills as a joint generative process by learning $p_\theta(\mathbf{x},\mathbf{z}\mid\mathbf{o})$, wherein a trajectory of actions $\mathbf{x}$ and a trajectory of predicate beliefs $\mathbf{z}$ aligned with PDDL predicates are generated together, conditioned on the current observation $\mathbf{o}$. The predicate-belief trajectory provides a planner-friendly abstraction for sequencing, monitoring, and triggering replanning, significantly broadening the compositional flexibility of learned skills.

Figure 1: The architecture comparison between Pacts's joint generative modeling and standalone/multi-task baselines isolating action and predicate prediction.

Methodology and Model Architecture

Pacts unifies generation of actions and predicate beliefs within a conditional trajectory generator, instantiated via DDPM denoising or conditional flow matching objectives. Unlike factorized multi-task methods, the output is a single temporally coherent trajectory of both modalities, enforcing consistency between executed actions and expected symbolic outcomes.

The architecture consists of an observation encoder (providing a latent representation) and either a temporal UNet or Transformer backbone for the joint trajectory generator. At each timestep, conditioned on $\mathbf{o}_t$, the policy samples $(\mathbf{x},\mathbf{z})$ and executes a short chunk of actions, re-observing and re-sampling. The design supports rollouts where each rollout inherently contains a symbolic trace for the planner interface.

Planning-Based Composition and Predicate-Belief Interface

A significant advance is leveraging the continuous predicate-belief trajectory as a symbolic interface compatible with standard planning frameworks (e.g., PDDL). During execution, the system uses sampled predicate beliefs to check skill effects, monitor progress, and trigger replanning upon discrepancies between expected and observed symbolic state.

A skill segmentation and labeling toolkit is introduced to convert monolithic demonstrations into skill-centric datasets with dense predicate labels, enabling efficient training and deployment of composable skills in realistic settings.

Figure 2: Evaluation environments, including compositional 2D simulations (PushBarrier), 3D visual manipulation tasks (Kitchen, Coffee Preparation), and real-world cube packing.

Empirical Evaluation

Experiments span controlled compositional benchmarks (PushBarrier), pragmatic 3D manipulation tasks (RoboMimic Kitchen and Coffee Preparation), and a real-world cube packing task. The study comprehensively evaluates policy performance, predicate classification, and predicate-action coherence, comparing Pacts to action-only, predicate-only, and multi-task baselines, sweeping across DDPM/CFM objectives and UNet/Transformer backbones.

Numerical Results

• On PushBarrier, Pacts consistently achieves the highest task reward and predicate classification accuracy among all baselines (macro F1 up to 0.9885, accuracy up to 93.78%) without degrading action performance.
• In RoboMimic domains, Pacts maintains competitive or superior predicate prediction (F1 up to 0.9451) and task completion relative to action-only models.
• Predicate-action coherence curves demonstrate that Pacts's predicted belief rollouts closely track outcomes realized under execution, outperforming base-rate or constant baselines, with near-perfect agreement and lower scoring rule losses over the horizon.

  Figure 3: Predicate–action coherence metrics showing superior agreement (CrossCons), lower Brier scores, and lower log loss for Pacts compared to baselines across prediction horizons.

Skill Recomposition and Recovery

The predicate-belief interface enables zero-shot recomposition and adversarial recovery. In PushBarrier, planners synthesize novel skill sequences to achieve subgoals not present in demonstrations. Online predicate monitoring allows immediate detection of unexpected effects (e.g., adversarial closure of the door), triggering replanning and recomposing the execution sequence to fulfill the desired goal.

Figure 4: Example of planner-driven skill recomposition and recovery; symbolic effects are monitored, replanning is triggered upon perturbation, and the system resumes goal-directed execution.

Limitations

The efficacy depends on quality and coverage of predicate sets; incomplete predicates restrict planning reliability. Symbolic labels must be well-aligned with task structure; partial observability, perception errors, and distribution shift remain challenging. The interface assumes manually defined symbolic domains; automated predicate and operator discovery is a promising but unsolved direction. Leveraging pretrained vision–LLMs to invent predicates from demonstration [athalye2026pixels] could potentially scale symbolic interfaces.

Implications and Future Prospects

This work positions joint generative modeling of action and predicate belief as a foundation for composable, planner-integrated robotic skills. Practical implications include rapid deployment of reusable skills with zero-shot recomposition, robust monitoring, and recovery—making robot policies more reliable and flexible for real-world multi-step tasks. Theoretically, the approach motivates richer abstractions and scaling symbolic vocabularies via foundation-model priors and automated predicate invention. Further research may focus on expanding abstraction learning, long-horizon planning, and integrating neuro-symbolic representations for generalizable manipulation [liang2024visualpredicator].

Conclusion

Pacts establishes joint predicate-action generation as a robust solution for composable skill deployment, offering strong empirical justification for treating skill models as distributions over temporally coherent action–outcome rollouts. The formal symbolic interface supports efficient planning integration, enabling zero-shot recomposition and recovery, and provides a promising substrate for further advances in modularity, abstraction, and autonomy in robot learning.