PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulation

Abstract: Diffusion policies have achieved remarkable success in robotic manipulation, yet they often fail to satisfy strict physical constraints required for safe deployment. Existing approaches impose safety either prematurely during training or reactively via external guardrails at test time, limiting policy expressivity and overall scalability. We propose Physical safety Alignment for Constrained Trajectories (PACT), a self-evolving post-training framework that projects pretrained diffusion policies onto constraint-feasible regions without accessing demonstration data or task rewards. PACT distills constraint gradients into the diffusion model through a reverse-KL objective with dense supervision across timesteps. It incorporates a curriculum that progressively tightens constraints while maintaining theoretically bounded policy shift and monotone improvement, mitigating the safety-performance trade-off from catastrophic forgetting. On simulated and real-world embodied manipulation benchmarks, PACT significantly reduces safety violations by 31.0% on average while improving task success by 30.7%.

• The paper introduces a novel post-training framework (PACT) that projects diffusion policies onto a physically safe set via constrained optimization.
• It employs curriculum-based distillation to iteratively inject dense constraint gradients, ensuring stable and monotonic improvements in safety performance.
• Empirical results show a 31% reduction in safety violations and a 30.7% increase in task success across simulation and real-world manipulation benchmarks.

PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulation

Motivation and Problem Definition

Diffusion policies have established themselves as state-of-the-art for robotic manipulation owing to their capacity for modeling multimodal action distributions and generalization across diverse tasks. However, their deployment in real-world robotic scenarios, especially in bimanual and precision-critical tasks, is hindered by the lack of formal enforcement of physical safety constraints—such as collision avoidance, force limits, and kinematic feasibility. Existing methods enforce safety either during policy training (limiting expressivity and requiring extensive engineering) or at inference via guardrails and backup policies (restricting flexibility and scalability), both insufficient for principled safety guarantees. Furthermore, post-training alignment for diffusion policies faces unique challenges due to the continuous, high-dimensional action spaces, the inaccessibility of task rewards and demonstration data, and the risk of catastrophic forgetting when enforcing constraints.

Formulation: Physical Safety Alignment as Constrained Projection

The authors formalize physical safety alignment as a projection of a pretrained diffusion policy $\mu_\phi$ onto the feasible set of policies defined by physical constraints in a Constrained Markov Decision Process (CMDP). This is formulated as a constrained optimization problem minimizing the KL divergence between the aligned and base policy while enforcing constraint satisfaction, which yields a Boltzmann-type solution: the optimal aligned policy is obtained through exponential tilting of the base policy by the weighted sum of safety costs. This constrained optimum is intractable for sampling, but its score function is directly computable by combining the base score and differentiable cost gradients.

Figure 1: PACT frames alignment as projection onto the CMDP feasible set, achieved via KL-regularized constrained optimization. The score function for the optimal aligned policy integrates base scores with differentiable cost gradients.

Methodology: Constraint Distillation and Curriculum-Based Alignment

PACT operates as an iterative, self-evolving post-training framework that distills constraint gradients—computed from differentiable safety cost functions—into the diffusion policy via a reverse-KL objective, using self-rollouts for data collection. Crucially, constraint signals are injected at every diffusion timestep (dense supervision), providing stable and continuous guidance, in contrast with RL approaches relying on sparse rewards.

A key technical innovation is curriculum distillation: constraints are progressively tightened along a predetermined schedule, bounding policy shifts and ensuring monotonic improvement in safety performance while mitigating catastrophic forgetting. This mitigates Irreversible Out-of-Distribution (OOD) Collapse, a major failure mode in direct distillation due to uncontrolled distributional drift.

Figure 2: Curriculum distillation controls intermediate policy shifts, preventing OOD collapse and enabling smooth transition to a safety-compliant policy.

Practical implementation includes training-free approximations for intermediate cost gradients, restricting constraint injection to late diffusion steps for computational efficiency with minimal loss of fidelity.

Empirical Results: Simulation and Real-World Benchmarks

Simulation studies on RoboTwin bimanual manipulation tasks demonstrate that PACT consistently elevates both task success and safety compliance. On four representative tasks, PACT reduces safety violations by 31% and increases task success by 30.7% relative to base policies; aligned behavior emerges as simultaneously more effective and physically compliant. Gains are pronounced in precision-sensitive tasks, validating its efficacy in reconciling the safety-performance trade-off.

PACT outperforms both on-policy and off-policy RL and imitation learning baselines, attaining higher and more stable success/safe rates, with markedly improved training efficiency (converges over 5× faster). The dense constraint distillation enables superior sample efficiency and stability compared to high-variance policy-gradient or value-based RL.

Figure 3: Training efficiency comparison against baselines. PACT demonstrates superior convergence and stability, achieving maximal success and safety rates.

Ablation studies show curriculum distillation and careful tuning of constraint multipliers are critical: direct distillation or excessive constraint strength induces OOD collapse. Injecting constraint gradients to a few late diffusion steps is optimal for fidelity and efficiency.

Qualitative and quantitative real-world evaluations on safety-critical tasks—such as pour water, transfer egg, nail insertion, and GPU assembly—show that PACT substantially reduces unsafe contacts and failure modes. On GPU Assembly, requiring millimeter-level precision, PACT induces emergent correction behaviors for accurate, safe insertions.

Figure 4: Physical safety alignment for diffusion-based manipulation. PACT resolves the safety-performance trade-off and enables precise, emergent corrective behaviors for challenging tasks such as GPU assembly.

Figure 5: Real-world qualitative comparison. PACT-corrected policies eliminate unsafe contacts and misalignments across four manipulation tasks.

Figure 6: Quantitative results in real-world tasks. PACT improves both normalized task progress and safe rate across all tasks, with the largest improvement on GPU Assembly.

Theoretical Guarantees

PACT provides theoretical guarantees on monotonic safety improvement and controlled policy shifts for each curriculum distillation step. The curriculum formulation ensures bounded total-variation change, preventing abrupt distributional collapse and ensuring smooth policy evolution. Distillation is provably optimal in the unlimited-data and unlimited-capacity regime, recovering the solution to the constrained CMDP projection.

Robustness and Further Ablation

Robustness to corrupted privileged state information is demonstrated: PACT sustains performance for moderate perception errors (up to 2 cm deviations), indicating resilience to upstream vision model inaccuracies commonly encountered in real-world deployment.

Figure 7: Robustness to noisy privileged state information; performance is maintained under significant corruption of privileged states.

Ablations confirm the approximate distillation and curriculum mechanisms are critical, supporting the design choices with empirical and theoretical evidence.

Implications and Future Directions

PACT exemplifies a general, scalable, reward-free paradigm for post-training alignment of diffusion policies with physical safety constraints in manipulation. Its decoupled framework allows adaptation to new constraint sets without retraining and eliminates the need for labor-intensive demonstrations, reward engineering, or test-time interventions. The approach is efficient, stable, and practically deployable in safety-critical robotic systems.

The practical implications are especially significant: PACT enables safer robotic manipulation without deployment-time overhead and is compatible with current and future generations of expressive diffusion policies. Theoretically, it advances post-training alignment techniques beyond discrete domains (e.g., LLMs) into continuous, high-dimensional control.

Extensions should address handling dynamic and force-sensitive constraints, improving upstream perception for noisy or ambiguous environments, and further scaling curriculum-based distillation into multi-agent and real-time settings. In the dual-use context, care is warranted to prevent misuse; PACT should be considered a risk-reduction mechanism, not a formal safety guarantee.

Conclusion

PACT delivers a principled and practical post-training safety alignment for diffusion-based manipulation policies. By curriculum distillation of differentiable constraint gradients, it achieves efficient, monotonic improvement in physical safety and task performance. Empirical and theoretical results indicate solid advantages over RL-based and imitation learning baselines. The approach is robust, deployable, and offers a clear pathway for scalable safety alignment in future embodied AI and robotics.