Inverse Manipulation through Symbolic Planning and Residual Operator Learning

Abstract: Inverting a robotic task requires more than reversing symbolic state transitions or rewinding motor trajectories. In robot manipulation tasks, symbolic inverse plans often fail to fully restore the effects of forward executions under continuous interaction dynamics. We present a hybrid framework for inverse manipulation that derives inverse-skill objectives from STRIPS-like operators automatically extracted from demonstrations through soft geometric predicates. For each extracted operator, we construct an inverse restoration objective that preserves preconditions, restores delete effects, and negates add effects. A task planner first attempts to satisfy this objective using available action primitives. Unresolved symbolic predicates then induce a residual operator learning problem solved through Reinforcement Learning (RL). We evaluate the framework on the ManiSkill3 PushCube task. For a forward pushing skill, the symbolic inverse performs a coarse pick-and-place restoration, while a residual Soft Actor-Critic policy refines the cube pose to satisfy the remaining inverse predicates. Our results show that predicate-derived residual control can turn an approximate symbolic inverse into a physically grounded inverse skill.

• The paper introduces a hybrid approach that integrates symbolic planning with residual RL to restore manipulated states with high precision.
• It employs STRIPS-like operator extraction from demonstrations and predicate-grounded reward shaping to bridge the symbolic-continuous gap.
• Experimental results on the PushCube task show a marked reduction in residual error and increased success rates compared to symbolic methods alone.

Inverse Manipulation via Symbolic Planning and Residual Operator Learning

Problem Formulation and Motivation

This work investigates the problem of skill inversion within robotic manipulation domains, asserting that task reversal encompasses more than simple temporal regression or trajectory flip. Instead, effective inversion must restore world states in the presence of continuous interaction dynamics, especially where non-trivial state transitions and contact-rich manipulations occur. Inverse plans, if constructed solely at the symbolic level, tend to yield only approximate or partial reversals due to the abstraction gap between high-level operators and continuous control.

Hybrid Symbolic-Continuous Inverse Skill Synthesis

The core system begins with the extraction of symbolic STRIPS-like operators from skill demonstrations. The environment state is represented through a scene graph, grounding a repertoire of soft, differentiable geometric predicates. Operator construction averages initial and final predicate values across demonstrations, thereby identifying predicates for the precondition, add, and delete sets. This enables robust operator extraction from limited observation data and accommodates uncertainty present in real-world demonstration variance.

Figure 1: System overview illustrating symbolic operator extraction, inverse target formation, and reward decomposition for RL-driven restoration.

Inverse manipulation is then formalized as a restoration objective: achieving the full set of the forward operator’s preconditions and delete effects, while negating add effects. Given a limited library of low-level action primitives, a symbolic planner (BFS-based) generates an executable macro-action sequence designed to achieve as much of the inverse target as feasible.

The symbolic plan produces a state handoff, $s_h$, at which point a split occurs: predicates satisfied by the symbolic policy are considered fence constraints, while unresolved predicates form the active set for subsequent RL-based residual correction. This closed-loop mechanism guarantees that RL is applied only to the irreducible portion of the restoration problem, focusing function approximation capacity where exact symbolic regress is impossible or impractical.

Reward Shaping and Predicate Grounding for RL

Soft predicate grounding yields differentiable scores, $V_p(s)$, mapping continuous state to [0,1], where soft margins provide robustness against small perception and execution perturbations. The RL reward for the residual skill is automatically synthesized from the active and fence predicate sets. Specifically, a bounded, signed margin (via $\tanh$) is used for each predicate, ensuring non-divergent value estimates and bounded, consistent reward landscapes during RL training.

Figure 2: Reward decomposition into active (residual restoration) and fence (violation penalty) components for RL at the symbolic plan handoff.

Active predicates contribute both positively (restoration) and negatively (if unattained), while fence predicates are penalized in a one-sided manner if violated. Importantly, this approach generalizes beyond the hand-engineered reward design typical in deep RL for robotics: reward assignment is structurally tied to the semantics of the symbolic plan, not the global task specification.

Experimental Results: PushCube Task

Evaluation on ManiSkill3’s PushCube benchmark (Franka Panda, pd-ee-delta-pos) demonstrates the system’s efficacy in a canonical manipulation inversion setting. The forward push operator, learned from demonstrations, is characterized in terms of pose, end-effector proximity, and gripper openness predicates. The inverse target—restoring the cube to its source position and ensuring removal from the goal state—was partially solvable via symbolic pick-and-place primitives. The unresolved residual (cube pose) was then targeted by RL.

The composite approach—symbolic + RL—yielded a mean residual position error of 1.4 mm (std 3.2 mm), with a 90% success rate at 1 cm tolerance. In contrast, using only the symbolic prefix (without RL) resulted in a residual error of 16.6 mm and 10% success rate, underscoring the necessity of residual policy refinement for precision restoration. These results were robust under handoff perturbation, confirming generalization and stability.

Figure 3: Sequential screenshots of (a) initial state, (b) post-forward manipulation, (c) after symbolic reversal, and (d) after RL-based pose refinement.

Reward Ablation and Structural Insights

Ablation studies on the residual reward structure showed that unbounded rewards (removal of $\tanh$) induced value divergence and task failure, while symmetric rewards for fence predicates promoted inaction. The specific design—bounded, one-sided penalties for fences—was necessary and sufficient for driving effective exploration and restoration without fence violation.

Implications and Future Directions

This framework operationalizes task inversion as a hybrid symbolic-continuous problem, leveraging direct operator extraction and restoration-based reward generation. The marriage of high-level symbolic planning and predicate-grounded RL supports generalization over explicit reward/task engineering and enables robust, modular composition of inverse skills. The approach may be extended to more complex manipulation domains, provided scalable predicate repertoires and generalizable primitive libraries.

However, explicit limitations remain: predicates, their margins, and reward scales are designed a priori and not yet learned from raw observation. Primitive action libraries are hand-scripted; extension to learning-from-demonstration for primitives, as well as large-scale skill transfer across diverse task families, is a research opportunity. Further, expanding evaluation to tasks with multi-object dependencies and high-dimensional action spaces will be essential for confirming the scalability.

Conclusion

The integration of symbolic planning and automatically structured residual RL provides a principled and effective pathway to inverting complex manipulation skills, especially where symbolic reversibility is insufficient. Enhanced by predicate-grounded reward shaping, this system offers a template for robust, interpretable inverse policy synthesis in robotics. Future developments will center on predicate and primitive learning autonomy and broader empirical validation.