Actionable 3D Object Flow

• Actionable 3D object flow is a dense, geometry- and task-aware representation that predicts per-point manipulation feasibility and concrete motion trajectories.
• It leverages visual priors and a curiosity-driven reinforcement learning framework to generate accurate 6-DoF motion predictions over detailed object surfaces.
• This approach bypasses traditional kinematic abstractions by directly coupling perception with action planning, enabling robust manipulation in diverse, real-world scenarios.

Actionable 3D object flow refers to dense, geometry- and task-aware predictions of how local surface points or parts of objects move and can be manipulated to achieve downstream goals. Unlike traditional kinematic abstractions or pose-based affordance representations, actionable 3D object flow encodes at the point or part level not only whether and how a region of the object is manipulable, but also provides concrete motion trajectories and success probabilities for diverse actions under varying task constraints. This approach is data-driven and tightly couples visual perception with action planning by explicitly representing what is physically doable—for example, opening a door by predicting 6-DoF trajectories for its handle, rather than just reporting the handle's position. Actionable 3D object flow thus links dense visual geometry directly to task-parameterized manipulation skills, yielding a fine-grained, generalizable interface between perception and control.

1. Object-Centric Actionable Visual Priors

The core innovation in actionable 3D object flow, as instantiated in VAT-Mart, is the use of object-centric actionable visual priors at the perception–action interface (Wu et al., 2021). Each point $p$ on an articulated surface receives a predicted actionability score $a_{p|O,T,\theta}\in [0,1]$ indicating its likelihood of supporting a successful action given the object $O$, task type $T$, and specification $\theta$. Beyond binary or categorical labelling, this representation attaches dense, per-point distributions over feasible action trajectories (expressed as sequences of 6-DoF waypoints) $\tau_{p|O,T,\theta}$, and a per-trajectory success likelihood $r_{\tau|O,p,T,\theta}\in[0,1]$.

This approach forgoes abstraction into global kinematic structures (e.g., joints, part axes), instead directly encoding interaction feasibility and expected outcomes at high spatial resolution. Such dense priors preserve subtle local features—curvature, edges, gripper accessibility—that are essential in real, cluttered, and unconstrained environments, but often lost in lower-dimensional abstracted models.

2. Interaction-for-Perception Framework

Actionable 3D object flow is realized in VAT-Mart within an "interaction-for-perception" paradigm, where a curiosity-driven RL policy and perception module mutually shape each other's outputs in a bidirectional supervision loop. The RL policy, based on TD3 (Twin Delayed DDPG), operates over a state space encompassing part pose, contact location, task specifications (angle of revolute joints or linear position of prismatic joints), and gripper pose, producing as output a residual trajectory comprising waypoints in SE(3). Rewards mix extrinsic task-completion terms with intrinsic curiosity feedback proportional to the perception module’s current uncertainty about trajectory success: $-500\cdot r_{\tau|O,p,T,\theta}$.

Simultaneously, the perception module ingests sampled partial point clouds, contact points, trajectories, and task specifications, outputting actionability, trajectory distributions, and success likelihoods for each surface point. The perception model’s cVAE learns a distribution over future action paths, supervised by observed RL rollouts (with L$_1$ losses on waypoints, 6D orientation loss, and KL divergence regularization), and computes average top-k success to regress pointwise actionability. The perception module's "curiosity feedback" critically drives the RL policy to explore interactions where its own predictions are currently least certain, resulting in a more diverse and informative actionable prior knowledge base.

3. Dense Trajectory Prediction and Evaluation

At test time, actionable 3D object flow enables querying every relevant object surface point for: (a) its manipulability with respect to the current task (encoded as a heatmap of actionability); (b) distributions over executable trajectories; and (c) the likelihood of success for sampled actions. In VAT-Mart, this is used within downstream planners to select optimal contact points and action sequences.

System evaluation is performed on the PartNet-Mobility dataset (SAPIEN), spanning hundreds of articulated CAD models over categories (doors, drawers, etc.), measuring per-point actionability classification (accuracy, F-score), coverage of the proposal set versus ground-truth trajectories, and overall manipulation success. VAT-Mart significantly outperforms both naive end-to-end RL and hand-tuned joint-parameter baselines—especially on tasks requiring geometric and interaction granularity (such as identifying subtleties in handle shape or edge accessibility).

Qualitative results show VAT-Mart generalizes dense actionable visual priors to unseen object classes and performs robustly on real-world scans and physical robots, exhibiting resilience to domain gaps between synthetic and real geometry.

4. Mathematical Formulation

The actionable 3D object flow interface, as formalized in VAT-Mart, is composed of:

• Actionability: $a_{p|O,T,\theta} \in [0,1]$
• Trajectory Distribution: $$\tau_{p|O,T,\theta} \sim \mathcal{P}_p(\cdot|O,T,\theta)$$
• Success Likelihood: $r_{\tau|O,p,T,\theta}\in [0,1]$

where the task $T$ is specified by an angle $\theta\in[-\pi, \pi]$ for revolute joints or a normalized value for prismatic joints. The cVAE-based trajectory proposal mechanism stabilizes predictions for high-variance motion spaces, while the per-point scoring enables spatially dense planning.

The policy is trained with both extrinsic task rewards and a curiosity-based penalty:

$$\text{reward} = \text{extrinsic} - 500 \cdot r_{\tau|O,p,T,\theta}$$

and actionability training uses the mean of top-k success scores over diverse action proposals.

5. Generalization and Real-World Transfer

VAT-Mart establishes that actionable 3D object flow learned via interaction-for-perception generalizes successfully along three axes:

• Unseen shapes: The system predicts manipulable sites and action proposals on CAD geometries and depth scans not encountered during training.
• Unseen categories: Learned actionable priors transfer across object classes (e.g., trained on drawers, generalized to doors).
• Physical domain: Policies and perceptions learned on synthetic, textureless geometries are empirically robust on real-world sensor observations and real robot execution.

This suggests that dense, data-driven actionable priors mitigate sim-to-real appearance mismatch by encoding interaction-centric rather than appearance-centric knowledge.

6. Applications and Implications

Actionable 3D object flow is directly applicable as an action interface for home-assistance, service, and industrial robots tasked with manipulating varied articulated objects. Given a new scene, a robot planning system can:

• Query surface points for predicted actionability under a given task
• Sample from diverse, geometry-adaptive trajectory proposals
• Select actions with maximal estimated success probability
• Bypass hand-engineered kinematic feature extraction or reliance on object CAD models

The modular and contextual nature of the interface supports complex, multi-step manipulation planning and is a promising foundation for research in temporally extended or multi-object interaction scenarios. The observed efficacy and generalization open directions for future research in leveraging longer action sequences, multi-frame perception, and integration with high-DoF articulated structures.

7. Visualizations and Interpretability

Visual depiction of actionable 3D object flow, as provided in VAT-Mart, typically includes:

• Per-point heatmaps overlaying the object mesh indicating spatial actionability priors
• Visualization of predicted trajectories (e.g., as curves or arrows) for key contact points, with color indicating success likelihood
• Architectural diagrams showing the interaction of RL and perception modules and their bidirectional supervision pipeline

Such visualizations clarify the operational semantics of actionable 3D object flow and provide transparent insight into the perception–action coupling that underpins dense, context-aware robotic manipulation.

---

Actionable 3D object flow, as operationalized in VAT-Mart, is a structured, geometry- and task-conditioned representation linking dense surface affordances, trajectory proposals, and success metrics in a perception–action loop. This approach enables robots to generalize manipulation skills across diverse objects, tasks, and environments by leveraging point-level priors directly grounded in interaction experience, marking a significant advance in bridging pixel-space perception to real-world manipulation.