Point2Act: Zero-Shot 3D Action Localization

• Point2Act is a zero-shot, language-conditioned framework that precisely localizes 3D action points by converting multimodal 2D cues into a compact 3D relevancy field.
• The method employs a NeRF-style 3D reconstruction and multi-view distillation to generate candidate 6-DoF grasps without task-specific training.
• Key innovations include efficient multi-view aggregation, rapid relevancy field convergence, and improved grasp accuracy even under occlusion and compositional language challenges.

Searching arXiv for the cited Point2Act-related papers to ground the article in current records. Point2Act denotes a zero-shot, language-conditioned manipulation framework that localizes the most relevant 3D action point for a contextually described task and extracts a physically feasible, context-aware manipulation pose in a novel scene without task-specific training. In its specific 2025 formulation, Point2Act directly retrieves the 3D action point relevant for a contextually described task by distilling multi-view, point-level outputs from a multimodal LLM into a compact single-channel 3D relevancy field, then uses that field to select 6-DoF grasps. The method is motivated by the observation that multimodal LLMs provide strong contextual understanding in 2D images but often produce blurry 2D regions that do not translate cleanly into precise 3D action locations required for physical interaction (Kim et al., 5 Aug 2025).

1. Problem setting and conceptual basis

Point2Act addresses a core gap in zero-shot, language-conditioned manipulation: multimodal LLMs are effective at semantic understanding in images but typically provide diffuse 2D guidance, such as heatmaps or bounding boxes, that is insufficient for precise 3D action localization. The target problem is defined as follows: given a natural language instruction, localize the most relevant 3D action point or points and extract a physically feasible, context-aware manipulation pose, such as a 6-DoF grasp, in an unseen environment without task-specific training (Kim et al., 5 Aug 2025).

The framework takes the position that, for manipulation, the end result is a localized 3D action point or region. On that basis, it bypasses dense high-dimensional 3D feature fields and instead distills sparse, point-level multimodal guidance across views into a compact scalar 3D field. This field, denoted $s(x) \in [0,1]$, is intended to encode where to act rather than to represent a general-purpose 3D semantic embedding. The practical motivation is twofold: high-dimensional 3D feature fields are characterized as redundant and computationally expensive, and viewpoint-dependent 2D activations are particularly problematic under occlusion and compositional language such as references to parts, spatial relations, or context-dependent object choice (Kim et al., 5 Aug 2025).

This design places Point2Act at the intersection of multimodal grounding, 3D reconstruction, and manipulation planning. The contribution is not an end-to-end policy trained from demonstrations, but a plug-and-play localization-and-grasping pipeline that uses multimodal reasoning for task semantics and a geometric field for actionable spatial grounding.

2. End-to-end pipeline

The pipeline begins with capture of posed multi-view images using a wrist-mounted RGB camera on a Franka Panda, with camera intrinsics $K_v$ and extrinsics $(R_v,t_v)$ supplied by the robot setup and communication handled via ROS. Depth is not used in the primary method, only in comparison baselines. Each captured image $I_v$ and the language instruction $L$ are then passed to Molmo, which is prompted to output the most relevant 2D point or points for the intended action, including part-aware, spatial, and abstract contextual instructions (Kim et al., 5 Aug 2025).

The resulting 2D points are converted into soft masks $M_{gt}^v$ via Gaussian blur, described as CornerNet-style and typically using $\sigma = 4$. In parallel, a NeRF-style geometry and appearance branch is trained for approximately $200$ iterations on downsampled images, for example $160 \times 90$ after $4\times$ downsampling. The reconstruction stage is explicitly optimized for speed, with downsampling used to accelerate optimization and reduce floater artifacts under few-iteration regimes (Kim et al., 5 Aug 2025).

A lightweight relevancy head, $K_v$0, then maps a 3D point $K_v$1 to a scalar relevancy value $K_v$2. Supervision is imposed through volumetric rendering: the rendered relevancy image $K_v$3 is required to match the soft 2D mask $K_v$4 across views. This multi-view distillation step is the central mechanism by which sparse 2D multimodal outputs are transformed into a view-independent 3D field. After training, RGB, depth, and relevancy are rendered from several views and unprojected into a colored point cloud with per-point relevancy (Kim et al., 5 Aug 2025).

The final stage is grasp proposal and selection. Point2Act uses AnyGrasp to generate candidate 6-DoF grasps from the point cloud. For each candidate grasp, the method identifies a contact neighborhood and selects the grasp whose contact region contains the point with highest relevancy. The result is a manipulation pose that is intended to satisfy both physical feasibility and instruction-conditioned semantic relevance.

3. Representation, rendering, and multi-view aggregation

The geometric core of Point2Act consists of a NeRF-style geometry and appearance model together with a separate lightweight relevancy head. For color rendering, a geometry MLP maps $K_v$5 to color $K_v$6 and density $K_v$7, and the rendered color along a ray $K_v$8 is

$K_v$9

with transmittance

$(R_v,t_v)$0

The relevancy head maps $(R_v,t_v)$1 to $(R_v,t_v)$2, and the rendered relevancy along a ray is

$(R_v,t_v)$3

Training minimizes the multi-view relevancy loss

$(R_v,t_v)$4

This formulation is important because the transmittance term $(R_v,t_v)$5 acts as a visibility model. In consequence, only visible surface regions along a ray substantially contribute to the rendered relevancy, which the method uses to attenuate occluded regions and enforce cross-view consistency (Kim et al., 5 Aug 2025).

Camera geometry is handled through the usual projective formulation. A 3D point is projected into view $(R_v,t_v)$6 by

$(R_v,t_v)$7

with $(R_v,t_v)$8 homogeneous. For RGB-D baselines or point-cloud extraction, back-projection is written as

$(R_v,t_v)$9

The paper also gives an intuitive alternative aggregation view in which a 3D relevancy function accumulates per-view signals using visibility, view weights, and radial kernels, but explicitly states that Point2Act itself relies on the differentiable volumetric renderer rather than on explicit back-projection-and-voting (Kim et al., 5 Aug 2025).

A central implication of this representation is that multi-view consistency is not imposed post hoc. Instead, the 3D field must explain all view-conditioned soft masks simultaneously. This suggests that the field is serving both as a geometric regularizer and as a semantic aggregation mechanism.

4. Grasp extraction, efficiency, and deployment characteristics

Once relevancy has been distilled into 3D, Point2Act renders RGB, depth, and relevancy from several views and unprojects them into a point cloud annotated by $I_v$0. AnyGrasp then proposes candidate grasps. For a candidate grasp $I_v$1 with contact center $I_v$2, the method finds the $I_v$3-nearest neighbors with $I_v$4 and scores the grasp by the maximum relevancy in that neighborhood. The selected grasp is therefore the one whose contact neighborhood contains the most relevant 3D point according to the distilled field (Kim et al., 5 Aug 2025).

The system is designed as a full-stack pipeline comprising capturing, MLLM querying, 3D reconstruction, and grasp pose extraction. Its reported end-to-end runtime is approximately $I_v$5 seconds, and it is also described as generating spatially grounded responses in under $I_v$6 seconds. MLLM querying takes about $I_v$7 to $I_v$8 seconds per view, 3D reconstruction requires about $I_v$9 seconds and is identified as the major bottleneck, and the scalar relevancy head converges in roughly $L$0 iterations. The implementation hides latency by preloading models, pipelining MLLM queries during scanning, and running grasp inference from an early point cloud at around iteration $L$1 while relevancy training continues (Kim et al., 5 Aug 2025).

The reported hardware includes an NVIDIA RTX 4090 for NeRF and AnyGrasp, an RTX 6000 Ada for Molmo, a Franka Panda arm, and an Intel RealSense wrist camera, although the primary pipeline uses only RGB from that camera. Practical deployment depends on known camera intrinsics and extrinsics, accurate posed captures, and diverse viewpoints to reduce occlusion and improve reconstruction stability.

5. Evaluation and empirical findings

Point2Act is evaluated on both localization and grasping benchmarks. The localization benchmark uses $L$2 real tabletop scenes and $L$3 language prompts spanning object-only, texture, part, spatial, and physical descriptions. Evaluation uses projection accuracy and 3D distance error to ground-truth object point clouds, with baselines including LERF, F3RM, and methods that unproject single-view MLLM points using measured or rendered depth (Kim et al., 5 Aug 2025).

On projection accuracy, the appendix results reported for Table A.1 are: at $L$4 iterations, LERF achieves $L$5, F3RM $L$6, and Point2Act $L$7; at $L$8 iterations, LERF reaches $L$9, F3RM $M_{gt}^v$0, and Point2Act $M_{gt}^v$1. The paper emphasizes that Point2Act achieves higher accuracy and lower distance error even with few iterations, and that multi-view distillation outperforms single-view unprojection baselines. It also notes that rendered NeRF depth can outperform measured depth near edges because it fills sensor holes (Kim et al., 5 Aug 2025).

The grasping benchmark uses $M_{gt}^v$2 real tabletop scenes and $M_{gt}^v$3 prompts with $M_{gt}^v$4 trials each, including part-level prompts such as “handle of mug” and “cap of marker” as well as context-level prompts involving spatial relations and commonsense choices. Baselines include F3RM, LERF-TOGO, and single-view MLLM 2D points with depth unprojection. The reported outcome is that Point2Act consistently achieves better zero-shot grasping, especially for compositional semantics and spatial context. Reported failure modes include recognition errors caused by occlusion or geometry artifacts on shiny objects, challenging grasp geometries such as scissor handles and kettle spouts, and cases where LERF-TOGO fails because high score thresholds filter valid grasps (Kim et al., 5 Aug 2025).

Ablations further characterize the method. Accuracy improves as Gaussian blur width increases up to $M_{gt}^v$5 and then declines due to over-smoothing, so $M_{gt}^v$6 is used throughout. Reliable contextual grounding in MLLM queries requires image resolutions at least on the order of $M_{gt}^v$7, whereas NeRF optimization benefits from downsampling during few-iteration training. The scalar relevancy objective is described as converging quickly, in about $M_{gt}^v$8 iterations, which is central to the system’s practical runtime (Kim et al., 5 Aug 2025).

6. Relation to adjacent “point-to-action” formulations, limitations, and future directions

The term “Point2Act” has acquired more than one usage in recent robot-learning literature. In the 2025 work discussed above, it denotes a language-conditioned, zero-shot grasping pipeline built around a compact 3D relevancy field (Kim et al., 5 Aug 2025). In the 2024 Track2Act paper, by contrast, Point2Act is presented as a paradigm: predict how points should move to realize a goal, convert those 2D motions into 3D object $M_{gt}^v$9 transforms, map object motion to end-effector motion, and refine execution with a residual policy. There, the emphasis is on goal-conditioned manipulation from web-video-derived point tracks rather than on MLLM-guided 3D action-point localization (Bharadhwaj et al., 2024). A separate 2026 work, PointACT, is explicitly not Point2Act; it is a dual-system, 3D-aware vision-language-action policy that integrates hierarchical 3D point cloud geometry directly into action decoding through multi-scale point-action interaction (Chen et al., 20 May 2026).

Method	Core intermediate representation	Primary manipulation setting
Point2Act	Single-channel 3D relevancy field $\sigma = 4$0 distilled from multi-view MLLM points	Zero-shot context-aware grasping
Track2Act	Goal-conditioned 2D point tracks lifted to object $\sigma = 4$1 transforms	Generalizable robot manipulation from web videos
PointACT	Hierarchical 3D point-cloud features coupled to evolving action tokens	3D-aware VLA policy learning

This comparison clarifies a common naming misconception. Point2Act and PointACT are distinct systems, and Track2Act uses “Point2Act” as a conceptual description of its point-to-action pipeline rather than as the name of the 2025 3D relevancy-field method.

The limitations of Point2Act are also specific. The system is not real-time because multi-view capture and NeRF optimization introduce latency, with reconstruction alone taking roughly $\sigma = 4$2 seconds. Robustness depends on capturing multiple posed views; this dependency cannot be omitted if occlusion is to be handled reliably. The field provides point-level guidance rather than full masks, which may be restrictive for tasks requiring region outputs or multiple action marks. The framework also inherits the compute cost of running Molmo, and it can fail under extreme occlusion, highly abstract language, textureless or shiny surfaces that degrade reconstruction, and difficult grasp geometries (Kim et al., 5 Aug 2025).

The future directions proposed in the paper are correspondingly geometric and systems-oriented: faster 3D reconstruction through RGB-D fusion, TSDF or Gaussian Splatting, NeRF-SLAM hybrids for continuous updates, combination of point-level relevancy with lightweight region retrieval, uncertainty-aware multi-view aggregation, and closed-loop control through coupling with real-time point tracking or keypoint-constrained control. A plausible implication is that Point2Act is best understood not as a complete manipulation policy, but as a high-precision semantic grounding front end that can be paired with broader reactive control systems when tasks require continuous correction or longer-horizon interaction (Kim et al., 5 Aug 2025).