Primitive-Aware Semantic Grounding (PASG)

• Primitive-Aware Semantic Grounding (PASG) is a framework that binds semantic language cues to explicit geometric primitives, enabling fine-grained control in robotic manipulation.
• It employs a closed-loop pipeline combining segmentation, geometric extraction, and multi-view semantic anchoring to continuously optimize task-specific operations.
• Empirical evaluations show PASG achieves competitive accuracy in manipulation tasks and extends to applications in radiance-field reconstruction and 3D affordance grounding.

Primitive-Aware Semantic Grounding (PASG) denotes a class of methods that bind semantically meaningful language descriptions to explicit sub-object geometric units and make those bindings operational for downstream computation. In its named formulation for robotic manipulation, PASG is a closed-loop framework that connects low-level geometric primitives such as keypoints, axes, and directions with high-level task semantics such as grasping, pouring, pressing, aligning, or transporting (Zhu et al., 8 Aug 2025). In related literature, the same general idea appears in radiance-field reconstruction, 3D affordance grounding, and controllable video generation: language, semantic parsing, or affordance descriptions are grounded not only to whole objects, but to planes, parts, points, or per-point physical fields that directly affect rendering, control, or simulation (Ying et al., 2023, Gou et al., 18 Mar 2026, Zhang et al., 25 Nov 2025).

1. Conceptual definition and scope

In the PASG formulation for robotic manipulation, the core idea is that a manipulation-relevant object should not be described only by geometry nor only by language. PASG explicitly binds them together into interaction primitives that combine

$(E, S, F),$

where $E$ is a geometric primitive entity, $S$ is a structural description, and $F$ is a functional role in manipulation (Zhu et al., 8 Aug 2025). This makes PASG more specific than generic semantic grounding: it assigns semantic and functional meaning to spatial primitives and makes those meanings usable for manipulation reasoning.

The named PASG paper identifies a persistent fragmentation between high-level task semantics and low-level geometric features. The framework is designed to address semantic asymmetry between high-level task semantics, such as “pour tea,” “press button,” and “lift bottle,” and low-level spatial control, such as the exact point or axis required for grasping, alignment, or actuation (Zhu et al., 8 Aug 2025). Existing approaches are described as suffering either from semantic reasoning without spatial grounding, or from spatial primitives without semantic context, while automated primitive detectors can be brittle if main axes and orientation cues are ignored.

The term also has broader use as an interpretive description of related methods. In PARF, “primitive-aware semantic grounding” refers to a representation in which semantic labels in a dense semantic volume do not denote object classes, but indicate whether a voxel belongs to empty space, ordinary volumetric space, or a specific primitive plane, thereby grounding radiance-field learning in explicit geometric structure (Ying et al., 2023). In PhysChoreo, although the paper is not framed with the term PASG, its part-aware reconstruction stage maps text prompts onto part-level physical fields so that semantics, geometry, and physics are aligned at a sub-object level and can be edited during simulation (Zhang et al., 25 Nov 2025). In part-aware open-vocabulary 3D affordance grounding, the same underlying idea appears as grounding natural-language questions to functionally relevant part-level regions in 3D objects through part-aware instructions, semantic–geometric alignment, and prototype-based abstraction (Gou et al., 18 Mar 2026).

A common misconception is to equate PASG with primitive detection alone. The cited work does not support that reduction. PASG requires not only detection of primitives, but also semantic anchoring, functional interpretation, and downstream usability. Another misconception is that PASG is restricted to rigid keypoints for robot grasping. The related literature shows that the primitive notion can include planes in indoor scenes, part tokens in 3D affordance grounding, or object parts carrying material and physical parameters in video generation (Ying et al., 2023, Gou et al., 18 Mar 2026, Zhang et al., 25 Nov 2025).

2. Primitive representations and formal structure

The PASG paper defines two major primitive families: point primitives $\mathcal{P}$ and axis primitives $\mathcal{A}$ (Zhu et al., 8 Aug 2025). Point primitives include Anchor Point $p_a$, Grasp Point $p_g$, and Actuation Point $p_{act}$. Axis primitives include Primary Axis $a_p$, Functional Axis $E$0, and Approach Axis $E$1. In the supplement, primitive types are also organized as Main, Anchor, Grasp, Actuation, and Hinge, with each primitive carrying both position and orientation.

The formal bridge from task semantics to spatial primitives is given through task decomposition into subgoals. Given manipulation tasks

$E$2

each task is decomposed into subgoals

$E$3

and, for each subgoal, primitive constraints are defined by

$E$4

where $E$5 and $E$6. The unified primitive set becomes

$E$7

This formulation makes the semantic role of a primitive task-conditional rather than purely geometric (Zhu et al., 8 Aug 2025).

Related PASG-style systems instantiate different primitive vocabularies. PARF uses a dense semantic volume $E$8 and a global primitive list $E$9, where each primitive is a plane

$S$0

satisfying

$S$1

The semantic volume contains E-voxels with $S$2, D-voxels with $S$3, and P-voxels with $S$4, the latter indexing explicit planes in the global primitive list (Ying et al., 2023). Here the primitive is not a manipulation point or axis, but an explicit planar region that controls rendering and editing behavior.

In PhysChoreo, the primitive is effectively an object part represented at the point level. Given an input image $S$5, the framework segments all instances, reconstructs a dense mesh for each instance using InstantMesh, and samples $S$6 surface points from each mesh to form a point cloud $S$7. It uses a global text prompt $S$8 embedded as $S$9 and optional part-level prompts $F$0 embedded as $F$1, so that points can be grounded to part-specific phrases and then to material class, Young’s modulus, Poisson’s ratio, and density (Zhang et al., 25 Nov 2025).

The part-aware 3D affordance grounding work adopts yet another primitive notion: the target is a per-point mask over a 3D object point cloud $F$2, predicted as

$F$3

where $F$4 is the affordance probability for point $F$5. A raw question is rewritten into a part-aware instruction, and a part-specific token embedding $F$6 becomes the semantic anchor used for geometric alignment (Gou et al., 18 Mar 2026). This suggests that PASG is better understood as a representational pattern—semantics anchored to explicit primitives—than as a single fixed architecture.

3. Closed-loop pipeline in robotic manipulation

PASG is described as a closed-loop framework because primitive extraction and semantic grounding feed back into each other. The loop is summarized as segmentation $F$7 primitive extraction $F$8 VLM semantic anchoring $F$9 confidence checking $\mathcal{P}$0 refinement $\mathcal{P}$1 updated primitives (Zhu et al., 8 Aug 2025). The paper explicitly describes this as a segment-align-detect-resample optimization loop.

The first stage is automatic primitive extraction via geometric feature aggregation. The object is rendered from multiple viewpoints

$\mathcal{P}$2

and these images are processed by Semantic SAM to obtain masks

$\mathcal{P}$3

Connected component analysis keeps the largest foreground component and removes background noise. The system then extracts centroids $\mathcal{P}$4 and corner feature points $\mathcal{P}$5. Corner detection uses $\mathcal{P}$6 and curvature analysis, while PCA estimates orthogonal axes and their boundary intersections. The raw keypoints $\mathcal{P}$7 are filtered by DBSCAN and then by farthest point sampling, yielding $\mathcal{P}$8. For standardized orientation, pre-aligned 3D object axes are used when available; otherwise the Z-axis is defined as the line connecting the geometric centroids of the top and bottom masks, with X/Y axes generated orthogonally (Zhu et al., 8 Aug 2025).

The second stage is VLM-driven semantic anchoring. Manipulation tasks are decomposed into subgoals, and a VLM uses multi-view object images together with detected primitives to map semantic descriptions to specific points and axes. The alignment procedure supports multi-candidate matching with confidence scores $\mathcal{P}$9, returns NONE if no candidate is found, and represents orientation either as symbolic axis labels such as $\mathcal{A}$0 or as point-pair directions such as $\mathcal{A}$1. Cross-view consistency is enforced so that point labels persist across viewpoints and can be projected between 2D and 3D (Zhu et al., 8 Aug 2025).

The third stage is dynamic self-refinement and semantic-affordance coupling. The paper gives an iterative algorithm that identifies low-confidence or unmatched semantic items,

$\mathcal{A}$2

and, when $\mathcal{A}$3, increases segmentation granularity $\mathcal{A}$4, re-runs Semantic SAM, re-aligns with the VLM, and replaces low-confidence matches. This continues up to $\mathcal{A}$5 (Zhu et al., 8 Aug 2025). The stated purpose is to make semantic-affordance relationships dynamic rather than static, allowing multiple possible grasp points on a bottle, emphasis on a hammerhead, or concurrent grasp and placement anchors on a cup.

This closed-loop design distinguishes PASG from one-pass annotation pipelines. Errors in geometry or semantics are not simply propagated downstream; they are actively corrected. A plausible implication is that PASG treats annotation as an optimization process over both geometric decomposition and semantic assignment, rather than as a fixed preprocessing step.

4. Benchmarks, fine-tuning, and empirical findings

PASG is also used to construct a spatial-semantic reasoning benchmark, Robocasa-PA (Zhu et al., 8 Aug 2025). Starting from RoboCasa and Objaverse, the dataset filters objects to keep those with meaningful spatial semantics. The final dataset contains 5,231 objects and 41,848 rendered images, with four horizontal orthographic views and four 45-degree oblique views. The benchmark contains 8,343 validated visual questions across three task types: Type Identification, Task Association, and Task-to-Primitive Mapping. The splits are a training set of 5,583 questions, an in-distribution test of 1,396 questions, and an out-of-distribution test of 1,364 questions, with evaluation by single-choice accuracy.

The same paper fine-tunes Qwen2.5VL with LoRA to create Qwen2.5VL-PA. The reported performance is 77.79% overall accuracy on the in-distribution test and 79.69% overall accuracy on the out-of-distribution test, described as a +33.9% absolute improvement over the base setting in the paper’s summary. Per-task results for Qwen2.5VL-PA are 86.63 / 89.02 for Task1, 83.48 / 81.73 for Task2, and 61.70 / 67.72 for Task3 (Zhu et al., 8 Aug 2025).

On benchmark comparison, Qwen2.5VL-PA is reported against GPT-4V, GPT-4o, GPT-4o-mini, LLaVA-1.5, Claude-3.5, Qwen-2.5VL, SpaceMantis, and RoboPoint. The notable overall accuracies reported are GPT-4V: 39.04 / 39.00, GPT-4o: 43.19 / 41.79, Qwen-2.5VL: 43.91 / 43.33, RoboPoint: 14.40 / 15.32, and Qwen2.5VL-PA: 77.79 / 79.69 (Zhu et al., 8 Aug 2025).

For robotic manipulation evaluation on RoboTwin, the authors test six tasks: Block Hammer Beat, Container Place, Dual Bottles Pick, Empty Cup Place, Pick Apple, and Messy Shoe Place. Each task is run 100 times with random seeds. The average success rates are Human Annotation: 84.67 and PASG: 77.83. Task-wise PASG results are 82 for Block Hammer Beat, 89 for Container Place, 70 for Dual Bottles Pick, 76 for Empty Cup Place, 81 for Pick Apple, and 69 for Messy Shoe Place (Zhu et al., 8 Aug 2025). The paper notes that PASG is competitive, and in some tasks it even outperforms human annotations, especially where richer or more flexible primitive sets help.

Human verification is reported at two scales. In an initial validation on 50 annotated objects, GPT-4o achieved 91.6% accuracy in semantic identification, 75.8% strict alignment accuracy, and 91.5% alignment effectiveness. In expanded verification on 200 objects, the reported metrics are keypoint accuracy 98.70% $\mathcal{A}$6 0.45%, direction accuracy 98.39% $\mathcal{A}$7 0.59%, consistency 92.69% $\mathcal{A}$8 1.15%, object-level accuracy 72.28% $\mathcal{A}$9 2.45%, positional completeness 92.52% $p_a$0 3.43%, functional completeness 97.00% $p_a$1 0.04%, and usage diversity 97.00% $p_a$2 0.04%. A hallucination analysis across 844 GPT-4o outputs on 100 objects reports invalid label index 0.40%, semantically unreasonable interaction 6.67%, missing functional labels 0.80%, and format violation 0.00% (Zhu et al., 8 Aug 2025).

5. Extensions and related formulations beyond manipulation

Related work shows that PASG-style grounding is not confined to manipulation. In 3D affordance grounding, the problem is to ground natural-language questions to functionally relevant regions in 3D objects. The cited method uses an LLM to rewrite a raw query into a structured instruction that contains object decomposition, part mention, and affordance-to-part linkage. The structured instruction $p_a$3 is encoded by RoBERTa as

$p_a$4

and a part-specific token embedding $p_a$5 is aligned to a grounded region embedding through

$p_a$6

The method further introduces Intra-Object Relational Modeling (IORM) for within-object structure and Affordance Prototype Aggregation (APA) for cross-object geometric consistency, with the total loss

$p_a$7

The main reported result on Open-set Full-view is aIoU 18.38, AUC 82.33, SIM 0.606, and MAE 0.110, with improvements also on Open-set Partial-view, Closed-set Seen, Closed-set Unseen, LASO, and 3D-AffordanceLLM (Gou et al., 18 Mar 2026). This indicates that PASG-like grounding can be formulated as part-token to region alignment together with prototype-based generalization.

In radiance-field reconstruction, PARF operationalizes primitive-aware grounding through a hybrid representation consisting of a dense semantic volume, a global primitive plane list, and a radiance MLP. The radiance field predicts density, color, and semantic primitive information,

$p_a$8

Rendering uses the standard NeRF-style compositing equation, but primitive regions are sampled by analytic ray–plane intersections,

$p_a$9

with fixed thickness $p_g$0 in experiments (Ying et al., 2023). The training objective combines RGB, depth, semantic, and opacity terms: $p_g$1 with $p_g$2, $p_g$3, and $p_g$4. The reported advantages include best reconstruction quality on Replica, ScanNet, and BundleFusion, significantly faster convergence than NeRF, InstantNGP, DVGO, Plenoxels, and NeRF-SLAM, robust extrapolation, on-the-fly incremental reconstruction in about 40 seconds per scene in the reported Replica setting, and direct semantic editing (Ying et al., 2023).

In controllable video generation, PhysChoreo grounds textual semantics to detailed 3D part structures, then predicts per-point material class probabilities $p_g$5 and continuous physical parameters $p_g$6 (Zhang et al., 25 Nov 2025). The core grounding mechanism is a soft assignment

$p_g$7

followed by hierarchical cross-attention,

$p_g$8

It includes prompt–part assignment supervision,

$p_g$9

wave continuity regularization over $p_{act}$0 and $p_{act}$1, and contrastive regularization based on elastic response descriptors $p_{act}$2 and $p_{act}$3. For video synthesis, the pipeline is reconstruct object/part physics from the image, simulate trajectories under editable dynamics, and feed trajectory video as conditioning to Wan2.2-Fun-5B-Control. The paper reports that PhysChoreo can generate videos with rich behaviors and physical realism, outperforming state-of-the-art methods on multiple evaluation metrics, and its ablations indicate that adding cross-attention, dual-stage design, soft assignment, and segmentation prior improves physical property prediction (Zhang et al., 25 Nov 2025).

Taken together, these works show that PASG can be instantiated with different primitives—points, axes, planes, parts, or affordance regions—so long as semantic information is localized onto explicit structural units that directly constrain inference, optimization, or control.

6. Limitations, interpretive issues, and research significance

The PASG paper explicitly notes two limitations: hallucinations from GPT-4o and a focus on rigid objects (Zhu et al., 8 Aug 2025). Despite multi-round refinement, semantic annotations can inherit hallucinated labels, and the iterative process is computationally expensive. The framework mainly handles rigid objects; deformable and articulated objects remain a challenge, and adapting primitive categories to dynamic shape variation is future work.

A second interpretive issue concerns what counts as a primitive. In robotic manipulation, primitives are keypoints, axes, and directions. In PARF, primitives are planes indexed by a semantic volume. In PhysChoreo, primitives are object parts represented by points and endowed with material and physical fields. In 3D affordance grounding, primitives are functionally relevant point-level regions tied to part-aware instructions. This suggests that “primitive” is not a single ontology, but a family of explicit structural units chosen to make semantic grounding actionable in a particular computational pipeline (Ying et al., 2023, Zhang et al., 25 Nov 2025, Gou et al., 18 Mar 2026).

A further misconception is that PASG is primarily about annotation efficiency. The evidence in the cited papers supports a broader interpretation. In the manipulation setting, PASG is used for automatic annotation, spatial-semantic reasoning, and policy-relevant primitive diversity (Zhu et al., 8 Aug 2025). In PARF, primitive grounding changes how rays are sampled and how scene editing is performed (Ying et al., 2023). In PhysChoreo, grounded parts become temporally editable physical control handles for MPM and rigid-body simulation (Zhang et al., 25 Nov 2025). In open-vocabulary 3D affordance grounding, part-aware instructions and prototype aggregation improve generalization to unseen affordances and partial views (Gou et al., 18 Mar 2026).

The broader significance of PASG lies in its unification of geometry, semantics, affordances, and downstream task structure. In the named PASG paper, this is expressed as a unified paradigm for bridging geometric primitives with task semantics in robotic manipulation (Zhu et al., 8 Aug 2025). In related domains, the same pattern links explicit structure to rendering, simulation, or open-vocabulary grounding. A plausible implication is that PASG is best understood not merely as a robotics technique, but as a general strategy for making semantic interpretation operational by attaching it to explicit primitives that downstream systems can optimize, simulate, edit, or control.