Assembly & Joint Annotations

• Assembly and joint annotations are formal representations that encode the structural and parametric details of multi-part systems in CAD, robotics, and human-machine interactions.
• They standardize data formats (e.g., JSON, graph models, and CAD schemas) to facilitate automated assembly planning, action recognition, and motion control.
• The annotations support precise metric evaluations (such as Chamfer distance and joint matching accuracy) and enable model-assisted workflows for enhanced synthesis and validation.

Assembly and joint annotations provide the formal backbone for representing, understanding, and automating actions, connections, and constraints within complex, multi-part domains—including robotic and human assembly, computer-aided design (CAD), psycholinguistic rating fusion, and vision-language instruction tasks. Annotation protocols encode both the compositional structure (how parts, joints, or constructs are organized and temporally deployed) and the parametric detail (types, locations, constraints, or multi-dimensional attribute values) of assemblies and their interactions. High-fidelity annotation is critical for supporting downstream tasks such as learning-based assembly planning, action recognition, CAD synthesis, robotic manipulation, and analytic modeling of human or machine understanding.

1. Foundational Annotation Schemas for Assembly and Joints

Annotation schemas vary by domain but consistently encapsulate part-level, joint/connector-level, and sequence-level information to enable downstream reasoning and automation.

Parametric CAD and Engineering Assemblies

In parametric CAD (e.g., Fusion 360, FreeCAD), assemblies are collections of component parts tied together by explicit joint constraints. Annotations are structured as JSON or attribute blobs, encoding for each part:

• Unique part ID (typically a UUID)
• Metadata: types (from controlled vocabularies), material attributes, and custom user-defined tags
• For each joint:
  • JointOrigin (frame placement: translation + quaternion)
  • Provided and required connection types (matching, e.g., “Hole3mm” vs. “BoltHead3mm”)
  • Joint parameters: axis vector, type (revolute, rigid, prismatic, ball, etc.), motion limits
  • Typed associations (e.g. “revolute” w/ axis and angle limits; “rigid” is constraint only)

This structure is used directly in tools such as CLS-CAD for automated assembly synthesis (Chaumet et al., 2023), JoinABLe for learning bottom-up joint and pose recovery (Willis et al., 2021), and ArtiCAD for agent-driven articulated model generation (Shui et al., 13 Apr 2026).

Vision-Language Manuals and Robotic Assembly

In assembly tasks informed by human manuals or robotic execution, annotation formalism elevates connection (joint) primitives to first-class entities:

• Parts and sub-assemblies are nodes in a hierarchical, edge-labeled assembly graph (Tie et al., 18 Oct 2025).
• Each connection (edge) annotates connector type (e.g. dowel, screw), specification (diameter, length, head type), quantity, and precise 3D placement (coordinates and normals in local part frames).
• The annotation process logs not only which parts connect, but granular attachment features, enabling geometric constraint optimization and motion planning grounded in physically meaningful parameters. Correspondence between manual image keypoints and actual 3D attachment locations is common (Wang et al., 2023, Tie et al., 18 Oct 2025).

Temporal Action and Multi-Agent Annotations

Action-based assembly datasets such as ATTACH (Aganian et al., 2023) annotate per-hand, per-frame action labels, enabling modeling of concurrent, temporally overlapping operations:

• For each frame and each hand: action label from an enumerated class set
• Synchronization across multi-camera views, with 3D body skeletons for spatial context
• Explicit modeling of overlapping actions (e.g., both hands active with non-null labels)

Multi-Dimensional and Psychological Constructs

For domains such as affective annotation where each instance carries multiple correlated dimensions (e.g., valence/arousal/dominance), joint annotation models represent:

• Annotator-specific confusion/mixing matrices for each dimension
• Time-series annotation as a block-filtered matrix
• Latent ground-truth constructs as multivariate vectors inferred via EM-based fusion of noisy ratings (Ramakrishna et al., 2020, Ramakrishna et al., 2020)

2. Formal Representation and Data Structures

Annotation approaches formalize assemblies and joints into data models suited for computational manipulation and learning.

Graph-Based Assembly and Joint Models

• Multi-level graphs: part-level (nodes = parts), joint-level (nodes = joints or connectors), and composition graphs (edges = connection relationships)
• Directed acyclic graphs (DAGs) for assembly plans: nodes represent subassemblies/steps, edges encode compositional sequences (post-order traversal for execution sequencing) (Wang et al., 2023)
• Joint-graphs: bipartite (pegs–holes), with edge weights encoding soft or hard matchings; features passed by PointNet or GNN-based message passing (Li et al., 2023, Willis et al., 2021)

Annotation File Formats

• CAD: JSON attributes per part/component, often as key-value or small object lists (Chaumet et al., 2023, Shui et al., 13 Apr 2026)
• Assembly plans: adjacency lists (parent/child), explicit step-level connection lists, attachment point coordinate arrays, and per-step pose transformations (4×4 SE(3) matrices)
• Vision-language: hierarchical JSON files grouping instance IDs, environment sections, and turn-wise navigation/assembly sub-objects (Kim et al., 2020, Wang et al., 2023)
• Joint attributes: connector label, axis vector, placement origin, and constraint parameters

Example Table: CAD Joint Annotation Fields

Field	Type / Format	Role
jointUUID	UUID string	Persistent joint key
type	Enum (e.g. revolute)	Joint class
transform	{tx,ty,tz, qx,qy,qz,qw}	Frame placement (SE(3))
axis	[float, float, float]	Unit vector for joint motion axis
limitLower/Upper	float	Joint bounds (radians or mm)
provided/required	string[]	Connection types for matching

3. Annotation Workflows and Quality Control

Annotation protocols span manual, semi-automated, and data-driven methodologies, with strong emphasis on unambiguous, reproducible mappings.

Manual and Interactive Annotation

• Human annotators bound temporal actions using synchronized multi-view videos (Aganian et al., 2023)
• Review via multiple AMT verification passes, enforcing instruction uniqueness, no disallowed terms/UI mentions, and linguistic referentiality (Kim et al., 2020)
• Assignment of types to parts and joints via interactive GUI in CAD plugins, enforcing subtype hierarchies and type-matching constraints (Chaumet et al., 2023)

Automated and Model-Assisted Annotation

• Vision-LLMs parse manuals/images to extract connector types, quantities, attachment locations, and pairings (connector-aware pipelines) (Tie et al., 18 Oct 2025)
• Keypoint annotation and EPnP-based pose extraction for 2D-3D correspondence (Wang et al., 2023)

Cross-Modal Validation

• Rollback validation: ArtiCAD employs VLM+LLM models for both part-level (geometry) and assembly-level (kinematics) validation, classifying failures as CODE or DESIGN and rolling back only the affected subset (Shui et al., 13 Apr 2026)
• Standardized file schemas (often JSON-Schema v4) to support parsing and downstream manipulation

4. Metrics and Evaluation of Annotated Assemblies

Evaluation protocols assess annotation quality, method performance, and suitability for learning or automation tasks using domain-specific metrics.

CAD and Physical Assembly

• Chamfer distance (shape, joint): quantifying geometric alignment between predicted/actual assemblies (Li et al., 2023, Willis et al., 2021)
• Part pose accuracy: % of parts/joints aligned within tolerances (e.g., <0.1 normalized CD)
• Joint matching accuracy: % of peg-hole pairs matched within strict geometric criteria (Li et al., 2023, Willis et al., 2021)

Action Recognition

• Mean class accuracy (mAcc), top-1/top-5 accuracy on per-clip and per-frame trimmed inputs (Aganian et al., 2023)
• Overlap ratio for concurrent per-hand actions,
• Framewise mean average precision (mAP) for action detection

Multi-dimensional Annotation Fusion

• Concordance correlation coefficient (ρ_c) and Pearson’s ρ comparing fused annotation estimates with reference ratings (Ramakrishna et al., 2020, Ramakrishna et al., 2020)
• Statistical comparison to independent baseline models (per-dimension independent EM)

Vision-Language Navigation & Assembly

• Normalized dynamic time warping (nDTW) for navigation trajectory similarity
• Placed Target Correctness (PTC): whether objects are placed in the prescribed grid location (Kim et al., 2020)
• Reciprocal Placed Object Distance (rPOD) to quantify proximity in placement tasks

5. Applications Across Domains

Assembly and joint annotations underpin a range of research and applications:

• Robotic Assembly and Manipulation: Connector-aware annotations enable precise motion planning, constraint satisfaction, insertion strategies, and real-world execution grounded in physical parameters (Tie et al., 18 Oct 2025, Shui et al., 13 Apr 2026)
• CAD Automation and Synthesis: Annotation-driven synthesis pipelines support automated component insertion, joint creation, and design variant generation in parametric environments (Chaumet et al., 2023, Shui et al., 13 Apr 2026)
• Human Action Understanding: Fine-grained, per-hand action labeling with explicit overlap modeling underpins action recognition in cobot-augmented industrial scenarios (Aganian et al., 2023)
• Natural Language Processing and Psychometric Modeling: Joint multi-dimensional fusion improves reliability of norm score aggregation, enabling sentence-level modeling not accessible via naive average-based protocols (Ramakrishna et al., 2020, Ramakrishna et al., 2020)
• Multimodal Instruction Interpretation: Joint navigation-assembly datasets (e.g., ArraMon) combine spatial and referential assembly annotation for embodied agent tasks (Kim et al., 2020)

6. Limitations, Ambiguities, and Future Research Directions

While current annotation schemes enable high-quality downstream outcomes, limitations persist:

• Ambiguity and Identifiability: Annotator confusion matrices (joint fusion) are identifiable only up to rotation/scaling; geometric ambiguities in 2D-to-3D keypoint mapping remain a challenge (Ramakrishna et al., 2020, Wang et al., 2023).
• Incompleteness in Hardware Typing: Some datasets abstract away low-level joint parameters (e.g., IKEA-Manual focuses on part pairing and pose, omitting screw/dowel specifics) (Wang et al., 2023).
• Cross-Modal Domain Gaps: Bridging vision-language data with physical geometry still involves intermediate manual or model-assist supervision (Tie et al., 18 Oct 2025, Wang et al., 2023).
• Scalability and Computational Complexity: EM-based fusion scales cubically with dimension, necessitating approximate or hard-EM inference in large time-series settings (Ramakrishna et al., 2020).
• Extension to Richer Joint Types and Topologies: Current CAD annotation pipelines (CLS-CAD, ArtiCAD) focus on kinematic-tree assemblies; non-tree structures and extended joint types (prismatic, spherical, etc.) are areas for ongoing development (Chaumet et al., 2023, Shui et al., 13 Apr 2026).

Annotations continue to become increasingly explicit, connector-centric, and structurally leveraged, moving from coarse per-entity labeling to graph-structured, parametric representations that directly support learning, synthesis, and robust physical execution.