ScrewSplat: Articulated Object Recognition

• ScrewSplat is an end-to-end method for articulated object recognition that uses multi-view RGB images to recover part-aware 3D geometry and joint parameters.
• It unifies kinematic screw modeling with 3D Gaussian splatting to estimate joint types, axes, angles, and part segmentation without category priors or depth data.
• The approach optimizes all variables in a single differentiable rendering framework, enabling accurate, real-time reconstruction and manipulation of complex articulated objects.

Searching arXiv for the named paper and closely related 3DGS/articulated-object work to ground the article. ScrewSplat is an end-to-end method for articulated object recognition from RGB-only observations that unifies kinematic screw modeling with 3D Gaussian Splatting in a single end-to-end optimization. From multi-view images across multiple articulation states, it recovers the object’s part-aware 3D geometry, the number and types of joints, the joint screw axes and positions, per-configuration joint angles, and part segmentation, without category priors, depth, pre-known joint counts, or multi-stage correspondence/clustering pipelines. The method assumes that all movable parts articulate with respect to a single static base and that multi-view RGB images are available for multiple configurations; camera intrinsics and extrinsics are assumed known during reconstruction (Kim et al., 4 Aug 2025).

1. Problem setting and representational scope

The inputs are multi-view RGB images $\{I_{vk}\}$ from $N_v$ camera poses $v = 1,\ldots,N_v$ and $n_a$ configurations $k = 1,\ldots,n_a$ of an unknown articulated object. For real-world data, images are masked with a pretrained segmenter, SAM. No depth is used. The outputs are a part-aware Gaussian scene representation for geometry and appearance; the set of active joints with types, screw axes $S = [\omega\ v] \in \mathbb{R}^6$, and per-configuration joint angles $\theta_k \in \mathbb{R}^{n_s}$; part segmentation as soft assignments of Gaussians to parts and a static base; and optional joint limits inferred post hoc as elementwise min–max of optimized $\theta_k$ (Kim et al., 4 Aug 2025).

Two assumptions define the main scope. First, all movable parts articulate with respect to a single static base; no kinematic chains are modeled in the main method. Second, the method operates on multi-view RGB observations across multiple articulation states. Within that scope, ScrewSplat explicitly avoids several common constraints: the number of parts and joints is unknown a priori; there are no category priors; there is no depth input; and there is no dependence on point correspondences or discrete pre-segmentation.

A plausible implication is that the method is positioned against articulated-object pipelines in which geometry reconstruction, correspondence estimation, part clustering, and joint fitting are separate stages. ScrewSplat instead treats part discovery and kinematic estimation as latent variables inside a single differentiable rendering objective.

2. Kinematic model: screw axes, joint types, and rigid motion

ScrewSplat uses screw theory as the motion parameterization. A screw axis $S \in \mathbb{R}^6$ is written as $S = [\omega\ v]$, with two special cases. For a revolute joint, the motion is pure rotation about a unit axis $N_v$0 with zero pitch $N_v$1, so $N_v$2 and $N_v$3, where $N_v$4 is any point on the axis. For a prismatic joint, the motion is pure translation along the axis, so $N_v$5 and $N_v$6 (Kim et al., 4 Aug 2025).

Let $N_v$7 be the $N_v$8 skew-symmetric matrix of $N_v$9,

$v = 1,\ldots,N_v$0

The associated $v = 1,\ldots,N_v$1 twist matrix is

$v = 1,\ldots,N_v$2

For a joint displacement $v = 1,\ldots,N_v$3, the induced rigid transform is

$v = 1,\ldots,N_v$4

For revolute motion,

$v = 1,\ldots,N_v$5

with

$v = 1,\ldots,N_v$6

and

$v = 1,\ldots,N_v$7

For prismatic motion, where $v = 1,\ldots,N_v$8,

$v = 1,\ldots,N_v$9

Although the main formulation attaches each movable part directly to the static base, the method notes an extension to kinematic chains through the product of exponentials formulation,

$n_a$0

where $n_a$1 is the part’s home pose. This suggests a natural path toward chain-structured articulated objects, but that extension is not the main operating regime.

3. Part-aware 3D Gaussian Splatting

The scene is represented by anisotropic 3D Gaussian primitives. Each Gaussian $n_a$2 has parameters $n_a$3, where $n_a$4 is the pose, $n_a$5 are scales, $n_a$6 is the covariance, $n_a$7 is opacity, and $n_a$8 denotes color through spherical harmonics coefficients. The per-Gaussian opacity field is

$n_a$9

Pixels are rendered by alpha compositing ordered Gaussians along the view ray,

$k = 1,\ldots,n_a$0

The rendering loss is

$k = 1,\ldots,n_a$1

with $k = 1,\ldots,n_a$2 (Kim et al., 4 Aug 2025).

ScrewSplat augments this representation with screw primitives and part probabilities. A screw primitive is

$k = 1,\ldots,n_a$3

where $k = 1,\ldots,n_a$4 is a screw axis and $k = 1,\ldots,n_a$5 is its confidence. A part-aware Gaussian is

$k = 1,\ldots,n_a$6

with $k = 1,\ldots,n_a$7 a probability simplex over $k = 1,\ldots,n_a$8 parts,

$k = 1,\ldots,n_a$9

Here $S = [\omega\ v] \in \mathbb{R}^6$0 is the probability of belonging to the static base, and $S = [\omega\ v] \in \mathbb{R}^6$1 for $S = [\omega\ v] \in \mathbb{R}^6$2 is the probability of belonging to the part moved by $S = [\omega\ v] \in \mathbb{R}^6$3.

For each part-aware Gaussian $S = [\omega\ v] \in \mathbb{R}^6$4, the method replicates $S = [\omega\ v] \in \mathbb{R}^6$5 standard Gaussians $S = [\omega\ v] \in \mathbb{R}^6$6, one for the static base and one per screw primitive. For configuration $S = [\omega\ v] \in \mathbb{R}^6$7:

• Base replica:

$S = [\omega\ v] \in \mathbb{R}^6$8

• Movable-part replica:

$S = [\omega\ v] \in \mathbb{R}^6$9

$\theta_k \in \mathbb{R}^{n_s}$0

This construction yields soft, probabilistic per-Gaussian part assignment via $\theta_k \in \mathbb{R}^{n_s}$1; continuous kinematic parameterization through screw axes $\theta_k \in \mathbb{R}^{n_s}$2 and angles $\theta_k \in \mathbb{R}^{n_s}$3; and joint-type discovery by spawning both revolute and prismatic screws and pruning unused types. Geometry $\theta_k \in \mathbb{R}^{n_s}$4 and appearance $\theta_k \in \mathbb{R}^{n_s}$5 are shared across replicas, while opacity is scaled by the soft part probability and further modulated by the screw confidence for movable parts.

4. Objective, initialization, and optimization dynamics

The unified objective is

$\theta_k \in \mathbb{R}^{n_s}$6

The $\theta_k \in \mathbb{R}^{n_s}$7 regularizer is a parsimony term that encourages using as few, confident screws as necessary. The reported settings are $\theta_k \in \mathbb{R}^{n_s}$8 in simulation and $\theta_k \in \mathbb{R}^{n_s}$9 in real-world experiments (Kim et al., 4 Aug 2025).

Initialization is deliberately overcomplete. The method initializes $\theta_k$0 part-aware Gaussians as in standard 3D Gaussian Splatting, with positions sampled uniformly in a workspace volume within the camera frustum. Part probabilities are initialized uniformly over the simplex. It also spawns $\theta_k$1 screw candidates: eight revolute and eight prismatic. For sampled $\theta_k$2 from $\theta_k$3,

• revolute screws use $\theta_k$4 and $\theta_k$5;
• prismatic screws use $\theta_k$6 and $\theta_k$7. All $\theta_k$8 are initialized to $\theta_k$9, and all joint angles are initialized to $S \in \mathbb{R}^6$0.

Optimization is by gradient descent on all variables. The reported learning rates are: geometry and appearance as in 3DGS; part probabilities $S \in \mathbb{R}^6$1 with learning rate $S \in \mathbb{R}^6$2 on logits before softmax; screw axes $S \in \mathbb{R}^6$3 with learning rate $S \in \mathbb{R}^6$4 on raw parameters before re-normalizing to satisfy revolute or prismatic constraints; screw confidences $S \in \mathbb{R}^6$5 with learning rate $S \in \mathbb{R}^6$6 on logits before sigmoid; and joint angles $S \in \mathbb{R}^6$7 with learning rate $S \in \mathbb{R}^6$8.

To escape local minima and stabilize discovery, the method periodically resets all $S \in \mathbb{R}^6$9 and all $S = [\omega\ v]$0 uniform. This reset period is asynchronous with the standard 3DGS opacity reset, and synchronizing the two was found detrimental. It also canonicalizes joint angles and Gaussian poses by selecting a random $S = [\omega\ v]$1 from $S = [\omega\ v]$2, setting $S = [\omega\ v]$3 for all $S = [\omega\ v]$4, and updating each $S = [\omega\ v]$5 by unwinding the dominant part $S = [\omega\ v]$6:

$S = [\omega\ v]$7

Model selection occurs through pruning. A screw is removed if $S = [\omega\ v]$8. It is also removed as a zero-motion screw if

$S = [\omega\ v]$9

with $N_v$00 for revolute joints and $N_v$01 for prismatic joints. After pruning, remaining variables are fine-tuned. During rendering and backpropagation, only screws with $N_v$02 are considered to reduce computation.

A plausible implication is that ScrewSplat converts unknown joint count and unknown joint type into an optimization-overparameterization problem, then resolves it through photometric consistency plus parsimony rather than through discrete structure search.

5. Segmentation, articulation recovery, and manipulation interfaces

Segmentation is induced by the per-Gaussian simplex variable $N_v$03. During training, each Gaussian softly participates in exactly one of $N_v$04 parts, and rigidity is enforced implicitly because each replicated movable Gaussian $N_v$05 moves rigidly with its single screw $N_v$06 through $N_v$07. Articulation consistency is enforced by a single $N_v$08 per screw across all images in configuration $N_v$09 and by one screw $N_v$10 per part; no explicit pairwise consistency or correspondence loss is used (Kim et al., 4 Aug 2025).

After optimization, a hard segmentation can be obtained by taking the $N_v$11 over $N_v$12. Joint types and counts are those that remain after pruning. Joint limits are estimated as elementwise min/max of the optimized $N_v$13. The learned part-aware Gaussian model can be meshed by depth rendering, TSDF fusion, and marching cubes.

ScrewSplat also functions as a differentiable renderer for downstream articulation inference. Given new RGB views, current-state estimation minimizes

$N_v$14

with respect to $N_v$15, using Bayesian optimization.

For zero-shot, text-guided manipulation, the method defines a renderer $N_v$16 mapping target joint angles $N_v$17 to an image $N_v$18. A CLIP-based directional loss aligns image-embedding change with text-embedding change:

$N_v$19

$N_v$20

$N_v$21

The optimization target is

$N_v$22

solved by Bayesian optimization through gp_minimize with Expected Improvement, 50 total evaluations, and 10 random initial samples. The search domain is the recognized joint limits inferred from the optimized training configurations.

Robot execution is described at the level of affordance and trajectory construction: choose an affordance point from the recognized movable part, such as Gaussian centers farthest from the revolute axis or nearest along the prismatic axis; move this point from $N_v$23 to $N_v$24 along the screw motion; lift the motion to an $N_v$25 gripper trajectory with fixed orientation; and solve IK for robot joint trajectories. This suggests that the representation is designed not only for recognition, but also for direct operational use in manipulation stacks.

6. Empirical performance, ablations, and limitations

The evaluation uses ten single-joint and three multi-joint objects from PartNet-Mobility. There are 48 cameras on a hemisphere, and ScrewSplat uses five configurations per object; baselines PARIS, PARIS* (depth), and DTA (RGB-D) use two configurations. Geometry is evaluated by bidirectional Chamfer-L2 distance in millimeters for static parts, movable parts, and the whole object. Motion is evaluated by axis angular error in degrees and axis position error for revolute joints. Appearance is evaluated by PSNR and SSIM on novel joint angles midway between training states. For multi-joint evaluation, bipartite matching between predicted and ground-truth axes minimizes total angular and position error (Kim et al., 4 Aug 2025).

On single-joint objects, the reported averages are:

• PARIS: CD-s 54.015, CD-m 18.032, CD-w 40.192, Ang 17.656, Pos 2.020, PSNR 28.64, SSIM 0.970.
• PARIS*: CD-s 49.706, CD-m 8.864, CD-w 33.856, Ang 16.287, Pos 1.742, PSNR 28.66, SSIM 0.970.
• DTA (RGB-D): CD-s 0.538, CD-m 0.528, CD-w 0.360, Ang 0.437, Pos 0.308.
• ScrewSpawn: CD-s 0.617, CD-m 11.566, CD-w 0.946, Ang 24.869, Pos 0.902, PSNR 29.11, SSIM 0.982.
• ScrewSplat (RGB-only): CD-s 0.319, CD-m 0.211, CD-w 0.261, Ang 0.084, Pos 0.010, PSNR 38.07, SSIM 0.993.

On multi-joint objects, the reported averages are:

• DTA (RGB-D): CD-s 0.568, CD-mm 5.647, CD-w 0.476, Angm 7.233, Posm 28.877.
• ScrewSplat: CD-s 0.675, CD-mm 0.096, CD-w 0.666, Angm 0.130, Posm 0.002, PSNR 36.76, SSIM 0.987.

The ablation study identifies the importance of the multi-screw formulation. ScrewSpawn, which spawns a single known screw, is insufficient to recover geometry and kinematics broadly. A two-configuration ablation shows competitive results for several objects, but failures on some cases, including stapler and USB, indicate that more articulation diversity and improved optimization help.

The practical profile is comparatively lightweight: approximately 7–9 minutes for 30,000 iterations on an RTX 4090, about 30 MB per object model, and peak GPU memory of about 2.8 GB. Real-world experiments use 16 feasible robot-mounted views. However, the method has several explicit limitations. It is sensitive to the parsimony weight $N_v$26: too low produces redundant or collapsed multiple screws per part, while too high misses articulation and explains everything as static. It assumes all movable parts articulate with respect to a static base rather than through chains. Shadows cast by moving parts can introduce view- and configuration-dependent appearance changes, and without explicit illumination modeling the method may spawn extra Gaussians to explain lighting. Camera poses are assumed known rather than estimated.

These results support the abstract claim that ScrewSplat achieves state-of-the-art recognition accuracy across a diverse set of articulated objects while operating solely on RGB observations. More specifically, the reported numbers indicate that its largest empirical advantage lies in movable-part geometry and kinematic accuracy, while the product-of-exponentials extension and illumination-aware modeling remain open directions for broader articulated-scene generalization.