UniStateDLO: Unified DLO Perception & Tracking

• UniStateDLO is a unified framework that employs conditional generative diffusion models to estimate and track the 3D state of deformable linear objects from partial point clouds.
• It integrates a two-branch coarse-to-fine estimator with diffusion-based fusion to recover missing geometry and ensure precise local details under extensive occlusions.
• The pipeline incorporates robust cross-frame tracking with zero-shot sim-to-real generalization, enabling stable closed-loop control in constrained robotic manipulation.

Searching arXiv for the target paper and closely related DLO perception/tracking work. UniStateDLO is a unified perception framework for estimating and tracking the three-dimensional state of deformable linear objects (DLOs) such as cables, ropes, and wires under severe occlusion in constrained robotic manipulation settings. It formulates both single-frame state estimation and cross-frame tracking from partial point clouds as conditional generative problems, using diffusion models to map highly partial, noisy observations to high-dimensional DLO state representations. The system is presented as the first complete DLO perception pipeline with deep-learning methods that covers both single-frame state estimation and cross-frame state tracking under substantial occlusions, while also exhibiting zero-shot sim-to-real generalization from synthetic-only training data (Lv et al., 19 Dec 2025).

1. Problem setting and motivation

A DLO is represented as an ordered chain of $M$ nodes,

$$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$

whose node positions capture the object’s spatial configuration at time $t$. The observation at each frame is a partial point cloud

$X_t \in \mathbb{R}^{N \times 3},$

extracted from an RGB-D camera. The estimation objective is to produce $\hat Y_t$ that minimizes $\|\hat Y_t - Y_t\|$, either from a single frame $X_t$ or by using the history $\{X_s, \hat Y_s \mid s < t\}$ for tracking (Lv et al., 19 Dec 2025).

The formulation is motivated by the failure modes of prior vision-based DLO pipelines in constrained environments. Earlier pipelines are described as three decoupled stages—segmentation, single-frame detection, and non-rigid registration-based tracking—and they often degrade when large portions of the DLO are occluded. The reported failure mechanisms are specific: skeleton extraction in 2-D or 3-D may fragment under occlusion; geometric registration methods such as CPD, CDCPD2, and TrackDLO require full or near-full visibility or expensive physics simulations; and purely data-driven single-frame lifting often lacks local precision in 3-D because DLO point clouds are feature-sparse. Constrained manipulation tasks including shape control in clutter, dual-arm cable routing, and knot tying therefore require a perception stack that combines global robustness with high local accuracy.

This framing places UniStateDLO at the intersection of DLO perception, generative modeling, and closed-loop robotic manipulation. A plausible implication is that the framework is intended not merely as an estimator, but as a front-end whose output quality must remain stable when visibility is poor and the downstream controller depends on temporally coherent geometry.

2. Unified conditional generative formulation

The central methodological decision in UniStateDLO is to cast both single-frame estimation and cross-frame tracking as conditional generative problems. Rather than directly regressing node positions with a deterministic map, the framework learns a conditional distribution $p(Y \mid \text{observations})$ using diffusion models, with the stated rationale that diffusion models capture complex, multi-modal, high-dimensional mappings between highly partial observations and DLO states (Lv et al., 19 Dec 2025).

For single-frame estimation, the model uses two coarse predictors—global direct regression and local point-to-point voting—and treats their outputs as node-wise conditions for a diffusion-based fusion module. The forward diffusion process is defined as

$$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$

or equivalently

$$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$0

with $$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$1 and $$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$2. The reverse process is parameterized as

$$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$3

and the network is trained via denoising score matching with fixed variance schedule.

The conditioning structure is explicitly node-wise. Features derived from the coarse estimates are assembled as

$$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$4

and combined with the noisy chain $$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$5, a step embedding $$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$6, and a node-index embedding $$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$7. These inputs are processed through a shared denoising MLP and three graph-convolution layers to predict $$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$8, with loss

$$Y_t = [y_{1,t}, y_{2,t}, \ldots, y_{M,t}]^T \in \mathbb{R}^{M \times 3},$$9

This unified generative view is significant because it treats missing geometry under occlusion as a conditional completion problem rather than a purely discriminative regression problem. The paper’s emphasis on globally smooth yet locally precise reconstructions suggests that the diffusion stage is intended to reconcile global shape completion with local detail preservation, especially when visibility is fragmented.

3. Single-frame state estimation architecture

The single-frame estimator is organized as a two-branch coarse-to-fine system followed by diffusion-based fusion (Lv et al., 19 Dec 2025).

The direct regression branch extracts a global feature

$t$0

and predicts

$t$1

with loss

$t$2

This branch supplies a coarse global hypothesis of the full DLO shape, including missing portions.

The point-to-point voting branch emphasizes local geometry. PointNet++ encodes local features $t$3, after which two heads predict a heatmap $t$4 and unit offsets $t$5:

$t$6

The ground-truth heatmap and unit offset are

$t$7

At inference, each point casts a vote

$t$8

and the method retains the top-$t$9 votes by $X_t \in \mathbb{R}^{N \times 3},$0 and computes a confidence-weighted average to obtain $X_t \in \mathbb{R}^{N \times 3},$1. The training objective is

$X_t \in \mathbb{R}^{N \times 3},$2

The diffusion fusion module then models $X_t \in \mathbb{R}^{N \times 3},$3 and iteratively refines the chain. The role division is explicit: global direct regression supports completion under heavy missingness, while local voting contributes point-wise geometric precision. The paper’s ablations indicate that this division is not incidental. End-to-end diffusion conditioned only on global features reduces error relative to pure regression, but remains inferior to the two-branch fusion design. This suggests that the architecture’s novelty lies not only in using diffusion, but in how the conditioning signal is structured at the node level.

4. Cross-frame tracking and full inference pipeline

Tracking is formulated with the previous estimate as a geometric prior. Given prior $X_t \in \mathbb{R}^{N \times 3},$4 and current cloud $X_t \in \mathbb{R}^{N \times 3},$5, the method first extracts point features

$X_t \in \mathbb{R}^{N \times 3},$6

then performs KNN aggregation around each prior node to obtain

$X_t \in \mathbb{R}^{N \times 3},$7

After multi-head self-attention,

$X_t \in \mathbb{R}^{N \times 3},$8

the diffusion backbone predicts the inter-frame motion $X_t \in \mathbb{R}^{N \times 3},$9. Sampling and training mirror the single-frame formulation, but the conditioning now embeds temporal continuity through the previous chain (Lv et al., 19 Dec 2025).

The pipeline includes explicit pre-processing, inference logic, and post-processing. Endpoint normalization translates so that $\hat Y_t$0, rotates so that $\hat Y_t$1 lies on the $\hat Y_t$2 axis, then rotates about $\hat Y_t$3 so the approximate midpoint lies in the $\hat Y_t$4–$\hat Y_t$5 plane; the result is centered to zero mean and scaled by the maximum side length of the bounding box. At $\hat Y_t$6, or after tracking failure, the system runs single-frame estimation; otherwise it normalizes $\hat Y_t$7 and $\hat Y_t$8, runs KNN aggregation and diffusion tracking, and then de-normalizes. Known endpoint positions can be appended and a low-smoothness B-spline fitted to ensure exact endpoint alignment and smooth node redistribution. Tracking failure is detected when the maximum per-node displacement exceeds a threshold, at which point the method reinitializes with single-frame estimation.

UniStateDLO is also evaluated as the front-end module of a closed-loop DLO manipulation system. The reported integration is dual-arm shape control with multiple obstacles, in which real-time feedback enables an MPC controller to avoid collisions and reach target shapes despite severe occlusions of endpoints and mid-sections. This system-level result indicates that the tracking component is intended to provide not only frame-wise accuracy but also sufficiently stable temporal behavior for feedback control in complex 3-D environments.

5. Data generation, training protocol, and empirical performance

Training is conducted solely on a synthetic dataset generated with Unity3D and Obi Rope, using two grippers that move endpoints to random targets within prescribed workspaces to generate diverse deformations. The dataset contains $\hat Y_t$9 sequences $\|\hat Y_t - Y_t\|$0 frames $\|\hat Y_t - Y_t\|$1K samples, with domain randomization over DLO length $\|\hat Y_t - Y_t\|$2–$\|\hat Y_t - Y_t\|$3, radius $\|\hat Y_t - Y_t\|$4–$\|\hat Y_t - Y_t\|$5, stiffness, camera pose, Gaussian jitter on depth, and random occlusion by masking image regions. The train/validation split is $\|\hat Y_t - Y_t\|$6 (Lv et al., 19 Dec 2025).

The model uses $\|\hat Y_t - Y_t\|$7 input points obtained by FPS and represents the DLO with $\|\hat Y_t - Y_t\|$8 nodes. PointNet++ has $\|\hat Y_t - Y_t\|$9 set-abstraction and $X_t$0 feature-propagation layers with $X_t$1. The voting branch uses radius $X_t$2 and $X_t$3. Diffusion uses $X_t$4 timesteps, cosine schedule, $X_t$5-dimensional embeddings, and $X_t$6-step DDIM at inference. Training proceeds in three stages: the two-branch network is trained for $X_t$7 epochs with Adam at $X_t$8; the branches are frozen and the diffusion fusion module is trained for $X_t$9 epochs with AdamW at $\{X_s, \hat Y_s \mid s < t\}$0; and tracking is trained end-to-end for $\{X_s, \hat Y_s \mid s < t\}$1 epochs with AdamW at $\{X_s, \hat Y_s \mid s < t\}$2 and cosine learning-rate decay.

The method is evaluated against several occlusion modes. Initial occlusion is handled by letting the fusion stage leverage global regression to complete missing shape; self-occlusion is addressed by local voting over visible segments followed by fusion-based reconnection; and multi-object occlusion is handled by KNN features that isolate local geometry despite clutter. Ablation results state that local node-wise conditioning plus local denoising with GCN gives the best performance, with MPNE $\{X_s, \hat Y_s \mid s < t\}$3 and PCN $\{X_s, \hat Y_s \mid s < t\}$4 under no occlusion, and MPNE $\{X_s, \hat Y_s \mid s < t\}$5 and PCN $\{X_s, \hat Y_s \mid s < t\}$6 at $\{X_s, \hat Y_s \mid s < t\}$7 occlusion. By contrast, end-to-end diffusion conditioned only on global features yields MPNE $\{X_s, \hat Y_s \mid s < t\}$8 under no occlusion, and fusion via MLP or registration is reported as weaker under occlusion than diffusion-based fusion.

Setting	UniStateDLO	Baseline comparison
Single-frame, no occlusion	MPNE $\{X_s, \hat Y_s \mid s < t\}$9, PCN $p(Y \mid \text{observations})$0, NSS $p(Y \mid \text{observations})$1	DLOFTBs: $p(Y \mid \text{observations})$2; Sun et al.: $p(Y \mid \text{observations})$3
Single-frame, $p(Y \mid \text{observations})$4 occlusion	MPNE $p(Y \mid \text{observations})$5, PCN $p(Y \mid \text{observations})$6	baselines $p(Y \mid \text{observations})$7
Tracking, no occlusion	MPNE $p(Y \mid \text{observations})$8, PCN $p(Y \mid \text{observations})$9, NSS $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$0	TrackDLO: $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$1; CDCPD2: $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$2
Tracking, $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$3 occlusion	MPNE $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$4, PCN $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$5	baselines $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$6

Latency is reported as approximately $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$7 for single-frame estimation and $$q(Y^k \mid Y^{k-1}) = \mathcal{N}\!\left(Y^k; \sqrt{1-\beta^k}\,Y^{k-1}, \beta^k I\right),$$8 for tracking on an RTX 4090. In real-world zero-shot transfer, the system is tested on four cables or wires with distinct stiffness and radii. The reported outcome is that single-frame estimation remains accurate under real occlusion, diffusion fusion completes hidden portions robustly, and tracking remains stable over long sequences while suppressing drift and preserving smoothness where registration-based methods fail. This suggests that the synthetic training regime, together with the point-cloud and diffusion design, captures geometric regularities that transfer beyond the simulator without real-world fine-tuning.

6. Limitations, operating regime, and distinction from similarly named frameworks

The reported limitations are specific. UniStateDLO assumes DLO segmentation is available, so extension to joint segmentation and tracking in clutter is left as future work. Extremely soft or knotted behaviors, including surgical sutures and complex knots, may require explicit physical constraints or simulation. Endpoint occlusions are handled implicitly, but prolonged complete endpoint occlusion may still break normalization. The overall system complexity—two branches plus two diffusion modules—is identified as a reason to explore more compact architectures, and tight integration of uncertainty estimates with planning and control remains open (Lv et al., 19 Dec 2025).

The intended operating regime is robotic perception and manipulation of cables, ropes, or wires in 3-D environments with limited views and heavy occlusions. In that regime, the framework’s defining elements are the two-branch coarse-to-fine architecture, node-wise conditioning, graph-convolutional diffusion fusion, and unified tracking. A plausible implication is that UniStateDLO is best understood as a perception stack specialized for constrained manipulation, rather than as a generic spatiotemporal learning model.

UniStateDLO is distinct from UniSTD, which addresses unified spatiotemporal learning across ten benchmark tasks from four disciplines using a frozen ViT-Base backbone, parameter-efficient spatiotemporal adapters, a temporal attention module, and rank-adaptive Mixture-of-Experts adaptation (Tang et al., 26 Mar 2025). The two systems share a “unified” design philosophy, but they target different problem classes: UniStateDLO focuses on partial point-cloud state estimation and tracking of deformable linear objects under occlusion, whereas UniSTD focuses on general spatiotemporal prediction across traffic, trajectory prediction and robot planning, driving scene prediction, and weather forecasting.