Wearing A Coat: Dual-Arm Robot-Assisted Dressing with Differentiable Clothing Simulation

Abstract: The development of assistive robots for dressing tasks serves to augment human convenience and improve the quality of life for individuals with physical impairments. However, due to the intricate contact interactions between garments and the human limbs during dressing, most robot-assisted dressing algorithms treat clothing as an assembly of discrete segments, thereby struggling to manage the partial worn garments under contact constraints. To overcome this challenge, we propose a novel robotic dressing control algorithm that integrates realtime differentiable clothing simulation. The simulation algorithm employs explicit iterative scheme with intentionally introduced higher-order perturbations to enhance computational efficiency while maintaining stability under large time-step conditions. Through simulation, we resolve the garment state under contact constraints, which then enables a multi-phase control strategy for successful coat dressing assistance. To further improve real-time performance, we introduce a constrained local model along with its corresponding optimization solver, permitting high-frequency local compensation for the differentiable simulation based global controller. Finally, we experimentally validate our approach through both simulated and physical dressing scenarios, conclusively demonstrating its feasibility and efficacy

• The paper introduces a stable, differentiable cloth simulator with frictional contact that remains real-time across large integration steps and runs about 3–10× faster than Projective Dynamics for draping at 0.01 seconds.
• The paper combines multi-stage dual-arm MPC with a constrained-DDP local compensator running at 10 Hz to coordinate sleeve coupling, avoid infeasible local minima, and reduce deformation forces during dressing.
• The paper demonstrates complete two-sleeve coat dressing across standing and sitting poses and passive or active human cooperation, while noting limitations including cloth self-collision, numerical damping, and cooperative-user testing.

Problem and motivation

Robot-assisted dressing of upper-body garments such as coats remains largely restricted to partial dressing: most prior systems don only the first sleeve and treat armholes as independent, ignoring the tight physical coupling that a donned sleeve imposes on the rest of the garment (Erickson et al., 2017), [Kapusta et al.], [Zhu et al.]. When a user's arm span exceeds the garment's shoulder width, trajectory-tracking formulations converge to infeasible local minima. The paper addresses this by coupling a real-time differentiable clothing simulation with a multi-stage dual-arm MPC strategy, so that the garment state under contact constraints is resolved online and both sleeves can be donned sequentially. The system targets the full task: driving two sleeve states, parameterized by a cylindrical "dressing coordinate" $(s_i, l_i, \theta_i)$ per arm (progress from hand to shoulder), to $s_i = 2$ for both arms.

Differentiable clothing simulation

The core algorithmic contribution is an explicit iterative integrator that remains stable under large time steps. Starting from a second-order Taylor expansion of position and velocity updates, the authors add distinct high-order bias (compensation) terms for position and velocity, constructed from the internal-force gradient $$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$ and the state increments $\Delta \boldsymbol{x}, \Delta \boldsymbol{v}$. The resulting increments are obtained by solving a linear system with matrix $\boldsymbol{I} - h^2 \boldsymbol{F}_x$, which the authors show is equivalent to a semi-definite augmentation of the mass matrix. As $h$ and stiffness grow, the increments shrink toward the equilibrium state with zero force, providing unconditional stabilization. The bias terms are $o(h^3)$, so accuracy matches second-order Taylor expansion, and the system is solved with a fixed positive definite preconditioner $$\boldsymbol{P} = \boldsymbol{M} + h^2 \sum_i w_i \boldsymbol{A}_i^T \boldsymbol{A}_i$$ (Cholesky-factorizable), with convergence proven via a Rayleigh quotient argument. In practice a single iteration per step suffices.

Frictional contact is handled through a Signorini-Coulomb formulation evaluated on mean per-step velocities. The authors first classify contact nodes (take-off, stick, slip) from a sticking-force hypothesis, then compute a feasible velocity $\boldsymbol{u}_f$ and force $\boldsymbol{r}_f$ that satisfy the contact constraints; tangential direction constraints on slipping nodes are ignored because forces and velocities are process means, and post-collision-detection projection corrects detaching contact points. Backward differentiation is derived in two regimes: without contact, gradients are propagated through a coupled linear map $s_i = 2$0 with transposed iterative solves matching the forward iteration count; under contact, Jacobians of the closed-form contact update are computed analytically, including derivatives of the contact force $s_i = 2$1.

Multi-stage control with local compensation

The controller is an MPC built on the differentiable simulator, with the task decomposed into stages per sleeve: (1) pull the armhole behind the hand with the sleeve direction facing the user, (2) slide the sleeve up the arm, plus a garment-body pose adjustment between sleeves. When armholes are geodesically farther than a threshold $s_i = 2$2, the sleeves are handled sequentially: the dressed sleeve is guided toward the other hand along the arm (cost $s_i = 2$3) while the free sleeve is donned, avoiding the stuck-between-arms local optimum. Safety is handled in the global problem via an actuator-force cost $s_i = 2$4, chosen because contact-area gradients are unavailable.

Because the global nonlinear MPC takes several seconds per solve, a linearized local model with state feasibility constraints provides high-frequency compensation. The local model maps end-effector velocities to sleeve-state velocities, with cross-coupling weights $s_i = 2$5 that interpolate between single- and dual-sleeve influence regimes, and enforces distance limits ($s_i = 2$6) plus tangential position/velocity bounds on dressed sleeves, which replace the force cost in the fast loop. The constrained local problem is solved by an extension of constrained DDP [Xie et al.]: because some fabric constraints are independent of the control input or exceed its dimension, the authors propagate null-space feasibility conditions forward through the horizon (via the matrix $s_i = 2$7), combine them into the backward-pass KKT system, and update the active set using decomposed slack variables $s_i = 2$8, with a line-search factor for infeasible iterates. The final control law is $s_i = 2$9, with local solves running at 10 Hz.

Perception

Garment reconstruction builds on GarmentNets [Chi and Song], with a precomputed canonical-space distance field $$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$0 added to the loss to correct the inconsistency of separable per-axis probability heatmaps. Sleeves are segmented in canonical space; armhole position and orientation come from cross-section keypoints and PCA. Tracking fuses deformation-flow-updated visible nodes with model predictions. Human perception uses RTMPose 2D keypoints lifted with depth, matched against motion-predicted priors to fill occlusions. Intent is classified as passive (compliant following) or active (independent motion) by an attention discriminator, and a Leapfrog diffusion model predicts arm trajectories from 20-step histories and sleeve states. Training data comprise 11,000 GarmentNets samples (CLOTH3D garments in Assistive Gym, point clouds only to reduce sim-to-real gap) and 2,000 sequences per intent class generated with the local model.

Simulation validation

Benchmarked against Projective Dynamics on a 6,121-node coat for draping and table-placement, the method's per-step cost is essentially independent of step size, whereas PD's cost grows with $$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$1:

Task / method	0.001 s	0.005 s	0.01 s	0.05 s	0.1 s
PD, draping	0.110 s	0.194 s	0.348 s	1.394 s	2.321 s
Ours, draping	0.036 s	0.033 s	0.034 s	0.026 s	0.029 s
PD, placement	0.110 s	0.184 s	0.421 s	2.938 s	5.249 s
Ours, placement	0.135 s	0.215 s	0.198 s	0.319 s	0.620 s

At $$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$2 s the method runs roughly 3–10× faster than PD in draping and remains real-time, while PD exceeds real time beyond $$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$3 s. The trade-off is explicit: the artificial damping from the high-order bias terms accelerates energy dissipation, so trajectories diverge from fine-grained PD simulation (peaking near $$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$4 s) before reconverging as PD itself decays; the authors argue this matches natural dissipation but concede unsuitability for highly dynamic scenarios. Contact update cost grows with contact-node count — full contact incurs cubic complexity in $$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$5 — though in dressing only sleeve nodes contact the body, so it does not dominate. A single-sleeve dressing experiment using the analytic gradients in a proportional control law tracked a target sliding from hand to shoulder with minimal steady-state error, validating the backward pass end-to-end.

Physical experiments

The platform uses two 7-DoF Franka arms grasping the coat shoulders, two Gemini 335 RGBD cameras, and a mesh simplified to ~1,800 nodes at 5 cm resolution. Dummy experiments with manually annotated keypoints (RTMPose was inaccurate on the dummy, so CoTracker served as a surrogate — a substitution that weakens the perception pipeline's validation) showed the expected three-phase progression and motivated an ablation: the local-only variant fails to don the second sleeve because it cannot capture sleeve-opening orientation changes; the global-only variant matches the full method on a stationary target but at lower control frequency; the full global-plus-local controller achieves faster sleeve-state convergence and, via the force cost, lower peak deformation forces.

Human trials covered two coats, standing and sitting poses, and both passive and active cooperation. In active trials, when the subject shifted stance at roughly 15 s after the first sleeve was donned, the system tracked arm positions through the intent discriminator and diffusion predictor and completed the second sleeve. These results substantiate the central claim: full dual-sleeve coat dressing is achievable with contact-aware simulation-based control, without garment-specific prior models.

Limitations and open questions

The authors identify two concrete limitations: the method does not handle cloth self-collision, and the intentionally introduced high-order bias causes pronounced energy dissipation, restricting applicability to quasi-static manipulation. Additional caveats bear on the evaluation: the contact solve's cubic complexity in contact-node count is acceptable only because dressing involves few contact nodes; the human experiments involve a cooperative, able-bodied participant with arms naturally opened, and the dummy trials bypass the learned pose-estimation pipeline; and the local model's fixed influence weights and thresholds ($$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$6) are garment- and task-specific. Whether the framework extends to garments with more complex topology, to non-cooperative users, and to contact-rich scenarios where the $$\boldsymbol{F}_x = \boldsymbol{M}^{-1}\nabla \boldsymbol{f}_{int}$$7 contact solve dominates, remains open.

Conclusion

The paper contributes a stable explicit differentiable cloth simulator with dry friction, a multi-stage bimanual dressing strategy that exploits simulated contact constraints to don both coat sleeves, and a constrained-DDP local compensator enabling 10 Hz control on top of a slower global MPC. Quantitative comparisons show step-size-independent per-step cost and real-time performance where Projective Dynamics does not, and physical trials demonstrate complete dressing across poses and human intent patterns, at the cost of numerical damping and absent self-collision handling.