Shape Servoing: Real-Time Control for Deformable Objects

Updated 7 July 2025

Shape servoing is a control paradigm that regulates the shape of deformable objects through closed-loop feedback.
It leverages low-dimensional feature extraction and mapping to convert sensory data into actionable control signals for soft materials.
Recent advances include both model-based and learning-based controllers that ensure stability and adapt to complex manipulation tasks.

Shape servoing is a robotic control paradigm in which a system manipulates an elastic or deformable object so that its shape, rather than merely its pose, converges to a specified goal configuration. Unlike traditional position-based servoing for rigid objects, shape servoing requires establishing a closed-loop feedback controller that continuously monitors the object’s deformation and computes appropriate actions to regulate its geometry in real time. This capability underpins a range of applications in manipulation of soft materials, surgical robotics, industrial assembly, and the emerging field of soft and swarm robotics.

1. Theoretical Principles and Modeling

At the core of shape servoing is the need to relate robot actions—typically end-effector motions—to quantifiable changes in the shape of a deformable object. This is achieved by defining a mapping from low-dimensional control features (such as the velocity or displacement of manipulated points) to relevant shape descriptors extracted from sensory data.

A standard modeling framework begins from a potential energy formulation of the deformable object’s mechanics. By expanding about equilibrium, small displacements in manipulated points (δp^m) and feedback (observation) points (δp^f) can be locally related by functions A and B, so that:

δp^m = D(δp^f), with D = A ∘ B⁻¹

To address the curse of dimensionality and measurement noise, shape information is typically reduced to a feature vector x = C(p^f). The control objective is then expressed as driving this vector towards a target x_d. This yields a feedback law such as

δp^m = H(η·(x_d – x)),

where H is the deformation-to-control mapping and η is a scalar gain. This formalism underlies the shape servoing feedback loop in contemporary robotic systems (1709.07218).

2. Feature Extraction and Shape Representation

Shape servoing relies on extracting informative, low-dimensional features from high-dimensional sensory observations (e.g., from images or point clouds). The extraction methods are tailored to the class of object and manipulation scenario:

Global features: Centroid positions, major axes, or pairwise distances between selected points (1709.07218, 1806.09618).
Surface descriptors: Surface variation indicators and extended local shape histograms (e.g., FPFH-based) capture local geometric complexity (1709.07218).
Image-based features: Histogram of Oriented Wrinkles (HOW) based on Gabor filter responses encode 2D deformation characteristics (1806.09618).
Parametric models: Regression-based curve or surface fitting (e.g., NURBS, Bézier) is used for linear/rod-like objects to reduce the object’s configuration to a set of parameters (2008.06896).
Contour moments: For composite objects, 2D moments and associated invariants (e.g., Hu’s moments, centroid, orientation) provide robust descriptors (2106.02424).
Point cloud embeddings and learned representations: Recent approaches use neural networks to produce low-dimensional embeddings that capture the 3D geometry from partial or full point clouds (2305.04449, 2110.04685, 2309.14463).

The choice of features is critical: it must balance expressiveness with tractability, and the feature space should be amenable to efficient online computation and robust to sensing noise and occlusion.

3. Feedback Control Laws and Learning-Based Controllers

Control architectures in shape servoing span from analytic, model-based approaches to learning-based, data-driven methods.

Model-based Control:

When a parametric relationship between robot action and shape change can be inferred (e.g., via an estimated or adaptive Jacobian), controllers employ feedback laws:

Δs_k = J_k Δr_k (parametric feature change) Δr_k = –Φ⁻¹ Ĵ_k e_k-1,

where J_k is the deformation Jacobian matrix (often estimated online), e_k-1 is the shape error, and Φ is a regularizer (2008.06896). Stability and convergence are often analyzed via Lyapunov theory, and adaptation laws (e.g., with Kalman filters or recursive least squares) dynamically update the Jacobian estimate (2008.06896, 2312.06340).

Learning-based Control:

For highly nonlinear or hard-to-model deformations, data-driven schemes are prevalent.

Gaussian Process Regression (GPR): The deformation function is learned online as a Gaussian process, modeling both a mean mapping and its uncertainty. Fast online GPR (FO-GPR) with incremental updates and selective forgetting ensures real-time performance (1709.07218, 1806.09618).
Imitation Learning and Random Forests: Offline-optimized tree-based controllers map observed visual features directly to control actions, and tree parameters are fine-tuned using expert demonstrations (1806.09618).
Deep Neural Networks: Networks such as DeformerNet process point clouds from current and goal shapes to produce robot motion commands, learning compact shape embeddings jointly with the control policy (2305.04449, 2110.04685).
Offline Reinforcement Learning: Goal-conditioned RL policies (e.g., TD3+BC) are learned directly from experimental data to achieve shape control under complex object dynamics, overcoming the limitations of local Jacobian-based methods (2403.10290).

The choice among these depends on application domain, the need for real-time responsiveness, and the nature of the available sensory feedback.

4. Computational Strategies for Online Control

A prevailing challenge in shape servoing is meeting real-time control requirements given the computational burden of inference and adaptation. Strategies include:

Fast Online Matrix Inversion: Block-matrix and Sherman–Morrison formulas are employed for rapid updates of the GPR covariance matrix to keep per-iteration computation between 2–5 ms (1709.07218).
Selective Forgetting: Limiting the training data set size and replacing redundant or least-informative samples avoids model overfitting and memory blowup (1709.07218, 1806.09618).
Auto-tuning and Adaptive Gains: Online adjustment of control gains using error- or cost-based criteria ensures convergence of the controller while preventing oscillations or excessive control effort (2008.06896).
Robustness to Occlusion and Noise: Adversarial neural networks and multi-resolution encoders compensate for partial observability in visual feedback, predicting the full shape even when the object is partly blocked (2205.09987).

For high-degree-of-freedom and high-dimensional observation spaces, feature dimension reduction and efficient mapping from observation to action are essential for tractable, robust control.

5. Experimental Validation and Application Domains

Shape servoing has been validated in diverse experimental setups:

Dual-arm manipulators: Tasks such as towel bending, sheet folding, plastic manipulation, and peg-in-hole on fabrics, with closed-loop timescales of 30 FPS (1709.07218, 1806.09618).
Soft arm positioning: Deep learning-based visual servoing of continuum arms using eye-in-hand cameras, robust to changes in load, lighting, and environment (2202.05200).
Non-prehensile shaping of materials: Using pushing and tapping actions to shape kinetic sand, demonstrating iterative error reduction in image-based error metrics (2001.11196).
Surgical sub-tasks: Retraction, tissue wrapping, and connecting tubular tissues, with direct closed-loop networks achieving high task success in both simulation and physical experiments (2305.04449, 2309.14463).
Swarm robots: Distributed shape actuation via collective extension and orientation of miniature robots for physical displays and tangible object manipulation (1909.03372).
Elastic rod and cable manipulation: Vision-guided, regression-based, and lattice-based frameworks for shaping linear, planar, or volumetric objects (2008.06896, 2209.01832).

Evaluations consistently report real-time or near real-time performance with robust convergence, as well as the ability to generalize across shapes, materials, and manipulation contexts.

6. Robustness, Limitations, and Future Research

Limitations and research challenges include:

Feature Engineering and Generalization: Choosing or learning shape features that faithfully represent the underlying deformation while being robust to environmental variation remains challenging, especially in high-dimensional or partially observable settings (1806.09618).
Model Dependence and Adaptation: Methods based on local models (e.g., Jacobians or GPR mappings) may fail in cases with significant nonlinearity or global shape transformations (e.g., curvature inversion). Learning-based RL and demonstration-driven controllers address some of these cases (2403.10290).
Actuation Constraints and Physical Limits: Accounting for actuator saturation, asymmetric bounds, and input non-smoothness requires specialized continuous and differentiable functions to maintain controller stability (2312.16048).
Real-World Deployments: Transfer to novel objects, integration with context-aware goal inference (e.g., in DefGoalNet for context-conditioned goal prediction from demonstrations), and handling sensor occlusions and disturbances remain active research areas (2309.14463, 2205.09987).
Planning Horizons and Multi-step Reasoning: Many systems rely on local, reactive maps from current to goal shape; future work may require multi-step planning and trajectory-level goal specifications to achieve complex maneuvers and satisfy higher-level task constraints (2309.14463, 2403.10290).

7. Representative Approaches and Comparative Summary

Approach	Feature Representation	Controller Type	Real-Time Capability	Generalization	Reference
Fast-Online GPR [FO-GPR]	Low-dim. hand-crafted, FPFH	Online learning (GPR)	Yes (2–5 ms/step)	Limited	(1709.07218, 1806.09618)
Regression/Parametric (e.g., NURBS, Arc)	Parametric model coefficients	Adaptive/Analytic (UKF/Jacobian)	Yes	Objects fit by model	(2008.06896, 2101.01889)
Sliding Mode (including finite-time)	Contour moments	Sliding mode (adaptive)	Yes	Mixed	(2106.02424, 2312.16048)
Point Cloud Neural Embedding (DeformerNet)	Learned point cloud embedding	Deep network, closed-loop	Yes	Strong (novel shapes, stiffness)	(2305.04449, 2110.04685)
Goal-conditioned RL (Offline)	ARAP or raw shape (18 points)	RL policy (TD3+BC)	Yes	Complex global shapes	(2403.10290)

Shape servoing has thus advanced from local, model-based feedback loops to data-driven controllers capable of learning robust shape manipulation policies. Continued development in generalizable learning representations, context-aware goal inference, and robust real-world adaptation remains central to the future of shape servoing research.