---
title: 'Shape Servoing: Real-Time Control for Deformable Objects'
url: https://www.emergentmind.com/topics/shape-servoing
type: topic
---

# Shape Servoing: Real-Time Control for Deformable Objects

Shape servoing is a robotic control paradigm in which a system manipulates an elastic or deformable object so that its shape, rather than merely its pose, converges to a specified goal configuration. Unlike traditional position-based servoing for rigid objects, shape servoing requires establishing a closed-loop feedback controller that continuously monitors the object’s deformation and computes appropriate actions to regulate its geometry in real time. This capability underpins a range of applications in manipulation of soft materials, surgical robotics, industrial assembly, and the emerging field of soft and swarm robotics.

## 1. Theoretical Principles and Modeling

At the core of shape servoing is the need to relate robot actions—typically end-effector motions—to quantifiable changes in the shape of a deformable object. This is achieved by defining a mapping from low-dimensional control features (such as the velocity or displacement of manipulated points) to relevant shape descriptors extracted from sensory data.

A standard modeling framework begins from a potential energy formulation of the deformable object’s mechanics. By expanding about equilibrium, small displacements in manipulated points (δp^m) and feedback (observation) points (δp^f) can be locally related by functions A and B, so that:

  δp^m = D(δp^f), with D = A ∘ B⁻¹

To address the curse of dimensionality and measurement noise, shape information is typically reduced to a feature vector x = C(p^f). The control objective is then expressed as driving this vector towards a target x_d. This yields a feedback law such as

  δp^m = H(η·(x_d – x)),

where H is the deformation-to-control mapping and η is a scalar gain. This formalism underlies the shape servoing feedback loop in contemporary robotic systems [1709.07218].

## 2. Feature Extraction and Shape Representation

Shape servoing relies on extracting informative, low-dimensional features from high-dimensional sensory observations (e.g., from images or point clouds). The extraction methods are tailored to the class of object and manipulation scenario:

- **Global features:** Centroid positions, major axes, or pairwise distances between selected points [1709.07218, 1806.09618].
- **Surface descriptors:** Surface variation indicators and extended local shape histograms (e.g., FPFH-based) capture local geometric complexity [1709.07218].
- **Image-based features:** Histogram of Oriented Wrinkles (HOW) based on Gabor filter responses encode 2D deformation characteristics [1806.09618].
- **Parametric models:** Regression-based curve or surface fitting (e.g., NURBS, Bézier) is used for linear/rod-like objects to reduce the object’s configuration to a set of parameters [2008.06896].
- **Contour moments:** For composite objects, 2D moments and associated invariants (e.g., Hu’s moments, centroid, orientation) provide robust descriptors [2106.02424].
- **Point cloud embeddings and learned representations:** Recent approaches use neural networks to produce low-dimensional embeddings that capture the 3D geometry from partial or full point clouds [2305.04449, 2110.04685, 2309.14463].

The choice of features is critical: it must balance expressiveness with tractability, and the feature space should be amenable to efficient online computation and robust to sensing noise and occlusion.

## 3. Feedback Control Laws and Learning-Based Controllers

Control architectures in shape servoing span from analytic, model-based approaches to learning-based, data-driven methods.

**Model-based Control:**  
When a parametric relationship between robot action and shape change can be inferred (e.g., via an estimated or adaptive Jacobian), controllers employ feedback laws:

  Δs_k = J_k Δr_k   (parametric feature change)
  Δr_k = –Φ⁻¹ Ĵ_k e_k-1,

where J_k is the deformation Jacobian matrix (often estimated online), e_k-1 is the shape error, and Φ is a regularizer [2008.06896]. Stability and convergence are often analyzed via Lyapunov theory, and adaptation laws (e.g., with Kalman filters or recursive least squares) dynamically update the Jacobian estimate [2008.06896, 2312.06340].

**Learning-based Control:**  
For highly nonlinear or hard-to-model deformations, data-driven schemes are prevalent.  
- **Gaussian Process Regression (GPR):** The deformation function is learned online as a Gaussian process, modeling both a mean mapping and its uncertainty. Fast online GPR (FO-GPR) with incremental updates and selective forgetting ensures real-time performance [1709.07218, 1806.09618].
- **Imitation Learning and Random Forests:** Offline-optimized tree-based controllers map observed visual features directly to control actions, and tree parameters are fine-tuned using expert demonstrations [1806.09618].
- **Deep Neural Networks:** Networks such as DeformerNet process point clouds from current and goal shapes to produce robot motion commands, learning compact shape embeddings jointly with the control policy [2305.04449, 2110.04685].
- **Offline Reinforcement Learning:** Goal-conditioned RL policies (e.g., TD3+BC) are learned directly from experimental data to achieve shape control under complex object dynamics, overcoming the limitations of local Jacobian-based methods [2403.10290].

The choice among these depends on application domain, the need for real-time responsiveness, and the nature of the available sensory feedback.

## 4. Computational Strategies for Online Control

A prevailing challenge in shape servoing is meeting real-time control requirements given the computational burden of inference and adaptation. Strategies include:

- **Fast Online Matrix Inversion:** Block-matrix and Sherman–Morrison formulas are employed for rapid updates of the GPR covariance matrix to keep per-iteration computation between 2–5 ms [1709.07218].
- **Selective Forgetting:** Limiting the training data set size and replacing redundant or least-informative samples avoids model overfitting and memory blowup [1709.07218, 1806.09618].
- **Auto-tuning and Adaptive Gains:** Online adjustment of control gains using error- or cost-based criteria ensures convergence of the controller while preventing oscillations or excessive control effort [2008.06896].
- **Robustness to Occlusion and Noise:** Adversarial neural networks and multi-resolution encoders compensate for partial observability in visual feedback, predicting the full shape even when the object is partly blocked [2205.09987].

For high-degree-of-freedom and high-dimensional observation spaces, feature dimension reduction and efficient mapping from observation to action are essential for tractable, robust control.

## 5. Experimental Validation and Application Domains

Shape servoing has been validated in diverse experimental setups:

- **Dual-arm manipulators:** Tasks such as towel bending, sheet folding, plastic manipulation, and peg-in-hole on fabrics, with closed-loop timescales of 30 FPS [1709.07218, 1806.09618].
- **Soft arm positioning:** Deep learning-based visual servoing of continuum arms using eye-in-hand cameras, robust to changes in load, lighting, and environment [2202.05200].
- **Non-prehensile shaping of materials:** Using pushing and tapping actions to shape kinetic sand, demonstrating iterative error reduction in image-based error metrics [2001.11196].
- **Surgical sub-tasks:** Retraction, tissue wrapping, and connecting tubular tissues, with direct closed-loop networks achieving high task success in both simulation and physical experiments [2305.04449, 2309.14463].
- **Swarm robots:** Distributed shape actuation via collective extension and orientation of miniature robots for physical displays and tangible object manipulation [1909.03372].
- **Elastic rod and cable manipulation:** Vision-guided, regression-based, and lattice-based frameworks for shaping linear, planar, or volumetric objects [2008.06896, 2209.01832].

Evaluations consistently report real-time or near real-time performance with robust convergence, as well as the ability to generalize across shapes, materials, and manipulation contexts.

## 6. Robustness, Limitations, and Future Research

Limitations and research challenges include:

- **Feature Engineering and Generalization:** Choosing or learning shape features that faithfully represent the underlying deformation while being robust to environmental variation remains challenging, especially in high-dimensional or partially observable settings [1806.09618].
- **Model Dependence and Adaptation:** Methods based on local models (e.g., Jacobians or GPR mappings) may fail in cases with significant nonlinearity or global shape transformations (e.g., curvature inversion). Learning-based RL and demonstration-driven controllers address some of these cases [2403.10290].
- **Actuation Constraints and Physical Limits:** Accounting for actuator saturation, asymmetric bounds, and input non-smoothness requires specialized continuous and differentiable functions to maintain controller stability [2312.16048].
- **Real-World Deployments:** Transfer to novel objects, integration with context-aware goal inference (e.g., in DefGoalNet for context-conditioned goal prediction from demonstrations), and handling sensor occlusions and disturbances remain active research areas [2309.14463, 2205.09987].
- **Planning Horizons and Multi-step Reasoning:** Many systems rely on local, reactive maps from current to goal shape; future work may require multi-step planning and trajectory-level goal specifications to achieve complex maneuvers and satisfy higher-level task constraints [2309.14463, 2403.10290].

## 7. Representative Approaches and Comparative Summary

| Approach                                    | Feature Representation          | Controller Type                 | Real-Time Capability | Generalization   | Reference          |
|:---------------------------------------------|:-------------------------------|:-------------------------------|:---------------------|:-----------------|:-------------------|
| Fast-Online GPR [FO-GPR]                     | Low-dim. hand-crafted, FPFH    | Online learning (GPR)           | Yes (2–5 ms/step)    | Limited          | [1709.07218], [1806.09618] |
| Regression/Parametric (e.g., NURBS, Arc)     | Parametric model coefficients   | Adaptive/Analytic (UKF/Jacobian)| Yes                  | Objects fit by model | [2008.06896], [2101.01889] |
| Sliding Mode (including finite-time)         | Contour moments                | Sliding mode (adaptive)         | Yes                  | Mixed            | [2106.02424], [2312.16048] |
| Point Cloud Neural Embedding (DeformerNet)   | Learned point cloud embedding  | Deep network, closed-loop       | Yes                  | Strong (novel shapes, stiffness) | [2305.04449], [2110.04685] |
| Goal-conditioned RL (Offline)                | ARAP or raw shape (18 points)  | RL policy (TD3+BC)              | Yes                  | Complex global shapes| [2403.10290]         |

Shape servoing has thus advanced from local, model-based feedback loops to data-driven controllers capable of learning robust shape manipulation policies. Continued development in generalizable learning representations, context-aware goal inference, and robust real-world adaptation remains central to the future of shape servoing research.

Source: https://www.emergentmind.com/topics/shape-servoing