Tactile-Reactive Multi-Agent MPC

• Tactile-Reactive Multi-Agent MPC is a control architecture combining multi-agent coordination, tactile sensing, and predictive optimization for bimanual grasps.
• It employs GelSight-based tactile image embeddings via a modified ResNet to capture contact state and enable latent dynamics prediction in MPC.
• The centralized QP formulation jointly optimizes two gripper motions to distribute load and maintain stable grasps for soft, fragile, and irregular objects.

A tactile-reactive multi-agent model predictive controller is a control architecture for contact-rich robotic manipulation in which tactile measurements are treated as part of the state used by a finite-horizon optimizer, and multiple manipulators or grippers are coordinated through shared costs and coupled dynamics. In the bimanual grasping formulation developed for collaborative grasping, two parallel-jaw grippers equipped with GelSight Mini sensors are controlled by a centralized, differentiable MPC layer that predicts tactile embeddings and gripper motion in closed loop so that soft, fragile, irregular, or elongated objects can be grasped and held stably (Giacobbe et al., 24 Sep 2025). Within the broader literature, this architecture sits between earlier single-agent tactile servoing from raw tactile images (Tian et al., 2019) and later single-agent tactile-reactive grasping with learned tactile embeddings and differentiable MPC (Xu et al., 2024).

1. Problem domain and control objective

The central problem is tactile-reactive bi-manual grasping: two grippers collaboratively grasp and hold an object while continuously reacting to tactile feedback. The motivating objects are not canonical rigid bodies but soft, fragile, irregular, or elongated objects such as a banana, a coffee bag, or a PVC pipe. In this regime, single-agent grasping is structurally disadvantaged. With one gripper there is generally one dominant contact region, so higher forces are needed to prevent slip, small pose errors or perturbations can destabilize the grasp, and torques about the grasp point become large for long or heavy objects (Giacobbe et al., 24 Sep 2025).

For compliant or delicate objects, rigid grasp planning assumptions break down. Fixed geometry, stable friction, and negligible deformation are poor approximations when the object deforms, contact area spreads, and the tactile gel may deform only modestly if the object is softer than the sensor. Excessive local pressure can damage the object or the sensor, whereas inadequate pressure leads to slip, and slip onset is subtle enough that geometric planning or force thresholds alone are insufficient. The resulting control problem is therefore not merely one of trajectory tracking; it is one of maintaining contact in a narrow regime between loss of grasp and over-compression (Giacobbe et al., 24 Sep 2025).

A tactile-reactive multi-agent MPC addresses this by combining three ingredients. Multi-agent control distributes load across two grippers and increases the spatial spread of contact points. Tactile reactivity uses dense tactile signals to infer grasp state, compliance, and incipient slip. Model predictive control predicts future contact and motion over a horizon and optimizes actions that preserve grasp stability while avoiding damage. Earlier deep tactile MPC work had already shown that tactile images can be used directly as a predictive control state, with goals specified in tactile space rather than by explicit object pose (Tian et al., 2019). The multi-agent formulation extends that logic from single-contact tactile servoing to cooperative grasp stabilization.

2. Tactile sensing, latent state, and contact dynamics

A canonical tactile-reactive multi-agent implementation uses two Robotiq 2F-series parallel-jaw grippers, each with 1 DOF in opening width, mounted on two manipulators, and each carrying a GelSight Mini tactile sensor near the fingertip. At each control step, each GelSight provides an RGB tactile image encoding surface texture and geometry through illumination patterns, local contact pressure through gel deformation magnitude, and contact coverage or partial-slip patterns. The paper explicitly distinguishes conditions such as heavy contact, partial slip, and empty sensor observations (Giacobbe et al., 24 Sep 2025).

The tactile image is mapped to a compact latent state by a modified ResNet-152. The convolutional backbone is frozen, and a two-layer MLP with ReLU activations and hidden width 128 outputs an embedding

$f^{(i)} \in \mathbb{R}^{20}, \quad i\in\{1,2\}.$

This embedding is the controller’s tactile state representation. It compresses contact location and area, normal deformation pattern as a proxy for normal force, deformation asymmetry and texture cues related to slip onset, and material differences such as foam, rubber, wood, or no contact. The learned representation is not decomposed into named physical variables inside the controller, but the training setup makes it support contact/no-contact discrimination, approach-to-slip discrimination, and implicit compliance estimation (Giacobbe et al., 24 Sep 2025).

The multi-agent state augments the tactile embeddings with mechanical variables. For each gripper,

$$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$

and the full tactile state is

$f_k = [f_k^{(1)}; f_k^{(2)}].$

The learned embedding dynamics are

$$\mathbf{f}_{k+1}^{(i)} = \mathbf{f}_{k}^{(i)} + \mathbf{A}_{f} V_{k}^{(i)} + \mathbf{C}_{f} V_{k}^{(-i)}, \quad i=1,2.$$

Here $\mathbf{A}_f$ models how a gripper’s own motion affects its tactile state, while $\mathbf{C}_f$ models cross-effects induced by object deformation and load sharing. This makes the latent dynamics explicitly cooperative: one gripper’s velocity can change the other gripper’s predicted tactile embedding because the object redistributes stress and contact geometry across both grasp points (Giacobbe et al., 24 Sep 2025).

This latent-dynamics construction is the distinctive tactile component of the controller. Instead of handcrafting analytic force or slip features, the system learns a local linear dynamics model in embedding space and then uses that latent model as the predictive state inside MPC. Single-agent LeTac-MPC had already shown that a learned embedding plus a differentiable MPC layer can regulate tactile-reactive grasping for one gripper (Xu et al., 2024); the multi-agent extension adds cross-agent tactile coupling directly into the prediction model (Giacobbe et al., 24 Sep 2025).

3. Centralized multi-agent MPC formulation

The mechanical model for each gripper is a 1D double integrator driven by acceleration input: $$\begin{bmatrix} p_{n+1}^{(i)} \ v_{n+1}^{(i)} \end{bmatrix} = A_g \begin{bmatrix} p_n^{(i)} \ v_n^{(i)} \end{bmatrix} + B_g a_n^{(i)}, \quad A_g = \begin{bmatrix} 1 & \Delta t \ 0 & 1 \end{bmatrix}, \quad B_g = \begin{bmatrix} \frac{1}{2}\Delta t^2 \ \Delta t \end{bmatrix}.$$ Equivalently,

$$p_{k+1}^{(i)} = p_k^{(i)} + \Delta t\, v_k^{(i)} + \frac{1}{2}\Delta t^2 a_k^{(i)}, \qquad v_{k+1}^{(i)} = v_k^{(i)} + \Delta t\, a_k^{(i)}.$$

The paper reports $\Delta t = 10$ in Table I (Giacobbe et al., 24 Sep 2025).

The controller is multi-agent in the strict MPC sense that both acceleration sequences are optimized jointly: $\{ a_k^{(1)}, a_k^{(2)} \}_{k=n}^{n+N-1},$ with prediction horizon $$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$0. The optimization is centralized rather than decentralized. A single QP includes the states and actions of both grippers, and the cost explicitly couples their tactile states. The stage cost is

$$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$1

and the full objective is

$$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$2

The matrix $$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$3 is a learnable quadratic penalty on tactile embeddings and includes off-diagonal blocks, so it can penalize inconsistent or unbalanced bilateral tactile states. $$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$4 penalizes velocity magnitude to encourage quasi-static contact, $$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$5 penalizes control effort, and $$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$6 weights terminal tactile stability (Giacobbe et al., 24 Sep 2025).

This architecture yields a learned tactile stability cost rather than an analytic grasp metric. Low cost corresponds to embeddings associated with stable contact near the slip threshold; high cost corresponds to no contact, excessive deformation, or slip. The formulation therefore treats grasp stability as a region in latent tactile space rather than as a closed-form frictional criterion. The optimization is cast in standard OSQP quadratic-program form and implemented as a differentiable optimization layer, making the controller trainable end-to-end with the tactile encoder (Giacobbe et al., 24 Sep 2025).

Physical constraints are imposed on opening width, velocity, and acceleration: $$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$7 These bounds prevent over-closing, excessive dynamic overshoot, and actuator saturation. The formulation does not include explicit friction-cone constraints; instead, slip avoidance and over-deformation avoidance are encoded as soft constraints through the tactile state dynamics and the learned tactile cost (Giacobbe et al., 24 Sep 2025). A common misconception is therefore that “multi-agent” here implies decentralized control. In the canonical formulation it does not: the controller is a centralized cooperative MPC with two agents, a single shared objective, and explicit cross-terms in both dynamics and cost.

4. Learning pipeline and differentiable training

The controller separates manually modeled structure from learned components. The double-integrator mechanics $$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$8, scalar weights on velocity and acceleration, and motion bounds are manually specified or tuned. The learned components are the tactile encoder, the embedding dynamics parameters $$p_k^{(i)} \text{ is opening width}, \qquad v_k^{(i)} \text{ is opening velocity},$$9 and $f_k = [f_k^{(1)}; f_k^{(2)}].$0, the tactile cost matrix $f_k = [f_k^{(1)}; f_k^{(2)}].$1, and a scalar $f_k = [f_k^{(1)}; f_k^{(2)}].$2 used to scale coupling blocks in $f_k = [f_k^{(1)}; f_k^{(2)}].$3 while retaining positive semidefiniteness (Giacobbe et al., 24 Sep 2025).

Training data are collected from real contact interactions rather than from simulation. The materials used are wood, soft gel, soft foam, hard rubber, and a no-contact baseline. Two grippers grasp a test block; one gripper gradually opens while the other remains stationary, and the process continues until the slippage opening, defined as the minimum width at which slip begins. The roles are then swapped. For each sub-trial, 25 pairs of tactile images are stored together with metadata including trial identifier, gripper openings, and slip threshold. The dataset explicitly includes both heavy contact and partial slip (Giacobbe et al., 24 Sep 2025).

The supervisory signal is not an object pose but the slip threshold. For each trial, the desired terminal opening $f_k = [f_k^{(1)}; f_k^{(2)}].$4 is the manually measured slippage opening. The model is trained so that its predicted opening trajectory ends near this threshold, which corresponds to stable grasping with minimal excess force. The training loss is

$f_k = [f_k^{(1)}; f_k^{(2)}].$5

with terminal scaling $f_k = [f_k^{(1)}; f_k^{(2)}].$6. The loss therefore emphasizes both trajectory fidelity and especially terminal accuracy at incipient slip (Giacobbe et al., 24 Sep 2025).

The forward pass is tactile image $f_k = [f_k^{(1)}; f_k^{(2)}].$7 CNN embedding $f_k = [f_k^{(1)}; f_k^{(2)}].$8 differentiable MPC layer $f_k = [f_k^{(1)}; f_k^{(2)}].$9 predicted trajectories and optimal accelerations. Gradients are backpropagated through the QP layer and the encoder, yielding an end-to-end learned controller. This design inherits the differentiable-MPC logic established for single-agent LeTac-MPC (Xu et al., 2024), but replaces single-gripper latent dynamics with bilaterally coupled latent dynamics and a shared multi-agent tactile cost (Giacobbe et al., 24 Sep 2025).

The training distribution contains varied materials and explicit contact/no-contact regimes, and the evaluation includes unseen objects such as banana and PVC pipe. The paper interprets the resulting controller as generalizing across different shapes, stiffness profiles, and textures because the embeddings become material-aware and the latent MPC learns how embedding dynamics vary with compliance (Giacobbe et al., 24 Sep 2025).

5. Closed-loop execution and empirical characteristics

At runtime, the controller operates in standard receding-horizon closed loop. Current tactile images from both GelSight sensors are encoded, current opening widths and velocities are measured, the joint MPC predicts future tactile embeddings and mechanical states over the horizon, OSQP solves for the optimal acceleration sequence, and only the first control input is applied before the whole optimization is repeated at the next step. In this loop, tactile feedback enters both the state dynamics and the objective, so the controller is explicitly tactile-reactive rather than merely tactile-conditioned (Giacobbe et al., 24 Sep 2025).

The paper characterizes three main closed-loop behaviors. First, the controller predicts grasp stability because the learned embedding dynamics and cost encode which tactile states evolve toward stable contact or toward slip. Second, it detects incipient slip and unsafe forces because partial-slip or over-compression patterns move the embeddings into regions of higher cost. Third, it redistributes effort cooperatively: if one side’s tactile state deteriorates, the joint optimizer can alter both agents’ accelerations rather than correcting only the disturbed side. The paper does not provide a Lyapunov-style stability proof; the stability guarantees are described as practical and numerical, supported by a positive-semidefinite tactile cost, positive scalar penalties, and convex QP solution through OSQP (Giacobbe et al., 24 Sep 2025).

Evaluation is performed on five objects: banana, pepperoni stick, Pringles can, PVC pipe, and coffee beans bag. A trial is successful if the object is grasped bimanually and held for at least 15 seconds with no slippage and no noticeable deformation beyond acceptable limits. Baselines are an independent two-gripper extension of single-agent LeTac-MPC and a single-agent PD controller using binary depth images from tactile sensing (Giacobbe et al., 24 Sep 2025).

Object	Multi-agent MPC	Baselines
Banana	10/10	single-agent MPC 9/10; PD 7/10
Pepperoni stick	10/10	single-agent MPC 7/10; PD 6/10
Pringles can	10/10	single-agent MPC 8/10; PD 5/10
PVC pipe	10/10	single-agent MPC 10/10; PD 6/10
Coffee beans bag	9/10	single-agent MPC 7/10; PD 6/10

Trajectory plots show that the independent baselines exhibit oscillations, mismatched openings between grippers, and occasional near-instability even in successful trials. By contrast, the multi-agent controller converges to near-constant openings for both grippers, keeps the difference in openings small, and maintains the stabilized opening with minimal fluctuation. Runtime measurements indicate that adding the second agent increases runtime by only about 45% on average, despite doubling input size and matrix dimensions, which the paper attributes to parameter sharing and efficient lifted-system construction (Giacobbe et al., 24 Sep 2025).

These results delimit the functional meaning of the term tactile-reactive multi-agent MPC in practice: a controller that reacts to tactile evidence of contact quality, reasons over multiple agents in one optimization problem, and uses prediction to maintain stable cooperative grasps over soft and rigid objects alike.

6. Broader context, misconceptions, and limitations

The architecture belongs to a broader progression in tactile MPC. Deep tactile MPC introduced tactile servoing from raw GelSight-style images by learning predictive tactile image dynamics and optimizing action sequences with CEM toward a goal tactile image; its setting was single-agent, non-prehensile manipulation, and its control loop ran at about 1 Hz (Tian et al., 2019). LeTac-MPC then moved to tactile-reactive grasping with a single gripper, a learned tactile embedding, and a differentiable linear MPC layer operating at 25 Hz (Xu et al., 2024). The bi-manual collaborative controller extends that lineage by adding agent coupling in both latent dynamics and cost, and by shifting the objective from single-contact stabilization to bilateral cooperative grasp maintenance (Giacobbe et al., 24 Sep 2025).

It also differs from classical multi-agent MPC formulations surveyed in the control literature. Standard multi-agent MPC emphasizes decomposition, assignment, and cooperation among subsystems, often using analytical models and structured communication schemes (0908.1076). More recent sequence-of-constraints and graph-of-constraints formulations show how partially ordered task constraints and disturbance recovery can be embedded in reactive multi-agent MPC (Toussaint et al., 2022, Manganaris et al., 19 Mar 2026). Those frameworks are highly relevant structurally, but the tactile-reactive controller departs from them by making high-dimensional tactile sensing the primary state input and by learning latent contact dynamics directly from interaction data rather than relying primarily on analytic contact models.

A second misconception is that tactile-reactive control necessarily requires explicit force estimation or explicit friction-cone constraints. The controller described here does not. It uses dense tactile images, a learned latent space, and a learned quadratic stability cost. This suggests that in some cooperative grasping problems, latent tactile states can substitute for hand-engineered force or slip variables, although the resulting guarantees remain empirical rather than analytical. A related adjacent development is CART-MPC for assistive transferring, which coordinates multiple devices through learned deformable dynamics and a contact-aware cost, but relies on visual keypoints and linking-number estimation rather than high-resolution tactile sensing (Ye et al., 19 Jan 2025).

The principal limitations are explicit. The formulation assumes quasi-static grasping, uses only the gripper opening DOF as the controlled mechanical variable, assumes accurate metric measurement of gripper positions and velocities, and relies on one GelSight per gripper with contact near the fingertip. Cross-sensor variability is significant, computation is more expensive than a hand-crafted controller, very soft objects that collapse before producing meaningful gel deformation remain difficult, and sharp or abrasive objects risk gel damage (Giacobbe et al., 24 Sep 2025). The future directions named in the paper are transfer and continual learning, decentralized learning and imitation learning, human-in-the-loop or human-robot collaboration, and better tactile simulation or dataset expansion. This suggests that current tactile-reactive multi-agent MPC is best viewed not as a finished general theory of cooperative touch-based manipulation, but as a concrete and technically mature control pattern: dense tactile sensing, learned latent contact dynamics, centralized finite-horizon optimization, and cooperative action selection for grasping under compliance and uncertainty (Giacobbe et al., 24 Sep 2025).