Multi-Resolution Tactile Sensing (MiTaS)

• Multi-Resolution Tactile Sensing (MiTaS) is a framework that fuses frame-based tactile, event-based tactile, and RGB vision data to capture both geometric details and rapid dynamic signals.
• It employs modality-specific CNN stems and transformer-based fusion to align heterogeneous sensor inputs within synchronized time windows for robust policy conditioning.
• MiTaS significantly outperforms vision-only and basic visual-tactile approaches, achieving up to 80% task success in challenging contact-rich manipulation scenarios.

Multi-Resolution Tactile Sensing (MiTaS) is a representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks (Krohn et al., 4 Jun 2026). In the reported formulation, it fuses an RGB camera stream, a vision-based GelSight Mini sensor, and a high-frequency event-based Evetac sensor through modality-specific convolutional stems and transformer-based fusion, and uses the resulting multi-sensor representation to condition a flow-matching policy. The central premise is that frame-based tactile sensing at $25\,\mathrm{Hz}$ captures shape and deformation but misses incipient slip and high-frequency vibrations, whereas event-based tactile sensing at $200\,\mathrm{Hz}$ captures rapid contact dynamics but has no spatial acuity.

1. Definition, scope, and motivation

MiTaS stands for Multi-Resolution Tactile Sensing. Its stated goal is to fuse heterogeneous touch modalities—high-spatial-resolution frame-based tactile sensing, high-temporal-resolution event-based tactile sensing, and egocentric RGB—into a single perceptual representation that jointly captures slow, geometry-rich contacts and fast, vibration-and-slip signals. The framework is therefore not a single tactile transducer; it is a multisensory representation-and-policy stack in which tactile sensing is central but not isolated from vision.

The motivation arises from failure modes in contact-rich manipulation. Vision-only policies often fail in tight insertion or heavy occlusion. Frame-based tactile sensing provides geometric contact information but under-resolves transient dynamics, while event-based tactile sensing captures impact, slip, and jamming signatures without spatial detail. MiTaS is designed to combine these complementary regimes in five contact-rich manipulation tasks. Reported average success is $80\%$ for the full system, compared with $31\%$ for a vision-only baseline and $54\%$ for a visual-tactile baseline (Krohn et al., 4 Jun 2026).

A useful conceptual clarification is that “multi-resolution” refers here primarily to different temporal resolutions across sensing modalities, not merely to a multiscale spatial pyramid within a single encoder. The framework aligns low-rate RGB and GelSight observations with a high-rate event stream and learns a shared tokenized representation for policy conditioning.

2. Sensor stack, observation model, and temporal synchronization

At each timestep $t$, MiTaS forms an observation

$$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$

The reported inputs are: $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$

The vision input consists of two $128\times128$ RGB frames sampled at $25\,\mathrm{Hz}$. The GelSight input consists of two $200\,\mathrm{Hz}$0 tactile images at $200\,\mathrm{Hz}$1. The Evetac input is a stack of $200\,\mathrm{Hz}$2 event-frames per $200\,\mathrm{Hz}$3 window from a $200\,\mathrm{Hz}$4 event stream. All raw pixels are normalized to $200\,\mathrm{Hz}$5, except Evetac, which is normalized to $200\,\mathrm{Hz}$6.

Temporal alignment is tied to the control loop. The robot controller replans at $200\,\mathrm{Hz}$7, and each timestep collects a $200\,\mathrm{Hz}$8 sensor window: two frames from vision, two frames from GelSight, and sixteen event-frames from Evetac. This synchronization step is central to MiTaS, because the fusion model is trained on time-aligned windows rather than on asynchronously processed modalities.

The hardware platform comprises a $200\,\mathrm{Hz}$9-DoF Franka Panda, a $80\%$0-finger gripper, a wrist RealSense camera, and finger-mounted GelSight Mini and Evetac sensors. Control is Cartesian impedance at $80\%$1, with replanning at $80\%$2 (Krohn et al., 4 Jun 2026).

3. Representation network and multimodal fusion

MiTaS uses three architectural components: modality-specific convolutional stems, learned positional and modality encodings, and a shared transformer-fusion module (Krohn et al., 4 Jun 2026).

For vision and GelSight, the stems are $80\%$3D CNNs with four sequential strided $80\%$4D convolutions, each with stride $80\%$5, padding $80\%$6, and ReLU activations. Channel dimensions progressively increase to $80\%$7. The final feature maps are reshaped into token grids: vision yields a $80\%$8 grid with $80\%$9 tokens of width $31\%$0, while GelSight yields a $31\%$1 grid with $31\%$2 tokens of width $31\%$3.

For Evetac, the stem is a $31\%$4D CNN over time and space. It uses four $31\%$5D convolutions with temporal kernels $31\%$6; the first three layers use stride $31\%$7. A final $31\%$8D convolution with kernel $31\%$9 collapses the temporal axis and produces a $54\%$0 grid of $54\%$1 tokens.

Each stem yields a flat sequence $54\%$2. MiTaS augments these features as

$54\%$3

where $54\%$4 is a learned $54\%$5D position embedding for grid cell $54\%$6, and $54\%$7 is a learned modality embedding for $54\%$8.

The augmented tokens are concatenated and processed by a standard Transformer encoder in pre-LN form. The reported configuration uses depth $54\%$9, $t$0 attention heads, embedding width $t$1, MLP ratio $t$2, and dropout $t$3. The output is the fused multisensory token set $t$4. This design makes the fusion stage modality-aware through explicit segment embeddings while remaining spatially structured through learned grid-position embeddings.

4. Flow-matching policy, action parameterization, and training

The policy predicts a chunk of $t$5 future delta-motions,

$t$6

where each action is

$t$7

Only the first $t$8 steps are executed before replanning. A notable design choice is that the policy is not given any robot state; it must infer everything from $t$9 (Krohn et al., 4 Jun 2026).

MiTaS follows a conditional flow-matching formulation. Noise is sampled as $$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$0, the data endpoint is the normalized expert chunk $$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$1, and the interpolation path is

$$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$2

The network $$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$3 is trained to regress $$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$4 under the mean-squared flow-matching loss

$$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$5

The policy head is a Diffusion-Transformer. The normalized action chunk is projected into $$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$6-dimensional tokens with chunk-position embeddings. A stack of $$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$7 DiT blocks then applies self-attention over action tokens, cross-attention from action tokens to $$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$8, a two-layer FFN, residual connections, and AdaLN-Zero time conditioning. Inference uses Euler integration with $$o_t = \bigl[\,O_t^{\mathrm{vision}},\;O_t^{\mathrm{gelsight}},\;O_t^{\mathrm{evetac}}\,\bigr].$$9 steps: $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$0 Reported execution speed on an RTX 4080 is approximately $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$1 per forward, or about $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$2 replanning.

Training uses $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$3 tele-operated demonstrations collected with a SpaceMouse, each approximately $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$4–$$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$5 timesteps. The optimizer is AdamW with $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$6, $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$7, weight decay $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$8, batch size $$O_t^{\mathrm{vision}} \in \mathbb{R}^{2\times 128\times 128},\qquad O_t^{\mathrm{gelsight}} \in \mathbb{R}^{2\times 120\times 160},\qquad O_t^{\mathrm{evetac}} \in \mathbb{R}^{16\times 120\times 160}.$$9, and gradient clip $128\times128$0. No auxiliary supervision is used; specifically, there is no explicit slip label, no contrastive loss, and no reconstruction loss. A multi-tactile co-training regime omits Evetac tokens with $128\times128$1 probability during training so that one network learns both $128\times128$2 and $128\times128$3 conditioning.

5. Tasks, baselines, and quantitative results

Evaluation uses five contact-rich tasks, each scored by binary task completion over $128\times128$4 trials (Krohn et al., 4 Jun 2026).

• Gear Assembly: insert a cylindrical gear between two fixed gears; the task requires sub-millimeter alignment and orientation.
• Board Wiping: maintain contact pressure with a sponge along a marked line on a board.
• Lamp Installation: screw a bulb into a socket in the FurnitureBench setting.
• Key in Lock: insert a metal key into a lock keyway under very tight tolerances.
• Lightbulb Connection: align a GU10 bulb’s two posts into socket slots and then twist.

The baselines are ViT-CNN, a vision-only CNN-stem plus transformer vision encoder; ViT, a standard vision transformer on RGB only; Sparsh-X in $128\times128$5 and $128\times128$6 variants; and MiTaS in $128\times128$7, $128\times128$8, and $128\times128$9 with co-training configurations. The reported results are highly asymmetric across task classes. Vision-only policies fail in tight insertions: for both ViT-CNN and ViT, Key and Bulb have $25\,\mathrm{Hz}$0 success. Averaged across tasks, ViT-CNN reaches $25\,\mathrm{Hz}$1, ViT reaches $25\,\mathrm{Hz}$2, Sparsh-X with $25\,\mathrm{Hz}$3 reaches $25\,\mathrm{Hz}$4, MiTaS with $25\,\mathrm{Hz}$5 reaches $25\,\mathrm{Hz}$6, Sparsh-X with $25\,\mathrm{Hz}$7 reaches $25\,\mathrm{Hz}$8, MiTaS with $25\,\mathrm{Hz}$9 and co-training reaches $200\,\mathrm{Hz}$00, and MiTaS with $200\,\mathrm{Hz}$01 and no co-training reaches $200\,\mathrm{Hz}$02.

The full MiTaS configuration reports per-task success rates of $200\,\mathrm{Hz}$03 on Gear Assembly, $200\,\mathrm{Hz}$04 on Board Wiping, $200\,\mathrm{Hz}$05 on Lamp Installation, $200\,\mathrm{Hz}$06 on Key in Lock, and $200\,\mathrm{Hz}$07 on Lightbulb Connection. The abstract further states that co-training a visuo-tactile model with multi-tactile data boosts performance by over $200\,\mathrm{Hz}$08 in certain tasks, without having access to the Evetac sensor during policy evaluation. In the tabulated results, the average improvement from MiTaS $200\,\mathrm{Hz}$09 to MiTaS $200\,\mathrm{Hz}$10 co-train$200\,\mathrm{Hz}$11 is $200\,\mathrm{Hz}$12, while certain task-wise changes are larger.

6. Sensor contribution analysis, interpretability, and relation to broader tactile sensing research

MiTaS includes a detailed sensor-reading and attention analysis intended to identify when different modalities dominate policy conditioning (Krohn et al., 4 Jun 2026). The reported analysis extracts the average cross-attention weight from the executed action token to each modality in $200\,\mathrm{Hz}$13 over time. In Lamp Installation, vision attention is high during the reach phase, GelSight attention spikes during screwing, and Evetac attention peaks during poking and insertion. The stated summary is modality-specific: vision dominates early coarse positioning, GelSight contributes sustained geometric contact and fine alignment, and Evetac contributes at instants of high-frequency dynamics such as impact, slip, and jamming.

The paper also defines an Evetac activation metric,

$200\,\mathrm{Hz}$14

described as the per-pixel $200\,\mathrm{Hz}$15 difference between the current Evetac base frame and a neutral gray frame. Reported trajectories show low activation during free motion, spikes at first contact, and sustained elevation during sliding or slipping. A plausible implication is that MiTaS uses event-based tactile sensing primarily as a detector of fast contact-state transitions rather than as a substitute for dense geometric touch imaging.

Within the broader tactile-sensing literature, MiTaS can be situated alongside data-driven approaches that exploit nontrivial mappings from sparse or heterogeneous tactile measurements to task-relevant latent variables. A distinct example is "Data-driven Super-resolution on a Tactile Dome" (Piacenza et al., 2018), which embeds $200\,\mathrm{Hz}$16 off-the-shelf pressure sensors in a soft hemisphere and learns a kernel ridge regressor $200\,\mathrm{Hz}$17 with Laplacian kernel $200\,\mathrm{Hz}$18 to localize contact on a curved surface. That system reports a real prototype median localization error of $200\,\mathrm{Hz}$19 over a hemisphere of approximately $200\,\mathrm{Hz}$20, using only five analog I/O channels. This suggests that MiTaS belongs to a broader shift toward learned tactile representations that trade dense handcrafted sensing layouts for data-driven inference over structured but heterogeneous signals.

The distinction between the two lines of work is substantive. The tactile dome study targets contact localization on a compliant three-dimensional surface under normal indentation, whereas MiTaS targets policy conditioning for contact-rich robotic manipulation with RGB, frame-based tactile images, and high-frequency event streams. The commonality is methodological: both rely on learned mappings to extract finer task-relevant information than would be apparent from the native sensing substrate alone.