Surformer v2: Efficient Multimodal Surface Classification

• The paper’s main contribution is the integration of in-model feature extraction with learnable decision-level fusion to enhance surface classification.
• It employs a pretrained EfficientNetV2-S for vision and a transformer-based tactile branch processing GelSight images, fusing logits late in the pipeline.
• Empirical results indicate a 30× reduction in inference latency with only a 2% accuracy drop, making it ideal for real-time robotic applications.

Surformer v2 is a multimodal classifier for surface understanding from touch and vision that integrates visual and tactile sensory streams through a late, decision-level fusion mechanism. It is presented as an enhanced successor to the earlier Surformer v1 framework, replacing v1’s handcrafted feature extraction plus mid-level fusion with in-model feature extraction for vision and a learnable weighted fusion of modality-specific logits. The system is evaluated on the Touch and Go dataset, a benchmark of synchronized first-person RGB images and GelSight tactile images spanning five surface classes, and is positioned for real-time robotic applications because it combines competitive surface classification performance with very low reported inference latency (Kansana et al., 4 Sep 2025).

1. Architectural definition

The Surformer v2 pipeline comprises four stages: input acquisition, two parallel feature extractors, decision-level fusion, and final softmax classification (Kansana et al., 4 Sep 2025). The inputs are a vision image, specified as an RGB image of size $224 \times 224$, and a tactile image, specified as an RGB GelSight close-up. Each modality is processed independently before fusion.

The vision branch is described as “Efficient V-Net,” implemented as an EfficientNetV2-S backbone pretrained on ImageNet followed by a custom multilayer perceptron head. This branch outputs vision logits $z_v \in \mathbb{R}^{N_{\text{classes}}}$. The tactile branch is described as an encoder-only transformer pathway that operates on handcrafted tactile features and outputs tactile logits $z_t \in \mathbb{R}^{N_{\text{classes}}}$ (Kansana et al., 4 Sep 2025).

A notable architectural distinction relative to Surformer v1 is the change in fusion level. Surformer v1 used handcrafted feature extraction followed by a mid-level fusion architecture with multi-head cross-attention layers, whereas Surformer v2 performs late fusion by combining classifier outputs rather than intermediate feature maps. The manuscript explicitly frames this as a move from feature-map fusion to decision-level fusion. This suggests that Surformer v2 prioritizes modular modality-specific optimization over explicit cross-modal interaction inside the latent representation (Kansana et al., 4 Sep 2025).

2. Modality-specific branches

The vision branch accepts a tensor of shape $B \times 3 \times 224 \times 224$. Its EfficientNetV2-S backbone begins with an initial $3 \times 3$ convolution with stride 2, producing a $112 \times 112$ feature map with approximately 24 channels. It then applies a sequence of MBConv and Fused-MBConv blocks with Squeeze-and-Excitation, using primarily $3 \times 3$ kernels, with some $5 \times 5$ kernels in early blocks. The stage configuration is reported as following Tan and Le (2021), with examples including Stage 1 as $2 \times$ MBConv1 with $k=3$, $z_v \in \mathbb{R}^{N_{\text{classes}}}$0; Stage 2 as $z_v \in \mathbb{R}^{N_{\text{classes}}}$1 MBConv4 with $z_v \in \mathbb{R}^{N_{\text{classes}}}$2, $z_v \in \mathbb{R}^{N_{\text{classes}}}$3; and Stage 3 as $z_v \in \mathbb{R}^{N_{\text{classes}}}$4 MBConv4 with $z_v \in \mathbb{R}^{N_{\text{classes}}}$5, $z_v \in \mathbb{R}^{N_{\text{classes}}}$6, continuing up to a final $z_v \in \mathbb{R}^{N_{\text{classes}}}$7 convolution yielding 1280 channels at $z_v \in \mathbb{R}^{N_{\text{classes}}}$8 spatial resolution. Activations are Swish (SiLU) throughout, followed by global average pooling to a 1280-dimensional feature vector (Kansana et al., 4 Sep 2025).

Fine-tuning in the vision branch is selective: all but the final 20 parameters of the backbone are frozen, so lower layers retain ImageNet-derived filters. The classification head is a compact MLP with dropout $z_v \in \mathbb{R}^{N_{\text{classes}}}$9, a linear layer from 1280 to 256, ReLU, another dropout $z_t \in \mathbb{R}^{N_{\text{classes}}}$0, and a final linear layer from 256 to $z_t \in \mathbb{R}^{N_{\text{classes}}}$1 (Kansana et al., 4 Sep 2025).

The tactile branch begins with preprocessing and handcrafted feature extraction. Each RGB tactile image is converted to grayscale using

$z_t \in \mathbb{R}^{N_{\text{classes}}}$2

From the grayscale image, seven handcrafted features are computed, described as including texture histograms and edge statistics. Each feature is normalized to zero mean and unit variance using statistics collected from approximately 1000 training samples, yielding a 7-dimensional feature vector per image (Kansana et al., 4 Sep 2025).

This tactile feature vector is linearly projected from 7 to 64 dimensions, then augmented with sinusoidal positional encoding of length 1 token using a scheme identical to Vaswani et al. The transformer encoder has a single layer, model dimension $z_t \in \mathbb{R}^{N_{\text{classes}}}$3, and multi-head self-attention with $z_t \in \mathbb{R}^{N_{\text{classes}}}$4 heads, giving per-head key, query, and value dimension $z_t \in \mathbb{R}^{N_{\text{classes}}}$5. The feed-forward network has inner dimension 256 and uses ReLU activation; LayerNorm is applied before attention, with residual connections “as usual.” The tactile classification head consists of LayerNorm, dropout with $z_t \in \mathbb{R}^{N_{\text{classes}}}$6, and a linear layer from 64 to $z_t \in \mathbb{R}^{N_{\text{classes}}}$7 (Kansana et al., 4 Sep 2025).

Although the abstract states that Surformer v2 “integrates the feature extraction process within the model itself,” the detailed architecture specifies that the tactile branch still uses a small handcrafted-feature extractor before the transformer encoder. The paper’s own detailed exposition therefore presents Surformer v2 as end-to-end with respect to the vision branch and classifier-level fusion, but not as a raw-pixel tactile transformer (Kansana et al., 4 Sep 2025).

3. Decision-level fusion and optimization objective

The central fusion mechanism operates at the logit level. Let $z_t \in \mathbb{R}^{N_{\text{classes}}}$8 denote the logits from the vision and tactile branches. Surformer v2 learns unconstrained scalar parameters $z_t \in \mathbb{R}^{N_{\text{classes}}}$9 and $B \times 3 \times 224 \times 224$0, normalizes them through a softmax to obtain non-negative fusion weights $B \times 3 \times 224 \times 224$1 and $B \times 3 \times 224 \times 224$2 that sum to 1, and then computes fused logits via a weighted sum: $B \times 3 \times 224 \times 224$3

$B \times 3 \times 224 \times 224$4

The final class probabilities are obtained by applying softmax to $B \times 3 \times 224 \times 224$5 (Kansana et al., 4 Sep 2025).

The fusion parameters are registered as model parameters and learned end-to-end. No additional weight-decay term or sparsity penalty is applied to the fusion weights beyond what the optimizer imposes. The paper states that auxiliary per-modality losses help prevent collapse to a single modality (Kansana et al., 4 Sep 2025).

Training uses a composite multi-objective cross-entropy loss: $B \times 3 \times 224 \times 224$6 where each $B \times 3 \times 224 \times 224$7 is standard CrossEntropy. The stated role of the auxiliary terms is to ensure that each branch remains discriminative on its own (Kansana et al., 4 Sep 2025).

This formulation distinguishes Surformer v2 from joint-representation approaches such as early fusion and cross-attention-based mid-fusion. Rather than encouraging representational entanglement, it learns a convex combination over final decision variables. A plausible implication is that the design reduces synchronization and co-adaptation pressures between modalities, which may help explain the reported latency behavior, although the paper does not present a formal decomposition of latency sources (Kansana et al., 4 Sep 2025).

4. Training protocol and data handling

Optimization uses Adam with $B \times 3 \times 224 \times 224$8, $B \times 3 \times 224 \times 224$9, and $3 \times 3$0. The learning rate is modality-specific: $3 \times 3$1 for the pretrained vision branch, $3 \times 3$2 for the tactile branch trained from scratch, and $3 \times 3$3 for the fusion weights. Learning-rate scheduling uses ReduceLROnPlateau on validation loss with patience 5 epochs and factor 0.5 (Kansana et al., 4 Sep 2025).

Dropout is set to $3 \times 3$4 in both classification heads. Weight decay is not stated in the paper. Batch size and number of epochs are also not specified (Kansana et al., 4 Sep 2025).

Vision images are resized to $3 \times 3$5 and normalized using ImageNet mean and standard deviation. Tactile inputs undergo grayscale conversion and handcrafted feature normalization as described above. No other augmentations, such as flips or crops, are mentioned (Kansana et al., 4 Sep 2025).

The evaluation dataset is the Touch and Go benchmark. It contains five surface classes: Concrete, Wood, Brick, Synthetic Fabric, and Grass. The dataset comprises approximately 200 samples per class, with vision-tactile pairs and an 80/20 stratified train/test split. The modalities are synchronized first-person RGB images and GelSight tactile images (Kansana et al., 4 Sep 2025).

5. Empirical results and comparative positioning

The reported test-set precision, recall, and $3 \times 3$6 scores place Surformer v2 below the listed multimodal baselines in absolute classification quality but well within the range considered operationally strong for the benchmark. The paper reports the following results (Kansana et al., 4 Sep 2025):

Model	Precision	Recall	$3 \times 3$7
Surformer v1 (mid-fusion)	99	99	99
Multimodal CNN (late CNN-fusion)	100	100	100
Multimodal CNN (early fusion)	100	100	100
Surformer v2 (decision-fusion)	97	97	97

A second comparison reports accuracy, inference time, and parameter count (Kansana et al., 4 Sep 2025):

Model	Accuracy (%)	Inference Time (ms/sample)	Parameters
Surformer v1	99.40	0.7271	0.673 M
Multimodal CNN	100.00	5.0737	48.311 M
Multimodal CNN (early fusion)	100.00	4.2674	24.100 M
Surformer v2	97.40	0.0239	20.660 M

The paper’s ablation-style comparison emphasizes fusion level. Mid-level fusion in Surformer v1 combines hand-crafted tactile features and PCA vision features via cross-attention and achieves $3 \times 3$8 with inference time $3 \times 3$9 ms. Early fusion through a 6-channel CNN yields $112 \times 112$0 but is “very slow” at roughly 4–5 ms. Decision-level fusion in Surformer v2, by contrast, yields $112 \times 112$1 and the fastest inference time at $112 \times 112$2 ms (Kansana et al., 4 Sep 2025).

The manuscript explicitly identifies a “significant latency reduction ($112 \times 112$3 vs. v1) at only a 2% drop in accuracy.” It also states that in scenarios where one modality is noisy or missing, the learnable fusion weights adapt by shifting emphasis dynamically, although per-class weight traces are not plotted. Because the standalone branch accuracies are not tabulated, the precise contribution of each modality to the fused performance is not numerically decomposed (Kansana et al., 4 Sep 2025).

A common misconception would be to assume that the late-fusion redesign improves classification accuracy over earlier Surformer variants. The reported results do not support that interpretation: on the Touch and Go benchmark, Surformer v2 is less accurate than Surformer v1 and both listed multimodal CNN baselines, while substantially faster (Kansana et al., 4 Sep 2025).

6. Real-time deployment, limitations, and research significance

Surformer v2 is presented as suitable for real-time robotics. Its reported total parameter count is approximately 20.66 million, broken down as roughly 20.0 million for the vision branch, roughly 0.6 million for the tactile branch, and the remainder for the fusion layer and overall system accounting. FLOPs are not reported, although the paper notes that EfficientNetV2-S at $112 \times 112$4 is approximately 8.8 GFLOPs (Kansana et al., 4 Sep 2025).

The reported inference speed is $112 \times 112$5 ms per sample, measured on a target GPU whose exact hardware is not detailed. The paper translates this to approximately 42 k FPS and regards this as suitable for real-time robotic use. It also notes that including feature extraction inside the model allows Surformer v2 to account for end-to-end latency, and that selective fine-tuning plus late fusion keeps adaptation lightweight and robust when ported to resource-constrained platforms (Kansana et al., 4 Sep 2025).

Several limitations are explicit. Transformer depth is not varied: the tactile encoder uses a single layer, and no deeper-versus-shallower comparison is reported. Weight decay, batch size, and number of epochs are unspecified. Single-modality baselines are discussed only indirectly through the auxiliary-loss rationale; their standalone accuracies are not tabulated. Per-class or per-instance analyses of learned fusion weights are likewise absent (Kansana et al., 4 Sep 2025).

Within the immediate Surformer lineage, Surformer v2 is best understood not as an accuracy-maximizing redesign but as a latency-oriented reformulation of multimodal surface classification. Its principal contribution is the combination of an EfficientNetV2-S vision expert, a compact encoder-only transformer for tactile cues, and a learnable weighted logit fusion mechanism. The reported evidence indicates that this design trades some benchmark accuracy for very large gains in inference speed. This suggests a deployment-oriented operating point in which decision-level fusion is preferred when runtime constraints dominate, whereas early or mid-level fusion may remain preferable when maximizing classification performance on the same benchmark is the primary objective (Kansana et al., 4 Sep 2025).