Haptic-Caption Retrieval Task

• The haptic-caption retrieval task is defined as mapping realistic 10-s vibration haptic signals to user-generated captions using supervised contrastive learning.
• The HapticCap dataset provides 92,070 haptic-text pairs from 2,736 signals, annotated across sensory, emotional, and associative categories.
• The framework employs AST for audio signal processing and T5 for text embedding, achieving robust retrieval performance with metrics like Precision@10 and mAP@10.

The haptic-caption retrieval task is a cross-modal retrieval benchmark crucial for advancing machine understanding of vibration haptic signals paired with human-written textual descriptions. Using realistic, temporally-extended signals that mirror smartphone or virtual reality vibration feedback, this task facilitates the mapping and alignment of user perceptions—sensory, emotional, and associative—to objective haptic waveforms. The framework is formalized in the HapticCap dataset, which presents the first large-scale, human-annotated collection of haptic-text pairs, serving as a rigorous ground for evaluating cross-modal alignment and similarity models (Hu et al., 17 Jul 2025).

1. Formal Task Definition

The haptic-caption retrieval task is defined over a set $C = \{ h_1, \ldots, h_n \}$ of vibration signals, each of fixed duration (10 s, $T$ samples) and $m$ channels, with each signal accompanied by multiple user-written text descriptions for each category $c \in \{s, e, a\}$ (sensory, emotional, associative).

For each haptic signal $h$ and description category $c$, the set of annotated texts $D^c = \{ d^c_1, d^c_2, \ldots \}$ defines a set of “haptic-text pairs,” with each labeled by $y = (\operatorname{id}(h), c)$. The retrieval objective is: given a query haptic signal $h^*$, rank a candidate pool of all textual descriptions to retrieve those most similar (by a learned embedding) for each category $c$ associated with $T$0. Evaluation is performed by measuring whether the original user captions for $T$1 (across categories) appear in the top $T$2 of the retrieved list (Hu et al., 17 Jul 2025).

2. HapticCap Dataset Characteristics

HapticCap operationalizes the retrieval task by providing:

• Volume: 92,070 haptic-text pairs (after “NA” cleaning) drawn from 2,736 unique vibration signals.
• Participants: 235 human annotators, each providing at least 10 descriptions per category.
• Description Categories:
  • Sensory (physical attributes, e.g., “strong, fast beats”)
  • Emotional (affective response, e.g., “makes me feel anxious”)
  • Associative (contextual association, e.g., “reminds me of a heartbeat after running”)
• Signal Provenance: Genuine vibration haptics (not tactile images), drawn from VibViz, sound-effect libraries, parametric designs, and data-driven transformations; all signals normalized to 10 s and further augmented via stretching, amplification, and noise processes, yielding a diverse set of 2,736 waveforms.
• Category Sampling: Each haptic signal is paired with at least several user captions per category.

3. Supervised Contrastive Learning Framework

The principal modeling paradigm is supervised contrastive learning, which encodes both vibration and text in a shared embedding space and applies a contrastive loss optimized for categorical alignment.

• Encoders:
  • $T$3 (e.g., AST, Wav2vec, EnCodec)
  • $T$4 (e.g., BERT, T5, Llama, Mistral)
  • Both are linearly mapped to a joint $T$5 space ($T$6).
• Supervised Contrastive Loss (cf. Khosla et al. 2020), for minibatch $T$7:

$T$8

where - $T$9 is the projection for example $m$0, - $m$1: set of batch positives (same $m$2, category), - $m$3: other batch samples, - $m$4: temperature parameter.

• Retrieval Similarity:

$m$5

with scaling $m$6.

4. Encoder Architectures and Modality Alignment

• Haptic Modality: The highest-performing encoder is the Audio Spectrogram Transformer (AST), which transforms each signal into a 128-dimensional log-Mel spectrogram, applies a Vision-Transformer architecture over spectrogram patches, and culminates in a 768-dimensional feature vector after projection.
• Text Modality: T5-base encoder, used in encoder-only mode and fine-tuned, produces a 768-dimensional embedding by mean-pooling the top-layer hidden states from the encoder stack.
• Alignment: Both AST and T5 encodings are projected through a single linear layer per modality into a shared 768-dimensional embedding space, with the supervised contrastive loss (above) encouraging modalities sharing signal id and caption category to cluster in latent space.
• Ablation Results: Encoders without audio pre-training or substituting BERT or Wav2vec/EnCodec for T5/AST show a decrease of 2–4 P@10 points in retrieval performance, indicating the advantage of the chosen architecture pairing (Hu et al., 17 Jul 2025).

5. Retrieval Metrics and Evaluation

Performance is assessed using established ranked retrieval metrics over test queries $m$7 (each a haptic signal); for each $m$8, candidate text captions are sorted by similarity.

• Recall@K: $m$9
• Precision@K: $c \in \{s, e, a\}$0
• mean Average Precision@K (mAP@K):

$c \in \{s, e, a\}$1

with $c \in \{s, e, a\}$2 if item $c \in \{s, e, a\}$3 is relevant.

• nDCG@K:

$c \in \{s, e, a\}$4

• Optional: Mean Reciprocal Rank (MRR), Median Rank.

These metrics assess the capability of models to surface relevant captions for a given haptic signal, with top-K values (e.g., $c \in \{s, e, a\}$5) being of particular interest (Hu et al., 17 Jul 2025).

6. Experimental Results and Baselines

Empirical benchmarking on the medium/high-agreement subset yields the following key performance values (for Precision@10, unless otherwise noted):

Model	P@10	R@10	mAP@10	nDCG@10
T5 + AST (overall)	16.66	19.14	30.62	0.5579
Llama + AST	16.54	19.03	—	—
T5 + AST (sensory)	16.85	—	—	—
T5 + AST (emotional)	17.30	—	—	—
T5 + AST (associative)	15.16	—	—	—

• Category-specific training increases P@10: Sensory-only (18.16), Emotional-only (17.45), Associative-only (17.01).
• Filtering low-agreement data improves all metrics by approximately 20%.
• Pretrained audio models and the T5 transformer are critical for superior performance. Training encoders from scratch or using BERT/Wav2vec/EnCodec results in 2–4 P@10 point drops.
• The associative category remains the most challenging retrieval category. This suggests richness and ambiguity in user associations compared to more direct sensory or emotional perceptions (Hu et al., 17 Jul 2025).

7. Implementation, Limitations, and Prospects

• Hardware: Single NVIDIA RTX A100 GPU employed.
• Optimization:
  • AdamW optimizer
  • Learning rate $c \in \{s, e, a\}$6
  • Temperature $c \in \{s, e, a\}$7, similarity scale $c \in \{s, e, a\}$8
  • Fine-tuning last $c \in \{s, e, a\}$9 layers of T5, $h$0 layers of AST
  • Batch size 128, 15 epochs, early stopping on validation set
  • Embedding dimension $h$1, retrieval cutoff $h$2
• Limitations:
  • Dataset annotations are monolingual (English) and geographically/culturally limited (Denmark/US).
  • Only supervised contrastive learning is explored; alternative alignment frameworks such as adversarial or generative models remain untested.
• Future Directions:
  • Extension to end-to-end haptic-to-text caption generation.
  • Exploration of domain adaptation for cross-category generalization.
  • Collection of multilingual or cross-cultural haptic descriptions, addressing potential cultural/geographic bias (Hu et al., 17 Jul 2025).

HapticCap—the dataset and its associated retrieval task—establishes a foundational benchmark for the cross-modal mapping of temporally rich vibration haptics to rich, user-generated language, and sets a supervised-contrastive pairing of T5 and AST as a strong baseline for ongoing research in multimodal haptic understanding.