HapticCap: Multimodal Haptic-Text Dataset

• HapticCap is a large-scale dataset with 92,070 human-annotated haptic-text pairs covering sensory, emotional, and associative dimensions.
• It uses systematic data augmentation and quality control to ensure high fidelity and reliable evaluation of haptic-caption retrieval tasks.
• The benchmark supports evaluation of models like HapticLLaMA through metrics such as Recall@K, BLEU, and METEOR, driving advances in generative haptic captioning.

HapticCap is a large-scale, multimodal dataset and benchmark task designed for the study of haptic-text alignment and understanding of user experiences of vibration haptic signals. Addressing the lack of annotated haptic datasets and limitations of existing modeling approaches, HapticCap enables computational research into how users perceive, describe, and associate meanings with vibrations from devices such as smartphones and virtual reality controllers. The dataset comprises over 92,000 verified human-annotated pairs, covering three dimensions of user experience—sensory, emotional, and associative. It serves as both a data resource and an experimental testbed for evaluating machine learning models on the novel haptic-caption retrieval task, and underpins the development of generative haptic captioning models such as HapticLLaMA (Hu et al., 17 Jul 2025, Hu et al., 8 Aug 2025).

1. Dataset Collection and Composition

Haptic Signal Acquisition

Vibration signals were produced via Meta Quest 3 and Meta Quest Pro controllers, held in the non-dominant hand under conditions controlling for audible cues (participants wore earplugs and headphones). Each waveform, normalized to 10 seconds, was constructed from 304 "seed" vibrations obtained from the VibViz library, sound-effect banks, programmatic parameterizations, and manual transformations. Eightfold data augmentation per seed (using signal elongation, amplitude scaling, noise injection, and their combinations) yielded a total of 2,736 unique vibrations.

Annotation Protocol

A cohort of 235 English-fluent participants from diverse backgrounds provided free-form textual descriptions after perceiving tactile signals. For each vibration, a minimum of 10 unique captions were collected in three categories:

• Sensory (physical perceptual qualities)
• Emotional (evoked affect)
• Associative (links to real-world objects/events)

A "no caption" (NA) response was permitted. Compensation was provided at $15 USD/hour.

Quality Control and Dataset Statistics

Captions underwent agreement filtering: each signal-category group was vectorized using a T5 model, after which cosine similarity was computed. Captions with average similarity below 0.5 were removed, yielding a "medium/high agreement" subset that improves experimental reliability. Human raters further verified augmentation fidelity (98.8% of augmented pairs judged perceptually similar).

	Full (no NA)	Med/High Agreement
# signals	2,736	2,709
# sensory pairs	32,202	28,134
# emotion pairs	30,762	25,092
# association pairs	29,106	15,295
Total pairs	92,070	68,521

Feature representations include 1D amplitude time-series and log-Mel spectrograms (128 bands) for modeling (Hu et al., 17 Jul 2025).

2. Haptic–Caption Retrieval Task

The core benchmark task is haptic-caption retrieval: ranking candidate captions by semantic similarity to a given vibration signal. Let $\{h_1,\dots,h_n\}$denote test signals and$D=\{d_j^c\}$the candidate captions for category$c\in\{\text{sensory},\text{emotion},\text{association}\}$$. The retrieval is correct if a ground-truth <a href="https://www.emergentmind.com/topics/caption" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">caption</a> is present in the top$$K$$ranked results.</p> <h3 class='paper-heading' id='evaluation-metrics'>Evaluation Metrics</h3> <p>The benchmark employs standard information retrieval metrics:</p> <ul> <li><strong>Recall@K ($$R@K$$):</strong> Fraction of true captions retrieved in the top$$K$.</li> <li><strong>Precision@K ($P@K$$):</strong> Fraction of retrieved captions that are correct.</li> <li><strong>mean Average Precision@K (mAP@K):</strong> Mean precision over ground-truth captions at rank$$\leq K$$.</li> <li><strong>normalized Discounted Cumulative Gain@K (nDCG@K):</strong> Rank-discounted and normalized measure of relevance.</li> </ul> <p>These enable direct comparisons between model architectures and ablations (<a href="/papers/2507.13318" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Hu et al., 17 Jul 2025</a>).</p> <h2 class='paper-heading' id='modeling-approaches'>3. Modeling Approaches</h2><h3 class='paper-heading' id='encoders-and-architecture'>Encoders and Architecture</h3> <p>The haptic encoder is a fine-tuned <a href="https://www.emergentmind.com/topics/audio-spectrogram-transformer" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Audio Spectrogram Transformer</a> (<a href="https://www.emergentmind.com/topics/average-secrecy-throughput-ast" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">AST</a>) operating on log-Mel inputs. For text, the T5-base transformer is fine-tuned separately or jointly for each caption category. Both encoders output 768-dimensional embeddings, concatenated and projected via a two-layer <a href="https://www.emergentmind.com/topics/multilayer-perceptron-mlp" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">MLP</a>.</p> <h3 class='paper-heading' id='supervised-contrastive-learning'>Supervised Contrastive Learning</h3> <p>A <a href="https://www.emergentmind.com/topics/supervised-contrastive-loss-supcon-loss" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">supervised contrastive loss</a> aligns embeddings in the shared latent space. For a minibatch of$$N$$(h_i, d_i)$$pairs, positives are defined by shared (signal ID, caption category). The loss:</p> <p>$$D=\{d_j^c\}$0

where $D=\{d_j^c\}$1 is the concatenated embedding, $D=\{d_j^c\}$2 is cosine similarity, and $D=\{d_j^c\}$3 is the temperature parameter.

Training Paradigms

• Combined-category: A single model jointly learns all description types, using a category token in the label.
• Single-category: Three models trained and evaluated independently, one per caption category.

Fine-tuning proceeds for 15 epochs (batch size = 128, $D=\{d_j^c\}$4, $D=\{d_j^c\}$5), early-stopped on validation loss. Data is split 70/10/20% for train/validation/test (Hu et al., 17 Jul 2025).

4. Experimental Results and Benchmarks

Retrieval Performance

Results on the medium/high agreement subset show T5 + AST outperforms BERT + AST and Llama + AST across all IR metrics:

Model	P@10	R@10	mAP@10	nDCG@10
T5 + AST	16.7	19.1	30.6	0.558
Llama + AST	16.5	19.0	30.5	0.554
BERT + AST	13.7	16.1	27.3	0.495

• Single-category training yields higher performance, especially for emotional captions (P@10 ≈ 18.5%).
• Filtering out low-agreement captions gives ≈ 20% relative improvement in metrics.
• Fine-tuning AST from audio-pretrained weights, rather than from scratch, is critical (+10–15% P@10 improvement).
• Emotional captions are the most readily retrieved, associative the most challenging.
• Substantial domain shift is observed across caption categories: single-category models generalize poorly out-of-domain (e.g., sensory→emotion P@10 drops from 18.2% to 12.5%) (Hu et al., 17 Jul 2025).

Limitations

• Dataset is limited to English and covers only participants in Denmark and the USA.
• Annotation is time-intensive, restricting diversity of signals and annotators.
• The approach is limited to contrastive retrieval; generative captioning remains unexplored within HapticCap.

5. Extensions: Haptic Captioning Models

Subsequent work leverages HapticCap for generative haptic captioning. In particular, HapticLLaMA introduces a multimodal sensory LLM that translates vibration signals ($D=\{d_j^c\}$6) and category $D=\{d_j^c\}$7 into a conditional probability $D=\{d_j^c\}$8, where $D=\{d_j^c\}$9 is a free-form caption (Hu et al., 8 Aug 2025).

Signal Tokenization

Two discrete tokenization schemes convert continuous haptic time series for transformer input:

• Frequency-based: Windowed STFT produces tokens indexed by log-spaced frequency bands and amplitude quantization, designed to match human just-noticeable differences.
• EnCodec-based: Raw waveforms are mapped via a convolutional encoder and vector quantizer into Q-ary codebook indices, flattened into a token stream.

LLaMA Integration and LoRA Fine-Tuning

Haptic tokens are appended to the input prompt and embedded jointly with standard vocabulary in the LLaMA transformer, using sinusoidal positional encodings. Model adaptation is accomplished by LoRA—training low-rank adapters added to query and value projections in each attention block (rank $$c\in\{\text{sensory},\text{emotion},\text{association}\}$$0).

Reinforcement Learning from Human Feedback

Direct Preference Optimization (DPO) is used to incorporate human Likert-score preferences at the caption level, further aligning outputs with perceptual judgments.

Evaluation and Scores

On the HapticCap test set, EnCodec-based HapticLLaMA (LoRA + RLHF) achieves:

• BLEU-4: $$c\in\{\text{sensory},\text{emotion},\text{association}\}$$1
• METEOR: $$c\in\{\text{sensory},\text{emotion},\text{association}\}$$2
• Over 61% of captions rated >3.5/7 by humans
• RLHF yields a +10% absolute increase in high-quality caption share

Ablations confirm incremental gains from haptic tokenization (+4 BLEU-4 over signal-agnostic LLaMA), LoRA adaptation (+16 BLEU-4 and +19 METEOR), and RLHF (+7.8 BLEU-4 and +5.9 METEOR).

6. Impact, Challenges, and Future Directions

HapticCap establishes a rigorous foundation for computational haptic perception research, enabling quantitative alignment of tactile signals with human language descriptors. Its utility extends to VR/AR accessibility, real-time feedback, and multimodal LLMs. Notable challenges persist, including the need for broader linguistic and cultural representation, increased coverage of haptic stimulus space, and migration from retrieval-based modeling to fully generative and end-to-end architectures. Future work proposes:

• End-to-end generative models for haptic-to-text translation
• Cross-lingual extensions and analysis of sociocultural perception variance
• Domain-adaptive methods to mitigate cross-category generalization failures
• Integration of haptic-captioning in designer-facing feedback tools

7. Related Benchmarks and Applications

HapticCap is the first dataset to pair extensive human-annotated captions with a large, systematically-augmented library of vibration signals at scale. It has catalyzed new architectures for sensory-language learning (e.g., HapticLLaMA), supports systematic ablation and pretraining studies, and provides a robust evaluation suite for information retrieval and generative haptic captioning. Plausible implications include the application of similar methodologies to other underexplored modalities (e.g., gustatory or olfactory signals), expanding the reach of multimodal representation learning (Hu et al., 17 Jul 2025, Hu et al., 8 Aug 2025).