Haptic Captioning: Bridging Touch and Text
- Haptic Captioning is a cross-modal process that converts visual scenes and vibration feedback into structured tactile cues for accessibility and interaction.
- Key systems like Foveated Haptic Gaze and HapticLLaMA employ both retrieval and generative methods to translate tactile signals into natural language descriptions.
- Robust evaluations using metrics such as BLEU, precision, and user studies highlight challenges in semantic accuracy and cross-device integration.
Haptic captioning is the translation of key visual scene information into structured, interpretable haptic cues so a user can understand and act within a scene without sight, and it is also the task of mapping vibration haptic signals to textual descriptions that capture how users experience the signals (Fakhri et al., 2020, Hu et al., 17 Jul 2025). In the literature represented here, the term therefore spans two closely related translational regimes: scene-to-haptics systems for accessibility and interaction, and haptics-to-language systems for retrieval or generation. A further extension appears in audio–haptic captioning for XR object selection and in mid-air Braille delivery, where haptics carries compact semantic or symbolic content rather than serving only as confirmatory feedback (Cho et al., 2024, Paneva et al., 2020).
1. Conceptual scope and task formulations
Recent work uses haptic captioning in several technically distinct but semantically aligned senses. In Foveated Haptic Gaze, it denotes the translation of visual scene content into a structured haptic representation organized around attention and action (Fakhri et al., 2020). In HapticCap and HapticLLaMA, it denotes the mapping from vibration haptic signals to language, with descriptions organized into three dimensions: sensory, emotional, and associative (Hu et al., 17 Jul 2025, Hu et al., 8 Aug 2025). In Grounding Emotional Descriptions to Electrovibration Haptic Signals, the central problem is language grounding: linking free-form sensory and emotional descriptions to signal features, which is directly relevant because a captioning system must translate between tactile signals and text (Hu et al., 2024).
| Work | Captioning target | Output form |
|---|---|---|
| Foveated Haptic Gaze (Fakhri et al., 2020) | Real-time 3D scene entities | Peripheral and foveal vibrotactile cues |
| HapticCap (Hu et al., 17 Jul 2025) | Vibration signals | Retrieved human-written captions |
| HapticLLaMA (Hu et al., 8 Aug 2025) | Vibration signals | Generated natural-language captions |
| SonoHaptics (Cho et al., 2024) | XR object attributes | Audio-haptic cursor cues |
| HaptiRead (Paneva et al., 2020) | Braille characters | Mid-air haptic information |
The sensory, emotional, and associative taxonomy has become a recurrent organizing structure in signal-to-language work. Sensory captions describe tactile qualities and physical attributes such as “rough,” “strong,” “pulse, fast,” or “regular.” Emotional captions describe affective responses such as “calm,” “energetic,” “anxious,” “uncomfortable,” or “pleasant.” Associative captions capture metaphors and real-world associations such as “alarm clock,” “car engine,” or “heartbeat” (Hu et al., 17 Jul 2025). HapticLLaMA adopts the same three-category output space, conditioning caption generation on a target category (Hu et al., 8 Aug 2025).
A central distinction is between captioning and retrieval. HapticCap explicitly separates the two: captioning generates a description for a signal, while retrieval returns the most relevant descriptions from a pool (Hu et al., 17 Jul 2025). This distinction matters because early progress in the area has been driven more by multimodal alignment and retrieval than by fully generative models, whereas HapticLLaMA moves to end-to-end generation (Hu et al., 8 Aug 2025). Taken together, these works suggest that haptic captioning is best understood as a broader cross-modal grounding problem rather than a single model family or evaluation protocol.
2. Scene-to-haptics captioning and attention-centric accessibility
Foveated Haptic Gaze operationalizes haptic captioning as an attention-centric accessibility interface for real-time 3-dimensional environments (Fakhri et al., 2020). Its core design is biologically inspired: a “foveated” channel conveys high-detail information at the user’s point of focus, while a “peripheral” channel conveys coarse but wide-field situational awareness. The user’s hand acts as a gaze proxy. A Leap Motion Controller tracks the hand in 3D, maps hand coordinates to avatar gaze coordinates, and routes identity information only when the gaze ray intersects an entity. ViZDoom provides the real-time scene and streams an entity map to a back-mounted haptic display.
The peripheral and foveal channels are explicitly partitioned. The back-mounted display provides “Haptic Peripheral Vision” by continuously captioning which entities occupy the field-of-view and their coarse locations, using pulsating spatial activations for entities and a solid vibration for current gaze position. The glove provides identity-level details on demand through distinct fingertip vibration patterns indicating whether the attended entity is a monster or an explosive barrel. A trigger button on the glove closes the action loop by firing along the current gaze direction (Fakhri et al., 2020).
This organization serves two functions documented in the paper. First, it reduces cognitive load and bandwidth by keeping identity information contingent on deliberate attention rather than presenting details for all objects simultaneously. Second, it aligns focus and peripheral cues: users “look” around by aligning the solid gaze cue with pulsating entity cues, then harvest identity through fingertip patterns. The paper explicitly frames this as avoiding overload while supporting purposeful action in an interactive FPS environment (Fakhri et al., 2020).
The study involved 5 with visual impairments and 10 sighted participants. After an introductory hand-tracking exercise and a demo room containing one monster and one barrel, participants played 7 runs of a 10-room hallway level of approximately 1.5 minutes per run, with 11 monsters and 5 barrels. The objective was to maximize
Both groups achieved workable scores without vision; sighted users scored slightly higher on average, but most visually impaired participants clustered near the middle of the sighted distribution. Accuracy was generally 70–80% for both groups in the first five runs, rising to approximately 90% for sighted participants in the last two runs. Many participants averaged below one mistake per game, and most maintained mistake ratio below $1/10$ for barrels shot relative to monsters killed, which the paper interprets as evidence that glove identity cues were clear and intuitive. The statistical presentation remains descriptive: no inferential statistics are reported (Fakhri et al., 2020).
The design implications are explicit. The system argues for partitioning information into a low-resolution, wide-field peripheral map and a high-resolution foveal channel; always showing current gaze position on the peripheral display; keeping identity patterns simple and learnable; and keeping audio sparse and confirmatory rather than dominant. The reported failure mode is also explicit: briefly appearing entities can be missed with coarse peripheral cues, suggesting a need for higher-resolution or larger-coverage back displays such as HaptWrap (Fakhri et al., 2020).
3. Language grounding, datasets, and semantic structure
The signal-to-language branch of haptic captioning depends on human-authored descriptions of vibration experience. HapticCap introduces the first large, fully human-annotated haptic-caption dataset for vibration feedback, containing 92,070 haptic–text pairs, 2,736 unique vibration signals, and 235 users (Hu et al., 17 Jul 2025). After filtering low-agreement descriptions, the medium/high-agreement subset contains 68,521 pairs across 2,709 signals. The category counts are reported separately: 32,202 sensory, 30,762 emotional, and 29,106 associative pairs in the full set, with 28,134, 25,092, and 15,295 respectively in the filtered subset.
The collection protocol is unusually labor-intensive. Signals were played through Meta Quest 3 and Meta Quest Pro controllers under controlled lab conditions in Denmark and the United States over approximately 11 months. Participants wore headphones and earplugs to mask device audio, held the controller in the non-dominant hand, experienced 16 signals in random order, and entered three free-text descriptions per signal. Each signal was annotated by at least 10 users. Signal creation began from an initial diverse set of 304 signals, including 78 from VibViz, 70 designed from sound effect libraries, 35 formed by varying amplitude, envelope frequency, carrier frequency, and rhythm, and 121 custom transformations. All signals were normalized to 10 seconds, and augmentation procedures produced 8 new signals per initial signal. Human validation on 5% of pairs confirmed that 98.78% of generated signals evoke similar haptic experiences as their originals (Hu et al., 17 Jul 2025).
Quality control is also formalized. Inter-annotator agreement was computed by embedding descriptions with T5 and measuring cosine similarity across annotators per signal-category. Descriptions with average similarity below 0.5 were filtered out in the medium/high-agreement subset. The paper reports diversity analyses across emotional categories and lexical distributions, as well as t-SNE visualizations of T5-encoded captions showing distinct but overlapping clusters for the three dimensions (Hu et al., 17 Jul 2025).
Grounding Emotional Descriptions to Electrovibration Haptic Signals addresses a smaller but more mechanistic question: how free-form language can be linked to electrovibration signal features (Hu et al., 2024). In a lab study with 12 users and 32 electrovibration signals displayed on a 3M MicroTouch electrostatic screen, participants verbally described sensory, emotional, and associative attributes while listening to white noise and sliding an index finger across the surface. The computational pipeline comprises keyword extraction using GPT-3.5 Turbo, semantic clustering with static word embeddings and agglomerative hierarchical clustering, and Pearson correlation between cluster counts and seven normalized signal features: Mean Amplitude, RMS, Pulse Count, Standard Deviation of Pulse Distance, Zero Count, Mean Onset Strength, and Spectral Centroid.
The keyword extraction comparison is concrete: rule-based extraction yielded , , ; POS-based extraction yielded , , ; and GPT-3.5 Turbo with fine-tuning yielded , 0, 1 (Hu et al., 2024). Using ConceptNet Numberbatch for final analysis, the authors obtained 14 positive clusters and 16 negative clusters. The strongest reported feature–concept links include RMS with “urgent,” Zero Count with “rhythmical,” Spectral Centroid with “aggressive,” and Pulse Count and Mean Onset Strength with lower constancy or steadiness. The paper characterizes these correlations as weak to moderate and does not report exact 2 values, 3-values, or multiple-comparison corrections. This suggests a viable but still early-stage grounding layer for future captioning systems.
4. Retrieval models and generative sensory LLMs
Haptic-caption retrieval in HapticCap is defined as follows: given a haptic signal 4, retrieve relevant textual descriptions 5 for category 6 from a corpus of human-written captions (Hu et al., 17 Jul 2025). The system learns encoders 7 for haptics and 8 for text, projects them into a shared 9-dimensional space, and scores similarity by scaled dot product with a softmax:
$1/10$0
with $1/10$1. Training uses supervised contrastive learning with labels composed of haptic ID and category so that multiple users’ captions for the same signal-category form one class. The framework evaluates Precision@10, Recall@10, mean Average Precision@10, and normalized Discounted Cumulative Gain@10.
The best overall result in combined-category training is reported for T5 + AST on the medium/high-agreement subset: $1/10$2, $1/10$3, $1/10$4, and $1/10$5 (Hu et al., 17 Jul 2025). The second-best overall model is Llama + AST with $1/10$6, $1/10$7, $1/10$8, and $1/10$9. Filtering low-agreement descriptions improves performance on average around 20% across metrics. Emotional captions yield higher performance than sensory, with associative lowest. AST is the strongest haptic encoder among those tested, and BERT lags T5 and Llama. When trained separately per category, the paper reports that T5 + AST consistently outperforms other combinations across metrics. Pretraining a haptic encoder from scratch underperforms relative to fine-tuning pretrained audio encoders; the paper gives the example that 0 drops from 16.66 to approximately 14.2 for T5+AST when trained from scratch (Hu et al., 17 Jul 2025).
HapticLLaMA extends the problem from retrieval to end-to-end generation (Hu et al., 8 Aug 2025). It formalizes haptic captioning as generation of a natural-language caption 1 from a vibration signal 2, first tokenized into a discrete sequence 3 by either a frequency-based tokenizer or an EnCodec-based tokenizer. The model then conditions a LLaMA-3.2-3B decoder on the haptic tokens and target category, defining
4
Stage 1 uses supervised fine-tuning with LoRA inserted into the query and value projections; Stage 2 uses reinforcement learning from human feedback implemented with direct preference optimization. The multimodal prompt is simple concatenation rather than cross-attention: “haptic signal: <haptic tokens>, its <category> description is: <caption>.”
The tokenization trade-off is explicit. The frequency tokenizer has vocabulary size 278, average sequence length 47.5, and min/max lengths 12 and 52; it is compact and spectral but underrepresents fine-grained temporal and rhythmic variation. The EnCodec tokenizer uses the pretrained EnCodec-24kHz audio codec, has vocabulary size 1,024, and fixed sequence length 1,379; it captures richer temporal detail at greater computational cost (Hu et al., 8 Aug 2025).
The main reported scores favor the EnCodec variant with LoRA and RLHF: BLEU-4 of 5, METEOR of 6, BLEU-1 of 7, and ROUGE-L of 8 (Hu et al., 8 Aug 2025). The frequency-based version reaches BLEU-4 of 9 and METEOR of 0. RLHF improves BLEU-4 by +7.78 points from pre-RLHF LoRA models, and more than 61% of generated captions receive human ratings above 3.5 on a 7-point scale, with an approximately 10% increase in the share of ratings above 3.5 after RLHF. Emotional captions reach the highest BLEU-4, with the EnCodec-based model reported at approximately 41.2 for that category. The paper nevertheless notes that BLEU, ROUGE, and METEOR are suboptimal for haptic perceptual alignment and that average human rating of approximately 4.8/7 remains insufficient for real-world deployment (Hu et al., 8 Aug 2025).
5. Alternative captioning interfaces: audio–haptic XR and mid-air Braille
SonoHaptics presents an audio–haptic captioning system for gaze-based 3D object selection in XR (Cho et al., 2024). As gaze hovers over scene objects, the system generates a multimodal caption whose parametric features encode color lightness, size, position, and material. The mappings are grounded in cross-modal correspondence and a dedicated perception study: lighter colors correspond to higher pitch, larger objects correspond to stronger haptic intensity, perceived position corresponds to auditory direction, and material corresponds to characteristic impact timbres. The global mapping equations are
1
for lightness-to-pitch and
2
for size-to-vibrotactile amplitude. The system also includes a local mode that redistributes pitch and amplitude within a cluster around the last-gazed object to maximize perceptual separation in clutter.
Its implementation combines a Unity-based VR scene, Meta Quest Pro eye tracking, headset audio, and a custom wristband with four linear resonance actuators. The evaluation compares five feedback conditions in cluttered living-room scenes: No feedback, Static, Text-to-speech, Visual, and SonoHaptics. Selection time is slowest for SonoHaptics among the nonvisual conditions at 3.65 s after outlier removal, whereas visual feedback remains fastest among guided conditions. Error rate, however, shows that SonoHaptics improves on Static and No feedback, especially under clutter and for small targets: Visual 7.8%, TTS 14.4%, SonoHaptics 15.3%, No feedback 18.3%, Static 20.3% (Cho et al., 2024). The system imposes higher mental demand than No feedback and Visual, which the paper attributes to learning the mappings. This makes clear that caption richness and immediate usability remain in tension.
HaptiRead addresses a different representational regime: symbolic haptic information delivered as Braille via mid-air ultrasound (Paneva et al., 2020). Using an Ultraleap Stratos Explore development kit with 256 ultrasonic transducers and a carrier frequency of 40 kHz, the system renders 6-dot Braille cells on the palm. Three stimulation schedules are compared: Constant, Point-By-Point, and Row-by-Row. Dot spacing is 30 mm between dot centers, and each Braille position is assigned a fixed modulation frequency within the 100–200 Hz range.
The principal result is that mid-air haptics is viable for presenting Braille characters, with the highest average accuracy achieved by Point-By-Point: 94% in the preliminary study and 88% in the user study with blind participants (Paneva et al., 2020). In the blind user study, Constant reached 81% and Row-by-Row 75%. Response times were similar across methods at approximately 7.2–7.3 seconds per trial, and the differences were not significant. The paper reports that 61% of errors were single-dot misperceptions. For captioning applications, the study translates these findings into schedule-limited throughput estimates, noting that Constant or Row-by-Row with short fixed dwells could support much higher raw character rates than those observed under first-exposure recognition conditions (Paneva et al., 2020).
These two systems demonstrate that haptic captioning need not be confined to natural-language output. In XR, captions can be compact audio–haptic descriptors supporting selection and disambiguation. In mid-air Braille, caption content can be symbolically encoded for private, contactless reading. This suggests a broader design space in which semantic compression, interaction requirements, and user expertise determine whether haptic captions should be analog, categorical, or fully linguistic.
6. Evaluation challenges, misconceptions, and future directions
One recurring misconception is that haptic captioning is synonymous with simple vibration alerts. The cited literature shows a richer picture: Foveated Haptic Gaze partitions presence/location and identity into separate channels for purposeful action in a 3D scene; HapticCap and HapticLLaMA treat haptics as a modality to be aligned with free-form human language; SonoHaptics encodes multiple object attributes through parametric audio–haptic mappings; and HaptiRead uses structured spatial-temporal stimulation to carry symbolic text (Fakhri et al., 2020, Hu et al., 17 Jul 2025, Cho et al., 2024, Paneva et al., 2020).
A second misconception is that captioning and retrieval are interchangeable. HapticCap explicitly distinguishes them, and the current literature reflects that distinction in methodology and maturity (Hu et al., 17 Jul 2025). Retrieval has benefited from shared-embedding formulations and established IR metrics, whereas generative captioning remains more difficult because haptic experience is ambiguous, multimodal, and highly variable across annotators. The relatively lower performance of associative captions in retrieval and the need for 8–10 references per vibration in HapticLLaMA evaluation both illustrate this instability (Hu et al., 17 Jul 2025, Hu et al., 8 Aug 2025).
Evaluation itself remains unsettled. Foveated Haptic Gaze reports encouraging playability and aiming accuracy but no inferential statistics and no explicit latency, refresh rate, or timing budget (Fakhri et al., 2020). The electrovibration grounding study reports weak to moderate correlations but not exact 3 values, 4-values, or multiple-comparison corrections (Hu et al., 2024). HapticLLaMA reports strong BLEU and METEOR values, yet also states that standard captioning metrics are suboptimal for haptic perceptual alignment (Hu et al., 8 Aug 2025). This suggests that the field still lacks consensus on how to measure semantic adequacy, perceptual fidelity, and action support in a unified way.
The limitations are correspondingly concrete. HapticCap is English-only, collected in Denmark and the United States, and does not specify signal file formats or sampling rates (Hu et al., 17 Jul 2025). HapticLLaMA focuses on vibration signals rather than force feedback or temperature and does not analyze device-level domain shift (Hu et al., 8 Aug 2025). Foveated Haptic Gaze identifies peripheral awareness as sometimes insufficient for fleeting targets and raises the possibility of slight sighted bias (Fakhri et al., 2020). SonoHaptics identifies cognitive load, color–pitch confusability in compound mappings, and limited tactile richness from amplitude-only coding (Cho et al., 2024). HaptiRead identifies the gap between first-exposure reading speed and practical streaming rates (Paneva et al., 2020).
Future directions in the cited work are consistent. They include increasing back-display resolution and coupling scene captioning with computer vision and wearable haptic displays in Foveated Haptic Gaze; end-to-end generative haptic captioning, better multimodal alignment across categories, and expansion beyond VR controllers in HapticCap; larger datasets, haptic word embeddings, and non-linear predictive models in electrovibration grounding; richer vibrotactile textures, personalization, and dynamic-scene handling in SonoHaptics; and better haptic-aware metrics, cross-device robustness, multimodal touch sensing, and improved tokenization or conditioning in HapticLLaMA (Fakhri et al., 2020, Hu et al., 17 Jul 2025, Hu et al., 2024, Cho et al., 2024, Hu et al., 8 Aug 2025). Taken together, these works suggest that the next phase of haptic captioning will depend less on proving feasibility than on improving semantic precision, perceptual calibration, evaluation validity, and deployment across heterogeneous devices and tasks.