HapticLLaMA: Multimodal Haptic Captioning

• HapticLLaMA is a multimodal model that translates vibration signals into natural language captions conditioned on perceptual categories.
• It employs dual tokenization—frequency-based and EnCodec-based—to convert continuous haptic data into discrete sequences for LLaMA integration.
• Using a two-stage training pipeline with LoRA and RLHF, the model enhances caption alignment with human perceptual evaluations for applications in VR, accessibility, and rehabilitation.

Searching arXiv for the HapticLLaMA paper and closely related haptic-language works to ground the article in current literature. Searching arXiv for HapticLLaMA. HapticLLaMA is a multimodal sensory LLM for haptic captioning that interprets vibration signals into natural-language descriptions conditioned on a perceptual category—sensory, emotional, or associative (Hu et al., 8 Aug 2025). It formalizes haptic captioning as the task: given a vibration signal $S$ and a category $$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$, generate a caption that describes the corresponding perceptual aspect of the haptic experience (Hu et al., 8 Aug 2025). Within the emerging literature on language-mediated haptics, HapticLLaMA is notable for combining discrete haptic tokenization, LLaMA-based caption generation, and reinforcement learning from human feedback (RLHF) in a single pipeline, thereby treating vibrotactile signals as first-class multimodal inputs rather than as auxiliary metadata (Hu et al., 8 Aug 2025).

1. Conceptual scope and task formalization

HapticLLaMA addresses haptic captioning, defined as generating natural-language descriptions from haptic signals such as vibrations for use in virtual reality, accessibility, and rehabilitation applications (Hu et al., 8 Aug 2025). The model conditions generation on one of three perceptual categories. In the sensory category, captions describe physical attributes of touch, including intensity, continuity, and pulse structure. In the emotional category, captions describe affective impressions such as calm, anxious, or urgent. In the associative category, captions describe familiar real-world experiences such as the buzzing of a bee or a ringing phone (Hu et al., 8 Aug 2025).

The input signal is represented as a continuous time series $x(t)$, but tokenization is performed offline so that continuous vibrations can be converted into discrete sequences consumable by LLaMA (Hu et al., 8 Aug 2025). This design places HapticLLaMA at the intersection of signal discretization and autoregressive language modeling: the haptic signal is not directly captioned as a raw waveform, but rather through a learned symbolic interface that allows vibration perception to be expressed in a language-model-native format.

This formulation distinguishes HapticLLaMA from adjacent systems that use LLMs to generate or render haptics rather than describe them. Scene2Hap, for example, is positioned as an LLM-centered system that automatically designs object-level vibrotactile feedback for entire VR scenes and estimates vibration propagation across objects in the scene (Jingu et al., 28 Apr 2025). By contrast, HapticLLaMA operates in the reverse direction: from vibrotactile signal to language (Hu et al., 8 Aug 2025). This suggests a broader bidirectional research program in which LLMs may both synthesize and interpret haptic content.

2. Haptic tokenization and signal representation

A central contribution of HapticLLaMA is the introduction of two haptic tokenizers: a frequency-based tokenizer and an EnCodec-based tokenizer (Hu et al., 8 Aug 2025). Both convert continuous vibrations into sequences of discrete units, enabling direct integration with the LLaMA model.

The frequency-based tokenizer is motivated by the observation that vibrotactile signals are effectively described by spectral content, with human frequency discrimination approximately logarithmic across the relevant range (Hu et al., 8 Aug 2025). The signal is analyzed in the spectral domain using the $N$-point discrete Fourier transform:

$X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}.$

The paper also gives the general short-time form,

$$X[k,m] = \sum_{n=0}^{N-1} x[n] w[n-mH] e^{-j 2\pi kn/N},$$

while emphasizing FFT-based processing (Hu et al., 8 Aug 2025). Frequencies are partitioned into logarithmically spaced bins aligned to haptic just-noticeable differences, amplitudes are normalized and quantized into discrete levels, and tokens are formed as unique frequency–amplitude pairs such as FREQ_i_AMP_j (Hu et al., 8 Aug 2025). For each signal, contiguous bins with non-negligible energy yield a token sequence. This tokenizer has a vocabulary size of 278 tokens, with sequence length average 47.5, minimum 12, and maximum 52 (Hu et al., 8 Aug 2025).

The EnCodec-based tokenizer instead uses the facebook/encodec_24khz neural audio codec (Hu et al., 8 Aug 2025). The waveform is passed through an encoder, quantizer, and decoder; the quantizer applies residual vector quantization to produce code indices that serve directly as haptic tokens (Hu et al., 8 Aug 2025). In this representation, all signals are encoded as a fixed-length sequence of quantized codes. The tokenizer has a vocabulary size of 1,024 tokens and sequence length fixed at 1,379 tokens per signal (Hu et al., 8 Aug 2025).

The two tokenizers imply different trade-offs.

Tokenizer	Vocabulary size	Sequence length
Frequency-based	278	average 47.5, min 12, max 52
EnCodec-based	1,024	fixed at 1,379

EnCodec tokens outperform frequency tokens on captioning, presumably due to richer temporal and rhythmic capture, whereas the frequency tokenizer is more compact and therefore more suitable for lightweight or potentially real-time pipelines, though with slightly inferior captioning performance (Hu et al., 8 Aug 2025). Because tokenization is performed offline in both cases, latency and bitrate are not explicitly reported (Hu et al., 8 Aug 2025).

3. Model architecture and training pipeline

HapticLLaMA uses LLaMA 3.2–3B as its backbone, chosen due to compute constraints (Hu et al., 8 Aug 2025). Haptic tokens are added to the LLaMA tokenizer vocabulary as special tokens whose embeddings are learned (Hu et al., 8 Aug 2025). The prompt format is:

$$I = \text{“haptic signal: <haptic tokens>, its <category> description is: <caption>.”}$$

During supervised training, the ground-truth caption is appended; at inference, it is omitted (Hu et al., 8 Aug 2025). Category conditioning is provided explicitly within the prompt through the text “sensory,” “emotional,” or “associative” (Hu et al., 8 Aug 2025).

Supervised fine-tuning is implemented with LoRA-based adaptation (Hu et al., 8 Aug 2025). LoRA is applied to the Query and Value projection layers in the Transformer, with parameterization

$$W_*^{lora} = W_* + \Delta W_*, \quad \Delta W_* = B_*A_*,$$

for $* \in \{Q,V\}$, where $A_* \in \mathbb{R}^{r \times d}$ and $$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$0, and only $$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$1 and $$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$2 are trained (Hu et al., 8 Aug 2025). The supervised objective is the generative cross-entropy loss

$$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$3

Training details reported are batch size 4, learning rate $$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$4, and NVIDIA RTX A100/H100 GPUs; haptic token embeddings are updated jointly with LoRA parameters (Hu et al., 8 Aug 2025).

A second training stage performs RLHF via Direct Preference Optimization rather than PPO (Hu et al., 8 Aug 2025). In Stage 1, captions are produced on VibRate signals in all three categories. Human ratings are then collected on a 1–7 Likert scale according to clarity and semantic accuracy of the caption and alignment between vibration experience and caption (Hu et al., 8 Aug 2025). Preference pairs are created by thresholding at 3.5: ratings above 3.5 form the positive set and ratings at or below 3.5 form the negative set, with each positive caption paired with all negatives for the same signal and category (Hu et al., 8 Aug 2025). The DPO objective is reported as

$$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$5

Implementation uses HuggingFace TRL DPOTrainer with default settings (Hu et al., 8 Aug 2025).

This two-stage design is important because HapticLLaMA is not trained only to maximize reference overlap under captioning metrics; it is also trained to shift generation toward human-perceived alignment with the vibration signal (Hu et al., 8 Aug 2025).

4. Datasets, supervision, and evaluation protocol

Two datasets are central to the training pipeline: HapticCap for supervised fine-tuning and VibRate for RLHF (Hu et al., 8 Aug 2025). HapticCap contains 92,070 haptic–text pairs with 8–10 human-written captions per vibration across sensory, emotional, and associative categories (Hu et al., 8 Aug 2025). Its splits are Train: 73,923, Valid: 8,031, and Test: 10,116, with 24,641, 2,677, and 3,372 examples per category respectively (Hu et al., 8 Aug 2025).

VibRate contains 16,896 tuples of <vibration, caption, rating> over 704 unique vibrations from four sources: parameterized signals (174), signals derived from sound effects via timing mimic or low-pass filtering (180), HapticGen-generated signals (176), and custom signals via operations like time reversal, repetition, and mixing (174) (Hu et al., 8 Aug 2025). Ratings were collected from 44 evaluators, each of whom rated captions for 32 vibrations (Hu et al., 8 Aug 2025).

Evaluation uses both automatic metrics and human judgments (Hu et al., 8 Aug 2025). Automatic metrics are BLEU-1, BLEU-4, ROUGE-L, and METEOR, with each prediction compared to multiple references and mean and standard deviation reported (Hu et al., 8 Aug 2025). The BLEU definition is explicitly given as

$$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$6

typically with $$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$7 and $$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$8 (Hu et al., 8 Aug 2025). Human evaluation uses a 1–7 Likert scale on semantic clarity and perceptual alignment (Hu et al., 8 Aug 2025).

The paper also evaluates category-specific behavior and ablations. Emotional captions achieve the highest BLEU-4, with EnCodec approximately 41.2, while sensory captions receive the highest average human ratings (Hu et al., 8 Aug 2025). The paper states that automated metrics and human judgments diverge, which is consistent with prior captioning literature (Hu et al., 8 Aug 2025). This is a significant methodological point: caption overlap and perceptual correctness are related but non-identical targets.

5. Empirical performance and ablation findings

The main reported results are strongest for the EnCodec variant of HapticLLaMA with LoRA and RLHF (Hu et al., 8 Aug 2025). On the test set, EnCodec HapticLLaMA achieves BLEU-4 $$c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$$9, METEOR $x(t)$0, BLEU-1 $x(t)$1, and ROUGE-L $x(t)$2 (Hu et al., 8 Aug 2025). The frequency-token variant reaches BLEU-4 $x(t)$3 and METEOR $x(t)$4 (Hu et al., 8 Aug 2025).

Ablations show that adding haptic tokens alone, without LoRA, yields BLEU-4 approximately $x(t)$5–$x(t)$6 and METEOR approximately $x(t)$7–$x(t)$8 (Hu et al., 8 Aug 2025). With LoRA but no RLHF, BLEU-4 improves to $x(t)$9 for Frequency and $N$0 for EnCodec, while METEOR improves to $N$1 and $N$2 respectively (Hu et al., 8 Aug 2025). RLHF via DPO further boosts the EnCodec model to BLEU-4 $N$3 and METEOR $N$4, corresponding to an absolute gain of $N$5 BLEU-4 over the LoRA-only setting (Hu et al., 8 Aug 2025).

Human evaluation confirms a comparable upward shift. After Stage 2, more than 61% of captions are rated above 3.5 on a 7-point scale (Hu et al., 8 Aug 2025). The rating distribution shows an approximately $N$6 increase in ratings above 3.5, with increases of approximately $N$7 in the $N$8 bin and approximately $N$9 in the $X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}.$0 bin, and decreases of approximately $X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}.$1 and $X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}.$2 in the $X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}.$3 and $X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}.$4 bins respectively (Hu et al., 8 Aug 2025). Statistical significance is not reported (Hu et al., 8 Aug 2025).

The baseline comparison reinforces that language fluency alone is insufficient. Random and signal-agnostic LLaMA/GPT-4.5 baselines perform poorly to modestly; GPT-4.5, for example, is reported at BLEU-4 4.28 and METEOR 28.20 (Hu et al., 8 Aug 2025). The intended interpretation is not simply that HapticLLaMA is a stronger captioning model, but that it exhibits haptic grounding rather than generic text generation.

6. Relation to adjacent haptic-language systems

HapticLLaMA belongs to a broader 2025–2026 line of work in which LLMs mediate haptic reasoning, generation, or action. Its nearest conceptual neighbors differ mainly in directionality and embodiment.

Scene2Hap uses a multimodal LLM to estimate object semantics and physical context in VR scenes, then generates or retrieves audio signals and converts them to vibrotactile signals, while also estimating propagation and attenuation of vibration signals across objects in the scene (Jingu et al., 28 Apr 2025). In that system, language-mediated semantics support haptic synthesis and rendering. HapticLLaMA instead interprets already-existing haptic signals into text (Hu et al., 8 Aug 2025). The two systems can therefore be viewed as inverse mappings over related representational spaces.

“Touched by ChatGPT: Using an LLM to Drive Affective Tactile Interaction” explores LLM-generated 10-second vibration patterns corresponding to 10 emotions and 6 touch gestures using a wearable sleeve integrated with a $X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}.$5 grid of vibration motors (Ren et al., 13 Jan 2025). That work shows that an LLM can generate emotional haptic data and that people are accurate at recognising intended emotions above chance (Ren et al., 13 Jan 2025). HapticLLaMA differs in that its output is language rather than motor-level vibrotactile control, but both systems share an underlying premise: textual semantic structure can be coupled to tactile perception (Hu et al., 8 Aug 2025, Ren et al., 13 Jan 2025).

Other contemporaneous systems broaden the modality set. VLH is a Vision-Language-Haptics Foundation Model that outputs a 7-dimensional action vector $X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}.$6 from dual-view vision and language, thereby making haptics a direct generative output of multimodal reasoning (Fuentes et al., 2 Aug 2025). HapticVLA instead argues that tactile-aware manipulation can be learned offline and deployed without direct tactile sensing at inference, achieving a mean success rate of 86.7% on real-world contact-rich tasks (Gubernatorov et al., 16 Mar 2026). These works extend language–haptics coupling from captioning into control and manipulation, whereas HapticLLaMA remains focused on semantic interpretation of vibration signals (Hu et al., 8 Aug 2025).

A plausible implication is that HapticLLaMA occupies the “perception-to-language” corner of a larger design space whose other corners include “language-to-haptics,” “vision/language-to-haptic action,” and “tactile-distilled manipulation.”

7. Applications, limitations, and open questions

The paper identifies applications in virtual reality, accessibility, and rehabilitation (Hu et al., 8 Aug 2025). For VR, the model may support tactile realism and guidance; for accessibility, it may support navigation aids using vibrotactile feedback; for rehabilitation, it may support wearable vibrotactile stimulation and training simulators (Hu et al., 8 Aug 2025). Because HapticLLaMA generates category-conditioned captions rather than control signals, its most immediate role is interpretive: documentation, indexing, explanation, and semantic mediation of vibrotactile content.

Several limitations are explicit. Evaluation metrics such as BLEU, ROUGE, and METEOR do not fully capture perceptual alignment, while human evaluation is costly and limited in scale (Hu et al., 8 Aug 2025). The data are focused on vibrations; other haptic modalities such as force feedback and temperature are not modeled (Hu et al., 8 Aug 2025). Device variability, temporal alignment, real-world latency, and actuator heterogeneity are not addressed, and tokenization is offline (Hu et al., 8 Aug 2025). The paper further states that performance is not yet sufficient for deployment, citing BLEU-4 approximately 32.06, average human rating approximately 4.8/7, and approximately 61% of captions above 3.5 as reasons for caution (Hu et al., 8 Aug 2025).

Error modes are also described. Automated n-gram metrics may underspecify perceptual alignment, the model may overgeneralize associations or miss subtle temporal patterns, and these issues are especially visible with the compact frequency tokenizer (Hu et al., 8 Aug 2025). EnCodec handles complex, rhythmic, or intermittent patterns better (Hu et al., 8 Aug 2025). This suggests that future work may depend as much on better haptic representation learning as on scaling the language backbone.

Within the broader literature, HapticLLaMA can be read as evidence that LLMs can process and adapt to sensory data beyond vision and audio (Hu et al., 8 Aug 2025). It does not demonstrate universal haptic understanding, nor does it establish a general multimodal foundation model for touch. What it does establish is narrower but technically consequential: with discrete haptic tokenization, parameter-efficient adaptation, and preference-based alignment, a LLaMA-family model can produce perceptually conditioned natural-language descriptions of vibrotactile signals at nontrivial quality (Hu et al., 8 Aug 2025).