--- title: 'HapticLLaMA: Multimodal Haptic Captioning' url: https://www.emergentmind.com/topics/hapticllama type: topic --- # HapticLLaMA: Multimodal Haptic Captioning Searching arXiv for the HapticLLaMA paper and closely related haptic-language works to ground the article in current literature. Searching arXiv for `HapticLLaMA`. HapticLLaMA is a multimodal sensory language model for haptic captioning that interprets vibration signals into natural-language descriptions conditioned on a perceptual category—sensory, emotional, or associative [2508.06475]. It formalizes haptic captioning as the task: given a vibration signal $S$ and a category $c \in \{\text{sensory}, \text{emotional}, \text{associative}\}$, generate a caption that describes the corresponding perceptual aspect of the haptic experience [2508.06475]. Within the emerging literature on language-mediated haptics, HapticLLaMA is notable for combining discrete haptic tokenization, LLaMA-based caption generation, and reinforcement learning from human feedback (RLHF) in a single pipeline, thereby treating vibrotactile signals as first-class multimodal inputs rather than as auxiliary metadata [2508.06475]. ## 1. Conceptual scope and task formalization HapticLLaMA addresses haptic captioning, defined as generating natural-language descriptions from haptic signals such as vibrations for use in virtual reality, accessibility, and rehabilitation applications [2508.06475]. The model conditions generation on one of three perceptual categories. In the sensory category, captions describe physical attributes of touch, including intensity, continuity, and pulse structure. In the emotional category, captions describe affective impressions such as calm, anxious, or urgent. In the associative category, captions describe familiar real-world experiences such as the buzzing of a bee or a ringing phone [2508.06475]. The input signal is represented as a continuous time series $x(t)$, but tokenization is performed offline so that continuous vibrations can be converted into discrete sequences consumable by LLaMA [2508.06475]. This design places HapticLLaMA at the intersection of signal discretization and autoregressive language modeling: the haptic signal is not directly captioned as a raw waveform, but rather through a learned symbolic interface that allows vibration perception to be expressed in a language-model-native format. This formulation distinguishes HapticLLaMA from adjacent systems that use large language models to generate or render haptics rather than describe them. Scene2Hap, for example, is positioned as an LLM-centered system that automatically designs object-level vibrotactile feedback for entire VR scenes and estimates vibration propagation across objects in the scene [2504.19611]. By contrast, HapticLLaMA operates in the reverse direction: from vibrotactile signal to language [2508.06475]. This suggests a broader bidirectional research program in which language models may both synthesize and interpret haptic content. ## 2. Haptic tokenization and signal representation A central contribution of HapticLLaMA is the introduction of two haptic tokenizers: a frequency-based tokenizer and an EnCodec-based tokenizer [2508.06475]. Both convert continuous vibrations into sequences of discrete units, enabling direct integration with the LLaMA model. The frequency-based tokenizer is motivated by the observation that vibrotactile signals are effectively described by spectral content, with human frequency discrimination approximately logarithmic across the relevant range [2508.06475]. The signal is analyzed in the spectral domain using the $N$-point discrete Fourier transform: $$ X[k] = \sum_{n=0}^{N-1} x[n] e^{-j 2\pi kn/N}. $$ The paper also gives the general short-time form, $$ X[k,m] = \sum_{n=0}^{N-1} x[n] w[n-mH] e^{-j 2\pi kn/N}, $$ while emphasizing FFT-based processing [2508.06475]. Frequencies are partitioned into logarithmically spaced bins aligned to haptic just-noticeable differences, amplitudes are normalized and quantized into discrete levels, and tokens are formed as unique frequency–amplitude pairs such as `FREQ_i_AMP_j` [2508.06475]. For each signal, contiguous bins with non-negligible energy yield a token sequence. This tokenizer has a vocabulary size of 278 tokens, with sequence length average 47.5, minimum 12, and maximum 52 [2508.06475]. The EnCodec-based tokenizer instead uses the `facebook/encodec_24khz` neural audio codec [2508.06475]. The waveform is passed through an encoder, quantizer, and decoder; the quantizer applies residual vector quantization to produce code indices that serve directly as haptic tokens [2508.06475]. In this representation, all signals are encoded as a fixed-length sequence of quantized codes. The tokenizer has a vocabulary size of 1,024 tokens and sequence length fixed at 1,379 tokens per signal [2508.06475]. The two tokenizers imply different trade-offs. | Tokenizer | Vocabulary size | Sequence length | |---|---:|---:| | Frequency-based | 278 | average 47.5, min 12, max 52 | | EnCodec-based | 1,024 | fixed at 1,379 | EnCodec tokens outperform frequency tokens on captioning, presumably due to richer temporal and rhythmic capture, whereas the frequency tokenizer is more compact and therefore more suitable for lightweight or potentially real-time pipelines, though with slightly inferior captioning performance [2508.06475]. Because tokenization is performed offline in both cases, latency and bitrate are not explicitly reported [2508.06475]. ## 3. Model architecture and training pipeline HapticLLaMA uses LLaMA 3.2–3B as its backbone, chosen due to compute constraints [2508.06475]. Haptic tokens are added to the LLaMA tokenizer vocabulary as special tokens whose embeddings are learned [2508.06475]. The prompt format is: $$ I = \text{“haptic signal: , its description is: .”} $$ During supervised training, the ground-truth caption is appended; at inference, it is omitted [2508.06475]. Category conditioning is provided explicitly within the prompt through the text “sensory,” “emotional,” or “associative” [2508.06475]. Supervised fine-tuning is implemented with LoRA-based adaptation [2508.06475]. LoRA is applied to the Query and Value projection layers in the Transformer, with parameterization $$ W_*^{lora} = W_* + \Delta W_*, \quad \Delta W_* = B_*A_*, $$ for $* \in \{Q,V\}$, where $A_* \in \mathbb{R}^{r \times d}$ and $B_* \in \mathbb{R}^{d \times r}$, and only $A_*$ and $B_*$ are trained [2508.06475]. The supervised objective is the generative cross-entropy loss $$ L_{SFT} = -\sum_t \log p_\theta(y_t \mid y_{` over 704 unique vibrations from four sources: parameterized signals (174), signals derived from sound effects via timing mimic or low-pass filtering (180), HapticGen-generated signals (176), and custom signals via operations like time reversal, repetition, and mixing (174) [2508.06475]. Ratings were collected from 44 evaluators, each of whom rated captions for 32 vibrations [2508.06475]. Evaluation uses both automatic metrics and human judgments [2508.06475]. Automatic metrics are BLEU-1, BLEU-4, ROUGE-L, and METEOR, with each prediction compared to multiple references and mean and standard deviation reported [2508.06475]. The BLEU definition is explicitly given as $$ \text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^N w_n \log p_n\right), $$ typically with $N=4$ and $w_n=1/4$ [2508.06475]. Human evaluation uses a 1–7 Likert scale on semantic clarity and perceptual alignment [2508.06475]. The paper also evaluates category-specific behavior and ablations. Emotional captions achieve the highest BLEU-4, with EnCodec approximately 41.2, while sensory captions receive the highest average human ratings [2508.06475]. The paper states that automated metrics and human judgments diverge, which is consistent with prior captioning literature [2508.06475]. This is a significant methodological point: caption overlap and perceptual correctness are related but non-identical targets. ## 5. Empirical performance and ablation findings The main reported results are strongest for the EnCodec variant of HapticLLaMA with LoRA and RLHF [2508.06475]. On the test set, EnCodec HapticLLaMA achieves BLEU-4 $32.06 \pm 1.2$, METEOR $59.98 \pm 1.8$, BLEU-1 $51.36 \pm 1.3$, and ROUGE-L $49.56 \pm 1.6$ [2508.06475]. The frequency-token variant reaches BLEU-4 $30.86 \pm 1.5$ and METEOR $58.95 \pm 2.1$ [2508.06475]. Ablations show that adding haptic tokens alone, without LoRA, yields BLEU-4 approximately $7.48$–$8.61$ and METEOR approximately $35.24$–$35.26$ [2508.06475]. With LoRA but no RLHF, BLEU-4 improves to $23.16$ for Frequency and $24.28$ for EnCodec, while METEOR improves to $50.16$ and $54.03$ respectively [2508.06475]. RLHF via DPO further boosts the EnCodec model to BLEU-4 $32.06$ and METEOR $59.98$, corresponding to an absolute gain of $+7.78$ BLEU-4 over the LoRA-only setting [2508.06475]. Human evaluation confirms a comparable upward shift. After Stage 2, more than 61% of captions are rated above 3.5 on a 7-point scale [2508.06475]. The rating distribution shows an approximately $+10\%$ increase in ratings above 3.5, with increases of approximately $+3\%$ in the $[5,6)$ bin and approximately $+4\%$ in the $[6,7)$ bin, and decreases of approximately $7\%$ and $3\%$ in the $[1,2)$ and $[2,3.5)$ bins respectively [2508.06475]. Statistical significance is not reported [2508.06475]. The baseline comparison reinforces that language fluency alone is insufficient. Random and signal-agnostic LLaMA/GPT-4.5 baselines perform poorly to modestly; GPT-4.5, for example, is reported at BLEU-4 4.28 and METEOR 28.20 [2508.06475]. The intended interpretation is not simply that HapticLLaMA is a stronger captioning model, but that it exhibits haptic grounding rather than generic text generation. ## 6. Relation to adjacent haptic-language systems HapticLLaMA belongs to a broader 2025–2026 line of work in which language models mediate haptic reasoning, generation, or action. Its nearest conceptual neighbors differ mainly in directionality and embodiment. Scene2Hap uses a multimodal large language model to estimate object semantics and physical context in VR scenes, then generates or retrieves audio signals and converts them to vibrotactile signals, while also estimating propagation and attenuation of vibration signals across objects in the scene [2504.19611]. In that system, language-mediated semantics support haptic synthesis and rendering. HapticLLaMA instead interprets already-existing haptic signals into text [2508.06475]. The two systems can therefore be viewed as inverse mappings over related representational spaces. “Touched by ChatGPT: Using an LLM to Drive Affective Tactile Interaction” explores LLM-generated 10-second vibration patterns corresponding to 10 emotions and 6 touch gestures using a wearable sleeve integrated with a $5 \times 5$ grid of vibration motors [2501.07224]. That work shows that an LLM can generate emotional haptic data and that people are accurate at recognising intended emotions above chance [2501.07224]. HapticLLaMA differs in that its output is language rather than motor-level vibrotactile control, but both systems share an underlying premise: textual semantic structure can be coupled to tactile perception [2508.06475; 2501.07224]. Other contemporaneous systems broaden the modality set. VLH is a Vision-Language-Haptics Foundation Model that outputs a 7-dimensional action vector $(V_x, V_y, V_z, H_x, H_y, H_z, H_v)$ from dual-view vision and language, thereby making haptics a direct generative output of multimodal reasoning [2508.01361]. HapticVLA instead argues that tactile-aware manipulation can be learned offline and deployed without direct tactile sensing at inference, achieving a mean success rate of 86.7% on real-world contact-rich tasks [2603.15257]. These works extend language–haptics coupling from captioning into control and manipulation, whereas HapticLLaMA remains focused on semantic interpretation of vibration signals [2508.06475]. A plausible implication is that HapticLLaMA occupies the “perception-to-language” corner of a larger design space whose other corners include “language-to-haptics,” “vision/language-to-haptic action,” and “tactile-distilled manipulation.” ## 7. Applications, limitations, and open questions The paper identifies applications in virtual reality, accessibility, and rehabilitation [2508.06475]. For VR, the model may support tactile realism and guidance; for accessibility, it may support navigation aids using vibrotactile feedback; for rehabilitation, it may support wearable vibrotactile stimulation and training simulators [2508.06475]. Because HapticLLaMA generates category-conditioned captions rather than control signals, its most immediate role is interpretive: documentation, indexing, explanation, and semantic mediation of vibrotactile content. Several limitations are explicit. Evaluation metrics such as BLEU, ROUGE, and METEOR do not fully capture perceptual alignment, while human evaluation is costly and limited in scale [2508.06475]. The data are focused on vibrations; other haptic modalities such as force feedback and temperature are not modeled [2508.06475]. Device variability, temporal alignment, real-world latency, and actuator heterogeneity are not addressed, and tokenization is offline [2508.06475]. The paper further states that performance is not yet sufficient for deployment, citing BLEU-4 approximately 32.06, average human rating approximately 4.8/7, and approximately 61% of captions above 3.5 as reasons for caution [2508.06475]. Error modes are also described. Automated n-gram metrics may underspecify perceptual alignment, the model may overgeneralize associations or miss subtle temporal patterns, and these issues are especially visible with the compact frequency tokenizer [2508.06475]. EnCodec handles complex, rhythmic, or intermittent patterns better [2508.06475]. This suggests that future work may depend as much on better haptic representation learning as on scaling the language backbone. Within the broader literature, HapticLLaMA can be read as evidence that large language models can process and adapt to sensory data beyond vision and audio [2508.06475]. It does not demonstrate universal haptic understanding, nor does it establish a general multimodal foundation model for touch. What it does establish is narrower but technically consequential: with discrete haptic tokenization, parameter-efficient adaptation, and preference-based alignment, a LLaMA-family model can produce perceptually conditioned natural-language descriptions of vibrotactile signals at nontrivial quality [2508.06475]. Source: https://www.emergentmind.com/topics/hapticllama