Audio-Driven Pseudo-Haptics

• Audio-driven pseudo-haptics is a technique that uses real-time auditory feedback mapped from user movements to simulate tactile sensations without physical haptics.
• It employs dynamic audio parameter mapping—such as pitch, amplitude, and spectral roughness—to evoke textures, resistance, and affective responses in digital environments.
• Empirical studies demonstrate that this approach can modulate user affect and force perception effectively, while lowering costs in XR and mobile applications.

Audio-driven pseudo-haptics refers to the use of auditory feedback, often dynamically modulated and temporally contingent on user action, to evoke or modulate haptic perceptions in the absence of actual physical feedback. Such approaches exploit human sensorimotor integration and cross-modal associations to induce sensations of texture, resistance, or affect on digital or extended-reality surfaces, typically where hardware haptic feedback is either infeasible or intentionally absent.

1. Principles of Audio-Driven Pseudo-Haptic Feedback

The foundational mechanism of audio-driven pseudo-haptics is the mapping of user-generated gestures (e.g., hand movement, finger stroke speed, or surface contact events) to parameters of synthesized or pre-recorded sound. Typical parameterizations include pitch ($f(t)$), amplitude ($A(t)$), spectral roughness, and grain density, with each parameter modulated by real-time kinematic variables such as velocity ($v(t)$), acceleration, or approach angle. These mappings exploit strong cross-modal correspondences, such as associating high-frequency noise with "rough" textures, or “boiling” crackle with hot touch.

Multi-modal pseudo-haptics expands this paradigm by pairing audio cues with visual (color glow, motion jitter) and proprioceptive feedback, enhancing the illusion through multi-sensory integration. Even without tangible forces, well-calibrated auditory feedback can bias perceived resistance, texture, or affective state, as established in mixed-reality and touchscreen studies (David et al., 27 Feb 2026, Martín et al., 2018, Gautam et al., 10 Oct 2025).

2. System Architectures and Signal Processing Pipelines

Audio-driven pseudo-haptic frameworks follow a closed-loop pipeline:

• Input acquisition: Spatial and kinematic hand/finger data are captured at sample rates typically ranging from 60 Hz (XR optical hand tracking) to 100 Hz (touchscreen surface).
  • Hand-Tracking: Position $p(t) \in \mathbb{R}^3$, velocity $v(t)$, optionally approximate pressure $d(t) = \max(0, -\mathbf{n} \cdot \dot{p}(t))$ (David et al., 27 Feb 2026).
  • Touchscreen: 2D position $x(t)$, velocity $v(t) = dx/dt$, with no pressure or area in baseline implementations (Martín et al., 2018).
• Feature extraction: Calculation of exploratory movement speed or gesture features (velocity, pressure proxies, direction).
• Audio parameter mapping:
  • Linear models: $f(t) = f_0 + \alpha v(t)$, $A(t) = A_0 + \beta v(t)$, $A(t)$0 for pitch, amplitude, noise gain modulation (David et al., 27 Feb 2026).
  • Granular synthesis selection: Input velocity selects among pre-segmented audio fragments representing real material rubs using a distance metric $A(t)$1 (Martín et al., 2018).
• Sound rendering: Real-time synthesis with end-to-end latencies typically constrained to $A(t)$2 ms for XR (David et al., 27 Feb 2026), but as high as 500 ms in some mobile tablet systems (Martín et al., 2018).
• Synchronization: Coupling of rendered audio (and, in some cases, concurrent visual effects) with user movement events on each frame, minimizing action-feedback latency to preserve contingency and strengthen the illusion (David et al., 27 Feb 2026).

3. Experimental Paradigms and Quantitative Outcomes

Empirical validation of audio-driven pseudo-haptics depends on psychophysical measurements, self-report scales, and objective behavioral indices:

• Affective modulation: In hand-tracked XR, “rough” and “hot” sounds produced systematic decreases in valence and increases in arousal, while “smooth” and “cold” sounds increased valence and reduced arousal (e.g., $A(t)$3 grid unit, $A(t)$4–$A(t)$5) (David et al., 27 Feb 2026).
• Force modulation: On tablet tasks, higher-pitch or broader-band rolling sounds induced greater finger pressure when interacting with virtual terrains: 0.4055 N (low band), 0.8077 N (mid band), 0.9022 N (high band) in audio-only conditions, with illusion strength scaling logarithmically with frequency band (Gautam et al., 10 Oct 2025).
• Immersion versus quality: Material sonification via dynamic granular synthesis increased user immersion/interaction time (mean: DA = 66.5 s vs. VI = 38.7 s) but did not improve subjective ratings of material qualities (agreement $A(t)$6 and classification accuracy unchanged vs. visual-only) (Martín et al., 2018).
• Multisensory integration: Simultaneous presentation of audio and visual cues can enhance or reduce illusion strength, contingent on semantic congruence; incongruent cues increased cognitive load and felt “weird” (David et al., 27 Feb 2026).

Study	Modality Tested	Main Quantitative Findings
(David et al., 27 Feb 2026)	A, V, AV (XR)	Arousal modulated by audio (ANOVA $A(t)$7, $A(t)$8); valence by color cues
(Gautam et al., 10 Oct 2025)	A, V, AV (Tablet)	Mean finger force rises with audio frequency (e.g., 0.41–0.90 N); significant main effect, $A(t)$9
(Martín et al., 2018)	SA, DA, VI, FM	Longer interaction in DA than VI/SA ($v(t)$0); no effect on material ratings

4. Audio Parameterization and Sound Design Strategies

Effective audio-driven pseudo-haptic cues are distinguished by the following synthesis and mapping principles:

• Spectral shaping: High-frequency, broad-band rolling sounds maximize perceived “roughness” and resistance. Looped rolling sounds with superimposed collision bursts (50–150 ms) communicate terrain discontinuities and enhance the illusion of stick–slip or increased force (Gautam et al., 10 Oct 2025).
• Movement–sound coupling: Dynamic modulation of tone pitch, amplitude, and noise grain density by instantaneous movement velocity provides fine-grained auditory contingency, supporting responsive user feedback (David et al., 27 Feb 2026).
• Texture-specific mapping: Distinct textures (smooth, rough, hot, cold) are encoded using specific oscillators (sine, triangle waves), granular noise, and sonification (e.g., “boiling” for hot, crystalline chime for cold), with audio feedback amplitude/gain modulated by stroke velocity (David et al., 27 Feb 2026).
• Granular synthesis with user-dependent selection: Velocity-contingent nearest-neighbor search among pre-annotated grains (pre-recorded rubs) ensures continuous, gesture-matched audio output on touchscreen systems (Martín et al., 2018).

5. Psychophysical and Cognitive Mechanisms

The effectiveness of audio-driven pseudo-haptics is mediated by several interactional and perceptual mechanisms:

• Affective substitution: Audio pseudo-haptics reliably shift users’ affective state (arousal/valence) without producing sustained tactile or thermal sensations; users interpret cues semantically or emotionally (“scratchy noise is irritating”) rather than physically (“I feel roughness”) (David et al., 27 Feb 2026).
• Force modulation illusion: Auditory cues can cause users to increase muscular effort when higher frequencies or “collisions” are played, with a normalized “illusion strength” quantified as $v(t)$1, $v(t)$2 (Gautam et al., 10 Oct 2025).
• Multisensory integration: Bayesian cue combination frameworks (MLE) predict a weighted averaging of force thresholds when congruent visual and auditory information is presented; in practice, combined cues can reduce the energy required to cross perceptual boundaries compared to either cue alone (Gautam et al., 10 Oct 2025).
• Engagement vs. realism: While interactive audio significantly boosts user engagement and time-on-task, it does not contribute measurably to the accuracy of perceived material quality unless paired with richer gesture or pressure mappings (Martín et al., 2018).

6. Limitations and Technical Constraints

Several technical and methodological limitations constrain the utility of audio-driven pseudo-haptics:

• Latency: Perceptually compelling pseudo-haptic illusions require end-to-end latencies below 30–50 ms; higher latencies (e.g., 500 ms in mobile granular synthesis) impair immediacy and weaken the illusion (Martín et al., 2018).
• Gesture diversity: Systems focusing solely on “rubbing” limit the richness of haptic cues; incorporating gestures like tapping, pinching, or scrunching, and leveraging additional contact features (pressure $v(t)$3, contact area) could improve realism (Martín et al., 2018).
• Acoustic overlap: Materials with similar acoustic signatures (e.g., leathers/fabrics) may be difficult to distinguish by rubbing sonification alone (Martín et al., 2018).
• Cognitive demand: Audio-inclusive conditions increase auditory resource demand (MReQ mean = 4.1 vs. 1.8 for visual-only); misaligned multimodal cues further elevate cognitive load (David et al., 27 Feb 2026).

7. Design Guidelines and Application Horizons

Recommendations derived from empirical studies include:

1. Audio as primary arousal modulator: Variations in tone roughness or amplitude affect emotional activation without encumbering visual processing or manual control (David et al., 27 Feb 2026).
2. Semantic congruence of multimodal cues: Robust pseudo-haptic illusions depend on congruent mapping between audio, visual, and proprioceptive channels (David et al., 27 Feb 2026).
3. Real-time synchronization: Co-scheduling audio and visual feedback within a single per-frame update loop ensures temporal alignment and sustains action–feedback contingency (David et al., 27 Feb 2026).
4. Immediate iteration and affordability: Pure-software implementations using commodity hardware (tablets, XR headsets, standard sensors) allow rapid prototyping across rehabilitation, training, and assistive domains without requiring actuators (Gautam et al., 10 Oct 2025).
5. Pipeline optimization: Reducing audio processing latency (e.g., via native C++/OpenSL ES) and integrating physically based synthesis or deep learning models may further enhance the richness and realism of the pseudo-haptic channel (Martín et al., 2018).

Audio-driven pseudo-haptics, while not a surrogate for true physical feedback, represent a robust strategy for modulating affect, engagement, and perceived resistance/texture in digital and extended-reality environments (David et al., 27 Feb 2026, Gautam et al., 10 Oct 2025, Martín et al., 2018). Their efficacy is particularly pronounced in controller-free or low-cost systems where direct haptic actuators are absent, substantiating their role as an affective communication channel in next-generation XR, virtual prototyping, and human–computer interaction.