AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistance

Abstract: Navigational aids for blind and low vision individuals struggle conveying dynamic real-world environments, leading to cognitive overload from continuous, undifferentiated feedback. We present AMAVA, a novel real-time video-to-audio framework that converts mobile device video into contextually relevant sound effects or text-to-speech descriptions. We propose a motion-aware pipeline using a lightweight AI classification model to distinguish between low and high-movement scenes followed by a real-time text-to-audio synthesis pipeline to enhance environmental perception more efficiently. In static environments, AMAVA generates spoken audio scene descriptions for situational awareness. In high-movement situations, it prioritizes safety by delivering sound cues, such as spoken hazard alerts and environmental sound effects. These audio outputs are produced by a decoder-only transformer-based vision-LLM with mixture-of-experts and cross-modal attention for visual understanding, in conjunction with neural text-to-speech and natural sound synthesis networks. The proposed framework uses prompt-based caching and category-specific throttling to avoid auditory clutter and minimize latency. We present a comprehensive evaluation of the system, including a real-time navigation study comparing a white cane alone versus with AMAVA, that shows a significant increase in user confidence and perceived safety.

• The paper introduces a novel V2A framework that adaptively synthesizes audio cues based on motion analysis to balance descriptive feedback and hazard alerts.
• It employs a bifurcated pipeline with an FCNN for motion classification and a transformer-based model for semantic audio synthesis, achieving a Fréchet Audio Distance of 1.71.
• User studies show enhanced perceived safety and reduced cognitive load, validating the effectiveness of adaptive multimodal feedback in real-time navigation.

AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistance

Introduction

The AMAVA framework addresses a persistent challenge in assistive technology for blind and low-vision users: the cognitive overload caused by undifferentiated and excessive real-time feedback in dynamic environments. Unlike prior video-to-audio (V2A) systems optimized for offline batch processing or static scene analysis, AMAVA integrates a real-time, motion-aware pipeline to conditionally synthesize audio cues. This approach aims to balance situational awareness and minimal auditory clutter by dynamically selecting between brief descriptive captions and urgent sound effects (SFX) or hazard warnings, adapting to the degree of visual scene motion.

Figure 1: System decision logic provides spoken descriptions in static settings and contextually relevant sound or alerts in dynamic scenes, reducing cognitive overload.

System Architecture

At the core of AMAVA is a bifurcated backend pipeline that receives a 2 Hz video stream from a browser-based mobile frontend. Frames are processed in batched pairs and classified by a lightweight fully connected neural network (FCNN) into low, medium, or high movement. The classifier consumes two features: mean absolute grayscale frame difference and average optical flow magnitude, both standardized using a pre-fit scaler. Importantly, "medium" and "high" are merged in downstream logic to trigger the dynamic audio branch, reinforcing the binary split between static and dynamic conditions.

Scene interpretation leverages a transformer-based vision-LLM (Gemini), employing prompt engineering to assign batches to one of four semantic categories: sound effect, hazard, descriptive caption, or none. Generated outputs are routed to either a text-to-speech (TTS) system or an ElevenLabs-based text-to-audio synthesis API. Caching and fine-grained category-specific throttling are integral to minimizing latency and preventing redundant or overwhelming feedback.

Figure 2: The AMAVA pipeline classifies each video batch by motion, thereby triggering distinct captioning and audio synthesis submodules.

Audio Synthesis and Semantic Fidelity

For static scenes (low-motion), AMAVA synthesizes a concise scene description using TTS, maximizing environmental awareness while minimizing auditory distraction via prompt caching and a 15-second repeat throttle. In dynamic or high-motion scenes, the logic prioritizes: (1) hazard alerts (TTS, 5s throttle, always played); (2) SFX cues representing relevant environmental events (3s throttle, played only if available in cache); and (3) scenario-specific extended descriptions.

The system demonstrates robust semantic alignment in generated audio, as measured with the CLAP joint language-audio embedding model. CLAP cosine similarity heatmaps confirm strong diagonal dominance, indicating that the synthesized audio consistently matches the intended prompt, though semantic or acoustic confusions are evident as higher off-diagonal similarities for certain classes.

Figure 3: CLAP similarity heatmap between SFX outputs and prompts, with pronounced diagonal strength indicating semantic alignment.

Quantitatively, the Fréchet Audio Distance (FAD) for synthesized sounds (1.71) outperforms prior V2A baselines including AudioGen and AudioLDM2, reflecting improved perceptual quality while maintaining semantic correctness.

System Performance and Latency

Processing latency in AMAVA is a tightly managed resource due to live navigation requirements. Increasing the batch size from 2 to 10 frames results in higher mean audio response times (1.3s to 2.2s) and increased rate of cue reordering due to processing pipeline delays. Accordingly, a two-frame batch (one-second effective window) is empirically optimal. Audio gap durations averaged less than five seconds, ensuring that crucial environmental changes are audibly communicated without excessive overlap or silences.

Figure 4: Processing latency, playback reordering rates, and mean audio cue gap durations under varied system configurations.

User Study and Ablation Analysis

A controlled navigation study with 12 blindfolded participants compared unaided white cane use to cane plus AMAVA support in both indoor and outdoor obstacle courses. Task completion times were statistically equivalent across conditions; however, subjective feedback highlighted substantial improvements in perceived safety (mean 4.8/5) and user confidence with AMAVA use. Other high-scoring metrics included ease of use, effective alert timing, low distraction, and system learnability.

An ablation variant, in which the motion classifier was disabled and TTS descriptions were always provided, exhibited diminished user preference and increased navigation time, suggesting the critical role of adaptive multimodal branching for cognitive load management.

Discussion and Limitations

Strong numerical results—100% binary motion classification accuracy, FAD superiority (1.71 vs. 2.85–3.12 for baselines), and high user survey scores—affirm the practical and technical validity of the AMAVA approach. However, several current limitations are acknowledged: (i) the absence of spatialized audio hinders directional awareness, (ii) real-time performance is contingent on cloud API latency and network stability, and (iii) scene understanding does not yet employ persistent context via SLAM or similar models.

Implications and Future Directions

AMAVA’s framework advances the field of real-time multimodal assistive technology, demonstrating that adaptive, motion-aware V2A mapping yields both qualitative and quantitative improvements for blind navigation. Integrating spatial audio (via individualized HRTF-based synthesis), on-device inference, and persistent scene modeling are immediate research avenues. Direct video-to-audio architectures—bypassing intermediate text—may further reduce latency. Large-scale longitudinal deployment is necessary for robust evaluation.

Conclusion

AMAVA presents an effective architecture for real-time assistive video-to-audio translation, using motion-aware branching to generate contextually relevant audio feedback that achieves high semantic fidelity and user acceptance. This work substantiates the importance of adaptive multimodal pipelines in next-generation assistive AI and highlights future opportunities in on-device spatial audio synthesis and persistent visual-semantic modeling (2604.23909).