VisionAId: AI-Driven Assistive Vision Systems

• VisionAId is a family of AI-driven assistive systems that convert visual data into actionable, non-visual guidance for people with visual impairment.
• These systems are built on diverse architectures—from offline smartphone apps to wearable edge devices—that integrate models for detection, OCR, and scene description.
• Evaluation studies highlight reduced latency, high accuracy, and user-centric interaction designs that balance privacy, cognitive load, and real-world usability.

VisionAId is a name used in recent arXiv literature for AI-driven assistive systems that transform visual input into actionable, non-visual guidance for people with visual impairment. In its most explicit recent form, it denotes an offline-first Android assistant that runs six on-device deep learning models through ONNX Runtime and uses Google Gemini Flash only for optional narrative scene description and automatic object labeling (Florea et al., 2 Jul 2026). Closely related systems under the same or adjacent design lineage include smartphone assistants such as AIDEN, wearable RGB-D and LVLM devices, RTC-based VLM agents, and acoustic spatialization systems, with recurring functions that include obstacle awareness, object and face recognition, OCR, scene description, and task-oriented navigation (Marquez-Carpintero et al., 8 Nov 2025, Baig et al., 2024, Zhao et al., 2 Nov 2025, Mehta et al., 2023). In current usage, the term denotes not a single canonical architecture but a family of multimodal assistive designs organized around low-latency perception, accessibility-first interaction, and increasingly personalized guidance.

1. Historical emergence and conceptual scope

Recent VisionAId work sits at the intersection of mobile assistive vision, wearable perception, and vision-language interaction. Earlier mobile systems already established the basic architectural pattern: a smartphone camera, on-device or server-assisted perception, and accessible feedback. A 2015 mobile face-recognition system for visually impaired users used on-device face detection and either local or server-side recognition, with TalkBack-based speech and vibrotactile feedback, showing that commodity phones could support assistive identification without dedicated hardware (Chaudhry et al., 2015).

Subsequent systems broadened the task set from person identification to integrated environmental assistance. VisBuddy combined image captioning, OCR, object discovery, and a news reader on a Raspberry Pi 4, with scene recognition based on an Inception-V3 encoder and LSTM decoder, object discovery based on RetinaNet, and OCR through Google Cloud Vision API (Sivakumar et al., 2021). AIris then moved toward a speech-driven eyewear form factor, coupling a point-of-view camera with local face recognition, server-side image captioning and object detection, OCR, money counting, barcode scanning, and note-taking (Brilli et al., 2024). AIDEN consolidated a similar multimodal agenda in smartphone form, unifying scene description, OCR, object detection, and interactive question answering inside a single cross-platform mobile application (Marquez-Carpintero et al., 8 Nov 2025).

Within this trajectory, VisionAId has become associated with systems that emphasize either personalized retrieval, context-rich narration, or real-time guidance. The hat-mounted VisionAId system integrates MobileNet SSD, FaceNet, GPT-4o-mini, and an ultrasonic collision alert, while the offline-first Android VisionAId emphasizes personalized object retrieval, calibrated metric depth, on-device face recognition, and a custom banknote detector (Baig et al., 2024, Florea et al., 2 Jul 2026). WalkVLM and VIA-Agent extend the same problem space toward streaming VLM assistance, focusing respectively on walking-specific temporal benchmarks and the reduction of cognitive load and task drift in real-time multimodal assistance (Yuan et al., 2024, Zhao et al., 2 Nov 2025).

The name has also appeared outside blindness assistance. In digital pathology, PathVis uses the term in the sense of an AI-augmented mixed-reality visual assistant for gigapixel whole-slide images on Apple Vision Pro, indicating that “VisionAId” has broadened into a more general label for multimodal AI assistance grounded in perception and interaction (Veerla et al., 5 May 2025). In the visually impaired-assistance literature, however, the dominant meaning remains a multimodal aid for autonomy in everyday environments.

2. Architectures and sensing configurations

VisionAId systems span three recurrent deployment patterns: smartphone-centered distributed systems, wearable edge devices, and hybrid edge-cloud streaming agents. AIDEN exemplifies the first pattern. Its mobile client is built with Ionic v6.20.1, Capacitor Core v4.7.0, and Vue.js v3.3.7, while a centralized server performs YOLOv8 and LLaVA inference; requests are handled in first-in-first-out order, and results are returned as audio, haptic, and visual feedback (Marquez-Carpintero et al., 8 Nov 2025). AIris adopts a similar split: a Raspberry Pi 4B handles speech I/O, local face recognition, and routing, while DeepDetect on a server hosts image captioning and YOLO inference (Brilli et al., 2024).

Other systems move critical functions entirely onto the device. The offline-first Android VisionAId uses Kotlin 2.0, Jetpack Compose, ONNX Runtime Mobile, Room, and CameraX, with all critical perception running locally on CPU and the cloud reserved for optional scene narration and semantic labeling (Florea et al., 2 Jul 2026). The augmented acoustic VisionAId runs fully offline on a Raspberry Pi 4+ with an Intel RealSense D415 or SR300 depth camera and headphones, translating downsampled RGB-D geometry into structured musical cues without any server dependency (Mehta et al., 2023). The IoT Cane similarly performs RGB-D fusion and RT-DETRv3-R50 inference on a Raspberry Pi 4B, with RealSense D435i sensing, custom PCB integration, and local audio/haptic feedback (Chandra et al., 22 Aug 2025).

A third pattern uses continuous streaming and cloud VLM reasoning. VIA-Agent replaces an earlier request-response wearable pipeline with an RTC mobile embodiment in which an iPhone streams continuous video and audio to a cloud gateway, a reasoning-capable VLM consumes both streams, and TTS returns incremental speech (Zhao et al., 2 Nov 2025). The hat-mounted VisionAId with GPT-4o-mini also uses the network selectively: local MobileNet SSD and FaceNet operate continuously on the Raspberry Pi, while images are sent to the LVLM only when the user presses the dark blue “Detail Request” button (Baig et al., 2024).

Sensor stacks vary accordingly. Smartphone systems rely primarily on monocular RGB, speech input, TTS, and haptics (Marquez-Carpintero et al., 8 Nov 2025, Florea et al., 2 Jul 2026). Wearables add depth cameras, ultrasonic range sensing, IMUs, or bone-conduction audio (Mehta et al., 2023, Baig et al., 2024, Chandra et al., 22 Aug 2025). This suggests that VisionAId is increasingly defined less by any single modality than by how modalities are fused under strict latency, privacy, and accessibility constraints.

3. Perception, reasoning, and personalized retrieval

The perception core of VisionAId systems is heterogeneous but converges on a small set of model families. AIDEN uses YOLOv8 for object detection across 80 standard categories extended with a “door” class, and LLaVA for both scene description/VQA and OCR via the prompt “Transcribe the text present in this image” (Marquez-Carpintero et al., 8 Nov 2025). The hat-mounted VisionAId uses quantized MobileNet SSD for edge object detection, Haar cascade plus FaceNet for face recognition, and GPT-4o-mini through Azure OpenAI for contextual description (Baig et al., 2024). AIris combines YOLO inference on the server, MAX Image Caption Generator with an Inception-v3 encoder, dlib ResNet-34 face embeddings, Tesseract OCR, and Pyzbar barcode decoding (Brilli et al., 2024).

The offline-first Android VisionAId is distinctive in its explicit six-model on-device stack: Depth Anything V2 Metric Small for metric monocular depth, YOLO11n-Seg for instance segmentation and detection, MobileCLIP2-S2 for 512-dimensional visual embeddings, YuNet for face detection, MobileFaceNet for facial embeddings, and YOLO26n for Romanian banknote detection (Florea et al., 2 Jul 2026). Metric depth is calibrated with a scale factor $f_c = 0.55$, and the paper reports below 1 cm mean error within 0.5–3 m after calibration. Personalized object retrieval is implemented through a few-shot registration pipeline: the user captures multiple views, masked embeddings are stored as compact binary blobs, an object-specific threshold $\tau^*$ is derived from intra-class similarity statistics, and real-time search combines centroid similarity, top-3 exemplar similarity, keyword-based category adjustment, EMA stabilization, and AR anchoring (Florea et al., 2 Jul 2026).

A parallel strand replaces verbal semantics with structured acoustic encodings. The augmented acoustic VisionAId uses RGB-D point clouds, 16×12 nearest-neighbor downsampling, and a chunk-based $O(n)$ flood-fill segmentation algorithm to group spatially proximate points without pairwise distance calculations, then maps depth to pitch, horizontal position to stereo pan, and vertical position to playback order (Mehta et al., 2023). The result is not OCR or captioning but rapid transmission of 3D scene geometry through musical notes.

Streaming VLM systems push perception toward task-conditioned reasoning. WalkVLM defines blind walking assistance as a sequential decision problem over video streams and uses hierarchical chain-of-thought planning to infer static attributes, danger level, scene summary, and concise reminders, together with temporal-aware adaptive prediction to reduce redundant warnings (Yuan et al., 2024). Scene-aware vectorized-memory systems go further still: one recent framework quantizes a 19B-parameter VLM from 38 GB to 16 GB through cross-modal differentiated quantization and augments it with ChromaDB-backed scene memory, enabling retrieval of similar historical scenes before answer generation (Wang et al., 25 Aug 2025).

4. Interaction design, guidance, and cognitive load

VisionAId systems are defined as much by their interaction policy as by their model stack. AIDEN’s most distinctive feature is active object search: the user vocally specifies a target, scans the environment with the phone, and receives continuous auditory and haptic guidance that prioritizes the largest detected instance, computes spatial position, and guides the user to center the object before moving toward it (Marquez-Carpintero et al., 8 Nov 2025). The offline-first Android VisionAId extends this with AR-based retrieval: once confidence exceeds a threshold, the system places a 3D marker via ARCore, derives a camera-local angle, and combines Romanian TTS, spatial audio panning, Geiger-style beeps, and distance-proportional haptics for object acquisition (Florea et al., 2 Jul 2026).

Wearable systems introduce further variations in feedback design. The hat-mounted VisionAId uses bone-conduction stereo headphones and a buzzer triggered when the ultrasonic distance falls below 20 cm, while also supporting one-click enrollment of new faces or objects through the green “Add Item” button (Baig et al., 2024). The acoustic VisionAId removes speech almost entirely from the primary spatial channel, relying on rapid, structured note sequences designed to minimize latency and cognitive load (Mehta et al., 2023). IoT Cane couples bone-conduction audio with PID-controlled vibration intensity, and exposes battery state, class counts, and feedback settings through a Swift-based iOS companion application (Chandra et al., 22 Aug 2025).

Recent work has made cognitive load an explicit optimization target. VIA-Agent argues that existing VLM assistants impose high cognitive load through verbose, unfocused narration and task drift, and responds with a goal-persistent “brain” plus an RTC “body.” The system caps responses at 128 tokens, keeps only the last two user-agent exchanges in sliding context, maintains one-day session memory, and applies multi-level confidence filtering so that uncertain items trigger reorientation requests or safe fallbacks rather than overconfident fabrication (Zhao et al., 2 Nov 2025). This suggests that VisionAId interaction design is moving from descriptive output toward constrained, user-steerable, action-first guidance.

An important corrective to this trend comes from CVI-focused work. A review and focus study on cerebral visual impairment reports that many visually impaired users with CVI prefer vision-first assistance, simplicity, decluttering, adaptive contrast and borders for figure-ground separation, and carefully dosed audio or haptics, because multimodal overload can worsen performance rather than improve it (Gamage et al., 29 May 2025). In that sense, VisionAId interaction is not reducible to “more modalities”; it is an accessibility policy that must be personalized to perceptual and cognitive phenotype.

5. Reported performance and evaluation

Reported performance varies widely across task classes and deployment choices. The offline-first Android VisionAId reports that INT8 quantization reduces Depth Anything V2 latency from ~1200 ms to ~491 ms on a Samsung Galaxy S21 Ultra, while the camera tab attains 7–8 FPS by caching depth every third frame. On the same device, the object-registration pipeline runs at ~550 ms, search at ~560 ms, people recognition at <80 ms, money detection at <15 ms, and color identification at <1 ms. Its custom Romanian banknote detector reaches $\mathrm{mAP}@50 = 0.986$, $\mathrm{mAP}@50\text{–}95 = 0.961$, precision 0.975, and recall 0.925 (Florea et al., 2 Jul 2026).

AIDEN reports substantially higher latencies for server-backed scene understanding than for object search. Over 20 trials per functionality, Scene Description and Question Answering took 7.34 ± 1.10 s on the server and 10.10 ± 1.29 s end-to-end on the smartphone; OCR took 7.09 ± 2.57 s on the server and 9.54 ± 2.94 s end-to-end; and Object Finder required 0.23 ± 0.18 s on the server, 0.51 ± 0.22 s end-to-end, with an average processing speed of 1.96 frames per second on the smartphone. In a TAM study with 7 visually impaired users, most responses fell between “Excellent” and “Best,” with highest scores for intuitiveness and perceived autonomous use (Marquez-Carpintero et al., 8 Nov 2025).

The hat-mounted VisionAId reports good-light precision 0.88, recall 0.92, F1 0.90, and accuracy 0.90; in low light, precision falls to 0.76, recall to 0.74, F1 to 0.75, and accuracy to 0.80. Average response time is ~1.5 s for core recognition and ~5.5 s for LVLM detail requests. Its 50-participant user study yields a System Usability Scale score of 85/100, with >90% reporting helpfulness and intent to adopt (Baig et al., 2024).

Other systems show the diversity of evaluation regimes. AIris cites model-level figures of ~99.38% face-recognition accuracy on LFW with ≈ 50 ms local inference per face, ~63.4% mAP for YOLO on COCO with ≈ 150 ms server-side inference, ~27% BLEU for scene captioning with ≈ 150 ms inference, and ~95% English speech-to-text accuracy with ≈ 200 ms response time (Brilli et al., 2024). The acoustic VisionAId reports Pearson’s $R = 0.866$ on detailed objects and 87.5% accuracy on a noisy outdoor night scene (Mehta et al., 2023). IoT Cane reports AP50 71.7%, mAP@[.5:.95] 53.4%, F1(person) 0.89, end-to-end latency around 150–200 ms, and correct mobility decisions in 92.1% of 48 field-trial situations (Chandra et al., 22 Aug 2025).

For streaming VLM guidance, the dominant metrics are no longer only detector AP or caption BLEU. WalkVLM introduces a dedicated walking benchmark with approximately 12,000 video-annotation pairs and evaluates not only reminder quality but also temporal redundancy and trigger timing (Yuan et al., 2024). VIA-Agent, evaluated with 9 visually impaired participants against BeMyAI and Doubao, matches or exceeds Doubao on success rate while reducing aggregate mean task time by 39.9% to 70.1 s from 116.7 s and reducing conversational turns to 4.3 from 5.8, while also improving perceived cognitive load and usability (Zhao et al., 2 Nov 2025). These results indicate a shift in VisionAId evaluation from model-centered recognition scores to task completion, cadence control, and trust calibration.

6. Limitations, open problems, and future directions

Across the literature, three limitations recur. The first is environmental fragility. AIDEN notes sensitivity to challenging lighting, occlusions, and unfamiliar objects (Marquez-Carpintero et al., 8 Nov 2025). The hat-mounted VisionAId explicitly degrades in low light (Baig et al., 2024). IoT Cane identifies reflective surfaces, partial occlusion, and fast motion as failure modes (Chandra et al., 22 Aug 2025). Even highly optimized offline mobile systems still report that motion blur, strong occlusion, and feature-poor AR surfaces degrade retrieval or anchoring quality (Florea et al., 2 Jul 2026).

The second limitation is the trade-off between privacy, latency, and capability. Offline-first designs preserve privacy and network independence but constrain model size and thermal budget (Florea et al., 2 Jul 2026). Cloud LVLM systems deliver richer descriptions but introduce variable latency and connectivity dependence (Brilli et al., 2024, Baig et al., 2024). RTC streaming agents reduce perceived delay through incremental output, yet continuous audio-video streaming raises stronger privacy and consent questions than single-image on-demand inference (Zhao et al., 2 Nov 2025). A plausible implication is that future VisionAId systems will increasingly combine local safety-critical perception with selective, explicitly user-triggered cloud reasoning.

The third limitation is the weakness of current evidence. Several papers rely on small user studies, informal trials, or upstream model benchmarks rather than task-specific real-world evaluation (Brilli et al., 2024, Chaudhry et al., 2015). CVI-focused work further argues that much of the existing assistive-vision literature is implicitly designed for ocular low vision or blindness and does not adequately address clutter intolerance, motion sensitivity, or sensory overload in cerebral visual impairment (Gamage et al., 29 May 2025). This widens the scope of VisionAId from generic “assistive AI” toward phenotype-specific accessibility engineering.

Current research directions follow accordingly. One line seeks richer reasoning on constrained hardware: scene-aware vectorized memory systems combine ChromaDB-backed retrieval with cross-modal differentiated quantization, reporting memory reduction from 38 GB to 16 GB and response latency of 2.83–3.52 seconds from scene analysis to initial speech output (Wang et al., 25 Aug 2025). Another line prioritizes human-centered communication policies, as in VIA-Agent’s goal persistence and calibrated conciseness (Zhao et al., 2 Nov 2025). A third line emphasizes personalized perception, exemplified by few-shot object retrieval and face registration pipelines that move beyond fixed category recognition (Florea et al., 2 Jul 2026, Baig et al., 2024).

Taken together, these works position VisionAId as a research program rather than a finished product category: a multimodal assistive stack in which perception, memory, interaction policy, and accessibility constraints are co-designed. Its central technical challenge is no longer merely to “recognize the scene,” but to decide what must be perceived, what must be said, when it must be said, and how that guidance can remain reliable, private, and cognitively sustainable in everyday use.