VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval

Abstract: Over 285 million people worldwide live with a visual impairment, for whom everyday tasks such as avoiding obstacles, locating personal belongings, recognizing familiar faces, or handling cash remain persistent obstacles to personal autonomy. Existing assistive applications are typically limited to recognizing predefined categories, depend heavily on cloud connectivity, or require dedicated hardware. We present VisionAId, an Android application that turns a commodity smartphone into a real-time visual assistant. The system integrates six on-device deep learning models (metric monocular depth estimation, instance segmentation, visual and facial embeddings, face detection, and a custom banknote detector) running entirely through ONNX Runtime, with an optional cloud LLM (Google Gemini Flash) used only for narrative scene description and automatic object labeling. A distinctive contribution is a few-shot pipeline for personal objects: the user photographs an object from several angles, and the system later locates that specific instance in the environment, guiding the user toward it with augmented-reality markers, spatial audio, and distance-proportional haptics. All feedback is multimodal (Romanian speech synthesis, voice commands, vibration). On a reference device (Samsung Galaxy S21 Ultra), INT8 quantization reduces depth latency from ~1200 ms to ~491 ms, the custom banknote detector reaches an mAP@50 of 0.986, and metric depth is calibrated to below 1 cm of error within 3 m.

• The paper presents a comprehensive offline-first assistant integrating six deep-learning models for real-time, personalized object retrieval and proximity estimation.
• It employs few-shot personalized object registration with adaptive similarity thresholds and AR guidance, achieving efficient instance detection at 7–15 FPS.
• The system demonstrates practical on-device performance through INT8 quantization, optimized embedding storage, and robust banknote recognition.

VisionAId: A Comprehensive Offline-First Multimodal Visual Assistant for the Visually Impaired

System Motivation and Contributions

VisionAId presents an Android-based, offline-first, multimodal assistant, emphasizing a system architecture entirely reliant on on-device execution for core vision tasks. This addresses practical limitations in current assistive technologies, notably the lack of personalized instance-level object retrieval, requirement for predefined object categories, heavy dependence on cloud connectivity, and/or specialized hardware. VisionAId's primary innovation is its comprehensive and personalized functionality, including real-time proximity estimation with monocular metric depth, personalized few-shot object registration and retrieval with AR guidance, face recognition, Romanian banknote detection, and multimodal (speech, haptics, AR) feedback.

Distinctive technical advances introduced by the system include:

• A modular integration of six deep-learning models targeting separate assistive functions, uniformly accessible through a layered Kotlin/Jetpack Compose architecture.
• Personalized object retrieval using a few-shot pipeline combining YOLO11n-Seg for segmentation, MobileCLIP2-S2 for instance embeddings, and adaptive similarity thresholds.
• Real-time AR guidance operating entirely on-device, by combining robust visual localization with Center-CLIP instance discrimination and marker placement with ARCore.
• A custom, lightweight detector for Romanian banknotes trained from scratch, addressing region-specific gaps in commercial applications.
• On-device model optimization, highlighted by INT8 quantization achieving a 2.4× reduction in depth inference latency and binary embedding storage for real-time search latency minimization.

This layered, self-contained architecture makes VisionAId unique among assistive technologies, providing robustness where network and hardware constraints are critical.

Figure 1: Overall layered architecture of VisionAId. All perception runs on-device; the cloud LLM is an optional component used only for narrative scene description and object labeling.

Core System Components and Methods

Proximity Estimation and Scene Narration

The proximity estimation module deploys an INT8-quantized Depth Anything V2 model to process metric depth maps at 7–15 FPS. It leverages a vertical "walking band" region of interest to extract minimum distances for obstacle detection, implementing a three-level alert system modulated by proximity, with haptic and auditory feedback. Calibration yields sub-centimeter accuracy (mean error $<$1 cm within 0.5–3.0 m).

On-demand scene description is facilitated using a cloud LLM (Gemini Flash) invoked only for narrative spatial feedback, maintaining privacy and local operability for all primary navigation tasks.

Figure 2: Camera module (Tab~0): real-time proximity estimation from the metric depth map, with warm colors indicating nearby obstacles in the central walking band.

Personalized Object Registration and AR Retrieval

Few-shot registration operates by capturing multi-angle, motion-triggered keyframes of user-selected objects. Instance segmentation removes contextual background, enabling MobileCLIP2-S2 to extract L2-normalized, 512-D embeddings encoding object-specific features. The system then calculates intra-class similarity distribution to derive an object-adaptive, conservative acceptance threshold $\tau^*$, improving robustness against spurious matches.

During AR retrieval, a combination of YOLO-based candidate narrowing and centroid-CLIP embedding comparison yields robust instance discrimination. The search operates at ~5 FPS with per-candidate average handling times of ~76 ms. An EMA tracker further stabilizes detection, and ARCore anchors are placed upon exceeding confidence thresholds, initiating spatial audio, haptic signaling, and distance/angle-based voice guidance routines.

Complementary Modules

• Face Recognition: Employs YuNet for face detection and MobileFaceNet for 512-D embeddings, with embeddings sampled from eight head orientations to maximize angular coverage—enabling real-time re-identification above $\tau=0.45$ cosine threshold.
• Color Identification: Achieves deterministic and low-latency classification using direct RGB-to-HSV conversions and rule-based logic over the central frame region, deliberately avoiding deep learning for this task due to its categorical simplicity.
• Banknote Detection: Utilizes a YOLO26n model fine-tuned on an author-constructed dataset covering all RON denominations. This model delivers mAP@50 = 0.986 and per-class precision/recall $>$0.92 on validation.

  Figure 3: Confusion matrix on the v3 validation set. The dominant diagonal confirms high accuracy; 500~RON (absent from the v1 evaluation) is now perfectly classified (16/16).

Performance and Validation

On a Samsung Galaxy S21 Ultra (Exynos 2100), all core modules maintain latency bounds compatible with real-time interaction for visually impaired users (depth: 491 ms, AR object search: 560 ms, banknote: $<$15 ms). INT8 quantization of the depth model results in a 2.8× size reduction and 2.4× latency improvement with negligible accuracy degradation. Embedding persistence as binary blobs permits $\sim$150× faster loading compared to JSON, fundamental for responsive instance retrieval.

Banknote detection achieves perfect classification for previously underrepresented 500 RON notes, and overall system accuracy/latency trade-offs are tuned for practical deployment, although only one high-end device has been profiled so far.

Implications and Future Directions

Practically, VisionAId establishes a blueprint for accessible, private, and robust multimodal assistants that do not require cloud access for critical functionality. The architecture demonstrates that standard commodity hardware can host multi-model vision pipelines suitable for real-world visual navigation, personalized object retrieval, and regional-currency support, without auxiliary hardware.

Theoretically, the work supports the efficacy of combining category-level detection with instance-level CLIP-like embeddings (using a hybrid-score and keyword-induced thresholding) to achieve reliable, personalized object discrimination under real-world constraints. The adaptive, data-driven thresholding and ex-memorized AR guidance represent concrete steps toward instance-level autonomy.

Potential future developments include:

• Migration to wider device classes by distilling models, porting to NNAPI where possible, and quantization-aware retraining, particularly for MobileCLIP2-S2.
• Integrating an LLM-based persistent assistant capable of semantic mapping, memory, and tool-calling over the local scene.
• Extending environmental support to outdoor scenarios and broadening language/region capabilities.
• Rigorous user studies to empirically validate effectiveness and impact on autonomy in blind and visually impaired populations.

Conclusion

VisionAId delivers a technically integrated, offline-first assistant that considerably expands the functional range of smartphone-based assistive technology for the visually impaired. By integrating modular, optimized deep-learning pipelines—from metric depth to instance-level retrieval and face/banknote recognition—behind a synchronous multimodal feedback system, and validating practical latency and accuracy, VisionAId lays the groundwork for future user-centric, privacy-respecting mobile vision systems.