OpenGlass: A Sensing-Computing Split Architecture for Local MLLM-Driven Real-Time Visual Assistance

Abstract: We present OpenGlass, an open-source, privacy-oriented, local-first system for low-latency multimodal visual assistance, with a primary focus on blind and low-vision users. Cloud MLLM assistants offer strong visual understanding, but often require uploading first-person visual data and can suffer multi-second network delays; wearable glasses are ideal for sensing, but cannot host large models under tight compute and power budgets. OpenGlass addresses this gap with a sensing-computing split: an ESP32-based glasses-side unit captures visual context, while a nearby consumer-grade device performs local MLLM inference and local speech output, reducing cloud reliance and keeping raw egocentric visual data on user-controlled devices by default. We evaluate response quality, query-ready-to-audio latency, safety-aware abstention, and auditable logs. Under real ESP32 Wi-Fi capture, OpenGlass reaches 993 ms median user-to-audio latency with resized payloads and 1625 ms with raw 1280 x 720 payloads; 97.5% and 93.3% of trials fall below 2 s, respectively. OpenGlass is a user-initiated visual-assistance reference platform for obstacle/hazard awareness, sign/object queries, and image-quality self-checking, rather than a certified navigation aid. We release source code, hardware instructions, prompts, evaluation data, and logs.

• The paper introduces a sensing-computing split design that offloads heavy computation to a host device, ensuring near-real-time response and privacy for BLV users.
• It demonstrates significant latency reductions via adaptive image resizing and quantized VLMs, achieving sub-second responses under many conditions.
• The study emphasizes safe, auditable, and extensible design with controlled abstention to prevent hazardous outputs in ambiguous visual scenarios.

OpenGlass: A Privacy-Oriented, Local-First Architecture for Real-Time Multimodal Visual Assistance

System Architecture and Deployment Paradigm

OpenGlass introduces an open-source reference pipeline for real-time visual assistance, particularly targeting blind and low-vision (BLV) users who require responsive and privacy-preserving multimodal feedback. The core architectural innovation is a strict sensing-computing split: an ESP32-based camera unit embedded in wearable glasses captures the egocentric visual scene, while a co-located consumer-grade host device (e.g., a laptop or smartphone) executes all compute-heavy modules, including VLM inference and on-device speech synthesis. This partition aligns with intrinsic resource constraints of wearable platforms and addresses user privacy by ensuring raw first-person data is processed locally by default.

The pipeline proceeds as follows: after user-initiated spoken input, OpenGlass performs ASR on the host, triggers camera frame acquisition over a Wi-Fi link, packages the frame with the transcribed query, and streams it to a locally hosted VLM backend. Responses are immediately segmented and dispatched to TTS for near-immediate audio feedback, minimizing actionable latency for the user.

Figure 1: System framework overview and latency decomposition. Glasses-side unit provides audio/image capture; a local host performs request processing, VLM inference, and TTS in a pipelined fashion.

The architecture efficiently exploits consumer device capacity for local inference, enabled by advances in quantized, open-source VLMs such as MiniCPM-V/o operating in vision-text modes via the llama.cpp stack. Modular design permits future extensibility towards end-to-end multimodal LMs or new wearable devices, while maintaining deterministic components for robust deployment.

Wearable Sensing and Edge-First Processing

The ESP32-S3/OV5640 camera orchestrates context capture, providing both on-demand still frames (for VLM invocation) and continuous low-bitrate MJPEG video streaming (for alignment and preview). This design explicitly manages bandwidth/latency trade-offs inherent to commodity wireless links by supporting dynamic adaptation of frame resolution and selective capture, governed by host control.

Figure 2: CameraWebServer interface and MJPEG preview on ESP32-S3/OV5640; OpenGlass requests snapshots on demand for inference.

A critical bottleneck in system-level responsiveness is the transport and preprocessing (resizing, encoding) of visual payloads. OpenGlass’s logging and latency decomposition confirm these dominate end-to-end user-to-audio delays. Controlled ablations further demonstrate the scaling effect of image size on backend TTFT: 1280×720 raw frames approximately double latency relative to resized images, while adaptive downsampling (e.g., to 448px on the short side) achieves order-of-magnitude reductions without catastrophic accuracy losses.

Streaming Multimodal Pipeline and Safety-Aware Feedback

By default, OpenGlass implements a pipeline in which VLM responses are streamed at sentence-level granularity into the TTS buffer. This streaming output architecture (1) substantially reduces perceived first-audio latency (TTFA) relative to a naive wait-for-all-response strategy, (2) supports rapid interruption and barge-in (verified to be $\sim$3.5ms median on stop, with zero content leakage), and (3) enables strict safety constraints by supporting conservative abstention on low-quality visual evidence.

Figure 3: OpenGlass live run in terminal, providing event-ordered logs for ASR, capture, packing, inference, and TTS; supports reproducibility and failure attribution.

Interaction modes encompass: walk-and-talk obstacle warnings, object localization, sign/text reading, proactive image-quality self-checks, and robust interruption switching. Task- and safety-conditioned prompts enforce that the system abstains from hazardous guidance or content hallucination and issues explicit user retake instructions on ambiguous, blurred, or occluded frames.

Evaluation Methodology and Results

Benchmarking and Latency Analysis

The custom benchmark comprises 120 real-world ESP32 captures, covering four primary task families (T1: obstacle/hazard, T2: object finding, T3: sign/QR understanding, H1: image quality/self-checking), reflecting the conditions encountered in situ (e.g., motion blur, indoor/outdoor transitions, dynamic occlusion).

OpenGlass’s local-first pipeline (MiniCPM-V/o 4.5 INT4 with llama.cpp) achieves:

• Median user-to-audio latency of 993ms (896×504); 97.5% of trials <2s.
• With uncompressed 1280×720 images, median latency remains 1.6s (93.3% <2s), highlighting the cost of large payloads.
• Overseas cloud VLMs (Gemini 2.5 Flash) exhibit $>$4.9s medians, with all trials above 2s; domestic cloud APIs (Qwen-VL-Max) reach 2.07s median with only 46.7% below 2s.
• Resizing/quantization is the primary determinant of responsiveness: end-to-end delays scale sublinearly with image size due to the interaction of Wi-Fi, host-side packing, and VLM chunking.

Figure 4: Evaluation images for (left to right) T1-obstacle, T2-finding, T3A-sign reading, H1-quality diagnosis; highlight typical challenging frames.

Response Quality, Safety, and Robustness

Comprehensive task-level breakdowns show:

• H1 (image quality guidance/scanning) achieves perfect rubric scores (2.0, 0 HCE) and T1 (obstacle awareness) has zero high-confidence errors, indicating successful conservative behavior under uncertain evidence.
• T2 (object finding) maintains safe fallback via abstention (100% abstain) to avoid false-positive localization in ambiguous scenes.
• T3A/T3B (sign/QR) are failure-prone: hallucination rates (HCE) can exceed 37.5%/66.7% in hard examples, underpinning the necessity for strict abstention rules and prompt engineering.
• All interaction events are logged and auditable, supporting posthoc traceability of mispredictions and facilitating systematic safety and reproducibility audits.

Figure 5: Qualitative input frames: (left) T1 walk-and-talk scene, (middle) T1 low-quality/blurred frame invoking abstention, (right) T3B QR frame showing a context likely to trigger hallucination.

Implications and Open Research Directions

OpenGlass demonstrates the feasibility of a practical, privacy-conscious, user-initiated visual assistance platform using purely local resources. The design effectively mitigates the privacy/utility trade-off endemic to cloud-based visual question answering for BLV users and provides empirical evidence that sub-second response times are achievable with commodity edge hardware and strong quantized VLMs—addressing a key barrier for real-world adoption.

Further, the results highlight that adaptive image resizing, explicit safety abstention, and interaction-grounded prompt/rubric engineering are critical levers for balancing latency, quality, and safety. Failures observed in sign/QR recognition are not due to system-level latency but to inherent model uncertainty and sensor limitations, motivating future integration of tool-based verification, dynamic prompt adaptation, and multi-frame confirmation.

From a systems perspective, OpenGlass makes a significant contribution as a reproducible reference platform. Public release of hardware/software instructions, prompt templates, task logs, and raw outputs provides a foundation for further research in local MLLM deployment, assistive interaction evaluation, and privacy-respecting egocentric vision.

On the theoretical side, opportunities remain for integrating robust intent inference, unified multimodal dialogue, adaptive per-frame resolution/throughput optimization, and tighter coupling with future low-power NPU-enabled mobile hosts.

Conclusion

OpenGlass establishes a technical reference point for local, low-latency, and privacy-oriented multimodal assistance for BLV users. By decoupling sensing and compute, leveraging open VLM architectures, and architecting an auditable, safety-first interaction pipeline, OpenGlass achieves response times and interaction quality unattainable by current cloud-centric systems, all while retaining user control over sensitive egocentric visual data. As the edge-AI ecosystem advances, this architecture is well-positioned for integration with next-generation compact VLMs, fully embedded wearables, and trust-centric human-AI collaboration in safety-critical assistive scenarios.