WorldScribe: Towards Context-Aware Live Visual Descriptions (2408.06627v1)

Abstract: Automated live visual descriptions can aid blind people in understanding their surroundings with autonomy and independence. However, providing descriptions that are rich, contextual, and just-in-time has been a long-standing challenge in accessibility. In this work, we develop WorldScribe, a system that generates automated live real-world visual descriptions that are customizable and adaptive to users' contexts: (i) WorldScribe's descriptions are tailored to users' intents and prioritized based on semantic relevance. (ii) WorldScribe is adaptive to visual contexts, e.g., providing consecutively succinct descriptions for dynamic scenes, while presenting longer and detailed ones for stable settings. (iii) WorldScribe is adaptive to sound contexts, e.g., increasing volume in noisy environments, or pausing when conversations start. Powered by a suite of vision, language, and sound recognition models, WorldScribe introduces a description generation pipeline that balances the tradeoffs between their richness and latency to support real-time use. The design of WorldScribe is informed by prior work on providing visual descriptions and a formative study with blind participants. Our user study and subsequent pipeline evaluation show that WorldScribe can provide real-time and fairly accurate visual descriptions to facilitate environment understanding that is adaptive and customized to users' contexts. Finally, we discuss the implications and further steps toward making live visual descriptions more context-aware and humanized.

• The paper introduces WorldScribe, a system that integrates vision-language and large language models to provide adaptive live visual descriptions for BVI individuals.
• It utilizes multifaceted context awareness—capturing user intent, dynamic visual changes, and acoustic cues—to tailor descriptions in real time.
• Evaluation with BVI participants demonstrated 75% intent coverage and 80.83% prioritization accuracy, highlighting its practical benefits and areas for refinement.

Context-Aware Live Visual Descriptions with WorldScribe

In the domain of Human-Computer Interaction (HCI) and specifically accessibility systems, the paper titled "WorldScribe: Towards Context-Aware Live Visual Descriptions" by Chang et al. presents a notable advancement. The research addresses the challenging problem of providing real-time, rich, and contextual visual descriptions to support blind and visually impaired (BVI) individuals in understanding their surroundings autonomously. WorldScribe is a system designed to generate live visual descriptions that are not only adaptive to the user's context but also customizable according to specific user needs.

Design and Implementation

Multifaceted Context Awareness

WorldScribe distinguishes itself by being context-aware on multiple fronts:

1. User Intent and Semantic Relevance: The system tailors descriptions to match users' specific intents, prioritizing information based on semantic relevance.
2. Visual Contexts: Adapts to dynamic versus stable scenes by providing succinct descriptions for fast-changing environments and more detailed descriptions when the visual scene stabilizes.
3. Sound Contexts: Adjusts the audio presentation by increasing volume in noisy settings or pausing when conversations are detected.

Technical Pipeline

WorldScribe's pipeline integrates multiple vision-LLMs (VLMs) and LLMs:

1. Intent Specification: Leverages GPT-4 to classify the user's intent and to generate relevant object classes and visual attributes. The system supports both general and specific intents.
2. Keyframe Extraction: Utilizes camera orientation and visual similarity to identify keyframes from video input, ensuring descriptions are provided for salient visual changes.
3. Description Generation: Balances latency and richness by deploying YOLO World for real-time, word-level descriptions, Moondream for short descriptions with spatial relationships, and GPT-4v for detailed descriptions.
4. Prioritization: Prioritizes descriptions based on their relevance to the user's intent and the proximity of described objects using a combination of semantic similarity and depth estimation techniques.
5. Presentation Layer: Adapts audio presentation in real time by detecting environmental sounds and manipulating the delivery of descriptions for better clarity and comprehension.

Evaluation and Findings

User Evaluation:

The paper presents an evaluation with six BVI participants across three different scenarios—specific intent, general intent, and user-defined intents. The feedback highlighted several key insights:

• Participants generally perceived the descriptions as accurate and beneficial.
• The adaptive nature of WorldScribe, switching between overview descriptions and detailed information, was particularly valued.
• The system's responsiveness to different contexts—such as increasing volume in noisy environments—was found to enhance usability.

However, some challenges were noted, such as occasional inaccuracies and the need for more colloquial language in descriptions. Users also expressed a need for integrating more concrete spatial information and incorporating a long-term memory of past interactions and environments.

Pipeline Evaluation:

• The system demonstrated an overall descriptive accuracy with 75% coverage of user-specified intents.
• Detailed descriptions from GPT-4v were found to be primarily accurate, although occasional errors due to low-quality image frames or contextual misinterpretations were observed.
• Description prioritization was largely effective, with 80.83% of descriptions aligned with user intent or prioritized by the proximity of described content.

Implications and Future Directions

The implications of this research extend both on a practical and theoretical front:

• Practical Implications: WorldScribe can significantly enhance real-time visual assistance systems for BVI individuals, offering a tool that adapts dynamically to a user's surroundings and needs.
• Future Developments: As VLMs and LLMs continue to evolve, incorporating more advanced models could further enhance the accuracy and richness of live descriptions. Future works should explore integrating spatial audio and developing datasets that reflect real-world complexities encountered by BVI users.
• Humanized Descriptions: Providing more colloquial and contextually relevant descriptions could improve user experience. Integrating user feedback to fine-tune descriptive language will be crucial.
• Integration with Other Assistive Technologies: Combining WorldScribe with navigation systems or wearable devices could offer a comprehensive assistive solution for BVI individuals, leveraging multimodal inputs for richer, more contextual support.

In conclusion, WorldScribe represents a meaningful step forward in context-aware live visual descriptions, effectively bridging the gap between emerging AI capabilities and the nuanced needs of BVI individuals. The research underscores the importance of context-awareness, real-time adaptation, and customizability in developing accessible technologies that genuinely enhance users' autonomy and understanding of their surroundings.