Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM

Abstract: Passive tracking methods, such as phone and wearable sensing, have become dominant in monitoring human behaviors in modern ubiquitous computing studies. While there have been significant advances in machine-learning approaches to translate periods of raw sensor data to model momentary behaviors, (e.g., physical activity recognition), there still remains a significant gap in the translation of these sensing streams into meaningful, high-level, context-aware insights that are required for various applications (e.g., summarizing an individual's daily routine). To bridge this gap, experts often need to employ a context-driven sensemaking process in real-world studies to derive insights. This process often requires manual effort and can be challenging even for experienced researchers due to the complexity of human behaviors. We conducted three rounds of user studies with 21 experts to explore solutions to address challenges with sensemaking. We follow a human-centered design process to identify needs and design, iterate, build, and evaluate Vital Insight (VI), a novel, LLM-assisted, prototype system to enable human-in-the-loop inference (sensemaking) and visualizations of multi-modal passive sensing data from smartphones and wearables. Using the prototype as a technology probe, we observe experts' interactions with it and develop an expert sensemaking model that explains how experts move between direct data representations and AI-supported inferences to explore, question, and validate insights. Through this iterative process, we also synthesize and discuss a list of design implications for the design of future AI-augmented visualization systems to better assist experts' sensemaking processes in multi-modal health sensing data.

• The paper introduces Vital Insight, a system that integrates visualization with LLM to generate contextual inferences from multimodal personal tracking data.
• It employs a methodology that combines time-series, discrete, and self-reported data with human-in-the-loop processing for refined anomaly detection and daily summaries.
• User studies with 21 experts validated the design, highlighting improvements in data exploration, evidence-based sensemaking, and AI trustworthiness.

Summary of the Paper

The paper "Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM" explores the development and user-centered design of a system that integrates visualization techniques with LLMs to assist experts in sensemaking of multi-modal personal tracking data. This research tackles the challenge of transforming raw sensor data into meaningful, high-level insights required for practical applications.

System Design and Functionality

The system, Vital Insight, comprises two main iterations — VI-1 and VI-2, aiming to facilitate experts' sensemaking through a combination of visualization and AI-generated inferences. The input data for this system includes multi-modal data types collected from smartphones, wearables, and voice assistants. These data include time-series data (e.g., physiological signals), discrete data (e.g., phone usage states), and self-reported data (e.g., surveys).

Visual Representations

The system provides a variety of visualization tools, enabling direct representation of sensing data. In VI-1, the use of small multiple-style time series plots enables experts to view diverse data streams like heart rate, location, and activity over time. This approach supports pattern recognition and data exploration, addressing the need identified in the initial user interviews for effective visual data representation.

AI Augmentation

Vital Insight leverages AI through LLMs to provide inferences at varying granularities. It generates daily and hourly summaries of activities, anomaly detection, and potential questions that could guide further investigation. In VI-2, the AI's anomaly detection process was refined to be more human-in-the-loop, addressing concerns about consistency and accuracy highlighted in the user studies.

Human-in-the-Loop Process

Incorporating expertise-derived insights into the AI workflow improves trustworthiness. VI-2 improves upon initial designs by embedding LLM-detected occurrences and their interpretations directly into the visualization layer, offering a more seamless and integrated experience for experts.

Figure 1: Eight components in VI-1 powered by visualization and LLM: time selection, main visualization, user profile, user check-in, LLM daily summary, LLM question, LLM anomaly detection, LLM hourly summary.

User Studies and Findings

The development of Vital Insight was informed by extensive studies with 21 experts over three phases. These experts were interviewed and participated in user testing to identify the challenges and opportunities in sensemaking of personal tracking data. Key findings from these studies emphasized the iterative nature of sensemaking, the importance of integrating multiple data modalities, and the role of AI in providing contextual inferences.

Integration of AI-Generated Insights

The AI components were generally well-received, particularly in terms of enhancing experts' understanding and trust. Experts appreciated the prototype's ability to connect data insights with contextual understanding, which aligns with human sensemaking processes. However, expectations for AI outputs included a need for clear evidence and explanation, which was addressed in subsequent system iterations.

Figure 2: Overview of the structure of LLM augmentation for VI-1 and VI-2.

Implications for Future Systems

Design Implications

1. Evidence-Based Sensemaking: Ensure systems facilitate validation of AI-generated insights by explicitly providing the underlying data and deduction processes.
2. Integration Across Modalities: Seamlessly integrate heterogeneous data streams for comprehensive analysis, particularly aligning data on time scales for easier comparison.
3. Iterative Support and Action Guidance: Support continuous refinement of insights and inspire stakeholder actions through multi-level data views and summarizations.

Addressing Bias and Trust

Understanding and mitigating biases in LLM outputs is crucial for enhancing trust among expert users. This includes improving narrative capabilities while ensuring consistency and transparency in how inferences are derived from raw data.

Conclusion

Vital Insight demonstrates a promising approach to bridging the gap between raw data collection and meaningful interpretation through a combination of visualization and AI-assisted inference. Its development highlights the importance of iterative, user-centered design in creating effective tools for expert users, ultimately aiming to augment human expertise rather than replace it. Further research and refinement, particularly in AI-assisted sensemaking and reducing biases, would enhance the utility and acceptance of similar systems in real-world applications.

Figure 3: Design of the new iteration of Vital Insight (VI-2). Top left: user testing input; bottom left: main visualization and LLM detected occurrences; top right: user profile; middle right: Day in a glance generated by LLM; bottom right: user check-in conversation.