PARSE-Ego4D: Proactive AR/VR Recs

• PARSE-Ego4D is an annotated extension of the Ego4D video corpus that formalizes proactive action recommendations for AR/VR contexts.
• The resource integrates LLM-based synthetic generation with rigorous human validation to produce nearly 10,700 gold-standard suggestion pairs across six action classes.
• Benchmark results demonstrate the method's potential for enhancing multimodal, context-aware action recommendation performance in dynamic AR/VR environments.

PARSE-Ego4D is an annotated extension of the Ego4D egocentric video corpus, introducing a large-scale resource for personal action recommendation. Targeted at equipping AR/VR assistants with the ability to proactively suggest context-appropriate actions, PARSE-Ego4D formalizes, annotates, and benchmarks the generation of personal, moment-specific recommendations that extend beyond traditional video understanding. The resource integrates synthetic action suggestion generation using LLMs with extensive human annotation, establishing new tasks and evaluation metrics for the development of intelligent proactive assistance in first-person video contexts (Abreu et al., 2024).

1. Purpose and Scope

PARSE-Ego4D’s principal aim is to bridge the gap between passive egocentric video annotation (e.g., activity recognition) and the active, context-driven action suggestion critical to AR/VR user assistance. Traditional datasets such as Ego4D provide rich labels (e.g., object tracking, activity segmentation) but do not specify actions an agent could proactively offer in the moment. PARSE-Ego4D introduces context-aware suggestions that an AR/VR assistant might propose, such as “Set a timer for 5 minutes” or “Show directions to the nearest coffee shop,” grounded in 3,670 hours (~9,600 clips) of head-mounted video with narrations, IMU, and gaze records.

The resource addresses the necessity of implicit, low-effort, and anticipatory system behavior in AR/VR headsets and smart-glasses, where seamless, proactive interaction can reduce cognitive overhead and increase the discoverability of device capabilities, especially in dynamic or safety-critical environments.

2. Annotation Pipeline

PARSE-Ego4D employs a two-stage annotation strategy combining synthetic generation with rigorous human validation for high-fidelity action recommendation labels.

2.1 LLM-Based Synthetic Generation

• Model: Gemini Pro (used in text-only mode)
• Prompting Pipeline: In-context prompts structure the LLM as a UX researcher, providing:
  • 10 s English narration segments as context
  • Enumerated API actions: Search, Assistant-search, Assistant-local, Language, Directions, Assistant-guide, and Others
  • JSON schema specifying “thoughts” (chain-of-thought reasoning), “query” (hypothetical user utterance + timestamp), “action” (API call), and optional “confidence” score
• Deduplication Methodology:
  • Start with 32,155 raw suggestions
  • Remove exact matches (−7,491)
  • Remove approximate matches using a normalized text-embedding similarity threshold $f(x_1,x_2)>0.9$ (−2,575)
  • Final: 19,255 unique (query, action) suggestions

2.2 Human Annotation and Validation

• Data Splitting:
  • 20% (3,672 suggestions) assigned to test set (each rated by 5 annotators)
  • Remaining 80% split to train (75%) and validation (5%); each suggestion rated once
  • Total: 36,171 ratings for 18,360 unique suggestions
• Annotation Protocol:
  • Each (clip, narration, LLM-generated query/action) unit rated along three axes (5-point Likert scale):
  • 1. Sensible? (1: nonsense, 5: spot-on)
  • 2. Helpful (for implicit recommendation)?
  • 3. Correct action for query?
  • Annotator quality controlled via prescreening (English fluency) and automated checks (JSON validity)
• Filtering Recommendations:
  • Task 1 (explicit query-action): retain suggestions with $\mu[\mathrm{sensible}]\ge4$ and $\mu[\mathrm{correct}]\ge4$
  • Task 2 (implicit): further restrict to $\mu[\mathrm{helpful}]\ge4$
• Reliability:
  • Intraclass Correlation Coefficient (ICC): Sensible=0.87, Helpful=0.74, Correct=0.81
  • Cohen’s kappa ($\kappa$): substantial to almost perfect agreement ($\kappa > 0.75$)

3. Dataset Composition and Organization

The PARSE-Ego4D dataset structure and statistics are as follows:

Component	Quantity / Classes	Notes
Ego4D video clips	~9,600	Avg. 15–30 min, includes IMU, gaze, narration
Synthetic pairs	19,255 unique (query, action)	Generated and deduplicated as above
Human-rated pairs	18,360	Total rated, following deduplication and filtering
Final gold pairs	~10,700	$\mu[\mathrm{sensible}]\ge4$, $\mu[\mathrm{correct}]\ge4$
Action classes	6	{Search, Assistant-search, Assistant-local, Language, Directions, Assistant-guide}
Suggestions/clip	$\approx$1.9	On average
Dataset splits	75% train, 5% val, 20% test	Test split: 3,672 suggestions, each with 5 ratings

This scheme supports standard ML development workflows for building and benchmarking action recommendation systems.

4. Benchmarks and Baseline Results

PARSE-Ego4D defines two novel benchmarks, accommodating text, video, or multimodal features as input modalities:

4.1 Explicit Query-to-Action Classification

Formal Problem: Given context $c$ (narration or video) and explicit user query $\mu[\mathrm{sensible}]\ge4$0, predict the action label $\mu[\mathrm{sensible}]\ge4$1. Function: $\mu[\mathrm{sensible}]\ge4$2.

• Primary metric: Classification accuracy ($\mu[\mathrm{sensible}]\ge4$3)
• Baselines:
  • Gemini Pro (zero-shot, text-only): Train Acc=55.95%, Val=54.43%, Test=63.57%
  • Constant baseline (majority class): 42.75% across splits

4.2 Implicit Query-to-Action Generation

Formal Problem: Given context $\mu[\mathrm{sensible}]\ge4$4 (no explicit query), generate $\mu[\mathrm{sensible}]\ge4$5. Function: $\mu[\mathrm{sensible}]\ge4$6.

• Primary metric: Negative log-likelihood (NLL) of ground-truth $\mu[\mathrm{sensible}]\ge4$7 under model output
• Baselines:
  • Gemini Pro (zero-shot): Train NLL=–43.43, Val=–43.46, Test=–42.50
  • Random (top 500 pairs): NLL ≈–44.8
  • Random (all train pairs): NLL ≈–53.4

Cost Considerations: Future work is encouraged to optimize inference latency and compute energy, with cost informally modeled as

$\mu[\mathrm{sensible}]\ge4$8

subject to hardware constraints.

5. Empirical Findings and Error Analysis

Human Preference and Agreement

• 65% of synthetic suggestions rated $\mu[\mathrm{sensible}]\ge4$9; 42% reach $\mu[\mathrm{correct}]\ge4$0
• Helpfulness (implicit) ratings lower than sensibleness, indicating stricter user standards for proactive recommendations

Error Patterns

• LLM Hallucinations: Generated out-of-domain or nonsensical suggestions are filtered by human annotation scores
• Redundancies: Deduplication (embedding-based) removes near-duplicate suggestions
• Resource trade-offs: Large LLMs outperform smaller models in accuracy/NLL, but deployment on-device is impeded by resource demands

Inter-rater Reliability

• Non-subjective axes (“sensible”, “correct”, “value”): ICC $\mu[\mathrm{correct}]\ge4$1 0.8, reflecting high annotation quality

6. Research Challenges and Future Opportunities

Open Problems

• Multimodal Fusion: Incorporation of video frames directly into LLM or sequence model prompts for enhanced contextual grounding
• Model Efficiency: Development of low-resource inference architectures (e.g., quantized LLMs, state-space models, RecurrentGemma) for on-device operation within energy and latency constraints
• Self-Supervision: Methods for automated validation and quality scaling of fresh data, including self-training, Self-Consistency, and Self-Reflection
• Dialogue and Personalization: Extension from single-shot recommendations to multi-turn, context-aware dialogues, and the ingestion of individualized user preferences, subject to rigorous privacy and safety protocols

Extensions

• Real-time AR/VR integration with adaptive user interfaces (e.g., gesture-based invocation, contextual hotwords)
• Advanced LLM prompt engineering (Chain-of-Thought, Tree-of-Thought) to produce richer, more transparent agent reasoning
• Expanded AI assistant functionality via integration with user calendars, contacts, and device APIs, raising new considerations for privacy-aware system design

The release of PARSE-Ego4D—with a rigorously constructed set of synthetic and human-validated action recommendation annotations, clearly defined benchmarks, and baseline code—provides a scaffold for the development and evaluation of next-generation proactive, context-aware intelligent assistance agents for egocentric video and AR/VR platforms (Abreu et al., 2024).