--- title: 'WAGIBench: Benchmark for Wearable Goal Inference' url: https://www.emergentmind.com/topics/wagibench type: topic --- # WAGIBench: Benchmark for Wearable Goal Inference Searching arXiv for the specified benchmark and closely related benchmark names to ground the article. WAGIBench is a benchmark for **egocentric multimodal goal inference** in **assistive wearable agents**, defined as agents embodied in wearable form factors such as smart glasses that observe a user’s surroundings and infer the user’s latent goal from passive contextual cues rather than from a fully articulated query. It is introduced to measure progress on the complementary problem to action execution or policy learning: before assistance can be delivered, a wearable agent often must infer what the user is trying to do from heterogeneous observations spanning vision, audio, digital state, and longitudinal history. The benchmark is built around **3,477 observation–goal pairs** from **348 participants** and **29 hours** of cleaned benchmark data, and it reports a persistent gap between human and model performance, including **93%** human multiple-choice accuracy on harder similar-distractor items versus **84%** for the best-performing vision-language model [2510.22443]. ## 1. Problem formulation and benchmark motivation WAGIBench targets the inference of a user’s **goal** from multimodal contextual observations in settings relevant to wearable assistance. The benchmark treats this as distinct from action recognition, narration, and downstream policy learning: the central task is not merely to identify what is visible or what action is being executed, but to infer what the user wants next from passive contextual evidence. The relevant observations include what the user sees, hears, is doing digitally, and has been doing over time [2510.22443]. The benchmark is motivated by a gap in prior egocentric datasets. Existing resources were described as often being built for narration, action recognition, or downstream re-annotation with LLMs, and as often lacking **ground-truth goals** or omitting the kinds of situations where wearable agents are most consequential. WAGIBench is therefore designed to be **ecologically valid** and to require reasoning over multiple modalities, including settings in which some modalities are informative while others are distractors [2510.22443]. This framing places goal inference at the interface of multimodal representation learning, contextual reasoning, and assistive agent design. A plausible implication is that progress on WAGIBench would indicate not just better egocentric perception, but improved inference over latent user intent under partial observability and heterogeneous signal quality. ## 2. Dataset construction and curation WAGIBench contains **3,477 observation–goal pairs** collected from **348 participants** over **29 hours** of cleaned benchmark data. The collection pipeline is explicitly large-scale: approximately **264 hours** of video were initially collected, filtering for quality reduced this to **155 hours**, and context-windowing to keep only task-relevant segments reduced it further to **29 hours** [2510.22443]. The data are scripted so that the ground-truth goal is known. Scripts cover common wearable-agent use cases across environments including **kitchen, office, outdoors, gym, transit, bedroom, living room, and social settings**. Script finalization used **Llama 3.2**, and each script includes variable slots intended to increase goal diversity. The resulting benchmark is not only multimodal but deliberately **modality-aware**, with modality subsets such as \(S_V\), \(S_{VA}\), \(S_{VD}\), and \(S_{VL}\) representing cases in which distinct modality combinations are necessary for accurate inference [2510.22443]. For visual and audio capture, participants recorded scripted scenarios using **Meta Aria glasses**. Each script was recorded multiple times, usually by about **6 participants**, and accepted recordings were filtered by human raters. A recording was retained only if **at least two of three raters marked it acceptable**, variable annotations agreed above a threshold, and the context windows overlapped sufficiently [2510.22443]. This construction strategy makes the benchmark unusually explicit about annotation validity and contextual relevance. The emphasis on context-windowing and acceptance thresholds suggests an attempt to avoid both underdetermined episodes and overly long contexts that dilute goal-relevant evidence. ## 3. Modalities and contextual representations Each WAGIBench sample is egocentric and includes four modalities: **Visual**, **Audio**, **Digital**, and **Longitudinal** [2510.22443]. | Modality | Representation | Construction details | |---|---|---| | Visual | Egocentric video | Recorded with Meta Aria glasses | | Audio | Transcribed speech from the recording | Derived from the recording | | Digital | Synthetic app-state context | Generated for seven apps via the Digital Context Generator | | Longitudinal | Historical context from prior episodes of the same user | Summarized into Socratic context | The **Digital Context Generator (DCG)** synthesizes realistic phone or app states for **Calendar, Messaging, Notes, Search, Videos, Maps, and Music**. It takes a sampled persona, other related personas, a timestamp, and optionally a cue, and produces structured app states. Most of this digital state is intentionally irrelevant noise; only some samples contain cue-conditioned relevant digital context. The benchmark reports **856** pairs with some relevant digital context, while the remaining samples contain digital context that is largely distractor information. The appendix describes a cue-annotated subset of **825** or **856** examples depending on the section, but the central design point is that the benchmark contains both relevant and irrelevant digital state [2510.22443]. For **Longitudinal context**, the dataset includes prior episodes from the same participant in the same environment. The authors create a history bank of about **5 support videos** per test example. One of these may be a positive support if the current script is longitudinal; otherwise, the supports are negative or distractor histories. Because raw histories would exceed model context limits, the videos are summarized into **Socratic context**, defined as detailed captions generated by VLMs and merged into a textual representation [2510.22443]. The multimodal design is consequential because it tests not only fusion but selectivity. WAGIBench is structured so that models must exploit informative modalities when present and suppress distractors when extra context is irrelevant or noisy. This suggests that the benchmark probes context arbitration as much as raw multimodal aggregation. ## 4. Task definition and evaluation protocol WAGIBench frames goal inference as an open-set language problem, but evaluates it under two complementary protocols: **discriminative multiple-choice evaluation** and **generative open-ended goal generation** [2510.22443]. In the **multiple-choice** setting, each sample is converted into a question containing the observation, the reference goal, and \(N = 3\) distractor goals. Two distractor regimes are used: **similar distractors**, sampled from goals semantically close to the reference, and **dissimilar distractors**, sampled from more distant goals. Sentence-BERT embeddings are used to map goals into a semantic space, after which distractors are sampled from percentile ranges of cosine similarity. A greedy diversity rule is then applied: \[ D_i = \begin{cases} sample(U(C)) & \text{if } i = 1 \\ \arg\min_{c \in C \setminus D_{)\). Several judge variants are compared: **Socratic**, **Cues**, **Reference**, **Socratic + Reference**, **Cues + Reference**, plus **Snap-MCQ** and **SBERT Similarity** baselines. The strongest judge is **Cues + Reference**, with **76.8% alignment**, effectively matching human-human agreement at **75.2%**; plain Socratic context is weaker, and both Snap-MCQ and SBERT similarity perform substantially worse [2510.22443]. The dual protocol is important because it separates closed-form recognition from open-ended intent articulation. A plausible implication is that WAGIBench evaluates not only whether a model can discriminate among plausible goals, but whether it can formulate a semantically relevant goal under the ambiguity and lexical variability of natural language. ## 5. Empirical findings The central empirical result is that **human performance exceeds model performance**, particularly when distractors are semantically difficult. In the multiple-choice setting, humans achieve **93% accuracy** on the harder similar-distractor MCQs, while the best vision-language model reaches **84%**. On dissimilar MCQs, human accuracy is near **97%**. Across the full benchmark, performance improves with model scale: small models are substantially weaker, medium models improve, and larger models such as **Qwen2.5-VL-72B** and **GPT-4.1** are strongest [2510.22443]. In the generative setting, larger models again perform better, but the benchmark reports that they remain far from reliable. The best models produce relevant goals only about **55%** of the time in the authors’ summary, and the top generative score in the reported table is around **0.55** for **GPT-4.1**, with large open models close behind [2510.22443]. A major contribution of WAGIBench is its **modality ablation** analysis. When a modality is genuinely informative, adding it improves performance over vision alone; the benchmark reports modality-specific gains as large as **35%** on MCQ and **30%** on generative evaluation. This effect is especially pronounced for **audio**, where speech transcripts can provide clues not recoverable from vision alone [2510.22443]. At the same time, extra irrelevant modalities hurt less than might be expected, particularly for larger models. The benchmark concludes that sufficiently large models can often ignore distractors. **Digital** and **Longitudinal** context yield smaller gains than audio because they have a lower signal-to-noise ratio. High-signal variants confirm this interpretation: \(D^*\), which retains only relevant digital sub-states, improves by up to **12%** over \(VD\), and \(L^*\), which retains only positive-support longitudinal history, improves by up to **5.6%** over \(VL\). The gap shrinks for large models, suggesting better noise filtering in complex contexts. When all modalities are provided together as \(VADL\), large models can exploit the full context and disentangle useful signals from distractors, whereas smaller models often suffer interference [2510.22443]. These results indicate that the benchmark is not saturated. The gap between strong human accuracy and materially weaker open-ended model relevance implies that current systems are substantially better at assisted recognition than at dependable latent-goal recovery in ecologically realistic wearable settings. ## 6. Significance, limitations, and benchmark identity WAGIBench is designed to move evaluation from generic egocentric understanding toward **assistive wearable goal inference**, where the challenge is to infer what the user wants next from **heterogeneous, partially noisy context** rather than merely to recognize an action or caption a scene. Its stated significance rests on four properties: it contains **ground-truth goals** rather than proxy labels, includes **multiple modalities** that a wearable agent would actually observe, covers **longitudinal user behavior**, and explicitly tests whether models can exploit relevant context while ignoring distractors [2510.22443]. The benchmark’s limitations are also explicit in its reported findings. Even the strongest systems remain below human performance, and generative relevance remains too low for robust deployment in assistive wearable agents. This suggests that the bottleneck is not only multimodal input availability but reliable latent-goal inference under ambiguity, noise, and open-ended language generation. WAGIBench should also be distinguished from other recent benchmarks with superficially similar names. **WBench** evaluates **interactive video world models** across video quality, setting adherence, interaction adherence, consistency, and physics compliance, rather than egocentric goal inference for wearable assistance [2605.25874]. **WGSR-Bench** evaluates **wargame-based strategic reasoning** for large language models via environmental situational awareness, opponent modeling, and policy generation, rather than multimodal intent inference from wearable observations [2506.10264]. The similarity in naming can obscure the fact that these benchmarks address different problem classes, data regimes, and evaluation targets. Within its own domain, WAGIBench establishes goal inference as a standalone benchmark problem for wearable agents. The reported results suggest that multimodal models can use additional context when it is informative and often suppress irrelevant context when sufficiently large, but they still fall short of practical, human-level goal inference in realistic open-ended settings [2510.22443]. Source: https://www.emergentmind.com/topics/wagibench