WAGIBench: Benchmark for Wearable Goal Inference

• WAGIBench is a benchmark for egocentric multimodal goal inference in wearable assistive agents, featuring 3,477 observation–goal pairs.
• It integrates visual, audio, digital, and longitudinal modalities to evaluate contextual reasoning and noise filtering in real-world settings.
• Empirical findings reveal that while large models improve performance, they still fall short of human accuracy in inferring latent user goals.

Searching arXiv for the specified benchmark and closely related benchmark names to ground the article. WAGIBench is a benchmark for egocentric multimodal goal inference in assistive wearable agents, defined as agents embodied in wearable form factors such as smart glasses that observe a user’s surroundings and infer the user’s latent goal from passive contextual cues rather than from a fully articulated query. It is introduced to measure progress on the complementary problem to action execution or policy learning: before assistance can be delivered, a wearable agent often must infer what the user is trying to do from heterogeneous observations spanning vision, audio, digital state, and longitudinal history. The benchmark is built around 3,477 observation–goal pairs from 348 participants and 29 hours of cleaned benchmark data, and it reports a persistent gap between human and model performance, including 93% human multiple-choice accuracy on harder similar-distractor items versus 84% for the best-performing vision-LLM (Veerabadran et al., 25 Oct 2025).

1. Problem formulation and benchmark motivation

WAGIBench targets the inference of a user’s goal from multimodal contextual observations in settings relevant to wearable assistance. The benchmark treats this as distinct from action recognition, narration, and downstream policy learning: the central task is not merely to identify what is visible or what action is being executed, but to infer what the user wants next from passive contextual evidence. The relevant observations include what the user sees, hears, is doing digitally, and has been doing over time (Veerabadran et al., 25 Oct 2025).

The benchmark is motivated by a gap in prior egocentric datasets. Existing resources were described as often being built for narration, action recognition, or downstream re-annotation with LLMs, and as often lacking ground-truth goals or omitting the kinds of situations where wearable agents are most consequential. WAGIBench is therefore designed to be ecologically valid and to require reasoning over multiple modalities, including settings in which some modalities are informative while others are distractors (Veerabadran et al., 25 Oct 2025).

This framing places goal inference at the interface of multimodal representation learning, contextual reasoning, and assistive agent design. A plausible implication is that progress on WAGIBench would indicate not just better egocentric perception, but improved inference over latent user intent under partial observability and heterogeneous signal quality.

2. Dataset construction and curation

WAGIBench contains 3,477 observation–goal pairs collected from 348 participants over 29 hours of cleaned benchmark data. The collection pipeline is explicitly large-scale: approximately 264 hours of video were initially collected, filtering for quality reduced this to 155 hours, and context-windowing to keep only task-relevant segments reduced it further to 29 hours (Veerabadran et al., 25 Oct 2025).

The data are scripted so that the ground-truth goal is known. Scripts cover common wearable-agent use cases across environments including kitchen, office, outdoors, gym, transit, bedroom, living room, and social settings. Script finalization used Llama 3.2, and each script includes variable slots intended to increase goal diversity. The resulting benchmark is not only multimodal but deliberately modality-aware, with modality subsets such as $S_V$, $S_{VA}$, $S_{VD}$, and $S_{VL}$ representing cases in which distinct modality combinations are necessary for accurate inference (Veerabadran et al., 25 Oct 2025).

For visual and audio capture, participants recorded scripted scenarios using Meta Aria glasses. Each script was recorded multiple times, usually by about 6 participants, and accepted recordings were filtered by human raters. A recording was retained only if at least two of three raters marked it acceptable, variable annotations agreed above a threshold, and the context windows overlapped sufficiently (Veerabadran et al., 25 Oct 2025).

This construction strategy makes the benchmark unusually explicit about annotation validity and contextual relevance. The emphasis on context-windowing and acceptance thresholds suggests an attempt to avoid both underdetermined episodes and overly long contexts that dilute goal-relevant evidence.

3. Modalities and contextual representations

Each WAGIBench sample is egocentric and includes four modalities: Visual, Audio, Digital, and Longitudinal (Veerabadran et al., 25 Oct 2025).

Modality	Representation	Construction details
Visual	Egocentric video	Recorded with Meta Aria glasses
Audio	Transcribed speech from the recording	Derived from the recording
Digital	Synthetic app-state context	Generated for seven apps via the Digital Context Generator
Longitudinal	Historical context from prior episodes of the same user	Summarized into Socratic context

The Digital Context Generator (DCG) synthesizes realistic phone or app states for Calendar, Messaging, Notes, Search, Videos, Maps, and Music. It takes a sampled persona, other related personas, a timestamp, and optionally a cue, and produces structured app states. Most of this digital state is intentionally irrelevant noise; only some samples contain cue-conditioned relevant digital context. The benchmark reports 856 pairs with some relevant digital context, while the remaining samples contain digital context that is largely distractor information. The appendix describes a cue-annotated subset of 825 or 856 examples depending on the section, but the central design point is that the benchmark contains both relevant and irrelevant digital state (Veerabadran et al., 25 Oct 2025).

For Longitudinal context, the dataset includes prior episodes from the same participant in the same environment. The authors create a history bank of about 5 support videos per test example. One of these may be a positive support if the current script is longitudinal; otherwise, the supports are negative or distractor histories. Because raw histories would exceed model context limits, the videos are summarized into Socratic context, defined as detailed captions generated by VLMs and merged into a textual representation (Veerabadran et al., 25 Oct 2025).

The multimodal design is consequential because it tests not only fusion but selectivity. WAGIBench is structured so that models must exploit informative modalities when present and suppress distractors when extra context is irrelevant or noisy. This suggests that the benchmark probes context arbitration as much as raw multimodal aggregation.

4. Task definition and evaluation protocol

WAGIBench frames goal inference as an open-set language problem, but evaluates it under two complementary protocols: discriminative multiple-choice evaluation and generative open-ended goal generation (Veerabadran et al., 25 Oct 2025).

In the multiple-choice setting, each sample is converted into a question containing the observation, the reference goal, and $N = 3$ distractor goals. Two distractor regimes are used: similar distractors, sampled from goals semantically close to the reference, and dissimilar distractors, sampled from more distant goals. Sentence-BERT embeddings are used to map goals into a semantic space, after which distractors are sampled from percentile ranges of cosine similarity. A greedy diversity rule is then applied:

$$D_i = \begin{cases} sample(U(C)) & \text{if } i = 1 \ \arg\min_{c \in C \setminus D_{<i}} \left( \max_{j=1}^{i-1} sim(D_j, c) \right) & \text{if } 1 < i < 4, \end{cases}$$

where $C$ is the candidate distractor pool, $U$ is uniform sampling, and $sim$ is cosine similarity of Sentence-BERT embeddings. Each sample yields one similar MCQ and one dissimilar MCQ, for roughly 7k MCQs total. Models are prompted to return only the answer option letter (Veerabadran et al., 25 Oct 2025).

In the generative setting, models generate a goal directly. Scoring is performed by an LLM-as-judge rather than exact string matching. The judge is shown the contexts, the model’s predicted goal, and in some variants the reference goal and/or script cues. It assigns one of three relevance levels: 1.0 / 2 = very relevant, 0.5 / 1 = borderline relevant, and 0 / 0 = irrelevant. The appendix notes that integer tokens $\{0,1,2\}$ are used in the prompt for efficiency and then normalized to $S_{VA}$0 (Veerabadran et al., 25 Oct 2025).

For MCQs, the metric is standard classification accuracy. For generative outputs, the benchmark additionally includes a meta-evaluation of automatic judges against human judgments using a pairwise comparison accuracy criterion. For pairs of predicted goals on the same video, the evaluation checks whether the judge agrees with the human ordering $S_{VA}$1. Several judge variants are compared: Socratic, Cues, Reference, Socratic + Reference, Cues + Reference, plus Snap-MCQ and SBERT Similarity baselines. The strongest judge is Cues + Reference, with 76.8% alignment, effectively matching human-human agreement at 75.2%; plain Socratic context is weaker, and both Snap-MCQ and SBERT similarity perform substantially worse (Veerabadran et al., 25 Oct 2025).

The dual protocol is important because it separates closed-form recognition from open-ended intent articulation. A plausible implication is that WAGIBench evaluates not only whether a model can discriminate among plausible goals, but whether it can formulate a semantically relevant goal under the ambiguity and lexical variability of natural language.

5. Empirical findings

The central empirical result is that human performance exceeds model performance, particularly when distractors are semantically difficult. In the multiple-choice setting, humans achieve 93% accuracy on the harder similar-distractor MCQs, while the best vision-LLM reaches 84%. On dissimilar MCQs, human accuracy is near 97%. Across the full benchmark, performance improves with model scale: small models are substantially weaker, medium models improve, and larger models such as Qwen2.5-VL-72B and GPT-4.1 are strongest (Veerabadran et al., 25 Oct 2025).

In the generative setting, larger models again perform better, but the benchmark reports that they remain far from reliable. The best models produce relevant goals only about 55% of the time in the authors’ summary, and the top generative score in the reported table is around 0.55 for GPT-4.1, with large open models close behind (Veerabadran et al., 25 Oct 2025).

A major contribution of WAGIBench is its modality ablation analysis. When a modality is genuinely informative, adding it improves performance over vision alone; the benchmark reports modality-specific gains as large as 35% on MCQ and 30% on generative evaluation. This effect is especially pronounced for audio, where speech transcripts can provide clues not recoverable from vision alone (Veerabadran et al., 25 Oct 2025).

At the same time, extra irrelevant modalities hurt less than might be expected, particularly for larger models. The benchmark concludes that sufficiently large models can often ignore distractors. Digital and Longitudinal context yield smaller gains than audio because they have a lower signal-to-noise ratio. High-signal variants confirm this interpretation: $S_{VA}$2, which retains only relevant digital sub-states, improves by up to 12% over $S_{VA}$3, and $S_{VA}$4, which retains only positive-support longitudinal history, improves by up to 5.6% over $S_{VA}$5. The gap shrinks for large models, suggesting better noise filtering in complex contexts. When all modalities are provided together as $S_{VA}$6, large models can exploit the full context and disentangle useful signals from distractors, whereas smaller models often suffer interference (Veerabadran et al., 25 Oct 2025).

These results indicate that the benchmark is not saturated. The gap between strong human accuracy and materially weaker open-ended model relevance implies that current systems are substantially better at assisted recognition than at dependable latent-goal recovery in ecologically realistic wearable settings.

6. Significance, limitations, and benchmark identity

WAGIBench is designed to move evaluation from generic egocentric understanding toward assistive wearable goal inference, where the challenge is to infer what the user wants next from heterogeneous, partially noisy context rather than merely to recognize an action or caption a scene. Its stated significance rests on four properties: it contains ground-truth goals rather than proxy labels, includes multiple modalities that a wearable agent would actually observe, covers longitudinal user behavior, and explicitly tests whether models can exploit relevant context while ignoring distractors (Veerabadran et al., 25 Oct 2025).

The benchmark’s limitations are also explicit in its reported findings. Even the strongest systems remain below human performance, and generative relevance remains too low for robust deployment in assistive wearable agents. This suggests that the bottleneck is not only multimodal input availability but reliable latent-goal inference under ambiguity, noise, and open-ended language generation.

WAGIBench should also be distinguished from other recent benchmarks with superficially similar names. WBench evaluates interactive video world models across video quality, setting adherence, interaction adherence, consistency, and physics compliance, rather than egocentric goal inference for wearable assistance (Ying et al., 25 May 2026). WGSR-Bench evaluates wargame-based strategic reasoning for LLMs via environmental situational awareness, opponent modeling, and policy generation, rather than multimodal intent inference from wearable observations (Yin et al., 12 Jun 2025). The similarity in naming can obscure the fact that these benchmarks address different problem classes, data regimes, and evaluation targets.

Within its own domain, WAGIBench establishes goal inference as a standalone benchmark problem for wearable agents. The reported results suggest that multimodal models can use additional context when it is informative and often suppress irrelevant context when sufficiently large, but they still fall short of practical, human-level goal inference in realistic open-ended settings (Veerabadran et al., 25 Oct 2025).