EgoTools-Bench: Tool-Centric Egocentric Video Evaluation

• EgoTools-Bench is a diagnostic evaluation tool suite consisting of 1,000 multiple-choice questions across four tracks that assess multimodal models' understanding of egocentric video, which includes visual perception, tool grounding, and procedural dynamics.
• The benchmark engages over four problem tracks, such as affordance and causality, perception, procedural dynamics and spatial reasoning, each with 222 to 363 questions specifically examining properties like hand-tool-object geometry and object-state transitions.
• The evaluation protocol shows substantial performance uplifts when supervised fine-tuning is applied to reasoning abilities regarding tool use and procedural dynamics.

EgoTools-Bench is the diagnostic evaluation component of EgoTools, a suite for tool-centric understanding in real-world egocentric video. It comprises 1,000 English, eight-way multiple-choice questions distributed across four tracks: Affordance & Causality, Perception & Grounding, Procedural Dynamics, and Spatial Reasoning. The benchmark evaluates whether multimodal models can connect tool identity, visual evidence, hand–tool–object geometry, functional affordances, procedural order, object-state transitions, and causal outcomes from localized first-person video clips (Tian et al., 30 Sep 2026).

1. Position within the EgoTools suite

EgoTools consists of two complementary components. EgoTools-Data contains 646 egocentric videos totaling 100.37 hours, with 361,332 hierarchical captions, 6,519 tool-centric narrations, and supplementary spatial annotations. EgoTools-Bench is the evaluation component, containing 1,000 questions designed to isolate the reasoning demands of tool-mediated activity.

The benchmark addresses a limitation of conventional egocentric-video evaluation. Activity recognition, object recognition, episodic memory, planning, and general video question answering may identify that a person is cooking, repairing, or conducting an experiment without determining which tool is being used, why it is appropriate, what it contacts, how it changes the target, or what spatial configuration enables the action.

EgoTools-Bench treats tool use as a coupled reasoning problem involving:

• visual perception and grounding;
• hand–tool–object geometry;
• functional affordances and tool choice;
• temporal and procedural organization;
• object-state transitions; and
• causal reasoning about action effects.

The benchmark is intended to evaluate both low-level visual grounding and higher-level interpretation. A model must often infer a tool’s role from its physical properties, the target object, the current procedural state, and the consequences of prior actions rather than from explicit textual descriptions.

2. Video corpus and benchmark construction

The full EgoTools-Data corpus was collected across seven domains: kitchen, classroom, research laboratory, repair workshop, craft, office, and household. The released canonical video is a rectified front-left RGB stream recorded with the HOMIE head-mounted multimodal recording device. Videos have a resolution of $1024\times1024$, a frame rate of 20 FPS, and synchronized audio. HOMIE also records multiple fisheye views and auxiliary motion, synchronization, pose, depth, and geometry-related signals, although raw fisheye videos are not publicly released.

Data collection used two regimes. Structured task-guided recording gave participants task goals and important procedural steps. Open-ended participant-driven recording provided only a broad goal, allowing participants to choose or substitute tools, recover from failed attempts, and adapt to changing object states. The combination was intended to provide controlled procedural coverage alongside natural variation in tool choice and manipulation.

Before instruction-tuning examples and benchmark questions were produced, recordings were partitioned at the source-video level. The benchmark-reserved pool contains 40.34 hours, while the remaining videos form the training pool. The separation was applied before derivative annotations were created, so benchmark-source videos and their captions, narrations, synthetic question-answer items, grounding annotations, and other derivatives were excluded from the training corpus.

The benchmark is not domain-balanced. Kitchen recordings account for 837 of the 1,000 questions, while classroom, research laboratory, repair workshop, craft, office, and household recordings account for substantially smaller portions. Consequently, the paper emphasizes capability-level rather than domain-level analysis.

Annotation pipeline

Dense captions were generated through a hierarchical streaming procedure using Gemini-3-Flash. Videos were divided into five-minute segments; 32 uniformly sampled frames were used to produce global context captions; sequential five-second chunks were captioned using previous captions as temporal memory; and the results were aggregated into one-minute and five-minute summaries.

Tool-centric narrations were separately provided for meaningful tool-use events. They describe the tool, possible alternatives, target object, performed action, reason for tool selection, and resulting state or outcome. A reference narration template is:

“I am using [the] [property] [tool] [location/identity cue] rather than [the] [property] [alternative tool] [location/identity cue] to [action] on/for [target] because [reason].”

Examples include using narrow chopsticks rather than a spoon to reach into a narrow jar opening, a flat spatula rather than chopsticks to lift a soft egg, and a pipette rather than a large container to transfer a small amount of liquid.

Selected focal tools and objects were annotated with 2D grounding points on keyframes. These points initialized longer-range tracking. The geometric pipeline combined HOMIE camera poses and depth with 3D Gaussian Splatting, SAM2 mask propagation, VLM-based tracking verification, segmentation reinitialization when tracking failed, and lifting of tracked masks into 3D. One hundred spatial questions were generated from these annotations and subsequently human-verified.

3. Benchmark format and evaluation tracks

Every item is an English, eight-way multiple-choice question with one correct answer and seven distractors. Each question is paired with a localized video clip centered on an anchor moment. The average temporal span is 4.19 minutes, and most clips last at least three minutes. The benchmark contains 900 human-crafted questions and 100 spatial questions generated from 3D annotations and human verified. Random-choice accuracy is $1/8=12.5\%$.

Track	Abbreviation	Questions	Primary capability
Affordance & Causality	AC	363	Functional and causal tool understanding
Perception & Grounding	PG	236	Precise visual grounding
Procedural Dynamics	PD	222	Temporal and procedural organization
Spatial Reasoning	SR	179	Egocentric geometry and spatial relations

Affordance & Causality

AC evaluates whether a model can connect a tool to a goal, material, object state, and outcome. Questions concern why a tool was selected, why one tool was substituted for another, what a tool affords, whether it is suitable for a material or target, how an action establishes a precondition, and what physical or procedural consequence follows.

Typical cases include explaining why narrow chopsticks are appropriate for a narrow opening, why a flat spatula is suitable for lifting a soft egg, or why a pipette is appropriate for transferring a small volume. AC therefore combines tool choice, affordance reasoning, substitution, and causal inference over object-state changes.

Perception & Grounding

PG measures grounding in directly visible evidence. It covers tool and object identity, visually similar tools, shape, size and material, contact relations, hand–tool–object contact points, quantities, local object states, and localized changes.

A model must distinguish which of two similar tools is actually being used, identify the object contacted by the tool, determine whether an object has been cut, opened, bent, loosened, or filled, and recognize which local portion has changed. General knowledge that a person is cooking is insufficient to answer these questions.

Procedural Dynamics

PD evaluates temporal organization and procedural progress. It covers tool–action ordering, preparation and workspace staging, events before and after an anchor moment, fine-grained manipulation sequences, task progress, next-step prediction, and state-dependent procedural changes.

Examples include determining whether an ingredient was cut before or after a pan was heated, identifying which tool was used first, selecting the next step, distinguishing preparation from execution, and recognizing a tool switch following an intermediate state change.

Spatial Reasoning

SR evaluates egocentric hand–tool–object geometry and spatial relations, including relative position, alignment, containment, support, contact, placement, depth, motion trajectories, and spatial state changes.

Questions may ask whether a tool is inside, above, beside, or aligned with an object; whether it supports or contacts an object; which object is nearer in depth; how a tool moves relative to a target; or whether the hand, tool, and object form the configuration required for an action.

4. Quality control and evaluation protocol

The benchmark uses a fix-first annotation and quality-control process. Collectors wrote task-aware procedural and causal questions, while external annotators identified visually inferable but difficult reasoning points. Domain-specific recordings, including wet-laboratory videos, were assigned to annotators with relevant expertise.

Quality control checked for malformed questions, missing fields, duplicates, temporal misalignment, inconsistent tool identity, missing visual evidence, weak distractors, answer–option mismatch, first-person leakage, personally identifying information, inappropriate full-video clip usage, and ambiguous or non-unique answers. Repairable issues were corrected through deterministic rules or minimal model-assisted edits; other items were routed to human review.

All items were normalized to exactly eight answer choices. Anti-shortcut checks addressed “all of the above” and “none of the above” options, duplicate or near-duplicate choices, token-set permutations, substring containment, excessive question–answer lexical overlap, answer-position imbalance, and systematic answer-length bias.

For an answer-length diagnostic, the standardized deviation is:

$$z=\frac{L_{\mathrm{ans}}-\mu_{\mathrm{dist}}}{\sigma_{\mathrm{dist}}},$$

where $L_{\mathrm{ans}}$ is the correct-answer length and $\mu_{\mathrm{dist}}$ and $\sigma_{\mathrm{dist}}$ are the mean and standard deviation of distractor lengths. Items with $|z|>1.5$ were flagged. The finalized benchmark has a corpus-level mean standardized deviation of approximately $\bar z\approx -0.15$, and every item satisfies $|z|\leq1.5$. Answer choices were shuffled using a deterministic seed derived from the question identifier.

An independent reliability study evaluated 250 questions using five annotators, with two independent judgments per question. Annotators saw only the video, question, and answer choices. Agreement with the original key was 89.8% across 500 judgments; AC agreement was 86.5%; exact agreement between the two assigned annotators was 88.4%; and 94.4% of items were judged to have a uniquely best answer. Fourteen items, or 5.6%, were flagged as ambiguous or multiply plausible and were manually reviewed.

The public evaluation does not provide corrected narrations, narration audio, dense captions, timestamps beyond the supplied clip, grounding annotations, or other annotation metadata. Audio-capable models receive synchronized non-narration audio.

Metrics

The primary metric is multiple-choice accuracy:

$$\operatorname{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat y_i=y_i].$$

Track-specific accuracy is computed over the questions in each track. Overall accuracy is item-level, or micro-averaged, rather than an unweighted average of the four track scores.

For error-overlap analysis, the benchmark uses Jaccard overlap between two models’ error sets:

$1/8=12.5\%$0

5. Model results and diagnostic findings

The evaluation includes human experts, proprietary multimodal models, open-source instruct models, and open-source thinking or reasoning variants. Most open-source instruct models receive 64 uniformly sampled frames. Thinking models receive 64 or 512 frames depending on the model, while Gemini models receive video at 1 FPS. Audio-capable models receive synchronized non-narration audio.

Model	AC	PG	PD	SR	Overall
Human Expert	82.1	85.2	83.3	82.7	83.2
Gemini-3.1-Pro	69.7	51.7	72.1	74.9	66.9
Gemini-3-Flash	64.5	50.0	73.0	72.6	64.4
Qwen3-VL-8B-Instruct	46.1	45.9	57.1	55.4	50.0
GLM4.1V-Thinking	46.1	50.0	57.1	53.3	50.7
LLaVA-OneVision-7B	37.6	37.1	42.9	37.0	38.9
EgoTools-8B, fine-tuned	55.4	59.7	68.1	44.4	60.9

Gemini-3.1-Pro achieves 66.9% overall accuracy but only 51.7% on PG. Its relatively higher AC, PD, and SR scores do not imply equally reliable low-level visual grounding. The best zero-shot open-source instruct model is Qwen3-VL-8B-Instruct at 50.0%, while the strongest open-source thinking result is GLM4.1V-Thinking at 50.7%. All remain below human performance of 83.2%.

Qwen3-VL-8B-Instruct scores 69.0% on EgoSchema and 61.0% on EgoThink, but 50.0% on EgoTools-Bench. This difference supports the distinction between broad first-person activity understanding and tool-centric physical reasoning.

The benchmark reveals a relative strength in high-level contextual and procedural interpretation. Models perform better on broad procedural organization, task-order cues, common tool-function priors, physical commonsense, coarse spatial relations, and familiar action consequences.

Persistent weaknesses include distinguishing visually similar tools, locating exact contact points, identifying which object changed, recognizing subtle local state transitions, integrating tool choice with material and geometry, inferring unstated affordances, and tracking hand–tool–object relations across time.

The low PG score is particularly diagnostic. A model may know that a spatula is generally used in cooking while failing to determine whether the visible object is a spatula, whether it is contacting the egg, or whether the egg has already changed state.

Supervised fine-tuning

The authors constructed 184,679 instruction-tuning examples exclusively from the non-benchmark training pool. These included AC, PG, PD, SR, episode-level multiple-choice, general video question-answering, dense-captioning or narration-completion, and single-image open-ended question-answering examples.

Full-parameter supervised fine-tuning of the Qwen3-VL-8B-Instruct backbone improved overall benchmark accuracy from 50.0% to 60.9% under source-video-disjoint evaluation. AC improved by 9.3 percentage points, PG by 13.8, and PD by 11.0, while SR declined by 11.0 points. The result indicates that tool-centric supervision improves several capabilities but does not automatically yield robust spatial reasoning.

Input-evidence ablations

Using the same Qwen3-VL-8B-Instruct backbone and deterministic decoding, ordered 64-frame input achieved 51.7% overall accuracy, compared with 49.3% for shuffled frames, 46.9% for a single middle frame, and 42.7% for text-only input.

Ordered video therefore improved over text-only input by 9.0 percentage points. PG showed the largest dependence on visual evidence, reaching 50.0% with ordered frames and 36.1% with text-only input. A single middle frame reduced SR by 11.9 points, indicating that spatial reasoning depends on multiple frames. Frame shuffling reduced overall accuracy by 2.4 points, while PD declined by 3.2 points, making procedural dynamics the most sensitive track to temporal order.

Text-only performance remained above the 12.5% chance level, suggesting residual commonsense or answer-choice cues despite anti-shortcut filtering. Nevertheless, visual input substantially improves performance.

6. Interpretation, limitations, and significance

EgoTools-Bench establishes a distinction between recognizing an activity and understanding how tools mediate that activity. Its central representational chain is:

$1/8=12.5\%$1

The benchmark shows that these components are not equally difficult. Models can often infer broad procedure or familiar tool functions while failing at precise visual grounding, local state changes, and contact geometry. Fine-tuning on EgoTools-Data improves overall performance but reduces SR, indicating that procedural and verbal supervision does not automatically produce reliable geometric representations.

The benchmark has several limitations. Its questions are heavily concentrated in kitchen videos, with 837 of 1,000 questions from that domain. The corpus contains approximately 100 hours, does not exhaustively cover tools, cultures, environments, professional practices, or participant populations, and includes subjectivity in human annotation despite high reliability. The source-video split prevents direct leakage from benchmark videos into training, but input formats differ across models: Gemini uses 1 FPS video, most open-source models use 64 frames, and thinking models may use 512 frames.

The benchmark also has possible residual language shortcuts, as indicated by text-only accuracy of 42.7%. Its spatial supervision remains insufficiently effective, since supervised fine-tuning reduces SR. Auxiliary annotations, including narration audio, grounding, and 3D trajectories, may be released incrementally subject to consent and privacy review. The project reports informed consent, anonymization, release restrictions, and participant-removal procedures, but no formal institutional ethics review was obtained.

EgoTools-Bench evaluates understanding from video rather than direct physical execution. It therefore does not establish whether a model can physically select, grasp, orient, manipulate, or safely use the inferred tool. Its primary contribution is diagnostic: it separates visual grounding, affordance and causality, procedural dynamics, and spatial reasoning within a tool-centric egocentric setting.

The benchmark supports future work on focal-tool and contact-point grounding, persistent object-state representations, depth-aware video reasoning, tool substitution and failed-attempt modeling, active temporal search, 3D supervision, and multimodal training that combines captions, narrations, trajectories, and causal explanations. Its results indicate that embodied agents should justify not only what action occurs, but why a tool is appropriate, what constraint it satisfies, and what physical effect it produces.