--- title: 'EgoTools: Dataset for Tool-Centric Reasoning' url: https://www.emergentmind.com/topics/egotools type: topic --- # EgoTools: Dataset for Tool-Centric Reasoning EgoTools is a dataset-and-benchmark suite for tool-centric reasoning in real-world egocentric video. Introduced in “EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos” [2609.39378], it treats tool-mediated interaction—not merely activity recognition—as the primary unit of analysis. The suite combines **EgoTools-Data**, a 100.37-hour corpus of multimodal egocentric recordings, with **EgoTools-Bench**, a source-video-disjoint diagnostic benchmark of 1,000 multiple-choice questions. Its target capabilities span visual grounding, affordance and causal reasoning, procedural dynamics, and spatial relations among hands, tools, and target objects. ## 1. Conceptual scope and motivation EgoTools addresses the distinction between recognizing an activity and understanding how tools enable that activity. Conventional egocentric video systems may identify cooking, repair, experimentation, or crafting, yet fail to determine which tool is being used, why it was selected, which object it contacts, what geometric relation holds among hand, tool, and target, or how the manipulation changes the target’s state. The suite formalizes tool use through a causal chain: $$ \text{tool choice} \rightarrow \text{manipulation} \rightarrow \text{object-state change} \rightarrow \text{task consequence}. $$ This formulation requires several forms of reasoning: - **Tool identity and selection**: identifying the tool and explaining why it is preferred over an alternative. - **Affordance reasoning**: relating tool properties to physical constraints, material properties, reachability, precision, or desired outcomes. - **Hand-tool-object geometry**: determining contact, support, containment, alignment, relative depth, and manipulation configuration. - **Procedural reasoning**: tracking preparation, step transitions, tool switching, recovery, and task progress. - **Causal reasoning**: connecting a tool action to force, contact, and a resulting change in the target object. The motivating examples include using narrow chopsticks rather than a spoon to reach into a narrow jar opening, using a flat spatula rather than chopsticks to support a soft egg without tearing it, and using a pipette rather than a larger container to transfer a small liquid quantity with finer volume control. These cases require more than object recognition because the relevant answer depends on affordances, physical constraints, alternatives, and intended outcomes. The first-person viewpoint is advantageous because hands, tools, target objects, and consequences often occupy the central visual field. It also introduces substantial difficulty through occlusion, rapid manipulation, changing viewpoints, small objects, and the need to integrate evidence over long temporal intervals. ## 2. EgoTools-Data ### Scale and domains EgoTools-Data contains: - **646 egocentric videos** - **100.37 hours** of video - **361,332 hierarchical captions** - **6,519 tool-centric narrations** - **Seven tool-use domains** | Domain | Videos | Hours | Captions | Narrations | |---|---:|---:|---:|---:| | Kitchen | 276 | 32.37 | 116,532 | 2,113 | | Classroom | 74 | 6.86 | 24,696 | 467 | | Research laboratory | 61 | 30.92 | 111,312 | 1,887 | | Repair workshop | 84 | 12.34 | 44,424 | 763 | | Craft | 67 | 10.17 | 36,612 | 657 | | Office | 30 | 2.83 | 10,188 | 213 | | Household | 54 | 4.88 | 17,568 | 419 | | **Total** | **646** | **100.37** | **361,332** | **6,519** | The corpus covers both everyday and expertise-intensive activities in kitchens, workshops, laboratories, classrooms, offices, craft spaces, and household environments. Data collection uses two complementary protocols. In **structured task-guided recordings**, participants follow predefined task goals and key steps while performing manipulations naturally. These recordings provide clearer task boundaries, observable state changes, and controlled coverage of hand-tool-object interactions. In **open-ended participant-driven recordings**, participants receive a broad objective and independently determine how to accomplish it. This setting captures spontaneous tool selection, substitution, failed attempts, recovery, adaptation to object changes, and individual procedural variation. ### Recording system and modalities Recordings use **HOMIE**, a head-mounted multimodal device providing: - four synchronized fisheye camera views; - audio; - auxiliary motion and synchronization signals; - geometry-related information supporting camera pose, depth, reconstruction, and tracking. The canonical released stream is a rectified front-left RGB view at $1024 \times 1024$ resolution and 20 FPS, synchronized with audio. Raw fisheye streams are not publicly released. Audio is synchronized with the video, and evaluation models that support audio receive synchronized non-narration audio, but not narration audio, corrected narrations, dense captions, or annotation metadata. Original narration audio is released only for examples covered by the relevant authorization and privacy review. Not all 3D and trajectory assets are available for every example; selected assets are released incrementally after annotation, authorization, and privacy review. ### Hierarchical captions The captioning pipeline uses Gemini-3-Flash in a streaming hierarchical process: 1. Each video is divided into 5-minute clips. 2. Thirty-two uniformly sampled frames generate a global context caption. 3. Each clip is captioned sequentially in 5-second chunks. 4. The first chunk uses the global caption. 5. Subsequent chunks use the previous caption as memory for temporal consistency. 6. Captions are aggregated into 1-minute windows and 5-minute summaries. Object recognition on key frames supports post-hoc hallucination checking. The resulting captions supervise visible action description, task-progress tracking, and long-context maintenance. ### Tool-centric narrations The reasoning-heavy narrations emphasize: - tool identity; - tool selection; - alternative tools; - intended use; - target object; - physical rationale; - effects on the target or task state; - tool switching; - adaptation to object changes; - recovery after unsuccessful attempts. A reference template is: > “I am using [the] [property] [tool] [location/identity cue] rather than [the] [property] [alternative tool] [location/identity cue] to [action] on/for [target] because [reason].” Selected narrations contain 2D grounding points on key frames for focal tools or target objects. Narrations may originate in English, Korean, or Chinese; non-English narrations are transcribed, translated, and normalized into corrected English, with language tags preserving the original language. ### Supplementary 3D and 4D information The geometry pipeline combines raw depth and camera-pose information from HOMIE with 3D Gaussian Splatting for multi-view-consistent scene reconstruction. Temporally coherent depth maps are generated from the reconstruction. Narration points initialize 2D object segmentation, and SAM2 propagates masks through videos. A vision-language model checks later grounding points and tracking consistency; tracking is reinitialized and refined when drift or semantic mismatch occurs. Masks are lifted into 3D using depth and camera poses, supporting: - 3D object boxes; - long-horizon object trajectories; - 4D object tracking; - spatial question generation; - object motion analysis; - relative position; - contact; - support; - containment; - alignment; - depth; - tool-object interaction dynamics; - state changes over time. ## 3. Instruction-tuning corpus and benchmark construction ### EgoTools-derived training corpus The authors derive an instruction-tuning corpus of **184,679 examples** exclusively from training-pool videos. Its composition is: - **69,760** examples supervising the four principal EgoTools capabilities: - Affordance & Causality: 14,175 - Perception & Grounding: 14,755 - Procedural Dynamics: 22,943 - Spatial Reasoning: 17,887 - **69,554** episode-level multiple-choice examples; - **31,281** general video QA examples; - **9,084** dense-captioning and narration-completion examples; - **5,000** single-image open-ended QA examples. The examples include grounding questions about tool identity, contacted objects, local state changes, and hand-tool-object arrangement. They also include reasoning questions about tool choice, causal effects, future actions, and procedural progress. Synthetic candidates are filtered for visual answerability, temporal grounding, a unique best answer, resistance to textual shortcuts, and balanced answer letters. ### EgoTools-Bench EgoTools-Bench contains: - **1,000 questions**; - exactly **eight answer choices per question**; - a chance accuracy of **12.5%**; - **900 human-crafted questions**; - **100 spatial questions** generated from 3D annotations and human-verified; - an average temporal span of **4.19 minutes**. The benchmark is constructed from a **40.34-hour source-video pool** reserved for evaluation. Training and evaluation are disjoint at the source-video level: no benchmark video, clip, caption, narration, synthetic QA pair, grounding annotation, or trajectory annotation derived from a benchmark-reserved video is included in training. The benchmark is domain-imbalanced: - Kitchen: 837 questions - Research laboratory: 99 - Craft: 23 - Repair workshop: 16 - Classroom: 15 - Office: 6 - Household: 4 Consequently, the authors emphasize capability-level rather than domain-level analysis. ### Quality control All questions undergo a fix-first quality-control process examining structural validity, video grounding, temporal alignment, tool identity, duplicates, distractor quality, missing visual evidence, text-only solvability, answer-position bias, answer-length bias, and lexical shortcuts. Option order is deterministically shuffled using a seed derived from each QA identifier. Answer-length control uses: $$ z=\frac{L_{\mathrm{ans}}-\mu_{\mathrm{dist}}}{\sigma_{\mathrm{dist}}}, $$ where $L_{\mathrm{ans}}$ is the character length of the correct answer, while $\mu_{\mathrm{dist}}$ and $\sigma_{\mathrm{dist}}$ are the mean and standard deviation of distractor lengths. Items with $\lvert z\rvert > 1.5$ are flagged for repair. An independent reliability study used 250 questions and five annotators. Agreement with the original answer key was 89.8%, agreement on Affordance & Causality was 86.5%, exact agreement between assigned annotators was 88.4%, and 94.4% of items were judged to have a uniquely best answer. Fourteen ambiguous items were revised or replaced. The principal evaluation metric is multiple-choice accuracy: $$ \text{Accuracy} = \frac{\text{number of correctly answered questions}} {\text{number of evaluated questions}} \times 100. $$ ## 4. Benchmark tracks EgoTools-Bench divides tool-centric reasoning into four tracks. ### Affordance & Causality This track contains **363 questions**. It evaluates whether a model can understand the functional and causal role of tools, including: - why a tool is chosen; - why a tool is replaced; - what a tool affords; - whether an alternative is feasible; - how a tool establishes a precondition; - what effect an action has on a target; - how a manipulation advances the task. The track requires a model to connect tool properties and physical constraints to the actor’s goal. It therefore distinguishes functional reasoning from generic knowledge about an object category. ### Perception & Grounding This track contains **236 questions** and evaluates: - tool identity; - target-object identity; - attributes; - object state; - quantities; - contact points; - local state changes; - the currently active tool. The track is grounded in direct visual evidence and is especially diagnostic of fine-grained perception. Gemini-3.1-Pro achieves only 51.7% on this track, despite achieving 66.9% overall. The discrepancy indicates that procedural or commonsense competence does not guarantee precise localization of tools, objects, contact points, or state changes. ### Procedural Dynamics This track contains **222 questions** concerning: - tool-action ordering; - step transitions; - preparation; - workspace staging; - fine-grained manipulation; - completed actions; - the current procedural step; - the next action; - changes following an object-state transition. The track requires temporal integration rather than frame-local recognition. Models must distinguish whether a tool is being selected, used, set aside, or prepared for a subsequent step. ### Spatial Reasoning This track contains **179 questions** addressing: - hand-tool-object relations; - position; - alignment; - containment; - support; - contact; - depth; - relative motion; - placement; - interaction geometry. Some questions use 3D-derived annotations and trajectories. The track is distinct from object recognition because the relevant geometric evidence may be distributed across several frames and may require inference of three-dimensional relations from an egocentric viewpoint. ## 5. Experimental results ### Human and model performance Human experts achieve an overall accuracy of **83.2%**. | System | Affordance & Causality | Perception & Grounding | Procedural Dynamics | Spatial Reasoning | Overall | |---|---:|---:|---:|---:|---:| | Human | 82.1 | 85.2 | 83.3 | 82.7 | **83.2** | | Gemini-3.1-Pro | 69.7 | **51.7** | 72.1 | 74.9 | **66.9** | | Qwen3-VL-8B-Instruct | 46.1 | 45.9 | 57.1 | 55.4 | **50.0** | | EgoTools-8B after SFT | 55.4 | 59.7 | 68.1 | 44.4 | **60.9** | Gemini-3.1-Pro is the strongest reported proprietary model, with 66.9% overall accuracy. Gemini-3-Flash achieves 64.4%, and Gemini-3.1-Flash-Lite achieves 55.4%. Representative open-source overall scores include: - GLM4.1V-Thinking: 50.7%; - InternVL3.5-8B-Instruct: 48.7%; - Qwen2.5-Omni-7B-Instruct: 47.1%; - InternVL3-8B-Instruct: 47.1%; - Qwen3-VL-4B-Instruct: 45.7%; - Qwen2.5-VL-7B-Instruct: 44.3%; - Qwen2-VL-7B-Instruct: 44.2%; - MiMo-VL-7B-SFT: 44.2%; - LLaVA-Video-7B-Qwen2: 39.8%; - LLaVA-OneVision-7B: 38.9%. Qwen3-VL-8B-Instruct reaches 69.0% on EgoSchema, 61.0% on EgoThink, and 42.3% on EgoPlan-Bench, but only 50.0% on EgoTools-Bench. This difference indicates that broad egocentric benchmark performance does not automatically transfer to tool-centric reasoning. ### Training-resource validation Full supervised fine-tuning of Qwen3-VL-8B-Instruct on the 184,679-example training corpus raises overall accuracy from **50.0% to 60.9%**, a gain of 10.9 percentage points under strict source-video separation. Track-level changes are: - Affordance & Causality: $+9.3$ points; - Perception & Grounding: $+13.8$ points; - Procedural Dynamics: $+11.0$ points; - Spatial Reasoning: $-11.0$ points. The largest gain occurs in Perception & Grounding, indicating that dense grounding and tool-state supervision can improve visual evidence alignment. Spatial Reasoning decreases despite explicit spatial training examples, suggesting that geometric integration remains unresolved. The fine-tuning configuration updates the language-model component while freezing the visual encoder and multimodal projector. It uses AdamW, a learning rate of $2.3\times10^{-6}$, a constant schedule without warmup, one epoch, per-device batch size 1, gradient accumulation over 16 steps, eight GPUs, an effective batch size of 128, 1,443 optimizer steps, bfloat16, DeepSpeed ZeRO Stage 3, FlashAttention, gradient checkpointing, 64 uniformly sampled frames, and a maximum multimodal sequence length of 8,192 tokens. On other benchmarks, the fine-tuned model reaches 67.6% on EgoSchema, 35.2% on EgoPlan-Bench, and 64.0% on EgoThink. These results do not establish uniform transfer improvements across egocentric tasks. ### Input-evidence ablations A matched Qwen3-VL-8B-Instruct evaluation compares different evidence conditions: | Input condition | Overall | Affordance & Causality | Perception & Grounding | Procedural Dynamics | Spatial Reasoning | |---|---:|---:|---:|---:|---:| | Ordered 64 frames | 51.7 | 47.4 | 50.0 | 57.1 | 57.6 | | Shuffled 64 frames | 49.3 | 45.5 | 47.4 | 53.9 | 56.5 | | Middle frame only | 46.9 | 44.7 | 45.9 | 51.0 | 45.7 | | Text-only | 42.7 | 39.0 | 36.1 | 51.4 | 47.8 | Ordered multi-frame input outperforms text-only input by 9.0 points. A single middle frame reduces overall accuracy by 4.8 points, while Spatial Reasoning falls by 11.9 points. Shuffling frames reduces overall accuracy by 2.4 points, with Procedural Dynamics showing a 3.2-point ordered-versus-shuffled difference. The text-only score remains above the 12.5% chance level, indicating residual language or answer-option cues. Nevertheless, the ordered-frame advantage demonstrates that the benchmark cannot be solved solely through tool commonsense or linguistic priors. For Gemini-3.1-Pro, accuracy is 65.2% on implicit questions and 72.9% on questions explicitly mentioning the relevant tool-use cue. For Qwen3-VL-8B-Instruct, the corresponding results are 49.1% and 53.1%. The gap indicates that models perform better when the question itself identifies the relevant relation, whereas implicit questions more strongly test autonomous discovery of tool-centric evidence. ## 6. Limitations, ethics, and research significance ### Model and benchmark limitations The suite has several limitations: - EgoTools-Bench is strongly concentrated in kitchen videos and is not domain-balanced. - The corpus does not cover all tools, cultures, participant populations, environments, or professional procedures. - 3D and trajectory assets are available only for selected examples. - A benchmark of 1,000 questions cannot represent the full diversity of tool-centric reasoning. - The text-only ablation remains above chance, indicating residual answer-option or commonsense shortcuts. - Explicit spatial supervision did not improve the fine-tuned model’s Spatial Reasoning score. - Evaluation input conditions differ across models: many open-source systems use 64 frames, Gemini uses 1 FPS input, and thinking models may use 512 frames. - The benchmark evaluates multiple-choice accuracy and therefore does not directly measure continuous control, real-time interaction, policy execution, or downstream task success. The observed decrease in Spatial Reasoning after fine-tuning is particularly significant. It suggests that adding textual and grounding supervision does not automatically produce robust geometric representations. Tool-centric reasoning may require explicit spatial memory, 3D scene representations, contact modeling, or architectures designed to integrate geometry and temporal state. ### Privacy and data governance The reported safeguards include voluntary participation, informed consent covering recordings and possible public release, exclusion of names and contact information, removal of sensitive or accidentally captured material, participant exclusion requests under the release policy, research-use licensing, and restrictions against participant identification, biometric profiling, and surveillance. Raw fisheye videos are not publicly released. Original narration audio, selected grounding annotations, and 3D trajectories are released only when authorization and privacy review permit. The authors state that no formal IRB or institutional ethics approval was sought for the EgoTools data collection, and that participants received no monetary compensation for recordings, narrations, or annotations. These conditions are material to interpretation of the dataset’s governance and to future extensions involving household, workplace, or professional environments. ### Research significance EgoTools establishes tool use as a compound diagnostic problem involving perception, affordances, geometry, procedures, and causality. Its results show that current multimodal video models are substantially better at broad activity understanding than at grounding tool use in precise visual evidence. Gemini-3.1-Pro reaches 66.9% overall accuracy but only 51.7% on Perception & Grounding, while human experts reach 83.2% overall. Supervised fine-tuning on EgoTools-derived data raises Qwen3-VL-8B-Instruct from 50.0% to 60.9% under source-video-disjoint evaluation, although Spatial Reasoning declines. The suite therefore supports two complementary research directions. EgoTools-Data provides multimodal supervision for training models to represent tool identity, affordances, hand-tool-object geometry, procedural state, and causal effects. EgoTools-Bench provides a controlled evaluation of whether those representations transfer to held-out tool-use episodes. Its principal methodological implication is that progress in egocentric understanding should be measured not only by activity recognition or caption quality, but also by the ability to explain how a tool is selected, how it interacts with a target, what state change results, and how that change advances the task. Source: https://www.emergentmind.com/topics/egotools