EgoTools-Data: Multimodal Corpus for Tool-Centric Reasoning

• EgoTools-Data comprises 646 videos and 100+ hours of multimodal data, which includes 361,332 hierarchical captions, 6,519 tool-centric narrations, and supplemental 3D/4D information.
• The corpus focuses on the relationship among tool selection, spatial arrangements, procedural progress, and causal consequences, specifically designed for tool-centric reasoning and affordance understanding in egocentric videos.
• EgoTools-Data is used to evaluate perception and grounding within instructions, procedural dynamics, spatial and geometrical awareness, toxicity analysis, and affordance–causality questioning.

EgoTools-Data is a multimodal corpus for tool-centric reasoning in real-world egocentric videos. It contains 646 videos totaling 100.37 hours, 361,332 hierarchical captions, 6,519 tool-centric narrations, and supplementary 3D and 4D information. Its organizing unit is not merely an activity or object, but the relationship among tool selection, affordance, hand–tool–object geometry, procedural progress, target-object state, and causal consequence. EgoTools-Data provides the training corpus for EgoTools-Bench, a diagnostic benchmark of 1,000 multiple-choice questions spanning affordance–causality, perception–grounding, procedural dynamics, and spatial reasoning (Tian et al., 30 Sep 2026).

1. Conceptual scope and research objectives

EgoTools-Data addresses the problem of understanding tool-mediated activity from an egocentric viewpoint. The dataset is designed around questions such as which tool is being used and why, what object or material it contacts, how the hand, tool, and target object are spatially arranged, how the target changes state, which procedural step follows, and what causal effect the manipulation produces.

The paper distinguishes tool-centric reasoning from conventional egocentric activity recognition and generic video question answering. Existing resources such as EPIC-KITCHENS, Ego4D, Ego-Exo4D, Assembly101, MECCANO, and HOI4D contain tools or tool-mediated actions, but primarily emphasize activities, objects, actions, hand–object interactions, or procedural events. EgoTools-Data instead makes the tool the explanatory unit connecting physical configuration, functional purpose, action, and outcome.

Two deficiencies motivate the corpus:

1. Evaluation gap: existing benchmarks rarely isolate tool-mediated reasoning from broader activity understanding.
2. Supervision gap: existing corpora rarely connect tool selection, target-object state, manipulation context, and causal consequences through dense annotations.

The dataset is intended to support perception and grounding, affordance reasoning, procedural understanding, spatial reasoning, next-action prediction, object-state tracking, tool substitution, failure recovery, and embodied-agent learning. Its central causal structure can be represented as:

$$\text{goal}+\text{material/object state}+\text{tool affordance} \rightarrow \text{tool choice and manipulation} \rightarrow \text{state change and consequence}.$$

EgoTools-Data and EgoTools-Bench serve different functions. EgoTools-Data is the large multimodal training corpus, whereas EgoTools-Bench is derived from source videos reserved before construction of training annotations and provides diagnostic evaluation.

2. Corpus scale, domains, and collection contexts

The appendix reports 646 videos with a total duration of 100.37 hours. The corpus contains 361,332 hierarchical captions and 6,519 tool-centric narrations. A separate 40.34-hour source-video pool is reserved for benchmark construction. The benchmark contains 1,000 questions, including 900 human-crafted questions and 100 human-verified spatial questions.

Statistic	EgoTools-Data
Videos	646
Duration	100.37 hours
Hierarchical captions	361,332
Tool-centric narrations	6,519
Collection domains	7
Benchmark-reserved duration	40.34 hours
Benchmark questions	1,000

The seven domains are kitchen, classroom, research laboratory, repair workshop, craft, office, and household. Their distributions are highly uneven: kitchen recordings dominate the corpus and the benchmark. EgoTools-Bench is therefore not domain-balanced and should not be treated as a representative sample of all seven domains.

Domain	Videos	Hours	Captions	Narrations	Benchmark questions
Kitchen	276	32.37	116,532	2,113	837
Classroom	74	6.86	24,696	467	15
Research laboratory	61	30.92	111,312	1,887	99
Repair workshop	84	12.34	44,424	763	16
Craft	67	10.17	36,612	657	23
Office	30	2.83	10,188	213	6
Household	54	4.88	17,568	419	4
Total	646	100.37	361,332	6,519	1,000

Data were collected in kitchens, workshops, laboratories, classrooms, offices, and daily-living spaces at universities and research sites in Asia. The corpus combines daily activities—such as cooking, household work, craft, repair, and office tasks—with expertise-intensive procedures, including laboratory, fabrication, educational, and specialized manual activities. Participants included graduate and undergraduate students from diverse disciplinary backgrounds; specialized recordings or reviews were performed by people familiar with the corresponding procedures.

Two collection settings provide complementary forms of variation. In structured task-guided recording, participants follow a predefined goal and sequence of key procedural steps without being scripted frame by frame. In open-ended participant-driven recording, participants receive only a broad topic or goal and choose how to complete it. The latter setting captures spontaneous tool selection, substitution, repeated attempts, error recovery, adaptation to changing object states, and alternative manipulation strategies.

The corpus does not report a total subject count, exhaustive tool-category count, or exhaustive object-category count. It documents an anonymized subset of contributors consisting of 16 participants or narrators, 10 question-answer annotators, two question-answer reviewers, three data curators, one grounding annotator, and two trajectory annotators. These role counts are non-exclusive and do not necessarily represent every contributor.

3. Recording system and multimodal representation

Recordings were made using HOMIE, a head-mounted multimodal recording device from Ropedia. HOMIE captures four synchronized fisheye camera views, audio, and auxiliary motion and synchronization signals. These data support view rectification, temporal alignment, camera-pose estimation, depth processing, scene reconstruction, hand and object tracking, and downstream 3D processing.

Although four fisheye streams are recorded, the canonical representation is the rectified front-left RGB stream. It is adjusted through zoom and pitch correction to produce a natural first-person perspective while preserving hand–tool–object interactions. The processed videos are single-view egocentric streams at 1024 × 1024 resolution and 20 FPS, synchronized with audio. A representative rectification configuration uses zoom 0.78125, a 90.0° field of view, pitch of −5.0°, yaw of 0.0°, and roll of 0.0°. These values are representative rather than universal settings for every recording.

Raw fisheye streams are not publicly released; the public release focuses on rectified canonical videos. Depending on consent and release authorization, synchronized audio may include environmental sound, participant speech, spoken narration, and other recording audio. Original narration audio is released only for consent-cleared examples that pass privacy review. During benchmark evaluation, models do not receive narration audio, corrected narration text, dense captions, or annotation metadata. Audio-capable models receive synchronized non-narration audio only.

The corpus preserves or derives several modalities:

• RGB video: canonical rectified egocentric views.
• Audio: environmental sound and, where authorized, speech or narration.
• Camera and motion data: camera poses and auxiliary synchronization signals.
• Depth: generated through the 3D processing pipeline.
• 2D grounding: keyframe points for focal tools and target objects.
• Object masks: propagated through long temporal ranges.
• 3D object boxes: lifted from tracked masks, depth, and camera poses.
• 4D trajectories: long-horizon object motion over time.
• Textual annotations: dense captions, tool-centric narrations, metadata, and question-answer pairs.

The paper does not report a universal inventory of tools, objects, or tracked trajectories. It also does not claim that all recordings contain complete hand–tool–object 3D annotations.

4. Textual, temporal, and geometric annotations

Hierarchical captions

The corpus contains 361,332 hierarchical captions describing visible actions and task progress at multiple temporal scales. The captioning pipeline divides each video into five-minute units, samples 32 uniformly distributed frames from each unit, and uses Gemini-3-Flash to generate a global context caption. The video is then captioned sequentially in five-second chunks. The global caption initializes the first chunk, while preceding captions provide memory for subsequent chunks.

Chunk captions are aggregated into one-minute and five-minute summaries. Object recognition on key frames is used for post-hoc hallucination checking. These annotations support action description, procedural summarization, temporal progression, object-state tracking, next-step prediction, and general first-person visual description.

Tool-centric narrations

The 6,519 tool-centric narrations differ from generic activity descriptions because they foreground the selected tool, target object or material, intended action, reason for tool choice, possible alternatives, resulting effect, and task-state change. A reference template is:

“I am using [the] [property] [tool] [location/identity cue] rather than [the] [property] [alternative tool] [location/identity cue] to [action] on/for [target] because [reason].”

Examples include using narrow chopsticks rather than a spoon to reach material inside a small opening, using a flat spatula rather than chopsticks to support a fragile egg, and using a pipette rather than a larger container for finer volume control.

Narrations target meaningful tool-use events rather than every frame. These include selecting or switching tools, adapting to a changing object state, preparing a tool for a later step, recovering from failure, and applying a tool in a physically appropriate way. Source narrations may be in English, Korean, or Chinese. Korean and Chinese narrations are transcribed and translated into English, then normalized and corrected by bilingual or English-proficient project members. Language tags preserve the original narration language.

Temporal localization

Narrations are linked to temporally localized video segments. The annotation hierarchy uses five-second chunks, one-minute aggregates, and five-minute summaries. Benchmark questions use localized temporal windows around an anchor moment; most benchmark clips last at least three minutes, with an average temporal span of 4.19 minutes per question-answer item. Temporal overlap is limited and the number of questions per source video is capped.

2D grounding and object states

Selected narrations contain 2D grounding points on keyframes for focal tools or target objects. These points link textual references to visible instances, identify focal entities, support focal-tool localization, and initialize long-horizon tracking. Grounding is available for selected instances rather than universally.

The annotations represent object-state changes through corrected narrations, dense captions, timestamps, focal-object annotations, and derived instruction examples. Examples include raw ingredients becoming cut, food becoming cooked, an object bending or loosening, a tool contacting a target, and an intermediate state enabling the next procedural step. The paper does not specify a single formal state ontology or exhaustive count of state labels.

3D and 4D information

The 3D annotation pipeline uses raw HOMIE geometry, 3D Gaussian Splatting, rendered depth maps, 2D grounding points, SAM2 mask propagation, VLM-based tracking verification, and camera poses. The process is:

1. Use raw HOMIE geometry.
2. Train a 3D Gaussian Splatting model to obtain multi-view-consistent scene geometry.
3. Reduce artifacts such as floaters.
4. Render temporally coherent depth maps.
5. Initialize object tracking with narration-linked 2D points and semantic descriptions.
6. Propagate object masks with SAM2.
7. Verify tracking consistency with a VLM when new grounding points appear.
8. Reinitialize SAM2 if drift or semantic mismatch is detected.
9. Lift tracked masks into 3D using depth and camera poses.
10. Produce 3D object boxes and long-horizon 4D object trajectories.

These annotations support object-motion queries, spatial relations, tool–object interaction dynamics, spatial question answering, and long-range object tracking. The paper does not report the total number of tracked objects, trajectories, tools, or 3D annotations. Selected trajectory and object-state assets are released incrementally rather than universally.

5. Instruction-tuning corpus and supported reasoning capabilities

The dataset is transformed into an instruction-tuning corpus containing 184,679 examples.

Supervised capability	Examples	Share
Affordance–Causality	14,175	7.7%
Perception–Grounding	14,755	8.0%
Procedural Dynamics	22,943	12.4%
Spatial Reasoning	17,887	9.7%
Episode-level multiple choice	69,554	37.7%
General video QA	31,281	16.9%
Dense captioning and narration completion	9,084	4.9%
Single-image open-ended QA	5,000	2.7%
Total	184,679	100.0%

The episode-level examples include 64,554 questions about action ordering, activity, and goals, together with 5,000 next-action questions paired with current observation frames. By answer format, the corpus contains 170,595 multiple-choice examples, 5,000 short open-ended examples, and 9,084 captioning or narration-completion examples. The next-action answer letters are balanced across four positions, with 1,250 examples per position.

Synthetic candidates are filtered for visual answerability, temporal grounding, single-best-answer validity, resistance to textual shortcuts, and balanced answer positions. The supported capabilities are organized as follows.

Perception and grounding

Perception–Grounding questions ask what is visibly present and where it is located. They include focal-tool identification, target-object identification, object attributes, tool–object contact, local state changes, visible-object counting, and grounding textual tool references to visual instances.

Spatial reasoning

Spatial Reasoning concerns hand–tool–object arrangement, relative position, alignment, containment, support, contact, depth, object motion, and tool–object trajectories. Some questions use the supplementary 3D and 4D annotations.

Procedural dynamics

Procedural Dynamics concerns tool–action ordering, step transitions, workspace staging, preparation, fine-grained manipulation, task progress, next-action prediction, and changes in tool use over time.

Affordance and causality

Affordance–Causality questions ask why a tool is appropriate, whether a substitute is feasible, what a tool affords, why a tool is switched, which precondition an action establishes, and what consequence an action produces. This category most directly distinguishes EgoTools-Data from ordinary activity captioning.

6. EgoTools-Bench, evaluation, and limitations

EgoTools-Bench contains 1,000 eight-way multiple-choice questions derived from source videos reserved before construction of the training corpus. The benchmark contains 363 Affordance–Causality questions, 236 Perception–Grounding questions, 222 Procedural Dynamics questions, and 179 Spatial Reasoning questions. Nine hundred questions are manually written by collectors and external annotators; 100 are generated from 3D annotations and human-verified.

Track	Questions
Affordance–Causality	363
Perception–Grounding	236
Procedural Dynamics	222
Spatial Reasoning	179
Total	1,000

The split is made at the source-video level before instruction data or benchmark questions are constructed. No derivative of a benchmark source video—including clips, captions, narrations, synthetic question-answer pairs, grounding annotations, or metadata-derived training examples—is used for model training. This prevents source-video overlap rather than merely removing exact benchmark clips from the training set.

The benchmark uses eight answer options, one correct answer, and seven distractors; chance performance is therefore 12.5%. Evaluated models generally receive 64 uniformly sampled frames, with 512 frames used by some thinking models and 1 FPS input used for Gemini models. Audio-capable models receive synchronized non-narration audio.

Model	AC	PG	PD	SR	Overall
Human Expert	82.1	85.2	83.3	82.7	83.2
Gemini-3.1-Pro	69.7	51.7	72.1	74.9	66.9
Gemini-3-Flash	64.5	50.0	73.0	72.6	64.4
Qwen3-VL-8B-Instruct	46.1	45.9	57.1	55.4	50.0
Qwen2.5-VL-7B-Instruct	40.1	43.3	50.2	47.8	44.3
InternVL3.5-8B-Instruct	45.8	45.9	53.4	53.3	48.7
LLaVA-OneVision-7B	37.6	37.1	42.9	37.0	38.9
GLM4.1V-Thinking	46.1	50.0	57.1	53.3	50.7
EgoTools-8B	55.4	59.7	68.1	44.4	60.9

Gemini-3.1-Pro achieves the strongest proprietary result at 66.9% overall, while its Perception–Grounding score is 51.7%. Human experts achieve 83.2%. Qwen3-VL-8B-Instruct obtains 50.0% on EgoTools-Bench, despite scores of 69.0% on EgoSchema and 61.0% on EgoThink.

Fine-tuning Qwen3-VL-8B-Instruct on the 184,679 EgoTools-derived examples improves overall accuracy from 50.0% to 60.9%. The improvement is 9.3 points for Affordance–Causality, 13.8 points for Perception–Grounding, and 11.0 points for Procedural Dynamics. Spatial Reasoning declines from 55.4% to 44.4%, a reduction of 11.0 points.

Track	Before fine-tuning	After fine-tuning	Change
Affordance–Causality	46.1	55.4	+9.3
Perception–Grounding	45.9	59.7	+13.8
Procedural Dynamics	57.1	68.1	+11.0
Spatial Reasoning	55.4	44.4	−11.0
Overall	50.0	60.9	+10.9

An input-evidence ablation shows that ordered multi-frame input produces 51.7% overall accuracy, compared with 49.3% for shuffled frames, 46.9% for a middle frame only, and 42.7% for text-only input. The largest gain over text-only input occurs in Perception–Grounding, which improves by 13.9 points.

Quality control uses a fix-first strategy and a 22-flag binary audit covering missing fields, malformed options, duplicates, first-person leakage, personally identifying information, answer–option mismatch, inappropriate full-video usage, and distractor anomalies. Anti-shortcut checks address duplicate choices, answer-position imbalance, lexical overlap, “all of the above” or “none of the above” options, and answer-length bias. Items with standardized answer-length deviation $|z|>1.5$ are flagged for repair.

A reliability study of 250 benchmark questions produced 500 judgments. Agreement with the original answer key was 89.8%, agreement for Affordance–Causality questions was 86.5%, and exact agreement between assigned annotators was 88.4%. Annotators judged 94.4% of questions to have a uniquely best answer supported by visible evidence; 14 questions, or 5.6%, were flagged as ambiguous or multiply answerable and were revised or replaced.

Important limitations include strong concentration in kitchen data, uneven domain coverage, the absence of an exhaustive tool or object vocabulary, selected rather than universal 2D grounding, selected rather than universal 3D trajectories, conditional availability of narration audio, and difficulty maintaining long-horizon spatial accuracy. The corpus does not report a total number of subjects, a complete tool taxonomy, a complete object taxonomy, or a conventional public train/validation/test split. It also does not provide universal dense annotations for tool-instance identities, hand–tool contact, grasp type, manipulation trajectories, or formal object-state ontologies.

The collection is primarily from universities and research sites in Asia and includes many students or academics, which may bias cultural practices, tool availability, workspace organization, language, and task execution styles. Raw fisheye video is not publicly released; personally identifying metadata is removed, sensitive content is screened, and selected narration audio and 3D assets are conditionally released. The dataset is distributed under a research-use license whose precise legal conditions must be obtained from the official release.

EgoTools-Data is therefore best characterized as a structured multimodal corpus for tool-centric egocentric understanding rather than as a conventional object-detection or manipulation dataset. Its distinctive contribution is the integration of tool choice, affordance, spatial configuration, procedural role, target-object state, and causal consequence with synchronized egocentric video, audio, hierarchical captions, narrations, visual grounding, and reconstructed 3D and 4D information.