EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates

Abstract: We present EgoFun3D, a coordinated task formulation, dataset, and benchmark for modeling interactive 3D objects from egocentric videos. Interactive objects are of high interest for embodied AI but scarce, making modeling from readily available real-world videos valuable. Our task focuses on obtaining simulation-ready interactive 3D objects from egocentric video input. While prior work largely focuses on articulations, we capture general cross-part functional mappings (e.g., rotation of stove knob controls stove burner temperature) through function templates, a structured computational representation. Function templates enable precise evaluation and direct compilation into executable code across simulation platforms. To enable comprehensive benchmarking, we introduce a dataset of 271 egocentric videos featuring challenging real-world interactions with paired 3D geometry, segmentation over 2D and 3D, articulation and function template annotations. To tackle the task, we propose a 4-stage pipeline consisting of: 2D part segmentation, reconstruction, articulation estimation, and function template inference. Comprehensive benchmarking shows that the task is challenging for off-the-shelf methods, highlighting avenues for future work.

• The paper introduces a function template paradigm to model cross-part interactions in simulation-ready 3D objects from egocentric videos.
• It employs a four-stage pipeline combining 2D segmentation, 3D reconstruction, and articulation estimation for robust object modeling.
• Benchmarking reveals current SOTA limitations in segmentation and reconstruction, highlighting paths for embodied AI improvements.

EgoFun3D: Structured Modeling of Interactive 3D Objects from Egocentric Video

Task Formalization and Function Template Abstraction

EgoFun3D introduces a rigorous computational framework for constructing simulation-ready interactive 3D object models directly from egocentric video. The core innovation is the formulation of function templates that precisely describe the cross-part functional dependencies inherent in real-world interactive objects. Instead of restricting to articulated state estimation, the method defines object interaction as a mapping from a receptor (the physically manipulated part) to an effector (the part exhibiting a result), coupling this mapping to a physical effect such as geometry, illumination, temperature, or fluid flow.

Figure 1: Human-object interaction is modeled as the state transition from receptor to effector, with the function template capturing cross-part mapping and physical effects.

These templates formalize part functionality as a two-component abstraction: a mapping type (binary, step, linear, or cumulative) and a physical effect. This representation not only enables structured comparison and evaluation but is also directly translatable to executable code for a variety of robotic and embodied AI simulators.

Methodological Pipeline

EgoFun3D decomposes the task into a four-stage pipeline:

1. 2D Part Segmentation: Identification of receptor and effector parts via VLM-prompted semantic segmentation and instance localization.
2. 3D Reconstruction: Per-part geometry reconstruction using multi-view and depth-based methods to create 3D meshes suitable for simulation.
3. Articulation Parameter Estimation: Extraction of part articulation properties (joint types, axes, origins, and motion ranges) from tracked interaction.
4. Function Template Inference: Automated inference of mapping and effect using VLMs operating on highlighted video segments.

   Figure 2: The four-stage framework leverages state-of-the-art models for segmentation, reconstruction, articulation, and function template induction, producing simulation-compatible interactive objects.

This modular architecture enables systematic benchmarking and error localization, providing insight into the specific failures and capabilities of current SOTA models on this complex, multi-modal task.

Dataset Design and Annotation Suite

The authors curate and release a comprehensive dataset comprising 271 egocentric videos spanning 88 object instances in 14 functional categories. The dataset integrates (and extends) videos from Ego-Exo4D and FunGraph3D, supplemented by novel captures to address long-tail distributions and class imbalance.

Figure 3: The dataset demonstrates long-tail distributions in object categories, function mappings, and physical effects, exposing diverse and challenging real-world interaction scenarios.

Each sample is exhaustively annotated with:

• Dense 2D segmentation for hands, receptor, effector, and whole object across all frames.
• 3D part segmentation on reconstructed meshes.
• Articulation parameters, distinguishing prismatic vs. revolute joints and supporting precise range annotation.
• Function templates covering all valid receptor-effector mappings and instantiated across four physical effect domains.
• Concrete simulator instantiations provided for Genesis, Isaac Sim, and BEHAVIOR.

  Figure 4: Rich annotation covers 2D/3D segmentation, joint parameters, and function templates, enabling end-to-end evaluation and simulator deployment.

Quantitative Benchmarking and Analysis

EgoFun3D benchmarks a suite of off-the-shelf and agentic models in each sub-task, exposing significant gaps in current model robustness and generalizability when faced with egocentric, dynamic settings.

• 2D Segmentation: SAM3 with Qwen3-VL achieves the highest receptor and effector IoU (30.0% and 47.9%, respectively), but is computationally inefficient and exhibits significant confusion, especially on small or ambiguous parts. Segmentation quality is a severe bottleneck for downstream tasks.

  Figure 5: Qualitative results indicate persistent challenges in accurate instance-part correspondence and temporal consistency for small/repeated parts.

• 3D Reconstruction: Depth Anything 3 is the superior model for mesh recovery (median Chamfer Distance $0.026\,m^2$ for receptors), outperforming ViPE and MapAnything. Nevertheless, all methods are afflicted by drift and incompleteness due to egocentric perspective changes, severe occlusions, and textureless surfaces.

  Figure 6: Even SOTA methods produce misaligned and partial meshes, with small and occluded parts particularly susceptible to failure.

• Articulation Parameter Estimation: ArtiPoint delivers higher joint type accuracy (74.2%) compared to iTACO (26.8%), but with a much higher failure rate (46.4%). Both methods struggle when tracking is hampered by occlusion, dynamic viewpoints, or poor part segmentation. Contradictorily, hand tracking (as opposed to part tracking) is hypothesized to afford greater reliability in such scenarios.

  Figure 7: Articulation inference is confounded by small part size and occlusions, with frequent misclassification of joint type.

• Function Template Inference: VLMs such as Gemini 3 Flash and GPT-5-mini exhibit strong function template classification, with Gemini achieving 95.2% physical effect accuracy, 97.6% mapping accuracy, and 92.9% overall joint accuracy. Open-source alternatives lag in mapping prediction.

Simulation Deployment and Qualitative Results

Function templates, together with predicted geometry and articulation, are compiled into simulator-executable code, enabling instantiation across heterogeneous platforms. Qualitative examplars demonstrate the feasibility of generating interactive assets for Genesis, Isaac Sim, and BEHAVIOR, though pipeline bottlenecks are manifest in mesh alignment and joint misclassification errors.

Figure 8: Example outputs show functional, interactive simulation objects; artifacts such as mesh misalignment or incorrect articulation propagate visibly.

Limitations and Theoretical Implications

The approach is currently restricted to single receptor-effector pairs and simple mappings, limiting expressiveness for compound or multi-stage interactions. Reliance on current segmentation and 3D reconstruction models introduces error cascades that degrade overall simulation fidelity. Function template abstraction, however, illustrates the utility of structured, executable representations for bridging data-driven perception and formal simulation modeling.

This work highlights that:

• High-fidelity end-to-end modeling from in-the-wild egocentric data remains unsolved, especially for small, occluded, or weakly-textured object parts.
• VLMs are effective in high-level functional reasoning and mapping induction, while geometric and kinematic inference from real-world data is substantially less reliable.
• In simulation pipeline design, error decoupling, intermediate annotation, and structured function representations are critical for scalable embodied AI asset generation.

Conclusion

EgoFun3D establishes a comprehensive task, benchmark, and reference baseline for direct modeling of simulation-ready, functionally-grounded 3D objects from egocentric video (2604.11038). The function template paradigm connects low-level perception to high-level executable semantics, offering a unifying abstraction for cross-simulator asset creation. Continued progress demands advances in agentic segmentation, robust multi-modal fusion for mesh recovery, and articulation inference robust to dynamic, cluttered scenes. Expansion to more complex interactions and richer task domains remains an open direction with substantive implications for embodied AI, robotics, and simulation-based learning.