FunRec: 3D Digital Twin Reconstruction

• FunRec is a training-free system that reconstructs functional 3D digital twins from single in-the-wild egocentric RGB-D videos.
• It integrates articulation-aware motion clustering, robust pose estimation, and canonical-space TSDF fusion to recover both static and dynamic geometries.
• The pipeline outputs simulation-compatible meshes with kinematic parameters, enabling realistic interaction modeling in virtual and physical environments.

Searching arXiv for the specified paper and closely related context. {"query":"(Delitzas et al., 7 Apr 2026) FunRec Reconstructing Functional 3D Scenes from Egocentric Interaction Videos", "max_results": 5} {"query":"FunRec Reconstructing Functional 3D Scenes from Egocentric Interaction Videos", "max_results": 10} FunRec is a training-free, optimization-driven system for reconstructing functional 3D digital twins of indoor scenes directly from single, in-the-wild egocentric RGB-D interaction videos. It recovers static geometry and articulated elements, automatically discovers articulated parts, estimates kinematic parameters, tracks 3D motion over time, reconstructs static and moving geometry in canonical space, and exports simulation-compatible meshes and joint definitions in URDF/USD suitable for physics-based interaction (Delitzas et al., 7 Apr 2026). In the formulation used by the work, a functional 3D digital twin is a physically consistent, interactable replica of an indoor scene that encodes static geometry, articulated elements such as doors, drawers, and lids, and joint models permitting continuous manipulation in simulation.

1. Problem formulation and representational scope

FunRec is designed around the setting of egocentric in-the-wild RGB-D interaction videos rather than controlled setups, multi-state captures, or CAD-prior pipelines (Delitzas et al., 7 Apr 2026). The stated motivation is that controlled setups, open/closed scans, and CAD priors either miss causal motion signals, require laborious data acquisition and annotation, or are only weakly tied to real geometry. By contrast, egocentric interaction videos naturally reveal what moves, how it moves, and volumes exposed during manipulation, including interiors of cabinets.

The representation is centered on canonical space. For each articulated part, FunRec reconstructs geometry in the part’s canonical coordinate frame, defined as the frame in which the part is “at rest,” while all motion across time is represented by joint parameters and per-frame rigid transformations that move the canonical mesh into the world frame (Delitzas et al., 7 Apr 2026). This canonicalization removes camera motion and occlusion effects and yields a single consistent mesh for the part that can be placed into any scene state by applying the estimated kinematics.

The output is explicitly simulation-compatible. TSDF-based fusion produces watertight meshes for the static scene and for each articulated part, organized hierarchically with joint definitions, axes, pivots or anchors, and motion limits. The resulting assets are directly exportable to URDF/USD with per-joint frames and constraints and can be loaded into physics simulators such as Isaac Sim (Delitzas et al., 7 Apr 2026). A plausible implication is that FunRec is positioned not merely as a reconstruction system but as a scene-to-simulation conversion pipeline.

2. End-to-end pipeline

The input is a single egocentric RGB-D video

$V=\{(I_i,D_i)\}$

with known camera intrinsics; the paper specifies head-mounted capture, such as Azure Kinect, for real scenes and photorealistic renders with logging for simulated scenes (Delitzas et al., 7 Apr 2026). The video is first decomposed into temporally contiguous fragments. A video-LLM, Gemini 2.5, classifies each fragment as static or dynamic and assigns an articulation type, revolute or prismatic, if the fragment is dynamic.

Camera motion estimation, articulated part discovery, dense segmentation, kinematic optimization, canonical-space fusion, and submap alignment are then executed in sequence. Camera poses $T_i^c \in SE(3)$ are estimated by robust 2D/3D matching using RoMA and SupeRANSAC, followed by fragment-level pose graph optimization. Sparse 3D trajectories are computed by TAPIP3D and filtered with hand and object masks from VISOR. These trajectories are clustered under articulation-aware motion hypotheses with HDBSCAN, producing candidate moving-part track sets (Delitzas et al., 7 Apr 2026).

Dense mask generation proceeds from keyframe over-segmentation and temporal propagation. On uniformly spaced keyframes, SAM’s automatic mask generator produces regions, a motion ratio is computed per region from moving and static trajectories, and regions whose ratio exceeds a threshold are labeled moving. SAM2 then propagates these masks through the fragment to obtain temporally consistent moving-part masks (Delitzas et al., 7 Apr 2026).

Part motion is estimated from 3D–3D correspondences built from moving tracks. Relative transforms are robustly estimated with SupeRANSAC, then refined jointly with articulation parameters through pose graph optimization and manifold optimization in Ceres. Canonical-space TSDF fusion integrates moving-part pixels in the part frame using the transform chain $(T_i^m)^{-1}T_i^c$, while a separate world-frame TSDF integrates static scene geometry with hand and moving pixels excluded. Marching cubes extracts the meshes, and PREDATOR aligns fragment-level submaps globally before final fusion and URDF/USD export (Delitzas et al., 7 Apr 2026).

3. Articulation discovery, kinematic parameterization, and optimization

FunRec supports two joint classes: prismatic and revolute. A prismatic joint is parameterized by a unit axis $a \in S^2$ and a per-frame scalar displacement $\lambda_i \in \mathbb{R}$, with homogeneous transform

$$T_{\text{pris}}(a,\lambda_i)= \begin{bmatrix} I & \lambda_i a\ 0^T & 1 \end{bmatrix}.$$

A revolute joint is parameterized by a unit axis $a \in S^2$, a pivot point $p \in \mathbb{R}^3$ on the axis, and an angle $\theta_i \in S^1$, with transform

$$T_{\text{rev}}(a,p,\theta_i)= \begin{bmatrix} R(a,\theta_i) & (I-R(a,\theta_i))p\ 0^T & 1 \end{bmatrix}.$$

Axes and pivots are initialized by least-squares fits to line or circle motion, and the pivot is chosen as the closest point on the axis to the scene origin subject to $T_i^c \in SE(3)$0 (Delitzas et al., 7 Apr 2026).

Articulated part discovery begins from TAPIP3D trajectories

$T_i^c \in SE(3)$1

with visibility

$T_i^c \in SE(3)$2

For each track, FunRec fits a line for prismatic motion or a circle for revolute motion, producing an axis, pivot, and per-frame motion state. A track is retained only if its average residual satisfies

$T_i^c \in SE(3)$3

and it is discarded if the maximum displacement is smaller than $T_i^c \in SE(3)$4 (Delitzas et al., 7 Apr 2026). Retained tracks are clustered by joint-parameter similarity using HDBSCAN. The selected cluster maximizes overlap with 2D interacted-object evidence through the score

$T_i^c \in SE(3)$5

which yields the moving and static track sets.

The refinement stage combines pose-graph consistency and track-level articulation fitting. With absolute part poses $T_i^c \in SE(3)$6, relative transforms $T_i^c \in SE(3)$7, loop-closure confidences $T_i^c \in SE(3)$8, and information matrices $T_i^c \in SE(3)$9, the residual is

$(T_i^m)^{-1}T_i^c$0

and the objective is

$(T_i^m)^{-1}T_i^c$1

with

$(T_i^m)^{-1}T_i^c$2

The track-level term is

$(T_i^m)^{-1}T_i^c$3

where $(T_i^m)^{-1}T_i^c$4, and the combined objective is

$(T_i^m)^{-1}T_i^c$5

subject to $(T_i^m)^{-1}T_i^c$6, $(T_i^m)^{-1}T_i^c$7, and $(T_i^m)^{-1}T_i^c$8 under manifold optimization in Ceres (Delitzas et al., 7 Apr 2026).

4. Canonical-space reconstruction and simulation export

The reconstruction stage explicitly separates static and dynamic geometry. The static scene TSDF is fused in the world frame using the camera poses $(T_i^m)^{-1}T_i^c$9 while excluding hand and moving-part pixels, that is, pixels in $a \in S^2$0. The articulated-part TSDF is fused in the canonical frame using only moving-part pixels after transforming depths by $a \in S^2$1, thereby removing both camera motion and part motion (Delitzas et al., 7 Apr 2026). The paper states that this yields a single canonical mesh per part and avoids ghosting and inconsistent surfaces under motion.

Surface extraction is performed with Marching Cubes, producing watertight meshes for the static scene $a \in S^2$2 and articulated parts $a \in S^2$3. Simulation-compatibility is enforced through consistent scaling and units, coherent normals from TSDF gradients, removal of redundant or overlapping triangles at cut boundaries via mask-based fusion, and the use of a single canonical mesh per part consistent across all states through the application of $a \in S^2$4 and kinematic constraints (Delitzas et al., 7 Apr 2026).

The export stage constructs a hierarchy of links and joints. The hierarchy is described as world $a \in S^2$5 static base link(s) $a \in S^2$6 part link(s), with one joint from each part link to the static base (Delitzas et al., 7 Apr 2026). For a revolute joint, the exporter records the unit axis $a \in S^2$7, origin frame at pivot $a \in S^2$8, and angle limits $a \in S^2$9; for a prismatic joint, it records the unit axis $\lambda_i \in \mathbb{R}$0, origin frame at anchor, and position limits $\lambda_i \in \mathbb{R}$1. Visual and collision geometry are specified per link, scaling is set to meters, and inertial properties are optional: mass and inertia may be estimated via a VLM and object priors if needed, otherwise default values can be used.

This export mechanism also supports downstream interaction tasks. The work describes hand-guided affordance mapping using off-the-shelf hand mesh estimation such as HaMeR or WILOR, followed by proximity queries between the hand mesh and the articulated-part mesh in canonical space across motion to produce contact heat maps. It also describes robot-scene interaction by transferring articulation parameters, motion limits, and inferred contact points to a manipulator controller in simulation or on a real robot (Delitzas et al., 7 Apr 2026).

5. Empirical evaluation

The evaluation spans three datasets and three baseline families (Delitzas et al., 7 Apr 2026). RealFun4D contains 351 egocentric videos from 60 apartments in four countries, captured with Azure Kinect RGB-D and annotated with interaction intervals, textual descriptions, per-frame hand and part 2D masks, articulation joints, and 2D part tracks. OmniFun4D contains 127 photorealistic sequences in 12 OmniGibson scenes with scripted interactions, logged poses, and rendered RGB-D and masks. HOI4D contributes 30 interactions across the categories “laptop,” “cabinet,” “safe,” and “trash can,” with kinematics post-processed from 6D part poses.

The baseline categories are Type 1 monocular 4D reconstruction methods, specifically MonST3R and SpatialTrackerV2; Type 2 6D object pose tracking, specifically BundleSDF with ground-truth masks; and Type 3 articulated object modeling from multi-state input, specifically ArtGS using two static states without video tracking (Delitzas et al., 7 Apr 2026).

The metrics include moving-part segmentation mIoU,

$\lambda_i \in \mathbb{R}$2

articulation error through axis direction error, axis position error, and state error, 6D part pose through ADD-S and ADD, and reconstruction accuracy via Chamfer Distance (Delitzas et al., 7 Apr 2026).

Benchmark	Moving-part mIoU	Reconstruction CD
OmniFun4D	77.9	3.2 cm
HOI4D	76.4	0.7 cm
RealFun4D	74.8	6.1 cm

For moving-part segmentation, FunRec reports 77.9 on OmniFun4D, 76.4 on HOI4D, and 74.8 on RealFun4D, compared with 23.6, 26.8, and 23.7 for MonST3R and 6.2, 5.8, and 13.4 for SpatialTrackerV2, respectively; the abstract summarizes this as up to +50 mIoU improvement (Delitzas et al., 7 Apr 2026). For 6D part pose accuracy, the reported ADD-S/ADD percentages are 78.96/71.28 on OmniFun4D, 79.43/69.85 on HOI4D, and 75.62/68.11 on RealFun4D. For articulated motion estimation, the reported errors are 5.3° revolute axis error and 5.4° prismatic axis error on OmniFun4D, 12.4° and 1.3° on HOI4D, and 5.6° and 5.5° on RealFun4D; revolute-axis position errors are 0.03 m, 0.06 m, and 0.05 m, while state errors are 5.0°/0.02 m, 9.1°/0.02 m, and 8.4°/0.03 m, with failure rates of 1.7%, 0%, and 0%, respectively (Delitzas et al., 7 Apr 2026). The paper characterizes these as 5–10 times lower articulation and pose errors than baselines and as order-of-magnitude position improvements over BundleSDF.

Ablations emphasize four effects. First, TAPIP3D trajectories with visibility-based weighting are described as crucial in occlusion-heavy videos; removing visibility weighting degrades pose graph stability and increases articulation state errors. Second, motion-ratio gating on SAM regions followed by SAM2 propagation substantially improves dense part masks relative to using only 2D interaction masks or raw tracks. Third, canonical-space fusion is reported to eliminate camera-motion-induced artifacts, whereas world-frame fusion of moving parts causes ghosting and inconsistent surfaces. Fourth, enforcing revolute versus prismatic constraints during clustering and fitting reduces ambiguity in axis estimation and stabilizes loop closures; joint-type misclassification leads to large axis and state errors (Delitzas et al., 7 Apr 2026).

6. Relation to prior paradigms, robustness, and limitations

FunRec is explicitly contrasted with methods that rely on multi-view or multi-state scans, CAD priors, object-centric reconstruction with static cameras, or pre-scanned object models (Delitzas et al., 7 Apr 2026). The paper argues that such methods either miss actual kinematics, require complex data collection, or do not generalize to scene-scale egocentric videos. FunRec instead operates on casual single-sequence RGB-D, discovers articulations from motion, reconstructs interiors revealed by interaction, and aligns articulated parts and static scene into a coherent simulation-ready twin. This suggests a shift from object-level articulated modeling toward scene-scale functional reconstruction from human interaction evidence.

The robustness claims are qualified. Robust estimators such as SupeRANSAC and TSDF fusion are said to mitigate sensor noise and shading artifacts. Visibility scores and pose graphs improve tracking through occlusions, while segmentation propagation maintains mask consistency. Fragment-wise processing and global alignment are stated to accommodate large viewpoint changes and long sequences (Delitzas et al., 7 Apr 2026). At the same time, non-rigid deformations are outside the current model, and multi-contact is handled only insofar as the articulated component is dominant and hand masks reduce contamination.

The limitations are specific. The current system supports only revolute and prismatic joints and does not model compound joints, sliders with coupled degrees of freedom, or deformable and continuous articulations (Delitzas et al., 7 Apr 2026). Very small joints, compliant elements, and soft materials challenge trajectory tracking and TSDF fusion. Accurate motion limits and interior reconstruction require sufficient interaction coverage, so short or partial interactions reduce fidelity.

The future directions listed in the work include broader articulation taxonomies such as universal joints and prismatic-revolute chains, learned affordance priors for automatic contact inference, uncertainty-aware kinematic estimation, multimodal inputs including IMU, audio, and tactile sensing, and real-time variants (Delitzas et al., 7 Apr 2026). Within the scope defined by the paper, FunRec’s principal contribution is the integration of articulation-aware motion clustering, manifold-based kinematic optimization, and canonical-space TSDF fusion into a single pipeline that outputs coherent functional scene twins rather than static geometry alone.