Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation
Abstract: Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging due to severe occlusions and ambiguous contact. Existing learning-based methods often struggle to generalise to in-the-wild scenes and are limited by the scarcity of supervision. We address these issues with two contributions. First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips (62.3K frames) with dense, bijective 3D hand-object contact correspondences and posed meshes. Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose in a single forward pass. A cross-attention decoder conditions object features on hand priors, producing robust pose estimation. We test HOPformer on the in-lab 3D dataset, ARCTIC, as well as our newly introduced EPIC-Contact dataset. HOPformer reaches 82.4% success rate on ARCTIC (+6.2 pts over current SOTA). On EPIC-Contact, it nearly doubles the success rate while reducing contact deviation by 75%. EPIC-Contact, HOPformer code and checkpoints are released: https://sid2697.github.io/epic-contact.
Summary
- The paper introduces HOPformer, a transformer-based architecture that jointly estimates 3D poses for hands and objects from RGB input, significantly reducing inference time.
- It leverages transformer encoders and CAD-guided scaling with symmetry-aware metrics to enhance pose accuracy and achieve a median inference time of 106.63 ms.
- The study presents the EPIC-Contact dataset, validated through qualitative and quantitative analysis, which improves annotation robustness in real-world scenarios.
Egocentric 3D Hand-Object Pose Estimation in-the-Wild: Contributions, Architectures, and Dataset Analysis
HOPformer Architecture and Computational Analysis
The paper introduces HOPformer, a transformer-based architecture for simultaneous estimation of 3D pose for both hands and manipulated objects directly from RGB input. The approach leverages feature encoders (DINOv2 for objects, WiLoR for hands), projects hand features to object embedding space via MLPs, and employs an aggregation module for reducing token dimensionality from 256 to 39—optimized for pose regression. Training involves extensive data augmentation including scaling, color jitter, and rotation. The inference is notably efficient, with a single forward pass taking a median of 106.63 ms on a GH200 120 GB GPU, substantially faster than optimization-based baselines that require several seconds per sample.
HOPformer handles occlusions and diverse manipulations robustly and is architecturally scalable as evidenced by improvements in prediction accuracy with an increasing number of object classes during training. The system generalizes effectively across object categories, benefiting from the diversity induced by additional training classes. Symmetry-aware metrics are employed for evaluation—avoiding penalization for equivariant prediction errors on symmetric objects, crucial for realistic classes such as bottles and plates.
Figure 1: Qualitative demonstration of HOPformer’s pose estimation under challenging occlusion scenarios and with both hands and small objects.
CAD-Free Comparisons, Failure Cases, and Model Assumptions
Qualitative analyses reveal fundamental limitations in current CAD-free methods, such as HOLD-Net and G-HOP, which operate in video-based domains and often generate implausible shapes and erroneous hand-object interaction representations. SAM3D, despite its advanced backbone and extensive training, introduces inconsistent mesh predictions under heavy occlusion and transparency (e.g., bottles misclassified as jugs, transparent glasses reconstructed as bowls, and coffee cups as implausible shapes). The explicit assumption of known CAD meshes in HOPformer is justified by such failures, enabling accurate pose prediction and facilitating robust annotation pipelines.
Figure 2: Failure cases and qualitative results from CAD-free baselines, demonstrating deviations from ground-truth interaction geometry.
Figure 3: Examples where SAM3D fails to reconstruct objects accurately in occluded and transparent scenarios.
EPIC-Contact Dataset: Collection, Annotation, and Validation
EPIC-Contact extends the state-of-the-art in egocentric hand-object datasets, curating challenging videos from EPIC-Grasps and leveraging stable grasps for efficient annotation propagation. The dataset pairs RGB video frames with 3D mesh annotations for nine object categories, using manual annotation supplemented by segmentation from VISOR. Crucially, the scale mismatch between mesh and visual instances is addressed via Gemini 2.5 VLM, producing scale estimates based on context prompts for each category—validated against real-world measurements, achieving a mean absolute error of 0.94 cm (5.9% relative error), with 82.5% of samples within ±10% of true dimensions.
Figure 4: Illustration of object mesh rescaling using VLM output, showing alignment of mesh with visual evidence in the input image.
Figure 5: Scale estimation verification comparing VLM-predicted object dimensions to physical measurements and visual input.
The annotation workflow features web interfaces for marking hand contact points (MANO mesh painting) and transferring the patch to the object (contact axis assignment). Inter-annotator agreement scores (, ) are high, evidencing reproducibility and annotation robustness.
Figure 6: Annotation interface for marking hand contact regions with mesh manipulation controls.
Figure 7: Inter-annotator contact agreement analysis for hand region annotations.
Figure 8: Annotation interface for transferring contact patches from the hand mesh to the object mesh with contact axis alignment.
Pose Fitting, Error Evaluation, and Robustness
Pose mesh fitting employs penetration loss based on signed distance fields computed for hand and object meshes. The error evaluation on ARCTIC demonstrates that the EC-fit pipeline achieves a pose L2 error of 1.9 mm and MRRPE of 8.0 mm, closely matching ground truth MoCap poses. Penetration metrics (volume, depth) further corroborate the quality of mesh fitting, with penetration depth in EPIC-Contact on par with MoCap reference. HOPformer’s performance remains robust against mesh fitting noise and penetration errors, indicating suitability of EPIC-Contact as a training resource for in-the-wild 3D hand-object pose estimation.
Figure 9: Pool of flat hand poses used for realistic finger separation during contact region transfer.
Figure 10: Inter-annotator agreement for object regions, showing consistency in finger, palm, and thumb annotations.
Analysis of Limitations and Prospective Extensions
Despite substantial annotation robustness, manual labeling remains time-consuming and can suffer from propagation errors due to hand pose estimation inaccuracies. HOPformer’s scalability to arbitrary object categories and real-world articulated interactions is yet to be explored fully. Future research directions include integration of advanced CAD reconstruction pipelines, leveraging automated contact region estimation, and expanding to articulated and more diverse object categories.
Conclusion
This work presents a comprehensive system for egocentric 3D hand-object pose estimation, integrating scalable transformer-based inference, validated annotation pipelines, and robust dataset curation. The explicit use of CAD models with context-driven scaling and the adoption of symmetry-aware metrics position the methodology at the forefront of in-the-wild hand-object interaction modeling. The demonstrated robustness, annotation reproducibility, and scalability underscore the practical applicability for embodied AI, robotics, and computer vision tasks involving dexterous manipulation and egocentric perception. The paradigm established herein is a foundation for advancing real-world multi-modal pose estimation, dataset curation, and interaction modeling.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
Overview
Imagine wearing a GoPro on your head while you cook, and a computer tries to figure out, in 3D, exactly how your hands and the objects (like a bottle or a pan) are positioned and moving—frame by frame. This paper builds two things to make that possible in real-life videos:
- HOPformer: a fast AI model that predicts the 3D pose (position and rotation) of both hands and the object from a single image.
- EPIC-Contact: a new dataset of everyday kitchen videos that includes information about where the hand touches the object and how both are posed in 3D.
The appendix you’ve read adds extra details about how the model works, how the data was built, how the team measured accuracy fairly, and what the results look like.
What questions does the paper try to answer?
- Can we quickly and accurately predict the 3D positions and poses of both hands and objects from first‑person videos taken “in the wild” (not in a lab)?
- How can we fairly judge accuracy when objects are symmetric (for example, a bottle looks the same when rotated)?
- Can we build a large, real-life dataset that tells us exactly where hands and objects touch, so models can learn realistic hand-object interactions?
- Will a model like HOPformer still perform well when trained on automatically “posed” hand-object meshes instead of expensive motion‑capture ground truth?
- How well does the method scale to more object types and different camera views?
How did they do it?
HOPformer (the model)
- What it is: A transformer-based model that takes a single egocentric RGB image (like a frame from a head-mounted camera) and outputs the 3D pose of both hands and the object in one go.
- How it works in simple terms:
- It uses powerful image encoders to extract features for the object and the hands.
- It learns from “contact” information—where fingers touch the object—to make more realistic predictions.
- It is “feed-forward,” which means it predicts in one pass without slow, per-frame optimization, making it fast.
EPIC-Contact (the dataset)
- What it is: A set of kitchen videos where people use everyday objects. For each clip, annotators mark where the hand touches the object and fit 3D meshes (digital blueprints of shapes) of both the hand and the object to match the video.
- How they annotated contact:
- Step 1: Annotators “paint” the areas on a 3D hand model where the hand touches the object (like a heatmap on fingers, palm, thumb).
- Step 2: They transfer those contact regions to the matching spots on the object’s 3D model, creating a one-to-one mapping between hand and object contact points.
- Getting the right object size:
- The team used a vision-LLM (Gemini 2.5) to estimate real-world dimensions (like the diameter of a pan or height of a glass) from an image, then scaled the object’s 3D mesh accordingly. They verified this with real measurements and found the estimates were usually within about 6% error on average.
- Fitting the 3D poses (EC-fit pipeline):
- They align hand and object meshes using the contact points and add a “no-overlap” rule so the hand and object don’t pass through each other. You can think of it like snapping Lego pieces together where they touch and making sure they don’t intersect.
Measuring performance fairly
Some objects are symmetric (e.g., rotating a bottle might look the same), so the team updated several accuracy metrics to be “symmetry-aware.” In everyday terms, the model doesn’t get unfairly punished if it places contact on the “equivalent” side of a symmetric object, or if it slides along a uniform surface where many spots are the same.
Speed and compute
- HOPformer runs in about 0.1 seconds per image on a high-end GPU, much faster than older methods that could take over 10 seconds per frame.
What did they find?
- Fast and accurate predictions:
- HOPformer predicts both hands and the object together in one pass and is much faster than optimization-heavy methods.
- It works even when parts of the hand or object are hidden (occluded) and with small objects like scissors or a phone.
- Scales to more object classes:
- As they trained HOPformer on more object types, its performance generally improved, showing it learns useful patterns from variety.
- Works in different views:
- Not only does it perform well on egocentric (first-person) videos, it also generalizes to exocentric (third-person) views.
- Robust to training labels with small errors:
- When trained on their fitted poses (from contact annotations) instead of perfect motion-capture data, the model performed about the same. This means their annotation and fitting process is good enough to train strong models.
- CAD-free approaches struggle in the wild:
- Methods that try to create object shapes from scratch (without known CAD models) often fail under heavy occlusion or when objects are transparent. Even large modern systems like SAM 3D had trouble producing accurate shapes for the videos here.
- Using a known pool of object meshes (CADs) currently works better for real-life, cluttered scenes.
- Object recognition:
- Their built-in object classifier gets the right object around half the time. When it does, HOPformer’s full pipeline gives results similar to those using the true (oracle) object class. If the class is wrong, they can’t compute object pose metrics fairly because the shape is different.
- Annotation quality:
- People who marked contact areas agreed fairly well with each other (moderate-to-strong agreement scores).
- Their posed hand-object meshes have small average errors and reasonable hand–object contact without unrealistic interpenetration, comparable in depth to motion-capture results.
Why does this matter?
- Practical uses:
- AR/VR: Better hand-object tracking can make virtual objects feel more realistic when you see your hands interact with them.
- Robotics: Robots can learn how people actually grasp and use objects in everyday tasks.
- Human-computer interaction: Systems can understand what you’re doing with your hands to assist or instruct you.
- Research impact:
- EPIC-Contact gives the community a real-world dataset with detailed contact information, which is rare and valuable.
- HOPformer shows a fast, practical way to estimate 3D hand–object poses from a single image in challenging, real-life scenes.
- Limitations and next steps:
- Annotations still take time (about 3–4 minutes per clip’s center frame), and scaling to many more object categories is future work.
- Estimating a brand-new object’s 3D shape directly from video (without a known CAD model) is still hard in the wild, especially with occlusions or transparent items. As those methods improve, they could be plugged into this pipeline to make everything fully automatic.
In short, this work brings us closer to truly understanding how people’s hands interact with everyday objects in real settings—quickly, accurately, and at scale—and lays the groundwork for smarter AR/VR, robotics, and human-centered AI.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, consolidated list of specific gaps and open questions that remain unresolved by the paper, focusing on what is missing, uncertain, or left unexplored.
- Reliance on oracle object classes for main results: end-to-end performance with predicted classes is not fully characterized, and no strategy is proposed for handling misclassified cases when CAD topology differs, preventing evaluation on the full test set.
- Low object classification accuracy (≈53%) limits practical deployment: no exploration of top‑K retrieval, cross-category pose transfer, or shape-agnostic evaluation metrics that remain computable under misclassification.
- Assumption of a fixed pool of known CAD meshes per category: unclear how the method would handle unseen object instances with topology deviating from the available CADs, beyond non-uniform scaling.
- No mechanism to adapt to unseen/new categories (open-vocabulary/generalized HOI): how to extend HOPformer beyond the nine EPIC-Contact categories or generalize to arbitrary objects is left open.
- Handling of articulated objects in the wild is not studied (only briefly shown on ARCTIC): estimating articulation parameters and joint states for real-world objects remains unexplored.
- Weak-perspective camera with a fixed focal length (different constants for ARCTIC and EPIC-Contact): sensitivity to unknown intrinsics, lens distortion, and egocentric camera variability is not analyzed; absolute scale recovery is ill-posed.
- Absence of temporal modeling: the method runs per-frame feed-forward inference; effects on temporal consistency, drift, or leveraging video context for robustness under occlusion/dynamics are not investigated.
- Symmetry-aware metrics trade-offs: CDev and SR become invariant to sliding along surfaces; the extent to which this masks real pose/motion errors or introduces metric gaming is not evaluated.
- Lack of symmetry-aware training losses: while evaluation is symmetry-aware, losses used during training are not discussed as symmetry-aware, raising potential train–test mismatch for symmetric objects.
- Limited failure analysis: no systematic breakdown of failure modes (e.g., heavy occlusion, severe clutter, truncation, glossy/transparent objects, extreme hand poses) or per-condition performance.
- Generalization to multi-object scenes and disambiguation when multiple similar objects co-occur is not considered.
- Dataset bias toward “stable grasps”: EPIC-Contact focuses on segments with stable contact; coverage of transient contacts, re-grasps, slips, and tool-use dynamics is limited and unquantified.
- Contact annotations’ reliability: inter-annotator agreement is measured on small samples (10 for hand, 10 for object) with moderate κ (≈0.6); large-scale agreement, per-class variability, and error propagation are not analyzed.
- Finger-distance transfer using a pool of 41 flat hand poses: coverage of the full spectrum of finger configurations and its impact on contact transfer accuracy is not ablated.
- EC-fit contact-to-pose pipeline increases CDev (7.0 → 12.2 mm) despite improving other errors: causes of contact deviation increase and potential refinements (e.g., contact-aware constraints) are not explored.
- Posed mesh quality vs. MoCap: EPIC-Contact shows higher penetration volume than ARCTIC (20.9 vs. 7.24 cm³); the implications of this additional penetration for model training and downstream generalization are not deeply studied.
- Robustness test to EC-fit noise is limited to ARCTIC subset: no ablation on EPIC-Contact itself to quantify how pose noise, contact inaccuracies, or penetration affect training/evaluation at scale.
- Scale estimation via VLM from a single image: ambiguity under perspective and occlusions remains; verification uses only 30 objects, leaving generalization across scenes, classes, and rarer object geometries underexplored.
- Non-uniform scaling captures size but not shape/topology variation within a class: the limits of scaling to approximate diverse instance shapes (e.g., different bottle profiles, mug handle geometry) are not quantified.
- No confidence-weighted training: while per-frame confidence scores are released, there is no demonstrated training strategy that weights or filters supervision by confidence to mitigate propagation noise.
- Use of segmentation masks from VISOR: robustness when such high-quality masks are unavailable in the wild is not addressed; sensitivity to segmentation errors (esp. with transparent/reflective objects) is not studied.
- Bimanual interactions in EPIC-Contact are not characterized: coverage, annotation quality, and HOPformer’s performance in true bimanual manipulation remain unclear.
- Cross-dataset/domain generalization: the paper reports strong ARCTIC performance, but the extent to which EPIC-Contact-trained models transfer to other datasets or real deployments is not systematically evaluated.
- Computational footprint: the model is large (≈1.83B params, ≈7 GiB peak memory) and measured on a GH200 GPU; latency, memory, and throughput on commodity hardware (e.g., mobile/edge devices) and end-to-end video throughput are not reported.
- Camera and scene variability: the impact of egocentric camera mounting variations, FOV changes, and motion blur on pose accuracy is not quantified.
- Handling ambiguous/invisible cases: evaluation and training strategies for frames with severely occluded or out-of-frame objects/hands are not detailed; how predictions are regularized in such cases is unclear.
- Evaluation blind spots for misclassification: since metrics are undefined when the wrong CAD is chosen, the community lacks a standard protocol to measure end-to-end robustness that includes classification errors (e.g., 2D silhouette IoU, category-agnostic pose proxies, or image-based reprojection metrics).
- Lack of ablations on backbone choices: contributions of DINOv2 and WiLoR, and potential gains from joint finetuning vs. frozen features, are not dissected.
- Limited geographic/cultural diversity: EPIC-Contact builds on EPIC datasets; potential bias in object types, kitchen layouts, and manipulation styles is not examined.
- No analysis of annotation throughput improvements: while future work suggests better initializations and active learning, the paper does not quantify how such strategies might reduce the 3–4 min/video annotation time.
Practical Applications
Immediate Applications
Below are actionable uses that can be deployed now, given the paper’s findings, tools, and released assets (HOPformer, EPIC-Contact, EC-fit, symmetry-aware metrics). Each item lists sectors, a short workflow/tool concept, and key assumptions/dependencies that influence feasibility.
- Real-time hand–object pose overlays in AR headsets
- Sectors: AR/VR, software
- Workflow/product: HOPformer-based SDK for Unity/Unreal to render posed hand-object meshes from egocentric RGB streams, enabling more precise occlusion, snapping, and affordance cues during manipulation (e.g., showing how to hold a mug or pan).
- Assumptions/dependencies: Known CAD meshes for supported categories; object classification accuracy (current paper reports 52.9%); GPU acceleration for ~9 FPS (median ~106 ms/sample on GH200, ~7 GiB memory); camera intrinsics/focal-length assumptions; limited to the released object classes.
- Third-person (exocentric) video analytics for manual tasks
- Sectors: Manufacturing, logistics, robotics, safety
- Workflow/product: Fixed-camera monitoring that uses HOPformer’s exocentric generalization to estimate hand–object interactions in workstation videos, enabling task progress tracking, ergonomic checks, and adherence to SOPs.
- Assumptions/dependencies: Coverage of handled objects by the CAD pool; controlled camera placement improves robustness; privacy/compliance for workplace monitoring.
- Contact-aware demonstration capture for robot learning
- Sectors: Robotics
- Workflow/product: Use EC-fit to convert in-the-wild videos into contact-annotated 3D demonstrations (poses + contact patches) as training data for grasp and manipulation policies; leverage symmetry-aware metrics for evaluation of data quality.
- Assumptions/dependencies: Human-object CAD availability; domain gap between human hand and robot gripper requires retargeting; policy training infra; stable grasp segments preferred.
- Skill training and guidance content generation
- Sectors: Education, AR/VR, enterprise training
- Workflow/product: Automatic extraction of contact regions and hand pose to create step-by-step 3D overlays (e.g., where to place fingers on a tool), supporting procedural training in kitchens, labs, or workshops.
- Assumptions/dependencies: Supported object categories and CADs; consistent camera viewpoints enhance UX; device with capable GPU or cloud offload.
- Rehabilitation and occupational therapy monitoring
- Sectors: Healthcare
- Workflow/product: Use hand kinematics and object-relative motion (ACC, MDev) to quantify smoothness and stability during prescribed tasks (e.g., grasping a cup), enabling remote progress tracking.
- Assumptions/dependencies: Clinical validation needed for medical use; reliable video capture conditions; coverage of relevant everyday objects by CAD pool.
- Video-based interaction retargeting for animation/VFX
- Sectors: Media/entertainment, software tools
- Workflow/product: Plugin for DCC tools (e.g., Blender, Maya) to import hand–object pose tracks from video, preserving contact patches and object-relative alignment for realistic animation.
- Assumptions/dependencies: Artist-provided or retrieved CADs; post-processing for temporal smoothing; current runtime adequate for offline pipelines.
- Benchmarking and QA using symmetry-aware metrics
- Sectors: Academia, industry R&D
- Workflow/product: Adopt the symmetry-aware CDev, MDev, ACCo, and Success Rate for fair evaluation of symmetric objects; integrate into CI for HOI model development.
- Assumptions/dependencies: Availability of GT or proxy GT for evaluation; symmetry definitions per class; metric code integration.
- Dataset creation with VLM-assisted object scaling
- Sectors: Academia, robotics, software tools
- Workflow/product: Reuse the paper’s VLM-based scaling prompts and verification protocol to produce instance-scaled CAD alignments from egocentric frames, lowering manual burden in new datasets.
- Assumptions/dependencies: VLM quality (reported ~5.9% relative error, 82.5% within ±10%); prompt design per category; failure cases for transparent/occluded objects need human oversight.
- Rapid prototyping of in-the-wild HOI models
- Sectors: Computer vision R&D
- Workflow/product: Train baseline/evaluation models with EPIC-Contact (posed hand-object meshes + contact labels) and ARCTIC; leverage EC-fit to expand to new videos with similar object categories.
- Assumptions/dependencies: Training infra (GPU memory for 1.83B params); legal rights for new video sources; annotation time (~3–4 minutes per center frame) if extending beyond released data.
- Contact-aware teleoperation interfaces
- Sectors: Robotics, telepresence
- Workflow/product: Estimate human hand pose and contact points from video and map to robot gripper control with contact constraints (e.g., maintain stable grasps), improving operator feedback.
- Assumptions/dependencies: Real-time throughput needs optimization beyond ~9 FPS; mapping from MANO to robot kinematics; robust CAD availability.
Long-Term Applications
The following applications need further research or engineering—for example, broader CAD coverage, robust CAD-free object modeling, model compression for edge devices, or clinical/operational validation.
- CAD-free, category-agnostic in-the-wild HOI systems
- Sectors: AR/VR, robotics, software
- Product vision: Integrate robust instance-level 3D reconstruction (future SAM3D-like models) with HOPformer-style pose estimation to handle arbitrary objects without a CAD pool.
- Dependencies: Advances in CAD-free mesh generation under occlusion/transparency; closed-loop fitting; scalable training data and evaluation protocols.
- On-device egocentric inference on wearables
- Sectors: AR/VR, consumer electronics
- Product vision: Compressed HOPformer variants running at 30–60 FPS on AR glasses or mobile SoCs for fine-grained interaction UX.
- Dependencies: Model distillation/quantization; hardware acceleration; power/thermal budgets; improved classification accuracy and multi-object support.
- Large-scale digital twins of human–object workflows
- Sectors: Manufacturing, logistics, enterprise digital twins
- Product vision: Automatically ingest egocentric/exocentric video to reconstruct 3D, contact-rich task episodes for simulation, what-if analysis, and process optimization.
- Dependencies: Broad object coverage; robust multi-camera fusion; privacy-preserving pipelines; standards for interaction semantics.
- Safety and compliance monitoring with contact semantics
- Sectors: Occupational safety, insurance, compliance
- Product vision: Detect unsafe grasps, slips (via MDev), and hazardous handling in real-time with alerts and logged evidence.
- Dependencies: High recall in cluttered scenes; policy and privacy frameworks; integration with facility IT/OT systems; expanded object and tool catalogs.
- Robot policy learning from in-the-wild demonstrations
- Sectors: Robotics, automation
- Product vision: Train general manipulation skills (pick, pour, twist) using contact-annotated egocentric datasets; leverage symmetry-aware success criteria in evaluation and reward shaping.
- Dependencies: Systematic domain adaptation from human to robot embodiments; sim-to-real pipelines; richer contact physics supervision.
- Personalized assistance and accessibility
- Sectors: Assistive tech, consumer software
- Product vision: Real-time guidance for users (e.g., “grasp here” overlays for tremor-compensated handling), including voice/visual prompts tailored to the object instance.
- Dependencies: Low-latency on-device inference; robust tracking during heavy occlusion; UX and clinical validation.
- Clinical-grade kinematic assessment from daily living
- Sectors: Healthcare, digital health
- Product vision: Longitudinal, at-home metrics of hand function using object-relative acceleration and motion smoothness during ADLs (e.g., drinking, cooking), feeding into rehabilitation plans.
- Dependencies: Regulatory validation; standardized test batteries; bias controls across environments and camera wear.
- Content creation: auto-synthesized interaction assets for games/VR
- Sectors: Media/entertainment
- Product vision: Generative pipelines that turn videos into reusable 3D interaction clips with parameterized contact patches and retargetable objects.
- Dependencies: CAD-free modeling maturity; consistent temporal coherence; tooling integration.
- Standardization of HOI evaluation with symmetry-aware metrics
- Sectors: Standards bodies, academia, industry consortia
- Product vision: Adopt symmetry-aware criteria as a standard benchmarking suite for hand–object tasks, reducing penalization on symmetric objects and surface sliding.
- Dependencies: Community consensus; open-source metric libraries; widespread dataset support.
- Semi-automatic dataset curation at scale
- Sectors: CV/ML infrastructure
- Product vision: Active-learning pipelines that pre-fill contact patches, pose, and scales using models + VLMs, with annotator-in-the-loop verification to reduce per-frame time.
- Dependencies: Robust initializers; UI/UX for fast corrections; quality scoring and propagation with confidence estimates.
Notes across applications:
- Current model and pipeline assume a fixed pool of known CAD models; extending to many categories or instance-specific geometry requires either large CAD libraries, reliable retrieval, or CAD-free reconstruction advances.
- Throughput and memory footprint indicate cloud or high-end edge deployment today; consumer wearables will require compression and/or hardware acceleration.
- Privacy and consent are essential for egocentric and workplace monitoring; deployment must comply with regional regulations and organizational policies.
- Transparent, reflective, or heavily occluded objects remain challenging for CAD-free pipelines; human oversight or multi-view input may be required in the near term.
Glossary
- ACC_h/o (Acceleration Error): A metric measuring discrepancy in predicted vs. ground-truth accelerations for hand (h) and object (o) trajectories; lower is better. Example: "Acceleration Error (ACC, m/s)"
- AAE (Average Angular Error): An object orientation error metric that reports the mean angular deviation in degrees; lower is better. Example: "AAE [] "
- Aggregation module: A network component that aggregates and reduces tokenized features for downstream regression tasks. Example: "Aggregation module is an MLP layer that reduces the number of tokens from $256$ to $39$"
- ARCTIC (dataset): A 3D hand–object interaction dataset used for training and evaluation in this work. Example: "ARCTIC dataset"
- articulated objects: Objects with movable parts (e.g., scissors) requiring pose estimation that accounts for internal joints. Example: "and for articulated objects."
- backbones (deep learning): Core feature extractors in vision models whose capacity affects performance. Example: "stronger backbones"
- bijective mapping: A one-to-one and onto correspondence between two sets; here, contact points on hand and object. Example: "we obtain bijective mapping of contact points between hand and object."
- CAD model (Computer-Aided Design model): A precise 3D model of an object used as a reference for pose estimation. Example: "calculating the object pose for another CAD model is undefined"
- CAD-free (methods): Approaches that do not rely on known CAD models to reconstruct object shape/pose. Example: "CAD-free methods"
- canonical CAD/shape: A fixed, standard reference shape/pose of an object used as the coordinate frame for pose estimation. Example: "relative to its canonical CAD/shape"
- color jitter: A data augmentation technique that randomly perturbs image color characteristics. Example: "color jitter ()"
- contact axis: A direction and anchor on the hand used to transfer contact regions onto the object consistently. Example: "``contact axis'' (blue ball and red line)"
- contact patch: A region of surface contact (e.g., on fingers, palm, or thumb) used to define hand–object interaction. Example: "the corresponding contact patch is transferred to the object"
- Contact Deviation (CDev): A metric (in mm) that measures average distances between corresponding hand–object contact vertices. Example: "Contact Deviation (CDev, mm)"
- CUDA synchronisation: Explicit coordination points to ensure GPU operations complete before timing or measurement. Example: "explicit CUDA synchronisation"
- data augmentation: Techniques applied to training images (e.g., scaling, rotation, color jitter) to improve generalization. Example: "data augmentation is applied to the images"
- diameter-normalised criterion: An evaluation rule that scales distance thresholds by the object’s diameter for fair comparison across sizes. Example: "the same diameter-normalised criterion"
- DINOv2: A self-supervised vision backbone used as the object encoder in the system. Example: "The feature dimension from the object encoder, DINOv2~\cite{oquab2023dinov2} () is $1536$"
- egocentric (view/split): A viewpoint from a head-mounted or first-person camera; split refers to the corresponding evaluation subset. Example: "egocentric split of ARCTIC dataset"
- exocentric (view/split): A third-person or external viewpoint; split refers to that evaluation subset. Example: "exocentric test split"
- feed-forward inference: Performing prediction in a single pass through the network without iterative optimization. Example: "HOPformer performs feed-forward inference"
- focal length (fixed): A camera intrinsic set to a fixed value for weak-perspective projection during training/testing. Example: "a fixed focal length of $1000.0$"
- FLOPs (Floating-Point Operations): A count of arithmetic operations indicating computational cost; often reported per forward pass. Example: "FLOPs per forward"
- geodesic distance: A distance on a curved manifold (here, in pose space), used to match hand pose configurations. Example: "geodesic distance"
- MANO mesh: A parametric 3D hand model used for representing and annotating hand shape and pose. Example: "MANO mesh"
- MLP (Multi-Layer Perceptron): A feed-forward neural network used here for feature projection and token aggregation. Example: "We use an MLP to project features"
- MoCap (Motion Capture): Ground-truth 3D pose capture technology used for evaluation or validation. Example: "3D MoCap poses are available."
- Motion Deviation (MDev): A metric (in mm) quantifying disagreement between hand and object motion over time. Example: "Motion Deviation (MDev, mm)"
- MPJPE (Mean Per-Joint Position Error): A metric (in mm) measuring average error over hand joints; lower is better. Example: "MPJPE [] "
- MRRPE (Mean Root-Relative Position Error): A metric (in mm) measuring positional error relative to a chosen root frame, e.g., for hand/object. Example: "MRRPE and MDev penalise sliding even when symmetry-aware."
- object root: The reference frame or global translation of the object that can be removed for relative comparisons. Example: "after removing the object root"
- occlusion: Visual obstruction of parts of hands/objects that complicates perception and pose estimation. Example: "high occlusion"
- oracle class knowledge: Using ground-truth object class labels during evaluation rather than predicted ones. Example: "Our main results are reported with oracle class knowledge."
- penetration loss: A term penalizing interpenetration between meshes by leveraging signed distance fields. Example: "A penetration loss is utilised to prevent hand-object penetration."
- photo-geometric cues: Combined photometric and geometric signals used to reconstruct 3D shape without strong priors. Example: "photo-geometric cues without learnt priors"
- Pose L2 Error: Average L2 distance between predicted and ground-truth vertices of a mesh; lower indicates better alignment. Example: "Pose L2 Error (average L2 error of predicted vertices)"
- P50/P95 (latency percentiles): Median (50th percentile) and 95th percentile latency statistics for runtime analysis. Example: "latency is summarised with P50 (median) and P95."
- shape priors: Learned statistical regularities about shape used to guide reconstruction or generation. Example: "learnt shape priors"
- Signed Distance Field (SDF): A scalar field giving the signed distance to a surface, positive inside and negative outside a mesh. Example: "Signed Distance Field (SDF)"
- stable grasp: A period where the same hand and object vertices remain in contact across frames. Example: "``stable grasp''"
- Success Rate ([email protected]/[email protected]): The percentage of object vertices predicted within 5%/10% of the object diameter from ground truth. Example: "Success Rate ([email protected]/[email protected], \%)"
- symmetry-aware (metrics): Evaluation variants that account for object symmetries to avoid penalizing equivalent predictions. Example: "symmetry-aware variants of the evaluation metrics."
- throughput: The number of processed samples per second during inference. Example: "Throughput"
- topology (object topology): The vertex/edge/face connectivity of a mesh; mismatched topologies hinder direct pose comparison. Example: "objects differ in their topology"
- tokens (Transformer tokens): Discrete feature vectors processed by the transformer; here reduced for regression heads. Example: "reduces the number of tokens from $256$ to $39$"
- vertex correspondences: Fixed mappings between vertices across meshes used for metric computation or alignment. Example: "vertex correspondences"
- VLM (Vision-LLM): A model combining vision and language to infer object properties such as scale. Example: "a VLM (Gemini 2.5"
- wall-clock timer: A real-time measurement of elapsed time for operations including all system overheads. Example: "end-to-end wall-clock timer"
- weak perspective camera: A simplified camera model assuming small depth variation, often parameterized by a single scale and translation. Example: "predicted weak perspective camera"
- WiLoR: A specific model used for 3D hand localization and reconstruction in the wild. Example: "WiLoR~\cite{Potamias_2025_CVPR_wilor}"