TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction

Abstract: Accurate 3D human keypoints localization is a critical technology enabling robots to achieve natural and safe physical interaction with users. Conventional 3D human keypoints estimation methods primarily focus on the whole-body reconstruction quality relative to the root joint. However, in practical human-robot interaction (HRI) scenarios, robots are more concerned with the precise metric-scale spatial localization of task-relevant body parts under the egocentric camera 3D coordinate. We propose TAIHRI, the first Vision-LLM (VLM) tailored for close-range HRI perception, capable of understanding users' motion commands and directing the robot's attention to the most task-relevant keypoints. By quantizing 3D keypoints into a finite interaction space, TAIHRI precisely localize the 3D spatial coordinates of critical body parts by 2D keypoint reasoning via next token prediction, and seamlessly adapt to downstream tasks such as natural language control or global space human mesh recovery. Experiments on egocentric interaction benchmarks demonstrate that TAIHRI achieves superior estimation accuracy for task-critical body parts. We believe TAIHRI opens new research avenues in the field of embodied human-robot interaction. Code is available at: https://github.com/Tencent/TAIHRI.

• The paper introduces TAIHRI, a vision-language model that precisely localizes 3D human keypoints in camera coordinates for robust close-range human-robot interaction.
• It employs a two-stage pipeline combining 2D reasoning and reinforcement fine-tuning on a synthetic CloseHRI dataset, achieving state-of-the-art G-MPJPE metrics.
• TAIHRI enables flexible, instruction-driven keypoint selection, facilitating safe and adaptive robotic manipulation in complex, occlusion-prone environments.

Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction: An Expert Analysis of TAIHRI

Introduction

The paper "TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction" (2604.08921) introduces TAIHRI, a Vision-LLM (VLM) for egocentric, metric-accurate 3D human keypoint localization tailored to close-range HRI. This model bridges the methodological and practical gap between standard root-relative whole-body human mesh recovery and the critical need for robust, instruction-driven, task-specific 3D keypoint localization in the camera coordinate frame for real-world robotic systems. Unlike conventional approaches that prioritize holistic body pose or root-relative estimation, TAIHRI focuses on perceiving interaction-relevant, semantically specified body landmarks with high spatial precision—addressing a pivotal requirement in manipulation, assistive, and safety-critical embodied robotics.

Figure 1: TAIHRI system overview, focusing on task-aware keypoint localization for close-range HRI.

Limitations of Existing Approaches

Canonical mesh-based human pose estimation pipelines (e.g., HMR, PARE, SMPL derivatives) optimize for full-body reconstructions and are evaluated using root-centric errors, which fails to accommodate the requirements of robotic reasoning and action planning in egocentric spatial frames. More recent works (CameraHMR, PromptHMR, SAM 3D Body) have migrated toward estimating human pose in global/camera coordinates by integrating camera intrinsics during both training and inference. However, these pipelines exhibit marked sensitivity to occlusions, truncations, and perspective distortion that are prevalent in close-range, robot-mounted camera setups, particularly impacting distal and task-critical joints such as wrists and ankles.

Figure 2: Differences between general human body estimation and HRI perception; TAIHRI shifts focus to camera-space, task-relevant landmarks.

Methodology

CloseHRI: A Synthetic Close-Range Egocentric Dataset

To surmount the data shortfall in close-range HRI, the authors synthesize CloseHRI, a 1.2M-image dataset featuring photorealistic, richly annotated, occlusion-diverse egocentric views. Using normal maps rendered with SMPL-X, camera viewpoints are sampled at 0.5–3 meters around the subject, and a generative pipeline (WildHuman + SDXL) produces RGB images with precise 3D keypoint ground truth. Rigorous quality filtering employs concept-based segmentation and 2D reprojection error culling to enforce geometric fidelity, facilitating robust metric-space learning.

Figure 3: CloseHRI dataset generation: motion sampling, rendering, photorealistic synthesis, and quality filtering.

Discretized Interaction Space and 2D Reasoning

TAIHRI discretizes the robot's interaction volume into a 3D voxel grid (1000 bins per axis), enabling the model to treat 3D coordinate prediction as autoregressive token generation—a strategic fit for Vision-Language Transformers. Keypoint localization proceeds via a decomposed, chain-of-thought style: first, the 2D pixel coordinates of relevant joints are inferred, which then condition the prediction of the corresponding discretized depth, supporting higher accuracy in depth-ambiguous monocular settings. Intrinsics conditioning is achieved by focal length unification and principal-point simulation during training, which ensures robust generalization to real-world optics without adapter layers or explicit online calibration.

Vision-LLM Architecture and Fine-tuning

TAIHRI extends Qwen3-VL with interaction-centric prompt conditioning. During training, both generic and task-specific prompts from an extensive prompt bank guide the model to focus on body regions pertinent to diverse HRI tasks. Supervised fine-tuning (SFT) establishes baseline capabilities, followed by reinforcement finetuning (RFT) via Group Relative Policy Optimization (GRPO) with a pose-structural reward blending robust Huber loss and PCK-based metrics. This hybrid training maximizes localization accuracy while mitigating the adverse effects of outliers and ensures policy stability.

Figure 4: TAIHRI inference workflow: egocentric image and instruction fusion, autoregressive token generation for keypoint names and 2D/3D locations, and downstream application integration.

Experimental Evaluation

Benchmarks and Metrics

Evaluation targets Harmony4D-Test and EgoBody—state-of-the-art egocentric datasets with high-fidelity 3D ground truth. The principal metric is Global Coordinate Mean Per Joint Position Error (G-MPJPE) in millimeters, computed without root alignment, directly reflecting the system's readiness for robot-perceived spatial reasoning and interaction safety.

Comparative and Ablation Results

TAIHRI establishes new SOTA performance across all body-part configurations and datasets, achieving statistically significant reductions in G-MPJPE—particularly for distal, task-critical regions:

• On Harmony4D-Egocentric, TAIHRI (4B) yields 93.83 mm G-MPJPE (Upper), outperforming SAM 3D Body (124.91 mm) and CameraHMR (167.50 mm).
• On EgoBody, TAIHRI delivers 75.77 mm (Upper), a substantial improvement over prior works.

Critically, standard VLMs and 3D-aware multimodal LLMs (e.g., GPT-5.2, Gemini-2.5 Pro, Qwen3-VL-235B) do not natively achieve reliable or physically plausible camera-space keypoint predictions, with errors an order of magnitude above TAIHRI. Depth-based compositional pipelines (VitPose + DepthXXX) also lag, indicating the limitations of 2D-to-3D backprojection without articulated structural priors.

Ablations confirm that camera intrinsics injection is essential (removal increases G-MPJPE to >400 mm), and the two-stage 2D reasoning + RFT pipeline is crucial for SOTA performance.

Figure 5: Qualitative comparison: TAIHRI vs prior SOTA; TAIHRI accurately localizes occluded/distal keypoints even under extreme truncation.

Practical Applications and Downstream Implications

TAIHRI enables semantically flexible, instruction-driven localization wherein robots can adaptively focus on any anatomical region specified in natural language—supporting a broad spectrum of interactive tasks, from assistive bimanual manipulation (wheelchair transfer, handshaking) to fine-grained affordance targeting (shoulder for massage). This is achieved via prompt-based control, as illustrated by its capacity to modulate attention across left/right, upper/lower body, or arbitrary joint sets.

The model's outputs serve as anchor points for mesh registration, thereby grounding any parametric mesh estimate into the robot's spatial frame for seamless mesh-to-world alignment. These features complement or replace standard root-based alignment, allowing HRI systems to function robustly in spatially ambiguous or multi-person scenes.

Figure 6: Top: Flexible language-driven keypoint selection. Bottom: Mesh repositioning from predicted anchors for accurate global alignment.

Theoretical and Future Directions

This work positions task-aware, instruction-conditioned VLMs as the core of embodied human understanding, highlighting the insufficiency of root-centric, reconstruction-oriented objectives for HRI and arguing for metric, task-relevant, camera-space estimation as the ground standard. The implications extend broadly:

• Semantically controlled perception for closed-loop robot controllers
• Enhanced transferability to diverse robotic platforms and camera geometries
• Modular integration with downstream mesh, tracking, or manipulation frameworks

Future directions may encompass unsupervised adaptation to robot-specific egocentric distributions, online real-time deployment under hardware constraints, and the extension to dynamic, occlusion-rich environments featuring concurrent multi-agent interactions and multi-object affordance reasoning.

Conclusion

TAIHRI is the first model to realize accurate, instruction-conditioned, metric-space 3D keypoint localization for close-range HRI, outperforming both human mesh recovery algorithms and foundational VLMs in this critical application regime. By fusing discretized space representations, 2D chain-of-thought reasoning, and reinforcement-enhanced training, the model redefines the operational paradigm for close-proximity robotic perception. These results substantiate task-aware VLM architectures as the logical foundation for future advances in safe, robust, and context-recognizing embodied AI.