TAIHRI: Task-Aware 3D Keypoint Localization

• TAIHRI is a task-aware, egocentric 3D keypoint localization system that outputs metric coordinates in the robot camera frame for close-range HRI.
• It leverages human prompts to select task-critical body parts and employs a 2D reasoning stage before discretized 3D token prediction to mitigate depth ambiguities.
• The model benefits from a tailored synthetic dataset and reinforcement finetuning with a pose-aware Huber reward, outperforming conventional global pose methods.

Searching arXiv for the TAIHRI paper and closely related context. {"query":"TAIHRI Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction", "max_results": 5} TAIHRI is a task-aware, egocentric 3D human keypoint localization model for close-range human-robot interaction (HRI). It is presented as the first Vision-LLM (VLM) tailored for close-range HRI perception, with the specific objective of understanding a user’s motion command, selecting the body parts relevant to the task, and predicting their metric-scale 3D coordinates directly in the robot’s camera coordinate system rather than in a root-relative body frame (Li et al., 10 Apr 2026). The central reframing is methodological: instead of optimizing for whole-body reconstruction quality relative to a root joint, TAIHRI treats HRI perception as task-conditioned spatial grounding of critical body parts under egocentric viewing, where truncation, self-occlusion, robot-induced occlusion, and small absolute metric errors are operationally consequential.

1. Problem formulation and conceptual scope

TAIHRI is motivated by a mismatch between conventional 3D human pose estimation and close-range embodied interaction. Standard methods are typically root-centric and evaluated by how accurately they reconstruct the whole body relative to a root joint. In the setting targeted here, that objective is insufficient, because the robot needs the absolute 3D locations of the body parts that matter for the current task in the robot camera frame. The examples emphasized include localizing a hand for a handshake, elbows and shoulders for lifting assistance, and torso or underarm keypoints for a transfer task (Li et al., 10 Apr 2026).

The task is defined as follows: given an egocentric RGB image $I$ and a human prompt $P$ describing the interaction task or desired body parts, predict a set of task-relevant 3D keypoints $\{J_i\}_{i=1}^N$ in the robot camera coordinate system, assuming known camera intrinsics $K_{int}$ at inference. This formulation is explicitly task-aware and coordinate-aware. It does not seek generic full-body pose recovery as its primary output; rather, it seeks an actionable subset of metric keypoints.

This suggests a different evaluation philosophy from conventional pose estimation. In close-range HRI, global reconstruction quality is secondary to whether the robot can identify and localize the specific contact-relevant or assistance-relevant joints with sufficient metric accuracy. A plausible implication is that errors tolerated in root-aligned benchmarks can be unacceptable in perception-action loops where end-effector motion depends directly on camera-frame joint coordinates.

2. Core method: language-conditioned keypoint selection and discretized 3D localization

TAIHRI is built around five stated design elements: task-aware prompting, discretized interaction-space localization, 2D keypoint reasoning before 3D, reinforcement finetuning with a pose-aware reward, and direct camera-coordinate output (Li et al., 10 Apr 2026).

The model uses the user’s motion or interaction command to identify which body parts matter. This is the semantic component of the system: language does not merely accompany the image, but conditions the selection of the output keypoints. The spatial component is implemented through a discretized interaction-space representation. The robot’s interaction space is modeled as a cuboid of width $W$, height $H$, and depth $D$, and each coordinate is quantized into the range $[0,999]$: $$\{ X_i, Y_i, Z_i \} = \left\{ \left\lfloor \frac{x_i}{W} \times 1000 \right\rfloor, \left\lfloor \frac{y_i}{H} \times 1000 \right\rfloor, \left\lfloor \frac{z_i}{D} \times 1000 \right\rfloor \right\}.$$ The discretized coordinates are then emitted autoregressively as tokens, aligning 3D localization with next-token prediction in a VLM.

A central technical feature is the sequential reasoning strategy described as “2D keypoint reasoning via next token prediction.” The model first reasons about image-plane keypoints and then predicts depth, emitting the keypoint name, then 2D position tokens, then 3D voxel tokens. The stated rationale is geometric: 2D localization is easier than direct 3D prediction under depth ambiguity, and anchoring the correct image evidence before inferring depth improves stability.

The paper also stresses the importance of camera geometry. Instead of introducing a separate camera adapter, TAIHRI follows intrinsic-conditioned augmentation: the focal length is unified to a fixed value, for example $1000$, by resizing the image, and principal-point variation is simulated using random crop augmentation. The intended effect is to let the VLM internalize camera geometry without disrupting pretrained representations.

3. Data construction and training pipeline

To support this problem setting, the work introduces CloseHRI, a large synthetic dataset tailored to close-range egocentric HRI, on the grounds that Human3.6M, MPI-INF-3DHP, AGORA, and BEDLAM do not adequately cover close egocentric robot viewpoints (Li et al., 10 Apr 2026). The data-generation pipeline is explicitly specified:

• motion sequences are sampled from AMASS;
• normal maps are rendered with SMPL-X in Blender;
• a virtual camera is placed at $P$0 to $P$1 meters, with randomized height and orientation to mimic robot-mounted cameras;
• the normal maps are used as control signals for SDXL to synthesize photorealistic images with diverse backgrounds and text prompts;
• low-quality samples are filtered using SAM3 with IoU $P$2;
• samples are further filtered by 2D reprojection error using VitPose, keeping only those with error below 15 pixels.

The resulting dataset contains over 1 million images, and the final training set is about 1.2 million images when combined with BEDLAMv1, BEDLAMv2, and PDhuman after filtering out samples farther than 3 meters. The system also uses a prompt bank of over 6,000 interaction-centric prompts for language-conditioned control.

TAIHRI is built on Qwen3-VL backbones, specifically 4B and 2B variants. The supervised finetuning stage uses batch size 4, 5 epochs, learning rate $P$3, and a prompt schedule in which prompts are sampled from the prompt bank with 50% probability and a default prompt asking for 1 to 6 keypoints is used for the other 50%. Reinforcement finetuning uses GRPO, with rollout size 8, KL penalty coefficient $P$4, batch size 32, 5 epochs, learning rate $P$5, and 10,000 sampled images during RFT. Training is reported on 4 NVIDIA H20 GPUs.

The reward is defined over visible joints $P$6 and uses a robust Huber penalty together with an indicator term. The same formulation is used for both 3D and 2D rewards by choosing corresponding units and hyperparameters. The ablation result reported for plain MSE is strongly negative, which the paper characterizes as very harmful.

4. Evaluation protocol and reported empirical performance

Evaluation is conducted on two first-view egocentric interaction benchmarks: Harmony4D-Test, consisting of 6,389 close human-human interaction frames from a head-mounted camera, and the EgoBody test set, consisting of 62,155 frames, with 5,000 close-range frames within 3 meters used for final evaluation (Li et al., 10 Apr 2026). Four body-part configurations are used to test task-aware selectivity: Upper, Lower, L-Upper, and R-Upper.

The metric is G-MPJPE in millimeters, defined as global-coordinate mean per joint position error in camera coordinates, without root alignment or rigid transform. This choice is integral to the paper’s argument: HRI requires absolute metric localization rather than relative pose correctness.

On Harmony4D-Egocentric, the reported G-MPJPE values for TAIHRI are $P$7 for Upper, Lower, L-Upper, and R-Upper. The corresponding values for TAIHRI-2B are $P$8. The listed baselines are CameraHMR, PromptHMR, 3DB-H, and 3DB-DINOv3, all with higher reported errors for each grouping. On the EgoBody test sample, TAIHRI reports $P$9, again outperforming the listed baselines, while TAIHRI-2B reports $\{J_i\}_{i=1}^N$0.

The paper highlights especially large gains on task-critical distal keypoints such as wrists, where previous global-pose methods struggle most. It also compares against general-purpose multimodal models and depth-based baselines on 50 Harmony4D-Egocentric samples with the upper-body task. The reported G-MPJPE values are 1298.3 for GPT-5.2, 436.9 for Gemini-2.5 Pro, 1298.3 for Qwen3-VL-235B-A22B-Instruct despite a reported 51/51 on 2D/3D columns, 352.2 for VP + Depth Anything 3, 282.3 for VP + DepthLM, and 97.2 for the full TAIHRI model.

These comparisons support the paper’s narrower claim that physically consistent 3D coordinate prediction for close-range HRI is not solved by generic VLM capability alone. A plausible implication is that the coupling of language, articulated body structure, camera geometry, and task-conditioned output space is more important here than broad multimodal generality.

5. Ablations and mechanistic interpretation

The ablation studies isolate the contribution of camera-intrinsic handling, 2D reasoning, reinforcement finetuning, and reward design (Li et al., 10 Apr 2026). For camera intrinsics, removing intrinsic handling causes substantial degradation: the reported values for “w/o Cam Int.” are $\{J_i\}_{i=1}^N$1, and the “w Ray Emb.” alternative remains poor at $\{J_i\}_{i=1}^N$2. The paper interprets this as evidence that explicit camera-intrinsic handling is crucial.

For training strategy, “w/o 2D” yields $\{J_i\}_{i=1}^N$3, “w/o RFT” yields $\{J_i\}_{i=1}^N$4, and “w $\{J_i\}_{i=1}^N$5” yields $\{J_i\}_{i=1}^N$6. The reported pattern is consistent: the 2D reasoning stage is beneficial, reinforcement finetuning adds further improvement, and naïve MSE-based RL is catastrophic.

These ablations are important because they provide a mechanistic account of where the gains arise. The results suggest that TAIHRI’s improvement is not reducible to using a larger backbone or prompt conditioning alone. Rather, the decisive components are geometric decomposition into 2D then depth, explicit treatment of camera intrinsics, and a reward shaped around visible-joint localization rather than a plain regression objective.

6. Downstream uses, scope conditions, and limitations

The paper presents two downstream adaptations and a proof-of-concept robot-control loop (Li et al., 10 Apr 2026). First, the prompt bank of over 6,000 interaction-centric prompts enables natural language control, allowing prompts such as “assist the person to stand from left” or “lift the person from the wheelchair” to select the relevant body parts and localize their keypoints. Second, TAIHRI’s keypoints are used as anchors to place a root-relative mesh into the robot’s camera coordinate system: given a predicted mesh $\{J_i\}_{i=1}^N$7 in normalized space and predicted keypoints $\{J_i\}_{i=1}^N$8, a transformation matrix $\{J_i\}_{i=1}^N$9 is computed to align mesh keypoints to predicted keypoints, and then applied as $K_{int}$0. Third, the system is used in a closed-loop robot setup for tasks including shaking hands and shoulder massage, where TAIHRI outputs guide robot end-effectors via inverse kinematics.

The broader implication identified by the paper is that HRI perception should be task-conditioned and metric-aware rather than merely pose-centric. This suggests a representational shift from “recover the body” to “localize the interaction-relevant parts in my own camera space.”

The paper also implies several limitations. Performance depends on known camera intrinsics. The method is tailored to close-range egocentric interaction rather than general scene understanding. Severe truncation or occlusion still makes 3D localization harder, especially for distal joints. The discretized coordinate representation trades continuous precision for token compatibility. The reported downstream control demonstrations are presented as proof-of-concept rather than full large-scale robotics validation.

Within those scope conditions, TAIHRI can be understood as a task-aware 3D keypoint grounding framework for embodied interaction: a VLM that interprets interaction commands, selects relevant body parts, reasons first in 2D and then in discretized 3D, and outputs metric keypoints directly in the egocentric camera frame.