Pose Indicator: Operational Role in Motion Systems

• Pose Indicator is a pose-derived representation that signals intended referents, visibility states, and correction cues in diverse motion-based systems.
• It is applied in human–robot collaboration, pose forecasting, and interactive annotation using geometric, learned, and binary techniques.
• Its operational role refines downstream estimations by converting body configurations into actionable variables while addressing challenges like occlusion and ambiguity.

Pose indicator denotes a pose-derived representation used to signal an intended referent, a visibility state, a correction cue, or a downstream semantic attribute. In recent arXiv literature, the term ranges from a shoulder–wrist extension intersected with a planar workspace in human–robot collaboration (Sassali et al., 27 Jun 2025), to a per-joint visibility variable in global pose forecasting (Adeli et al., 2021), to hand-intent tokens derived from 21 three-dimensional hand keypoints for egocentric question answering (Choi et al., 13 Mar 2026). Across these usages, body configuration is not treated as an end in itself; it is converted into an operational variable that another module can consume for selection, prediction, control, annotation, or assessment.

1. Conceptual scope and terminological variation

The term is not standardized across subfields. In some works, a pose indicator is a deictic signal that resolves which object or region is intended. In others, it is a binary observability flag, a learned correction vector, an action-conditioned prior, or a clinically meaningful posture label. This heterogeneity reflects the fact that “indicator” refers less to a fixed mathematical object than to the role pose plays inside a larger decision system.

Context	Pose indicator	Operational role
Human–robot pointing	Shoulder–wrist extension or finger–object relation	Target localization or object selection
Pose forecasting	Per-joint visibility variable $s_t^p(k)$	Forecast observability; mask regression loss
Pose refinement	Per-joint 2D correction vector	De-bias 2D pose before 3D reconstruction
Interactive annotation	Corrected keypoints, tracklets, trajectory memory	Propagate sparse edits through time
Assessment and monitoring	Milestone pose, gait representation, mounting posture	Infer health, development, or estrus state

This variability is explicit in the literature. The planar HRC system treats pointing as a pose-based indicator that is sensed, reduced to a directional estimate, intersected with a known workspace, and then snapped to candidate targets or areas (Sassali et al., 27 Jun 2025). TRiPOD defines the indicator as a binary joint-level visibility variable that is part of both the observed and predicted state (Adeli et al., 2021). PoseRN uses a learned 2D correction signal that can be interpreted as a pose-quality indicator or refinement stage before final 3D reconstruction (Sayo et al., 2021). IMPose broadens the notion further by organizing corrected keypoints, identity embeddings, and trajectory-bank history into temporally coherent pose trajectories (Ge et al., 3 Jun 2026).

2. Deictic and referential pose indicators

A prominent line of work uses pose indicators for deictic reference, especially pointing. In a planar human–robot collaboration setting, the indicator is defined geometrically from RGB-D pose extraction. OpenPose-based 2D keypoints are deprojected through RealSense depth and intrinsics to three-dimensional shoulder and wrist points $P_s$ and $P_w$. With a planar workspace $ax + by + cz + d = 0$, the pointed location is estimated by line–plane intersection,

$P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$

The system buffers valid estimates and publishes a running average of five samples; downstream snapping computes the mean of $N=15$ gesture samples and only accepts it if their radial distance from the mean remains under $5$ cm. In tabletop experiments, average pointing accuracy was $3.0$–$3.3$ cm for the dominant right hand and $6.4$–$P_s$0 cm for the left hand, with about $P_s$1 cm improvement after transformation into the workplane frame (Sassali et al., 27 Jun 2025).

Other systems replace explicit line–plane geometry with learned or panoramic formulations. DeePoint treats pointing as a full-body spatio-temporal gesture rather than a local hand pose. Using OpenPifPaf joints, ResNet-34 features, a Joint Encoder, and a Temporal Encoder, it outputs a pointing probability $P_s$2 and a camera-coordinate unit vector $P_s$3. On the DP Dataset, the full model achieved $P_s$4 angular error and precision/recall $P_s$5, outperforming head-and-hands-only and hands-only variants; temporal context improved both detection and direction estimation, with $P_s$6 frames giving $P_s$7 and $P_s$8 (Nakamura et al., 2023).

In omnidirectional imaging, Point Anywhere models the pointing cue as a great circle on the sphere rather than a Euclidean image line. It localizes the user with YOLOv5, estimates pose with OpenPose on perspective projections, slides $P_s$9 field-of-view ROIs every $P_w$0 along the pointing direction, and ranks candidate objects by geometric and learned cues. The candidate feature vector is built from distance to the pointing direction $P_w$1, category frequency $P_w$2, detector confidence $P_w$3, area $P_w$4, and horizontal distance $P_w$5; a linear SVC improves directed-object estimation over pure distance-based selection, with best reported TOP-1 accuracy $P_w$6 (Kotani et al., 2023).

Monocular RGB systems often introduce approximate 3D geometry to disambiguate referents. In pointing-based object recognition, MediaPipe Pose provides shoulder, elbow, and wrist, MoGe lifts body joints and object centroids into camera-centered 3D, and target selection uses cosine similarity between arm direction and shoulder-to-object direction,

$P_w$7

Across detectors, 3D lifting improved object recognition, especially in hard scenes with overlapping objects; for example, with YOLOv8s in hard cases, accuracy rose from $P_w$8 in 2D to $P_w$9 with MoGe, and recall rose from $ax + by + cz + d = 0$0 to $ax + by + cz + d = 0$1 (Hajdúch et al., 16 Mar 2026).

Transformer-based formulations make the pose–object relation explicit. MM-ITF represents the hand by 21 MediaPipe landmarks, the scene by object centroids from OWLv2, and each hand–object pair by an angle

$ax + by + cz + d = 0$2

where the finger direction is approximated from the index fingertip and DIP joint. Inter-modality attention uses pose tokens as queries and object tokens as keys and values; the decoder then scores relationship tokens against the pose–object memory. On the NICOL tabletop dataset, the full three-modality model reached $ax + by + cz + d = 0$3 accuracy and $ax + by + cz + d = 0$4 F1, whereas the two-modality pose-plus-object variant reached $ax + by + cz + d = 0$5 accuracy, showing that explicit pose–object angular encoding materially sharpens target estimation (Müller et al., 5 Sep 2025).

Egocentric multimodal reasoning extends the same idea to question answering. EgoPointVQA frames pointing as the evidence needed to resolve deictic expressions such as “this” and “the second thing I pointed at.” HINT maps WiLoR-reconstructed 21-keypoint camera-space hand pose $ax + by + cz + d = 0$6 into a single frame-aligned token,

$ax + by + cz + d = 0$7

and inserts it only when detection confidence exceeds $ax + by + cz + d = 0$8. On the real test set of 672 questions over 300 videos, HINT-InternVL3-14B achieved $ax + by + cz + d = 0$9 average accuracy over six tasks, improving on base InternVL3-14B and yielding the largest gains on Reference and Temporal tasks (Choi et al., 13 Mar 2026).

3. Visibility, correction, and action-conditioned indicators

A second major usage defines the pose indicator as an internal state variable for forecasting or refinement rather than as a deictic referent. TRiPOD models global pose forecasting with per-joint tuples $P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$0, where $P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$1 is a binary visibility score that is $P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$2 if the joint is invisible. The full state is

$P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$3

The visibility indicator is part of both input and prediction, is trained with binary cross-entropy, and gates coordinate regression because invisible joints do not contribute $P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$4. The paper also introduces visibility-aware evaluation, including VAM and direct IoU/F1 evaluation of future visibility prediction. On PoseTrack, where $P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$5 of joints are invisible in observation frames and $P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$6 in future frames, TRiPOD improves substantially over SC-MPF on both VIM and VAM (Adeli et al., 2021).

PoseRN uses “indicator” in a different sense: a learned 2D bias correction field that reveals systematic mismatch between image annotations and the projection of MoCap-defined joints. The discrepancy is modeled as

$P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$7

PoseRN takes an initial triangulated 3D pose and per-view 2D keypoints, predicts a per-joint 2D correction vector, and re-runs multi-view reconstruction. The model is a lightweight MLP with two linear layers, trained with MSE on the target bias. On Human3.6M, the de-biasing step improves MPJPE from $P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$8 mm to $P_i = P_s + t\cdot\Vec{P_sP_w}, \qquad t = \frac{-(\langle \mathbf{n}, P_s \rangle + d)}{\langle \mathbf{n}, \Vec{P_sP_w}\rangle}.$9 mm over the authors’ vanilla multi-view optimization, and on a 4-joint subset it reaches $N=15$0 mm (Sayo et al., 2021).

ActionPrompt shifts the indicator from observability or correction to prior structure. Its Action Prompt Module uses Action-related Text Prompt (ATP) and Action-specific Pose Prompt (APP) to inject action-conditioned semantic and position-aware information into video-based 2D-to-3D human pose estimation. ATP aligns shallow pose features with CLIP-based action prompts; APP uses a bank $N=15$1 of action-specific pose prompts and refines the deep pose feature through a transformer decoder. Across VPose, A3DHP, and MixSTE on Human3.6M, the module consistently reduces P1, P2, and Tail D-MPJPE; the paper states that APM achieves an average gain of more than $N=15$2 in MPJPE and is especially helpful on depth-axis error and hard actions (Zheng et al., 2023).

These formulations show that a pose indicator may be geometric, binary, corrective, or prior-like. What remains constant is its operational role: it modifies the effective state space seen by the downstream estimator.

4. Interactive and annotation-oriented indicators

Interactive annotation systems recast pose indicators as user-corrected signals that can be propagated and regularized. Click-Pose begins from the observation that pose errors are structured across joints. Built on ED-Pose, it edits keypoint position queries directly, reinitializes content queries from a learnable codebook $N=15$3, and loops the Human-to-Keypoint decoder so that one trusted correction can update the full pose. Its pose error modeling explicitly simulates four error types—jitter, miss, swap, and inversion—and trains the decoder to reconstruct the correct pose. On COCO, Click-Pose reaches NoC@95 $N=15$4 versus $N=15$5 for ViTPose; on Human-Art, NoC@95 is $N=15$6 versus $N=15$7. Without clicks, Click-Pose-C0 improves ED-Pose from $N=15$8 AP to $N=15$9 AP on COCO and from $5$0 AP to $5$1 AP on Human-Art (Yang et al., 2023).

IMPose generalizes this idea to multi-person video annotation. It treats corrected keypoints as priors rather than final outputs: after a one-frame edit, CoTracker propagates corrected keypoint and box references forward, the detector refines them on subsequent frames, and an instance-level tracker maintains identity consistency through keypoint-aware embeddings and a trajectory bank. The framework represents pose through raw 2D keypoints, relative geometry, Fourier features, tracklets, and historical trajectory windows. Empirically, IMPose requires only 27 clicks per 1,050-frame video on 3DPW and 3 clicks per tracklet per 84-frame video on PoseTrack21. At one click on PoseTrack21, it reaches $5$2 mAP and $5$3 HOTA/DetA, far above the compared annotation tools; it also enabled an extension of PoseTrack21 by 188K pose instances and 3.55M keypoints with 10 annotators in 10 hours (Ge et al., 3 Jun 2026).

This body of work suggests that in annotation systems the pose indicator is not a static estimate but a mutable, high-value correction signal. Its utility lies in how efficiently it reorganizes the remainder of the pose state.

5. Pose indicators for diagnosis, assessment, and monitoring

Several domain-specific systems use pose indicators as interpretable proxies for latent biological or health-related variables. In infant motor assessment, AIMS pose recognition treats milestone postures as developmental indicators. The proposed pipeline combines a ResNet-50 image branch with LMMD-based unsupervised domain adaptation, an HRNet-plus-SMPLify pose branch, and a hierarchical infant pose classifier that maps single infant images into 4 coarse classes—Prone, Supine, Sitting, Standing—and 12 fine classes such as Forearm Support, Rolling, and Sitting w/o Support. On real data, Image Branch (w/ LMMD) reaches $5$4 coarse-level accuracy, Pose Branch reaches $5$5 fine-level accuracy, and the full hierarchical model reaches $5$6 fine-level accuracy (Yang et al., 2022).

Human health indicator prediction from gait video uses pose pretraining rather than explicit posture labels. GlanceNet is first trained for SMPL-based video pose estimation using GLANCE, a Global-Local Aware aNd Centrosymmetric Encoder, together with a bidirectional GRU and a VIBE-style SMPL generator. The pretrained spatial-temporal encoder is then transferred to MoVi, where the last-frame sequence feature is average-pooled and fed into an SVM regressor for BMI, age, height, and weight. The reported errors are BMI MAE $5$7, age MAE $5$8, height MAE $5$9, and weight MAE $3.0$0; height is the easiest in relative terms, with MAPE $3.0$1 (Li et al., 2022).

In precision livestock monitoring, FSMC-Pose treats mounting posture as a visual estrus indicator in dairy cattle. The MOUNT-Cattle dataset contains 1,176 mounting instances in COCO format with 16 keypoints and three visibility labels. FSMC-Pose combines a lightweight backbone, CattleMountNet, with SFEBlock and RABlock, and a Spatial-Channel Self-Calibration Head. On the combined benchmark, the model reaches AP $3.0$2, AP$3.0$3, AP$3.0$4, AR $3.0$5, and $3.0$6 FPS, while using about $3.0$7 GFLOPs and $3.0$8M parameters according to the detailed tables (Li et al., 17 Mar 2026).

These application-specific systems differ in modality and target variable, but they share an important property: the indicator remains interpretable. Whether the output is “Forearm Support,” BMI, or mounting posture, the intermediate pose representation is intended to preserve semantic structure rather than collapse it into an opaque latent vector.

6. Recurring limitations and open problems

Across pointing systems, the most persistent failure mode is occlusion. In planar HRC, overlapping arms can cause the wrong wrist to be estimated and produce ghost gestures on the workplane (Sassali et al., 27 Jun 2025). DeePoint reports its worst errors for low-pitch cases when the person points down while facing away from the camera, so the arm is occluded by the body (Nakamura et al., 2023). EgoPointVQA attributes residual errors to unreliable hand reconstruction under motion blur or partial occlusion, and to rapid viewpoint drift that breaks hand–object linkage (Choi et al., 13 Mar 2026). Pointing-based object recognition adds another geometric limitation: long user–object distances amplify small angular errors, while representing objects by lifted box centroids remains coarse for elongated or partially occluded targets (Hajdúch et al., 16 Mar 2026).

A second recurring issue is that many indicators remain deliberately simple. TRiPOD’s visibility score is binary rather than a richer uncertainty model (Adeli et al., 2021). The tabletop HRC localizer uses a shoulder–wrist extension with running averages and thresholding, but no explicit gesture-onset model or trajectory-level interpretation (Sassali et al., 27 Jun 2025). ActionPrompt depends on action labels during training and becomes more effective as the sequence length increases to 243 frames, so the action-as-indicator assumption weakens for short windows (Zheng et al., 2023). IMPose currently addresses only 2D pose and still identifies very long temporal gaps, prolonged disappearance, and extremely crowded scenes as difficult cases (Ge et al., 3 Jun 2026).

A third limitation is deployment dependence on upstream estimators. MM-ITF presupposes MediaPipe hand landmarks and OWLv2 detections, and its patch confusion matrix shows systematic confusions when multiple objects lie on the same 2D pointing direction (Müller et al., 5 Sep 2025). PoseRN cannot recover from catastrophic 2D detector failure because it refines rather than replaces the initial 2D and 3D estimates (Sayo et al., 2021). Click-Pose is highly effective as a corrective system, but its quantitative evaluations use simulated user clicks, so practical performance depends on actual click quality and correction strategy (Yang et al., 2023).

These patterns suggest two broad directions. First, many current pose indicators are strong enough for controlled settings yet remain brittle under occlusion, ambiguity, or missing depth. Second, the most successful systems often combine pose with a second structure—object candidates, language, visibility masks, action priors, trajectory memory, or domain knowledge—rather than relying on pose alone.