---
title: 'EgoPoint-CoT: Egocentric Visual Chain-of-Thought'
url: https://www.emergentmind.com/topics/egopoint-cot
type: topic
---

# EgoPoint-CoT: Egocentric Visual Chain-of-Thought

Searching arXiv for the cited papers to ground the article in current records.
arxiv_search(query="PointVG-R EgoPoint-CoT egocentric pointing visual grounding", max_results=10)
arxiv_search(query="Beyond Language Grounding Referring Expressions with Hand Pointing in Egocentric Vision", max_results=10)
EgoPoint-CoT is a visual Chain-of-Thought dataset for egocentric pointing-based visual grounding, introduced as the “cold-start” supervised corpus for PointVG-R. It provides richly annotated egocentric images together with fine-grained geometric reasoning traces, so that training supervision covers both the target object and the intermediate reasoning process used to localize it. The dataset is built on EgoPoint-Ground, which established the underlying egocentric deictic grounding benchmark, and it operationalizes a four-stage Visual Chain-of-Thought format oriented around hand detection, keypoint extraction, pointing geometry, and target alignment [2606.24539][2603.26646].

## 1. Position within egocentric deictic grounding

EgoPoint-CoT arises from the problem setting in which an agent must localize an intended target object by interpreting a pointing gesture in an egocentric scene. In the associated line of work, traditional visual grounding is described as predominantly text-based, whereas pointing-based grounding requires explicit handling of spatial geometry and gestural cues. EgoPoint-Ground formulated this setting as Egocentric Deictic Grounding (EDG), with input consisting of an image $I$, a pointing hand box $B_{hand}^{2D}$, and an underspecified linguistic expression $S$, and output given by the intended target box $B_{obj}^{2D}$ through the maximum-a-posteriori estimate
$$
\hat B_{obj}=\arg\max_B P(B\mid I,\;B_{hand}^{2D},\;S).
$$
EgoPoint-CoT specializes this general benchmark by supplying explicit reasoning trajectories that supervise geometric interpretation inside a multimodal model rather than leaving that process implicit [2603.26646].

A common misconception is to treat EgoPoint-CoT as merely another bounding-box dataset. Its defining property is not only that it labels the hand and target, but that it encodes the intermediate reasoning variables as a structured sequence. This makes it a dataset for supervised reasoning traces as well as localization targets [2606.24539].

## 2. Composition, provenance, and corpus statistics

EgoPoint-CoT contains 15 455 egocentric images, split 7 : 1 : 2 into train (10 837 samples), validation (1 546), and test (3 072). It is built on EgoPoint-Ground and combines real-world smart-glass captures and synthetic renders to ensure diverse lighting, clutter and object types. The corpus includes 345 distinct object categories and 136 468 object instances. On average, there are 8.83 annotated objects per image, with $\sigma = 12.14$ and a maximum of 76, indicating both sparse and crowded scenes. Hands cover on average 11.4 % of image area, with median 9.2 %, while targets cover 19.8 %, with median 13.8 %. The hand–target center distance averages 374 px, normalized to 0.203 of the image diagonal, which is reported as showing typical close-range pointing [2606.24539].

These statistics differ from the underlying EgoPoint-Ground benchmark, which comprises 15 338 egocentric images spanning 140 object categories in over 20 everyday scenarios, with an average of 2.8 candidate objects per image, 63.7 % of samples containing two or more instances of the same category, and 42.5 % exhibiting moderate or severe occlusion. EgoPoint-Ground also includes 2 171 negative samples, corresponding to 14.15 %, where no valid target lies along the fingertip ray. This relationship is important: EgoPoint-CoT extends a benchmark for egocentric deictic grounding into a corpus with explicit reasoning supervision [2603.26646].

## 3. Annotation workflow and quality control

The annotation process is organized to capture not just “what” the model should predict but “how” it should think. Initial auto-annotation uses MMPose to extract candidate hand bounding boxes and two keypoints, namely the finger root and tip. Two annotators then independently validate or correct hand boxes and keypoints, and disagreements are adjudicated by a senior annotator. On a subset, this process achieved Fleiss’ $\kappa = 0.91$, indicating high inter-annotator agreement. After validation, a 2D pointing ray is computed from the verified root and tip keypoints. A small LLM draft-writes step-by-step reasoning captions, and human experts edit and align them into the four-stage V-CoT template [2606.24539].

Negative samples are explicitly incorporated. EgoPoint-CoT includes 1 052 images without any visible pointing gesture, paired with a blocking CoT, “No hand detected; no target.” Quality control is described as expert cross-validation and manual correction at every stage, covering bounding boxes, keypoints, rays, and caption spans. The dataset also reports that negative samples were injected to prevent hallucinated tool calls. This suggests that the corpus is designed not only to supervise positive grounding behavior, but also to constrain failure modes in structured reasoning outputs [2606.24539].

## 4. Visual Chain-of-Thought representation

Each EgoPoint-CoT sample provides ground-truth variables $R^*=\{r_1 \ldots r_4\}$ and target box $b^*$. The Visual Chain-of-Thought is divided into four reasoning stages.

| Stage | Variable | Tokenization / output |
|---|---|---|
| 1. Hand Detection | $r_1 = b_{hand}$ | `<|box_start|> [x₁,y₁,x₂,y₂] <|box_end|>` |
| 2. Keypoint Extraction | $r_2 = (P_{root}, P_{tip})$ | `<|point_start|> [x_root,y_root] [x_tip,y_tip] <|point_end|>` |
| 3. Pointing Geometry | $r_3 =$ `draw_ray` call | `<tool_call> {"name":"draw_ray","start":[x_tip,y_tip],"end":[x’,y’]} </tool_call>` |
| 4. Target Alignment | $r_4 = b_{object}$ | optional caption span, then final box |

The third stage internally produces an image-ray region $I_{ray}$ for re-encoding. The target alignment stage may optionally be preceded by a caption span and ends with the final object box. A representative sequence includes `<think> ... </think>` spans, a `tool_call` for `draw_ray`, and the final `<|box_start|> ... <|box_end|>` output. All annotations are stored in JSON files, one per split, with fields including `image_id`, `image_path`, `hand_bbox`, `finger_root`, `finger_tip`, `ray_start`, `ray_end`, `object_bbox`, `object_category`, `cot_sequence`, and `is_negative` [2606.24539].

This representation is closely related to the staged reasoning paradigm introduced in SV-CoT. In EgoPoint-Ground, SV-CoT reformulates grounding as a structured latent chain with directional reasoning, candidate-path reasoning, and referent selection. EgoPoint-CoT differs in notation and supervision granularity, but it serves an analogous purpose: materializing intermediate geometric reasoning steps rather than performing a one-shot fusion of gesture and language [2603.26646].

## 5. Role in PointVG-R training

PointVG-R uses EgoPoint-CoT in a two-stage training pipeline consisting of Supervised Fine-Tuning and Adaptive Group-Relative Policy Optimization. In Stage 1, the SFT objective is
$$
L_{\mathrm{SFT}}(\theta)=-\mathbb{E}_{(I,R^*,b^*)\sim D}\big[\log P(R^*, b^* \mid I;\theta)\big].
$$
LoRA adapters are used on the LLM to inject the structured V-CoT syntax and geometric priors. In Stage 2, the policy gradient objective is
$$
J(\theta)=\mathbb{E}_{I\sim D,\; b,R\sim\pi_\theta}\big[w_g \cdot R_{\mathrm{total}}(b,R)\cdot \nabla_\theta \log P(b,R\mid I;\theta)\big].
$$
The reward is decomposed as
$$
R_{\mathrm{total}} = R_{\mathrm{base}} - \mathrm{repetition\_penalty},
$$
with
$$
R_{\mathrm{base}} = R_{\mathrm{acc}} + \lambda_{\mathrm{fmt}} R_{\mathrm{fmt}},
$$
and
$$
R_{\mathrm{acc}}=\lambda_{\mathrm{iou}}\cdot \mathrm{IoU}(b_{\mathrm{pred}}, b_{\mathrm{gt}}) + (R_{\mathrm{hand}} + R_{\mathrm{ray}} + R_{\mathrm{kpt}}).
$$
The component rewards are
$$
R_{\mathrm{hand}}=\mathrm{IoU}(b_{hand}^{pred}, b_{hand}^{gt}),
$$
$$
R_{\mathrm{kpt}}=\mathrm{clip}(1-\mathrm{MPJPE}_{norm}, 0, 1),
$$
$$
R_{\mathrm{ray}}=\frac{\cos\theta(pred,gt)+1}{2},
$$
and
$$
R_{\mathrm{fmt}}=1 \text{ if final token structure is valid else } 0.
$$
The repetition penalty is proportional to $N_{\mathrm{toolcalls}}$ and $N_{\mathrm{boxes}}$ [2606.24539].

Adaptive importance weighting is based on Group Variance. For each prompt, the method collects $G$ rollouts with rewards $\{r_i\}$, computes group variance $\sigma_g^2$, maintains global variance $\sigma_{glob}^2$ through EMA update, and defines the importance weight
$$
W_i=\frac{\sigma_g^2}{\sigma_{glob}^2+\epsilon},
$$
which is then clipped to $[W_{min}, W_{max}]$. The scaled advantage is
$$
\hat A_i = \mathrm{clip}(W_i, W_{min}, W_{max}) \cdot \frac{r_i-\mu_g}{\sigma_g}.
$$
The final GRPO loss clips policy ratios and uses $\hat A_i$ for stability. Within this training design, EgoPoint-CoT functions as the supervision source that introduces both structured syntax and geometric priors into the model [2606.24539].

## 6. Empirical significance and relation to adjacent datasets

The reported experimental outcome is that PointVG-R achieves SOTA performance, outperforming the baseline by 15.86 points in mIoU. Extensive ablation studies are reported to validate the efficacy of the proposed modules. The article-level significance of EgoPoint-CoT is therefore inseparable from its role as the dataset that warm-starts explicit geometric reasoning through supervised V-CoT traces before RL refinement [2606.24539].

Its closest antecedent is EgoPoint-Ground. That earlier benchmark introduced the first large-scale multimodal dataset dedicated to egocentric deictic visual grounding, with annotations including hand-target bounding box pairs, dense semantic captions, category labels, visual QA pairs, and 2D hand keypoints. It also introduced SV-CoT, a baseline framework that decomposes grounding into directional inference, candidate pruning along the finger ray, and semantic alignment. On EDG, the strongest open-source baseline, Qwen3-VL-8B, reached $P@0.5 = 0.701$ on the real split, while SV-CoT achieved $P@0.5 = 0.818$, a +11.7 ppt absolute gain; at $\tau = 0.7$, SV-CoT achieved 0.727 versus 0.593. In ablations, language-only D-REC improved from 0.684 to 0.750, and pointing-only POG improved from 0.612 to 0.639. EgoPoint-CoT can be read as a further step in the same program: the earlier work benchmarked explicit staged inference, whereas the later work packaged that style of reasoning into a tokenized supervisory corpus for SFT and RL [2603.26646].

A plausible implication is that EgoPoint-CoT marks a shift from evaluating whether multimodal models can resolve egocentric deictic expressions to directly supervising the intermediate geometric operations by which they do so. In that sense, its defining contribution is not simply expanded annotation density, but the codification of a reasoning protocol that is machine-readable, trainable, and auditable within PointVG-R [2606.24539].

Source: https://www.emergentmind.com/topics/egopoint-cot