EgoPoint-CoT: Egocentric Visual Chain-of-Thought

• EgoPoint-CoT is a visual Chain-of-Thought dataset that explicitly encodes intermediate reasoning steps for egocentric pointing-based visual grounding.
• It comprises 15,455 annotated egocentric images from real and synthetic sources, detailing hand detection, keypoint extraction, and target alignment.
• The dataset underpins the PointVG-R training pipeline by providing structured supervision that enhances geometric reasoning and improves localization accuracy.

Searching arXiv for the cited papers to ground the article in current records. arxiv_search(query="PointVG-R EgoPoint-CoT egocentric pointing visual grounding", max_results=10) arxiv_search(query="Beyond Language Grounding Referring Expressions with Hand Pointing in Egocentric Vision", max_results=10) EgoPoint-CoT is a visual Chain-of-Thought dataset for egocentric pointing-based visual grounding, introduced as the “cold-start” supervised corpus for PointVG-R. It provides richly annotated egocentric images together with fine-grained geometric reasoning traces, so that training supervision covers both the target object and the intermediate reasoning process used to localize it. The dataset is built on EgoPoint-Ground, which established the underlying egocentric deictic grounding benchmark, and it operationalizes a four-stage Visual Chain-of-Thought format oriented around hand detection, keypoint extraction, pointing geometry, and target alignment (Li et al., 23 Jun 2026, Li et al., 27 Mar 2026).

1. Position within egocentric deictic grounding

EgoPoint-CoT arises from the problem setting in which an agent must localize an intended target object by interpreting a pointing gesture in an egocentric scene. In the associated line of work, traditional visual grounding is described as predominantly text-based, whereas pointing-based grounding requires explicit handling of spatial geometry and gestural cues. EgoPoint-Ground formulated this setting as Egocentric Deictic Grounding (EDG), with input consisting of an image $I$, a pointing hand box $B_{hand}^{2D}$, and an underspecified linguistic expression $S$, and output given by the intended target box $B_{obj}^{2D}$ through the maximum-a-posteriori estimate

$$\hat B_{obj}=\arg\max_B P(B\mid I,\;B_{hand}^{2D},\;S).$$

EgoPoint-CoT specializes this general benchmark by supplying explicit reasoning trajectories that supervise geometric interpretation inside a multimodal model rather than leaving that process implicit (Li et al., 27 Mar 2026).

A common misconception is to treat EgoPoint-CoT as merely another bounding-box dataset. Its defining property is not only that it labels the hand and target, but that it encodes the intermediate reasoning variables as a structured sequence. This makes it a dataset for supervised reasoning traces as well as localization targets (Li et al., 23 Jun 2026).

2. Composition, provenance, and corpus statistics

EgoPoint-CoT contains 15 455 egocentric images, split 7 : 1 : 2 into train (10 837 samples), validation (1 546), and test (3 072). It is built on EgoPoint-Ground and combines real-world smart-glass captures and synthetic renders to ensure diverse lighting, clutter and object types. The corpus includes 345 distinct object categories and 136 468 object instances. On average, there are 8.83 annotated objects per image, with $\sigma = 12.14$ and a maximum of 76, indicating both sparse and crowded scenes. Hands cover on average 11.4 % of image area, with median 9.2 %, while targets cover 19.8 %, with median 13.8 %. The hand–target center distance averages 374 px, normalized to 0.203 of the image diagonal, which is reported as showing typical close-range pointing (Li et al., 23 Jun 2026).

These statistics differ from the underlying EgoPoint-Ground benchmark, which comprises 15 338 egocentric images spanning 140 object categories in over 20 everyday scenarios, with an average of 2.8 candidate objects per image, 63.7 % of samples containing two or more instances of the same category, and 42.5 % exhibiting moderate or severe occlusion. EgoPoint-Ground also includes 2 171 negative samples, corresponding to 14.15 %, where no valid target lies along the fingertip ray. This relationship is important: EgoPoint-CoT extends a benchmark for egocentric deictic grounding into a corpus with explicit reasoning supervision (Li et al., 27 Mar 2026).

3. Annotation workflow and quality control

The annotation process is organized to capture not just “what” the model should predict but “how” it should think. Initial auto-annotation uses MMPose to extract candidate hand bounding boxes and two keypoints, namely the finger root and tip. Two annotators then independently validate or correct hand boxes and keypoints, and disagreements are adjudicated by a senior annotator. On a subset, this process achieved Fleiss’ $\kappa = 0.91$, indicating high inter-annotator agreement. After validation, a 2D pointing ray is computed from the verified root and tip keypoints. A small LLM draft-writes step-by-step reasoning captions, and human experts edit and align them into the four-stage V-CoT template (Li et al., 23 Jun 2026).

Negative samples are explicitly incorporated. EgoPoint-CoT includes 1 052 images without any visible pointing gesture, paired with a blocking CoT, “No hand detected; no target.” Quality control is described as expert cross-validation and manual correction at every stage, covering bounding boxes, keypoints, rays, and caption spans. The dataset also reports that negative samples were injected to prevent hallucinated tool calls. This suggests that the corpus is designed not only to supervise positive grounding behavior, but also to constrain failure modes in structured reasoning outputs (Li et al., 23 Jun 2026).

4. Visual Chain-of-Thought representation

Each EgoPoint-CoT sample provides ground-truth variables $R^*=\{r_1 \ldots r_4\}$ and target box $b^*$. The Visual Chain-of-Thought is divided into four reasoning stages.

Stage	Variable	Tokenization / output
1. Hand Detection	$r_1 = b_{hand}$	`<
2. Keypoint Extraction	$B_{hand}^{2D}$0	`<
3. Pointing Geometry	$B_{hand}^{2D}$1 draw_ray call	<tool_call> {"name":"draw_ray","start":[x_tip,y_tip],"end":[x’,y’]} </tool_call>
4. Target Alignment	$B_{hand}^{2D}$2	optional caption span, then final box

The third stage internally produces an image-ray region $B_{hand}^{2D}$3 for re-encoding. The target alignment stage may optionally be preceded by a caption span and ends with the final object box. A representative sequence includes > ... spans, a tool_call for draw_ray, and the final <|box_start|> ... <|box_end|> output. All annotations are stored in JSON files, one per split, with fields including image_id, image_path, hand_bbox, finger_root, finger_tip, ray_start, ray_end, object_bbox, object_category, cot_sequence, and is_negative (Li et al., 23 Jun 2026).

This representation is closely related to the staged reasoning paradigm introduced in SV-CoT. In EgoPoint-Ground, SV-CoT reformulates grounding as a structured latent chain with directional reasoning, candidate-path reasoning, and referent selection. EgoPoint-CoT differs in notation and supervision granularity, but it serves an analogous purpose: materializing intermediate geometric reasoning steps rather than performing a one-shot fusion of gesture and language (Li et al., 27 Mar 2026).

5. Role in PointVG-R training

PointVG-R uses EgoPoint-CoT in a two-stage training pipeline consisting of Supervised Fine-Tuning and Adaptive Group-Relative Policy Optimization. In Stage 1, the SFT objective is

$B_{hand}^{2D}$4

LoRA adapters are used on the LLM to inject the structured V-CoT syntax and geometric priors. In Stage 2, the policy gradient objective is

$B_{hand}^{2D}$5

The reward is decomposed as

$B_{hand}^{2D}$6

with

$B_{hand}^{2D}$7

and

$B_{hand}^{2D}$8

The component rewards are

$B_{hand}^{2D}$9

$S$0

$S$1

and

$S$2

The repetition penalty is proportional to $S$3 and $S$4 (Li et al., 23 Jun 2026).

Adaptive importance weighting is based on Group Variance. For each prompt, the method collects $S$5 rollouts with rewards $S$6, computes group variance $S$7, maintains global variance $S$8 through EMA update, and defines the importance weight

$S$9

which is then clipped to $B_{obj}^{2D}$0. The scaled advantage is

$B_{obj}^{2D}$1

The final GRPO loss clips policy ratios and uses $B_{obj}^{2D}$2 for stability. Within this training design, EgoPoint-CoT functions as the supervision source that introduces both structured syntax and geometric priors into the model (Li et al., 23 Jun 2026).

6. Empirical significance and relation to adjacent datasets

The reported experimental outcome is that PointVG-R achieves SOTA performance, outperforming the baseline by 15.86 points in mIoU. Extensive ablation studies are reported to validate the efficacy of the proposed modules. The article-level significance of EgoPoint-CoT is therefore inseparable from its role as the dataset that warm-starts explicit geometric reasoning through supervised V-CoT traces before RL refinement (Li et al., 23 Jun 2026).

Its closest antecedent is EgoPoint-Ground. That earlier benchmark introduced the first large-scale multimodal dataset dedicated to egocentric deictic visual grounding, with annotations including hand-target bounding box pairs, dense semantic captions, category labels, visual QA pairs, and 2D hand keypoints. It also introduced SV-CoT, a baseline framework that decomposes grounding into directional inference, candidate pruning along the finger ray, and semantic alignment. On EDG, the strongest open-source baseline, Qwen3-VL-8B, reached $B_{obj}^{2D}$3 on the real split, while SV-CoT achieved $B_{obj}^{2D}$4, a +11.7 ppt absolute gain; at $B_{obj}^{2D}$5, SV-CoT achieved 0.727 versus 0.593. In ablations, language-only D-REC improved from 0.684 to 0.750, and pointing-only POG improved from 0.612 to 0.639. EgoPoint-CoT can be read as a further step in the same program: the earlier work benchmarked explicit staged inference, whereas the later work packaged that style of reasoning into a tokenized supervisory corpus for SFT and RL (Li et al., 27 Mar 2026).

A plausible implication is that EgoPoint-CoT marks a shift from evaluating whether multimodal models can resolve egocentric deictic expressions to directly supervising the intermediate geometric operations by which they do so. In that sense, its defining contribution is not simply expanded annotation density, but the codification of a reasoning protocol that is machine-readable, trainable, and auditable within PointVG-R (Li et al., 23 Jun 2026).