---
title: Video Pointing Dataset Overview
url: https://www.emergentmind.com/topics/video-pointing-dataset
type: topic
---

# Video Pointing Dataset Overview

Video pointing datasets constitute a foundational resource for training and benchmarking algorithms designed to spatially and temporally localize instances, actions, or human interactions within videos via discrete pixel-level annotations. These datasets depart from traditional mask- or bounding-box-centric paradigms by emphasizing concise, spatio-temporal point annotations for grounding and tracking tasks, thus aligning with current trends in interactive localization, multimodal reasoning, and efficient annotation in computer vision.

## 1. Definition and Taxonomy

A video pointing dataset is defined by its provision of pixel-level temporal annotations (points, clicks, or directions) identifying objects, actions, or relevant semantic events in video frames. The overarching taxonomy includes:

- **Temporal localization with spatial granularity:** e.g., Molmo2-VideoPoint delineates each instance by a point (frame index/timestamp, (x, y) location), not a dense mask or box [2601.10611].
- **Gesture or interactive pointing:** datasets where human pointing (fingertip, gaze, or head orientation) specifies intent, interaction, or saliency, such as EgoGestAR’s FPV gesture corpus [1904.09843] or Panonut360’s eye/head tracking [2403.17708].
- **Sparse object or region annotation:** as in Point-VOS, where annotation protocol samples points for weak-supervision segmentation and grounding [2402.05917].
- **3D scene-point tracking:** TAPVid-3D records camera-space coordinates ((X, Y, Z)), mapping pixel queries to scene geometry for dynamic video content [2407.05921].

A plausible implication is that video pointing datasets serve dual roles as both ground truth for multimodal grounding tasks and as efficient, scalable annotation regimes reducing the burden of dense mask labeling.

## 2. Data Collection and Annotation Protocols

Annotation methodologies in video pointing datasets are highly specialized for efficiency, clarity, and generalizability:

- **Molmo2-VideoPoint:** Annotators select the relevant frame and submit pixel-level coordinates per query (normalized to [0, 1000] scale), guided by free-form referring expressions and subject to LLM and segmentation-based validation [2601.10611].
- **VideoMolmo:** Semi-automatic annotation tools use mask-driven point selection, favoring interior points via Euclidean distance and optimizing for maximal IoU with SAM2-generated segmentation [2506.05336].
- **Point-VOS:** Sparse annotation protocol samples up to 14.7 human-verified points (foreground, background, ambiguous) per object per frame, with temporal sparsity achieved by annotating only 10/frames per object [2402.05917].
- **GIFT:** An automated pipeline samples surface points on 3D models, projecting them into 2D for real/synthetic video rendering, and tracks each point’s visibility under complex camera trajectories [2503.12944].
- **Panonut360:** Head and gaze pointing are captured via HMD and eye tracker at 120 Hz; points encode instantaneous 3D directional vectors, further transformed to projected image coordinates [2403.17708].
- **EgoGestAR:** Fingertip or hand trajectory logging via tablet/camera is used for gesture definition; video runs employ a fingertip regressor as a surrogate for dense annotation [1904.09843].

Table 1 summarizes annotation characteristics of major datasets:

| Dataset              | Annotation Mode      | Typical Point Density/frame | Temporal Coverage                |
|----------------------|---------------------|-----------------------------|----------------------------------|
| Molmo2-VideoPoint    | 2D point/click      | 6.0 per query               | 1.8 frames per query             |
| VideoMolmo           | Mask-derived point   | 1 per object per frame      | All frames, with prompt fusion   |
| Point-VOS            | Sparse points       | ∼14.7 per frame/object      | 10 frames/object                 |
| TAPVid-3D            | Camera-space 3D pts | 50–1024 per clip            | Full clip (25–300 frames)        |
| EgoGestAR            | Fingertip traj      | Variable                    | Full gesture duration @30FPS     |
| Panonut360           | Directional vector  | 1 per sample (120Hz)        | Full video duration              |

## 3. Dataset Structure, Coverage, and Domains

Video pointing datasets exhibit broad coverage in domains, scale, and point annotation schemes:

- **Molmo2-VideoPoint:** 250,000 training videos, 450,000 frame–point pairs, spanning eight query categories (objects, animals, actions, spatial, comparative, generative artifacts) [2601.10611].
- **VideoMolmo:** 72,000 video–caption pairs, ~100,000 object points; real-world scenarios include cell tracking, egocentric vision, autonomous driving, GUI interaction, and robotics [2506.05336].
- **Point-VOS:** Aggregates 19M annotations across 133K objects in 32K videos, with open-vocabulary labels and diversified action/taxonomy via PV-Oops and PV-Kinetics splits [2402.05917].
- **TAPVid-3D:** Over 4,569 clips, three data sources (Aria Digital Twin, DriveTrack, Panoptic Studio), with dense camera-space 3D point tracks and visibility flags [2407.05921].
- **GIFT:** Synthetic indoor scenes, 1,800 videos, 5–10 points per video, stratified by texture intensity and camera motion complexity [2503.12944].
- **Panonut360:** 15 panoramic videos, 50 participants; head/gaze vectors and projected coordinates for user attention localization [2403.17708].
- **EgoGestAR:** 2,500 gesture trajectories (touch), 240 egocentric video clips; FPV pointing for AR/MR [1904.09843].

A plausible implication is that researchers can select datasets aligned to video type (real/synthetic), annotation density, and grounding granularity required for their application.

## 4. Annotation Formats, Storage, and Access

Standardized annotation formats foster accessibility and interoperability:

- **Molmo2-VideoPoint:** Each record is a JSON object: includes video ID, query string, annotations (frame_index, timestamp_sec, object_id, x_norm, y_norm), directory split by train/valid/test [2601.10611].
- **VideoMolmo:** Point annotations per frame/object, stored alongside referring text and masks, with representative pixel coordinates [2506.05336].
- **Point-VOS:** JSON with frame size, objects, frame indices, points (x, y, type), open-vocabulary object nouns [2402.05917].
- **TAPVid-3D:** Numpy arrays of 3D tracks (tracks_xyz), query_xyt, visibility flags, camera intrinsics; RGB frames remain in original formats [2407.05921].
- **EgoGestAR:** CSV or plain-text logs for touch trajectories, raw .mp4 for video clips [1904.09843].
- **Panonut360:** Head/eye CSV files, converted to pixel coordinates by scripting, with per-second saliency map outputs [2403.17708].

Licensing is typically permissive (Apache, MIT, CC BY-NC 4.0), though non-commercial restrictions may apply in some cases.

## 5. Supported Tasks and Evaluation Protocols

Video pointing datasets support a spectrum of localization, grounding, and tracking tasks via standardized metrics:

- **Molmo2-VideoPoint:** Primary task is spatio-temporal pointing by instance; metrics are precision, recall, F1, valid-parsing accuracy (based on click inclusion within ground-truth masks) [2601.10611].
- **VideoMolmo:** Evaluation on spatio-temporal pointing (precision, recall, F1) and mask fusion (region Jaccard, boundary F-measure, J–F average), with counting metrics (EMA, MAE) [2506.05336].
- **Point-VOS:** Assesses weakly supervised segmentation via J&F=(Jaccard+F)/2, using sparse points for initialization and mask derivation [2402.05917].
- **TAPVid-3D:** Tracks Average Precision-Distance (APD₃D(δ)), Occlusion Accuracy (OA), 3D Average Jaccard (AJ₃D), with scale ambiguity correction and spatio-temporal smoothness [2407.05921].
- **GIFT:** Uses δ_{avg}^{vis}(τ) (visible point threshold), AJ, OA, evaluated across texture-level subsets; baselines include RAFT, TAPIR, CoTracker [2503.12944].
- **EgoGestAR:** Gesture classification accuracy (LSTM/Bi-LSTM), latency (on-device), trajectory-based validation [1904.09843].
- **Panonut360:** Head/gaze offset statistics, saliency map generation by aggregated fixations, assessment of centrality and anisotropy of attention regions [2403.17708].

A plausible implication is that the “point-driven” paradigm enables benchmarking of both pure geometric localization and multimodal reasoning under weak supervision, supporting both classical vision and vision-language model training.

## 6. Current State, Baselines, and Limitations

Summary baselines from recent benchmarks articulate current performance ceilings and error modes:

| Model                  | Precision (%) | Recall (%) | F1 (%)         | Dataset                |
|------------------------|--------------|------------|----------------|------------------------|
| Molmo2-4B              | 39.4         | 42.7       | 39.9           | Molmo2-VideoPoint [2601.10611] |
| Molmo2-8B              | 38.7         | 39.3       | 38.4           | Molmo2-VideoPoint      |
| Gemini 3 Pro           | 19.8         | 27.4       | 20.0           | Molmo2-VideoPoint      |
| CoTracker+ZoeDepth     | 14.8         | —          | 8.7 (AJ₃D)     | TAPVid-3D [2407.05921] |
| Static baseline        | 9.4          | —          | 4.9 (AJ₃D)     | TAPVid-3D              |

Key limitations:

- Datasets such as GIFT and TAPVid-3D suffer from domain-constrained coverage (e.g., synthetic indoor scenes, driving, staged actions). Textureless regions and depth scale ambiguity remain significant error modes [2503.12944, 2407.05921].
- Annotator variability and runtime regressors induce noise (EgoGestAR's finger positions, Panonut360's eye vector estimation) [1904.09843, 2403.17708].
- Many datasets (Point-VOS, Molmo2-VideoPoint) rely on sparse spatial and/or temporal sampling for scalability, which limits applications requiring dense trajectory or segmentation supervision [2402.05917, 2601.10611].

Recommended practices include augmentation with real data, joint training on auxiliary modalities (depth, normals, flow), and use of rich annotation schemas for self-supervised approaches.

## 7. Applications and Future Directions

Video pointing datasets are increasingly utilized for:

- **Multimodal grounding in VLMs:** Open datasets (Molmo2-VideoPoint, VideoMolmo) empower progress in vision-language pixel grounding, counting, and reasoning, unattainable with closed proprietary corpora [2601.10611, 2506.05336].
- **Point-driven video object segmentation:** Sparse point annotations bootstrap competitive performance in VOS with efficiency gains (Point-VOS achieves ≈80% full-mask baseline directly, >90% using pseudo-masks) [2402.05917].
- **Robotics and AR/VR interaction:** Finger and gaze datasets (EgoGestAR, Panonut360) enable calibration and learning of intent, saliency, and control interfaces for edge deployment [1904.09843, 2403.17708].
- **Robust point tracking in challenging conditions:** Synthetic benchmarks like GIFT stress-test algorithms in textureless, occluded, or motion-complex scenes, facilitating development of tracking models robust to real-world phenomena [2503.12944].
- **3D point tracking for dynamic scenes:** TAPVid-3D extends trajectory annotation to three dimensions, supporting research in motion, deformation, and manipulation in complex environments [2407.05921].

Future directions include extension to dense scene flow, semantic segmentation, dynamic scene representation, cross-domain adaptation, and integration with language-driven interactive systems. Expansion into underrepresented domains (sports, wildlife, in-the-wild scenes) is indicated as a research need [2407.05921, 2503.12944].

Source: https://www.emergentmind.com/topics/video-pointing-dataset