ChildPlay-Gaze Dataset

• ChildPlay-Gaze is a benchmark dataset designed to model children's gaze behavior in uncontrolled child–adult play and interaction settings.
• It provides extensive per-frame annotations including head bounding boxes, 2D gaze targets, seven gaze classes, and person type labels from YouTube clips.
• The dataset supports tasks like gaze target localization and in/out-of-frame classification while introducing the P.Head metric to assess socially meaningful gaze behavior differences between children and adults.

Searching arXiv for the ChildPlay/ChildPlay-Gaze benchmark and closely related work. ChildPlay, also referred to in later work as ChildPlay-Gaze, is a video benchmark for modeling and analyzing children’s gaze behavior in uncontrolled child–adult interaction settings. It was introduced to address the mismatch between adult-centric public gaze-following datasets and the markedly different visual ecology of children’s play, especially in scenarios involving nearby toys, floor-level interaction, and reduced face-looking relative to adult social video. The dataset consists of short RGB clips curated from YouTube and annotated per frame with head bounding boxes, 2D gaze targets, gaze classes, and adult/child identity labels for selected persons, enabling gaze target prediction, in/out-of-frame classification, and higher-level analyses of behaviors such as eye contact and shared attention (Tafasca et al., 2023, Farkhondeh et al., 2024).

1. Identity, scope, and scale

ChildPlay was introduced as a benchmark for understanding children’s gaze behaviour “in the wild,” rather than in private or laboratory-only datasets. Its stated motivation is that publicly available gaze target prediction datasets largely contain adults, making adult-trained models less applicable to scenarios involving young children and child–adult interaction (Tafasca et al., 2023).

Attribute	Value
Source material	95 source “shows” / 401 clips
Annotation scale	120,549 frames / 257,928 gaze annotation instances
Population balance	Approximately 62% of instances correspond to children

The dataset is video-based rather than image-only. This matters because per-frame annotation supports both frame-wise gaze-following and temporally extended analyses such as gaze shifts, fixations, shared attention, and mutual gaze. The benchmark is centered on realistic play and caregiving scenes rather than screen-looking or adult conversational footage, and it explicitly distinguishes child and adult annotated persons, which enables cross-population analysis within the same scenes (Tafasca et al., 2023).

A key feature of the benchmark is that it is designed not only for localization of a gaze target in image space, but also for studying the semantic content of gaze. The paper emphasizes that children’s gaze targets are often nearby toys or other play objects, while adults in the same clips often look downward toward children. This produces a different distribution of target semantics and geometry from adult TV- or web-derived gaze datasets (Tafasca et al., 2023).

2. Data sources, environments, and participants

ChildPlay is explicitly an uncontrolled / in-the-wild dataset. Its clips were retrieved from YouTube using queries such as “children playing toys,” “childcare center,” and “kids observation.” The environments include childcare facilities, preschools, schools, homes, and therapy centers. The content is mainly indoor and free-play oriented, with children playing with toys, interacting with adults, and in some clips participating in structured behavioural therapy exercises (Tafasca et al., 2023).

The source videos have aspect ratio 16:9 and resolution 720p or 1080p. Audio is downloaded but is not used for gaze modeling. There is no native depth capture; instead, depth is later inferred from RGB within the proposed modeling framework. The benchmark is therefore fundamentally an RGB video dataset with dense gaze annotations rather than a multimodal capture system with synchronized sensors (Tafasca et al., 2023).

Each clip contains at least one child. Often there are one or two adults and multiple children. For annotation, up to three people per clip are selected, typically the main interacting child and key adults. Each selected person is labeled as either adult or child. The dataset focuses on children from toddlers to pre-teens, but exact age labels are not provided. This absence of precise age metadata constrains direct developmental stratification, even though the scenes themselves clearly span multiple developmental stages (Tafasca et al., 2023).

The benchmark’s design differs materially from adult-focused gaze datasets. In ChildPlay, children and adults are frequently sitting or playing on the floor, leaning over, or occupying non-standard poses relative to the camera. This yields a markedly different distribution of head pose, target distance, and target category. The paper explicitly links this to poorer transfer from adult-trained models, particularly for face-looking prediction on children (Tafasca et al., 2023).

3. Annotation schema and protocol

For each selected person in each annotated frame, ChildPlay provides four core labels: a head bounding box, a 2D gaze point, a gaze class, and a person type label indicating child or adult. The 2D gaze point is annotated only when the target is meaningfully localizable inside the image; otherwise the frame is assigned to one of several non-point classes (Tafasca et al., 2023).

The gaze classes are seven mutually exclusive categories:

• inside-frame: the gaze target is visible inside the image and a 2D point is annotated.
• outside-frame: the person is looking outside the camera field of view.
• gaze-shift: the person is shifting attention from one target to another over at least two frames.
• occluded: the target is within the view but completely occluded.
• uncertain: annotators cannot confidently determine the target.
• eyes-closed: the person closes their eyes.
• not-annotated: none of the above applies, or the frame is not annotated for that person.

Empirically, inside-frame accounts for 85.3% of instances, outside-frame for 5.4%, and the remaining classes together for 9.3%. This explicit treatment of ambiguous and transitional cases is a distinguishing design choice. It prevents the common collapse of semantically different failure modes into a single “outside” or “unknown” category and is intended to improve consistency relative to prior datasets (Tafasca et al., 2023).

The representation used for training and evaluation includes a ground-truth gaze heatmap derived from the annotated gaze point. For inside-frame instances, the heatmap is defined as a 2D isotropic Gaussian centered at the target: $$H_{\text{gt}}(x, y) = \exp\left(-\frac{(x - x_{\text{gaze}})^2 + (y - y_{\text{gaze}})^2}{2\sigma^2}\right).$$ The paper notes that semantic consistency is central to annotation: the 2D gaze point should lie on the actual object being looked at, not merely somewhere along a plausible 2D gaze ray. This is important because two nearby 2D points can correspond to distinct objects at different depths, only one of which is semantically correct (Tafasca et al., 2023).

Annotation was performed in LabelBox by seven annotators. For 2,000 instances, double-coding was used to assess agreement. The resulting human–human agreement, evaluated with the same metrics used for models, was reported as similar to that of other gaze datasets. This agreement is treated as an approximate upper bound for model performance on the benchmark (Tafasca et al., 2023).

4. Benchmark tasks and evaluation methodology

ChildPlay defines a gaze-following benchmark with two primary tasks and one semantically targeted evaluation protocol. The first task is gaze target localization for inside-frame cases: given the RGB scene, the head bounding box, and a head crop, the model predicts a 2D gaze location through a heatmap. The second task is in-vs-out-of-frame classification, which predicts whether the target lies inside the image or outside it. The third component, looking-at-heads detection, is not a separate model but a semantic evaluation that tests whether a prediction falls on another person’s head (Tafasca et al., 2023).

Localization is evaluated with AUC and normalized Euclidean distance. Let the predicted point be the argmax of the heatmap,

$(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$

and let the image be normalized to size $1 \times 1$. The distance metric is

$$\text{Dist} = \sqrt{(x_p - x_{\text{gt}})^2 + (y_p - y_{\text{gt}})^2}.$$

For in/out-of-frame classification, the benchmark uses Average Precision (AP) over the predicted inside-frame probability $\hat{y}_{\text{in}}$. These are standard gaze-following metrics, but the paper argues that they are insufficient for characterizing child gaze behaviour on their own (Tafasca et al., 2023).

To address this, the benchmark introduces P.Head, a semantic metric for “looking at heads.” Ground truth is built by running a YOLOv5-based head detector on each image, followed by manual validation and correction. A ground-truth gaze is considered head-directed if the gaze point lies inside one of the detected head boxes. A model prediction is then reduced to a binary decision—inside a head box or not—and precision is computed over predictions of “looking at head” (Tafasca et al., 2023).

The motivation for P.Head is that distance can be biased by target proximity. Children in ChildPlay often look at nearby toys, which can yield low localization error even when a model is poor at predicting socially significant face-looking behavior. The benchmark statistics make this asymmetry explicit: the percentage of ground-truth gaze instances classified as “looking at a head” is 23.0% for GazeFollow, 69.0% for VideoAttentionTarget, 15.7% for ChildPlay children, and 44.4% for ChildPlay adults. This confirms that ChildPlay is semantically less dominated by face targets than adult TV-derived benchmarks, especially for children (Tafasca et al., 2023).

5. Geometrically grounded 3DFoV modeling and empirical findings

The paper introduces a new gaze target prediction model built around a geometrically grounded 3D field of view (3DFoV) representation. The architecture has two pathways. The Gaze Pathway takes a head crop and predicts a gaze embedding $g$, a 3D gaze direction vector $\hat{\mathbf{d}}$ in an eye-centered coordinate system, and a 3DFoV heatmap. The Scene / Visual Attention Pathway takes the scene image, a head-location mask, the 3DFoV heatmap, and the gaze embedding, and predicts the final attention heatmap $H_p$ and the in/out probability $\hat{y}_{\text{in}}$ (Tafasca et al., 2023).

The geometric component begins with monocular depth inference using a geometry-preserving mono-depth model, followed by focal-length estimation and reconstruction of a 3D point cloud. The camera intrinsics are modeled as

$$K = \begin{bmatrix} f & 0 & W/2 \ 0 & f & H/2 \ 0 & 0 & 1 \end{bmatrix},$$

under assumptions of square pixels, zero skew, and principal point at the image center. A 3D point in camera coordinates is recovered from pixel coordinates and inferred depth, then transformed into an eye-centered coordinate system whose origin is the eye location and whose $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$0-axis points from camera to eye (Tafasca et al., 2023).

The head crop is encoded with a ResNet-18 backbone, and a two-layer MLP with tanh predicts the unit gaze vector $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$1. For each point $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$2 in the eye-frame point cloud, alignment with the predicted gaze direction is computed as

$(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$3

A piecewise exponential mapping then converts this alignment score into a 3DFoV intensity, which is projected back into the image plane as the 3DFoV heatmap. This heatmap is concatenated with the RGB scene and a binary head mask, processed by an EfficientNet-B1 encoder with an FPN decoder, and passed to a dilated-convolution attention head that predicts the final gaze heatmap (Tafasca et al., 2023).

Training uses three losses: $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$4 with $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$5, $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$6, and $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$7. The heatmap loss is an $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$8 loss on $(x_p, y_p) = \arg\max_{x, y} H_p(x, y),$9, the direction loss is

$1 \times 1$0

and the in/out term is binary cross-entropy. The pseudo ground-truth 3D gaze direction $1 \times 1$1 is computed by projecting the annotated 2D gaze point into 3D using the inferred depth and then normalizing in eye coordinates (Tafasca et al., 2023).

On the ChildPlay test set, after fine-tuning on ChildPlay, the proposed model obtains the following results. For children: AUC 0.939, Distance 0.098, AP 0.989, P.Head 0.604. For adults: AUC 0.928, Distance 0.121, AP 0.983, P.Head 0.704. For all instances: AUC 0.935, Distance 0.107, AP 0.986, P.Head 0.663. The central empirical finding is that children exhibit slightly better distance scores but substantially worse P.Head than adults. The paper interprets this as evidence that pixel-distance metrics alone can obscure the difficulty of predicting socially meaningful face-looking in children (Tafasca et al., 2023).

The study also reports that fine-tuning on ChildPlay improves child performance more than adult performance. This supports the broader claim that adult-centric gaze datasets are insufficient for modeling children’s gaze behaviour, especially for clinically or socially salient behaviors such as looking at faces (Tafasca et al., 2023).

6. Derivative resources, related datasets, and terminological boundaries

Later work explicitly uses ChildPlay and ChildPlay-Gaze interchangeably. In particular, “ChildPlay-Hand” is described as being derived from the ChildPlay-Gaze videos and adds dense annotations for person boxes, per-hand interactions with objects, involved object boxes, and manipulation stages on the same clips and for the same people that have gaze annotations. It retains the underlying 401 video clips and the gaze labels as an accompanying resource, with the stated goal of enabling future joint modeling of manipulation and gaze from a third-person view (Farkhondeh et al., 2024).

This derivative relationship is significant because it repositions ChildPlay from a single-task gaze-following benchmark to a multimodal substrate for social perception and hand–object interaction research. ChildPlay-Hand itself does not yet perform joint gaze–manipulation modeling, but it is explicitly framed as a complementary annotation layer that enables future study of the coordination between manipulations and visual attention (Farkhondeh et al., 2024).

A plausible source of confusion is that the phrase “child-play gaze dataset” is also used descriptively for other resources that are not the ChildPlay benchmark. One distinct example is the ASD therapy dataset used for automated mutual gaze detection in home-based intervention sessions, which contains 28 observations, 84 video clips, and 21 hours of child–therapist interaction, but is centered on mutual gaze analytics rather than gaze-following and is not presented under the ChildPlay name (Guo et al., 2023). Other child-play gaze resources include head-mounted toddler–caregiver eye-tracking corpora transformed into gaze-centered visual streams for self-supervised learning (Yu et al., 2024), and multimodal child–caregiver datasets built around eye-tracking, third-person video, and automated annotation toolchains such as GBAT (Baig et al., 21 May 2026).

This suggests a useful boundary. ChildPlay-Gaze properly denotes the specific 2023 benchmark introduced for understanding children’s gaze behaviour in uncontrolled third-person video (Tafasca et al., 2023), whereas child-play gaze dataset can also function as a broader descriptive category for datasets of gaze during child play. The distinction matters because the data modality, annotation target, and benchmark task differ substantially across these resources: ChildPlay emphasizes third-person gaze target prediction with 2D target annotations; the ASD therapy dataset emphasizes mutual gaze episodes; the toddler egocentric work emphasizes gaze-contingent crops from head-mounted eye tracking; and GBAT emphasizes scalable annotation of gaze targets, pose, and hand action in multimodal child–caregiver recordings (Guo et al., 2023, Yu et al., 2024, Baig et al., 21 May 2026).

ChildPlay’s authors state that the dataset and models will be made publicly available, and the benchmark is built from public YouTube material rather than private clinical recordings (Tafasca et al., 2023). At the same time, the paper notes limitations that remain salient for research use: precise demographic metadata are not provided, depth is inferred rather than measured, and the scene distribution is mainly indoor free-play. These constraints do not diminish its role as the first large public benchmark specifically organized around children’s gaze behaviour, but they do delimit the kinds of developmental and clinical inferences that can be made directly from the released annotations (Tafasca et al., 2023).