FPI-Det: Face–Phone Interaction Benchmark
- FPI-Det is a specialized benchmark that focuses on detecting face–phone interactions by analyzing the relational context between faces and phones in real-world scenes.
- It provides 22,879 images with synchronized annotations for faces, phones, and phone-use behaviors, supporting both object detection and binary classification tasks.
- The dataset challenges models with diverse conditions such as scale variation, occlusion, and varied capture environments, crucial for safety-critical applications.
Searching arXiv for the FPI-Det paper to ground the article in the cited preprint. FPI-Det, short for Face–Phone Interaction Detection, is a specialized face–phone interaction dataset and benchmark designed for phone-use detection and understanding in realistic scenes. It was introduced to address the fact that detecting phone use is not merely an object-recognition problem: reliable inference depends on the relationship between faces and phones, rather than on isolated object presence alone. The benchmark contains 22,879 images with synchronized annotations for faces and phones, together with phone-use behavior labels and additional relation information such as pixel coordinates and pairwise distances between face and phone instances. It is positioned as a benchmark for fine-grained human–device interaction detection, supporting both object detection and binary phone-use classification (Gao et al., 11 Sep 2025).
1. Concept and problem formulation
FPI-Det is centered on face–phone interaction as the semantic basis for deciding whether a person is using a phone. The underlying premise is that a phone in an image does not automatically imply phone use, and a face alone does not indicate behavior; only the interaction between the two supports reliable phone-use inference. The benchmark therefore differs from generic object-detection datasets that mainly label isolated objects, because it emphasizes the relationship between a face and a device in behaviorally meaningful scenes (Gao et al., 11 Sep 2025).
The dataset was created for environments in which phone misuse or distraction is consequential, specifically workplaces, education, transportation, and public spaces. The motivation described for the benchmark is that a person may be holding a phone but not using it, looking at a phone, talking on the phone, using a phone while seated, or using a phone sideways or obliquely. This makes the task dependent on contextual reasoning about relative placement, visibility, and behavioral configuration rather than on category detection alone.
A central distinction in the benchmark is between object detection and phone-use classification. The former concerns detecting faces and phones; the latter is a binary decision based on co-detection and interaction context. This suggests that FPI-Det occupies an intermediate position between conventional detection benchmarks and behavior-understanding datasets: it uses bounding-box supervision but is explicitly designed to support inference about human–device interaction.
2. Dataset composition and annotation design
FPI-Det contains 22,879 images, 29,279 face annotations, and 10,255 phone annotations. The dataset averages 1.28 faces/image and 0.45 phones/image, with a maximum density of up to 37 faces in a single image. These statistics indicate that the benchmark includes both sparse and crowded scenes, which is relevant to the difficulty of face detection under interaction-heavy conditions (Gao et al., 11 Sep 2025).
The image-level composition and split are reported as follows:
| Item | Value |
|---|---|
| Total images | 22,879 |
| Total face annotations | 29,279 |
| Total phone annotations | 10,255 |
| Double (both face and phone present) | 9,888 |
| Face-only | 12,164 |
| Phone-only | 367 |
| Null | 460 |
| Training | 18,800 |
| Validation | 1,730 |
| Test | 2,349 |
The annotations are described as synchronized labels for face bounding boxes, phone bounding boxes, face–phone coexistence, and coarse behavior categories. The benchmark also provides pixel coordinates for faces and phones, absolute pairwise distances, and a CSV-formatted annotation file for evaluating final classification performance on validation and test sets. In addition, it includes fine-grained labels indicating whether the smartphone-use behavior is genuine.
The coarse behavior categories listed for the dataset are Calling, Using, Seated using, and Using obliquely. These labels define a behavior-aware annotation regime that goes beyond box-level localization. A plausible implication is that the dataset can support future work on structured interaction modeling, even though the baseline evaluation described in the benchmark focuses on detection and binary classification.
3. Scenario coverage and benchmark difficulty
The benchmark covers four major scenarios: Workplace, Education, Transportation, and Public spaces. These settings were selected because phone use is common in them and often consequential. The intended application areas include traffic safety and driver distraction monitoring, workplace productivity and compliance, classroom attention monitoring, and human–computer interaction and behavior-aware vision (Gao et al., 11 Sep 2025).
FPI-Det is explicitly described as intentionally difficult. Three challenges are highlighted. The first is scale variation: phones can be as small as 20 × 20 pixels, while faces can exceed 800 × 800 pixels. The second is occlusion, including phones partially covered by hands, faces partially hidden, and frequent side views or partial visibility. The third is diverse capture conditions, since images come from surveillance cameras, mobile devices, and online videos, introducing low light, compression artifacts, motion blur, cluttered scenes, and varied viewpoints.
These properties are important because the dataset is not organized around canonical, high-quality imagery. Instead, it is built around real-world, behavior-critical scenes in which detection and interpretation are entangled. The benchmark’s difficulty profile also explains why face detection may be harder than phone detection: faces occur in dense crowds, with frequent occlusion, pose variation, and lighting changes, whereas phones, although often small, may present a more constrained appearance when visible.
The paper contrasts FPI-Det with several benchmark families. Generic datasets such as COCO and PASCAL VOC focus on many object categories and typically treat object-level detection as the end goal. Face datasets such as WIDER FACE and FDDB do not include synchronized phone annotations. Interaction datasets such as HICO-DET are broader and do not target the specific safety-critical problem of face–phone interaction. Accordingly, FPI-Det is designed to answer not only “what objects are in the image?” but also “is this person using a phone?”
4. Tasks and evaluation protocol
The benchmark defines two formal tasks. The first is object detection of faces and phones. The second is binary phone-use classification based on detector outputs and interaction context (Gao et al., 11 Sep 2025).
For detection, the reported metrics are [email protected] and [email protected]. Performance is additionally analyzed by object category, object size, occlusion level, and environment. This evaluation design is aligned with the dataset’s central challenges, because overall mean average precision alone would not reveal the effects of scale, visibility, and scene type.
For phone-use classification, the benchmark defines a parameter-free binary classifier with the following rule: phone-in-use if a face and a phone are co-detected in the same frame, and not-in-use otherwise. The evaluated metrics are Accuracy, Precision, Recall, F1-score, and Specificity. The class convention is stated explicitly: the positive class is phone use (label 0), and the negative class is no phone use (label 1).
This classification setup is deliberately minimal. The binary decision depends mainly on co-occurrence rather than highly precise localization. The benchmark therefore exposes an important distinction: precise box regression matters strongly for detection metrics, especially at IoU 0.95, whereas the binary interaction decision can remain comparatively stable once both relevant object categories are detected.
5. Baseline detectors and benchmark results
The benchmark evaluates representative models from two detector families: YOLOv8-n, YOLOv8-s, YOLOv8-x, YOLOv11-n, YOLOv11-s, YOLOv11-m, YOLOv11-x, and the transformer-based DETR and Deformable DETR. This establishes a comparison between fast one-stage detectors and transformer-based global-reasoning detectors (Gao et al., 11 Sep 2025).
The principal detection findings are summarized below.
| Setting | Reported result |
|---|---|
| Best face [email protected] | YOLOv8-x, 56.5% |
| Best face [email protected] | YOLOv8-x and YOLOv11-x, 35.8% |
| Best phone [email protected] | YOLOv8-x, 92.4% |
| Best phone [email protected] | YOLOv8-x, 71.0% |
| Fastest model | YOLOv8-n, 416.7 FPS on a single NVIDIA Tesla V100 |
The benchmark reports that the YOLO family performs strongly overall, and that phones are generally detected more accurately than faces. Among the transformer baselines, DETR is substantially weaker than both YOLO and Deformable DETR, while Deformable DETR consistently improves over DETR. The reported improvement from DETR to Deformable DETR is Face [email protected]: 39.5 → 45.4, Face [email protected]: 21.2 → 29.7, Phone [email protected]: 57.3 → 68.9, Phone [email protected]: 26.9 → 40.6, [email protected]: 48.4 → 57.2, and FPS: 43.0 → 50.2. The larger gains at IoU 0.95 are noted as indicating much tighter localization for Deformable DETR.
The phone-use classification results are comparatively compressed across models. The best overall classifier is YOLOv11-x, with Accuracy: 89.5%, Precision: 85.9%, Recall: 89.1%, F1: 87.5%, and Specificity: 89.8%. Other notable results include YOLOv8-x with 89.2% accuracy and 87.0% F1, YOLOv11-s with the highest Recall: 92.2%, and YOLOv11-m with the highest Specificity: 90.6%. Among the transformer classifiers, DETR attains 89.2% accuracy and 87.0% F1, whereas Deformable DETR attains 88.4% accuracy and 85.3% F1.
These results support the benchmark’s observation that classification performance is relatively robust once both objects are detected. A plausible implication is that future gains in binary phone-use classification on this protocol may depend less on sophisticated relational heads than on improving recall for difficult face and phone instances under severe scale variation and occlusion.
6. Error analysis, practical uses, and benchmark significance
The benchmark’s qualitative and summary analysis identifies several consistent trends. Workplace scenes are reported as the easiest, likely due to more stable lighting and clearer compositions, while transportation scenes are the hardest. Small phones and heavily occluded objects remain the most difficult cases. Common failure modes include false positives on tablet-like objects, missed detections for very small phones, and missed detections for heavily occluded phones or faces (Gao et al., 11 Sep 2025).
These findings clarify that FPI-Det is not simply a narrow dataset about two categories. Its significance lies in making visible the operational bottlenecks of interaction-aware detection: failures arise from scale, clutter, and incomplete visibility rather than from a lack of category diversity. This is especially relevant for deployment settings in which missed detections may undermine downstream behavior analysis.
The benchmark is described as useful for driver distraction detection, workplace monitoring, classroom attention analysis, public safety surveillance, edge-deployable phone-use detection systems, and human-centered behavior understanding. It is therefore intended not only as an evaluation set but also as a development resource for models that need to move beyond generic object detection and reason about interaction.
The code and dataset are available at https://github.com/KvCgRv/FPI-Det. The images were collected from https://www.datafountain.cn/competitions/506, and the faces were de-identified to protect privacy. In this respect, FPI-Det combines a behavior-oriented detection task with explicit privacy handling, which is notable for a dataset constructed from realistic scenes.
7. Position within interaction-aware vision
FPI-Det is best understood as a large-scale, interaction-aware benchmark for detecting phone use through face–phone relationships. Its defining contribution is the combination of synchronized face/phone annotations, diverse real-world scenarios, challenging occlusion and scale variation, and a dual evaluation regime spanning both detection and classification (Gao et al., 11 Sep 2025).
A common misconception would be to treat FPI-Det as merely a dataset for finding phones or faces. The benchmark’s formulation rejects that interpretation: it is designed around the premise that phone-use understanding depends on the spatial and contextual relationship between a face and a phone. The addition of geometry and relation information, including pixel coordinates and pairwise distances, reinforces this orientation toward interaction modeling.
Within the broader landscape of computer vision benchmarks, FPI-Det represents a task-specific move from category recognition toward behavior-critical relational reasoning. It retains the operational simplicity of detection-style annotations and evaluation metrics, but directs them toward a higher-level question: whether the observed configuration constitutes genuine smartphone use. In that sense, its significance lies not in category breadth but in the formalization of a narrowly defined, safety-relevant interaction problem.
