FRUITS: Face Recognition Under Inference-Time Constraints

• The protocol evaluates face recognition systems within fixed inference time ceilings by integrating face detection, alignment, feature extraction, and matching.
• End-to-end measurements on resource-constrained hardware highlight the trade-off between speed and accuracy across mobile, edge, and cloud applications.
• Benchmarks include both identification and verification metrics, guiding model selection for reliable performance in practical deployment scenarios.

Face Recognition Under Inference Time conStraint (FRUITS) Protocol is a rigorous evaluation and benchmarking framework for face recognition systems that places a strict upper bound on end-to-end inference latency, including face detection and alignment, feature embedding, and matching. By enforcing explicit inference-time ceilings and reporting accuracy together with latency—rather than considering accuracy in isolation—the protocol enables deployment-focused, reproducible comparison of face recognition pipelines under real-world resource and responsiveness constraints (Zhu et al., 2021, Zhu et al., 2022, Zhu et al., 2021, Almadan et al., 2021).

1. Motivation and Scope

FRUITS was developed to address the disconnect between state-of-the-art face recognition models, which often achieve high accuracy but at prohibitive computational cost, and the realities of deployment on mobile, edge, or cloud hardware. Its core purpose is to assess whether a face recognition system can deliver reliable results within prescribed (and limited) wall-clock time budgets, such as those found in law enforcement body-worn cameras, mobile devices, airport checkpoints, and cloud-based large-scale identity verification (Zhu et al., 2021, Almadan et al., 2021, Zhu et al., 2021).

The protocol is distinguished by its end-to-end perspective: all subprocesses—from face detection through to similarity scoring—must be completed within the specified latency window, and all evaluations are conducted on resource-constrained hardware (typically a single CPU core, excluding GPU or specialized accelerator access) (Zhu et al., 2021, Zhu et al., 2022).

2. FRUITS Tracks and Latency Budgets

The protocol defines three primary inference time tracks, each corresponding to a canonical deployment scenario (Zhu et al., 2022, Zhu et al., 2021):

Track	Time Budget (ms)	Use Case
FRUITS-100	100	Mobile/IoT
FRUITS-500	500	Edge/Local surveillance
FRUITS-1000	1000	Cloud/Kiosk

The time budget encompasses the entire processing pipeline—face detection, five-point landmark alignment, feature extraction, and matching—and is enforced per image (or image-pair for verification). No component may be excluded or amortized beyond the batch-level (if allowed); wall-clock time is measured over multiple runs and typically the maximum per-sample latency is reported (Zhu et al., 2021, Zhu et al., 2022, Zhu et al., 2021).

3. Latency Measurement Protocol

Latency is measured on a single physical core (e.g., Intel Xeon E5-2630-v4 @ 2.20 GHz), with standard CPU clock governors and without turbo/frequency scaling. The submitted pipeline—including the detection, alignment, and embedding modules, executed in a self-contained Docker container—is timed start-to-finish on raw unseen images. Warmup runs (typically 100 iterations) are conducted to stabilize caches and I/O, followed by several hundred/thousand timed executions. The throughput is computed as the reciprocal of the mean per-sample inference time:

$\text{Throughput} = \frac{1}{E[t(x)]}$

where $t(x)$ is the wall-clock time for processing image $x$ (Zhu et al., 2022, Zhu et al., 2021). In challenge contexts, both overall pipeline time and individual component times ($T_\text{det}$ for detection+alignment, $T_\text{rec}$ for feature extraction) are logged for further analysis (Zhu et al., 2021).

4. Evaluation Metrics and Datasets

FRUITS employs both identification and verification metrics. For identification tasks (body-worn camera use cases, e.g., BWCface dataset), rank-$k$ identification accuracy is defined as:

$$\text{Rank-}k\text{-Accuracy} = \frac{1}{P}\sum_{i=1}^P I_i \times 100\%$$

where $P$ is the number of probe images, and $I_i = 1$ if the correct identity appears among the top-$k$ gallery matches for probe $t(x)$0 (Almadan et al., 2021).

For verification tasks on WebFace260M and related benchmarks, performance is characterized by False Non-Match Rate (FNMR) at a fixed False Match Rate (FMR), typically $t(x)$1, with threshold $t(x)$2 set accordingly (Zhu et al., 2021, Zhu et al., 2022). Additional fairness metrics such as Subgroup Error Ratio (SER) and per-group FNMR variance (STD) are reported for unbiased face recognition (Zhu et al., 2022). Datasets used in FRUITS spans controlled and in-the-wild images, with attributes for age, race, gender, and mask status, supporting evaluation across standard, masked, and unbiased face recognition tasks (Zhu et al., 2021, Zhu et al., 2022, Zhu et al., 2021).

5. Pipeline Structure and Reporting

A typical FRUITS-compliant pipeline includes:

• Face Detection & Alignment: RetinaFace-based detectors (MobileNet-0.25 or ResNet-50 backbones) for detection and five-point alignment.
• Embedding/Recognition Module: Lightweight or heavyweight CNNs such as MobileFaceNet, EfficientNet-B0, ResNet (depth 14 to 200), SENet, LightCNN, RegNet, and VGG variants.
• Similarity Computation: Cosine similarity (CNN features) or Euclidean distance for hand-crafted features (Zhu et al., 2021, Almadan et al., 2021).

Reporting under FRUITS includes comprehensive tables and Pareto frontier plots summarizing, for each model:

Model	Params (M)	$t(x)$3 (ms)	Rank-1 [%] or FNMR@FMR
MobileNet-V2	3.4	240	85.5 (Almadan et al., 2021)
EfficientNet-B0	5.3	329	80.95 (Almadan et al., 2021)
LightCNN-9	5.5	171	93.35 (Almadan et al., 2021)
LightCNN-29	12.6	430	94.48 (Almadan et al., 2021)
ResNet-50	23.5	834	96.33 (Almadan et al., 2021)
VGG-16	138	3747	81.65 (Almadan et al., 2021)

Additionally, accuracy-latency Pareto plots, rank-$t(x)$4 curves, and attribute-stratified results are recommended (Almadan et al., 2021, Zhu et al., 2021, Zhu et al., 2022).

6. Trade-Offs and Model Selection

FRUITS formalizes the speed–accuracy trade-off by fixing a maximum allowable inference time ($t(x)$5) and reporting the best achievable accuracy (e.g., lowest FNMR or highest rank-1) within that constraint. For each deployment tier, the protocol surfaces a clear performance frontier:

• Mobile/FRUITS-100: Only extremely lightweight models (e.g., RegNet-800MF, MobileFaceNet, EfficientNet-B0) can satisfy $t(x)$6 ms, but typically incur FNMR > 5% (Zhu et al., 2021, Zhu et al., 2022).
• Edge/FRUITS-500: Mid-size backbones (ResNet-50, ResNet-100, SENet-50/RegNet-8GF) can reach FNMR ≈ 1–2% within 500 ms (Zhu et al., 2022).
• Cloud/FRUITS-1000: Deep models (ResNet-200, SENet-152, AttentionNet-152, RegNet-16GF) achieve near 0.8% FNMR, maximizing accuracy at the expense of high latency and resource use (Zhu et al., 2021, Zhu et al., 2022).

Model selection in FRUITS is guided by the accuracy-latency Pareto front: choose models on or near the front that meet operational accuracy requirements under the desired latency ceiling (Almadan et al., 2021, Zhu et al., 2021).

7. Deployment Guidelines and Impact

Best practices identified through FRUITS deployments include:

• Joint optimization of detection and recognition, as detector latency dominates for lightweight backbones (Zhu et al., 2022, Zhu et al., 2021).
• End-to-end pipeline export into ONNX to minimize language-level and I/O overhead at inference (Zhu et al., 2021).
• Use of large, cleaned training data (e.g., WebFace42M) and augmentation (e.g., synthetic mask overlay) for robustness, especially in masked recognition scenarios (Zhu et al., 2022, Zhu et al., 2021).
• Cautious trade-off between speed and accuracy, accepting minor accuracy loss on easy subsets to improve performance in challenging masked or cross-condition cases (Zhu et al., 2021).
• Explicit latency budgeting for each use case: e.g., ≤100 ms for mobile (FNMR ≈ 12%), ≈500 ms for edge (FNMR ≈ 3%), ≤1000 ms for cloud (FNMR ≈ 1%) (Zhu et al., 2022, Zhu et al., 2021).

The FRUITS protocol has been adopted in large-scale benchmarking, including masked face recognition challenges during COVID-19, and has enabled granular fair and real-world relevant assessments unattainable by accuracy-only protocols (Zhu et al., 2021, Zhu et al., 2022, Zhu et al., 2021).

---

References:

• (Almadan et al., 2021) Towards On-Device Face Recognition in Body-worn Cameras
• (Zhu et al., 2021) WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition
• (Zhu et al., 2022) WebFace260M: A Benchmark for Million-Scale Deep Face Recognition
• (Zhu et al., 2021) Masked Face Recognition Challenge: The WebFace260M Track Report
• (Karlekar et al., 2019) Deep Face Recognition Model Compression via Knowledge Transfer and Distillation