Facial Similarity (F-Sim) Analysis

• F-Sim is the quantitative and qualitative assessment of human face resemblance using biometric scores, perceptual metrics, and deep learning techniques.
• Recent advances incorporate robust techniques such as image quality normalization, regional statistical analysis, and crowdsourced triplet ranking to address real-world variability.
• F-Sim evaluation bridges technical methods and human perception, driving improvements in biometrics, forensic analysis, and social research.

Facial Similarity (F-Sim) is the quantitative and qualitative assessment of how alike two or more human face images appear, either to automated systems or to human observers. While facial similarity overlaps with face recognition, which focuses on identity discrimination, F-Sim shifts the problem to evaluating resemblance independent of identity—addressing applications as diverse as biometric uniqueness scoring, lookalike retrieval, kinship analysis, and the match quality of facial synthesis. Advances in F-Sim span several subdomains including impostor score analysis, perceptual and structural metrics for face sketches, deep metric learning for subjective similarity, biologically inspired codes, evolutionary hypotheses, and the benchmarking of dataset difficulty.

1. Impostor Scores and Facial Uniqueness

Impostor scores are central to classical biometric approaches for F-Sim. For a given facial image $i$ and a gallery $J$ of impostor images from different subjects, similarity scores $S = \{ s(i, j_k) \}$ quantify how much $i$ matches other faces. Klare and Jain’s uniqueness measure (Dutta et al., 2013) formalizes facial uniqueness as:

$$u(i, J) = \frac{S_{max} - \mu_S}{S_{max} - S_{min}}$$

where $S_{max}$, $S_{min}$, and $\mu_S$ are the maximum, minimum, and mean scores in $S$, respectively. A larger $u(i, J)$ denotes a more unique face in the biometric "face space". However, empirical evidence demonstrates this metric is highly sensitive to image quality factors—pose, noise, and blur can dramatically shift the impostor score distribution, decoupling the measure from actual identity uniqueness. Correlation of $u(i, J)$ across high-quality and degraded images drops from $\approx0.68$ to as low as $0.13$, highlighting instability under typical real-world conditions.

This instability implies that impostor-based F-Sim metrics must explicitly incorporate image quality normalization or identify robustness to outliers, since unmitigated quality variation can obscure the underlying similarity structure, leading to degeneracy in uniqueness evaluation.

2. Perceptual and Structure-Based Sketch Metrics

Moving beyond pixel-level metrics such as SSIM or FSIM, recent approaches leverage regional statistics and perceptually relevant structure for F-Sim in face sketches. The Scoot-measure (Fan et al., 2018, Fan et al., 2019) quantifies similarity by

• Dividing an image into grids of blocks.
• Computing co-occurrence matrices for blockwise quantized intensity levels and multiple directions.
• Extracting robust statistics such as Contrast $(\mathcal{C})$ and Energy $(\mathcal{E})$,
• Aggregating these features to form the style descriptor.

Similarity is then the inverse of the Euclidean distance between feature vectors:

$$E_s = \frac{1}{1 + \| \Psi(X'_s) - \Psi(Y'_s) \|_2 }$$

Meta-measures (stability to resizing, rotation, content capture, and human judgment agreement) validate that Scoot-measure maintains ranking consistency under mild geometric and photometric changes, with human correlation reaching as high as 78.8% (compared to $\approx$58.6% for prior metrics). These findings reveal that perceptually inspired F-Sim metrics that honor block-level structure and local statistical texture are more robust and human-aligned than pixel-based metrics.

The large-scale human-ranking database (over 152k judgments) further substantiates Scoot’s efficacy in evaluating facial sketch synthesis, providing a standard for model calibration and benchmarking.

3. Deep Metric Learning for Perceived Face Similarity

Facial similarity as perceived by humans does not always correlate with identity labels. "Finding your Lookalike" (Sadovnik et al., 2018) establishes that F-Sim is a subjective metric, often misrepresented by recognition-oriented systems when two faces are not identical but "look alike." Data collection involves crowdsourced triplet ranking (anchor, more similar positive, less similar negative), using an 80% consensus threshold to ensure annotation reliability. The Lookalike network fine-tunes VGG-Face using triplet loss:

$$L = \sum_i \max(0, \|f(x_i^a) - f(x_i^p)\| - \|f(x_i^a) - f(x_i^n)\| + \alpha)$$

where $f(x)$ is the embedding of image $x$, designed to align positive anchors closer than negatives in similarity space. Augmenting with "easy" triplets prevents collapse of global structure.

This method achieves hard-triplet accuracy up to 81.36% and elevates top-1 precision from 21.6% to 33.2%, empirically separating similarity from identity classification. F-Sim therefore requires dedicated training objectives and data, not mere reuse of recognition tasks.

4. Biological and Psychological Models of Similarity

Gabor wavelet filtering (Lyons et al., 2020) and the Linked Aggregate Code (LAC) (Lyons et al., 2020) extend F-Sim by modeling localized, multi-scale filter responses in topographically aligned facial grids. The similarity between filter response vectors at corresponding locations is aggregated to form a biologically plausible similarity space, aligning with the circumplex model of affect—affective similarity emerges along orthogonal axes of pleasantness and arousal.

Notably, dimensions such as sex and race arise naturally in mixed-category tasks with LAC, without supervised labels. However, LAC does not reproduce all social biases observed in human similarity judgments (for example, racial clustering), indicating that certain perceptual biases may originate beyond primary visual coding and require higher-order or learned mechanisms.

5. Quantitative Evaluation, Benchmarking, and Dataset Effects

The challenge of validating F-Sim metrics is addressed by benchmarking on large datasets of twins and non-twin lookalikes (Sami et al., 2022) and facial expression datasets (Gaya-Morey et al., 26 Mar 2025). Siamese networks trained with contrastive loss on twin pairs set a baseline for "maximum" non-identity similarity, achieving AUC = 0.9799. This enables quantitative assessment of the prevalence and impact of highly similar impostor pairs on recognition systems. Correlation analyses show that while high similarity scores often align with recognition scores, significant scatter indicates that additional discriminative factors mediate identity decisions; thus, F-Sim and recognition must be decomposed as related but not identical constructs.

In expression recognition, novel metrics—Local, Global, and Paired Similarity—quantify within-dataset difficulty, cross-dataset generalization, and transferability. Large, variable datasets such as AffectNet and FER2013 excel in generalization, while smaller, controlled datasets demonstrate higher within-dataset performance but weaker transfer characteristics. Paired Similarity ratios greater than 1 reveal training datasets with inherent redundancy beneficial for transfer, guiding effective dataset selection.

Metric	Measures	Context
Impostor Score	Identity-dependence but quality sensitive	Biometrics
Scoot-measure	Structure+Texture, block-level robustness	Sketch F-Sim
Contrastive Loss	Learned twin-level similarity	Embedding

6. Applications and Implications

Applications of F-Sim span the matching of masked and unmasked faces using cosine similarity (Abdullah et al., 8 Jan 2025), forensic voice-to-face synthesis with embedding fusion (Bai et al., 2020), and enhanced recognition via facial symmetry regularization (Prakash et al., 18 Sep 2024). In masked face matching, feature extraction via transfer-learned VGG16 paired with cosine similarity and K-NN classification achieves up to 95% identification accuracy, directly addressing the occlusion problem that degrades classical recognition systems.

Symmetric embedding loss functions on vertically split faces result in superior intra-class compactness and inter-class separation, validated across benchmarks such as LFW and AgeDB.

Evolutionary and social implications arise from studies linking facial similarity to parental investment, with experimentally measured decreases in father-son resemblance correlating with increased paternal promiscuity (Stone, 2021).

7. Directions for Future Research and Challenges

Persistent challenges in F-Sim include sensitivity to image quality, unstable metrics under real-world conditions, and limited alignment between machine-generated and human-perceived salient regions in expressive face analysis (Gaya-Morey et al., 22 Jan 2024). Quantitative analysis with Intersection over Union (IoU) and correlation coefficients consistently demonstrates that CNN-based attention heatmaps only partially align with ground-truth human facial action unit masks, regardless of network depth or pre-training.

Research trajectory points to normalization approaches for quality variations, refinement of outlier-insensitive uniqueness measures, incorporation of human perceptual benchmarks, and cross-modal similarity learning. Furthermore, inquiry into dimensions that emerge naturally in biologically inspired codes, the role of social bias, and data-driven generalization across demographic groups continues to shape the landscape of robust F-Sim evaluation.

---

Facial Similarity (F-Sim) is an evolving, multifaceted field that integrates biometric verification, perceptual metrics, psychological coding, deep metric learning, and comprehensive benchmarking. Technical progress depends on the development and validation of metrics that both correspond to human perceptual reality and maintain robustness under diverse acquisition conditions, with significant implications for security, forensics, affective computing, and social understanding.