Cluster and Aggregate: Face Recognition with Large Probe Set (2210.10864v3)

Abstract: Feature fusion plays a crucial role in unconstrained face recognition where inputs (probes) comprise of a set of $N$ low quality images whose individual qualities vary. Advances in attention and recurrent modules have led to feature fusion that can model the relationship among the images in the input set. However, attention mechanisms cannot scale to large $N$ due to their quadratic complexity and recurrent modules suffer from input order sensitivity. We propose a two-stage feature fusion paradigm, Cluster and Aggregate, that can both scale to large $N$ and maintain the ability to perform sequential inference with order invariance. Specifically, Cluster stage is a linear assignment of $N$ inputs to $M$ global cluster centers, and Aggregation stage is a fusion over $M$ clustered features. The clustered features play an integral role when the inputs are sequential as they can serve as a summarization of past features. By leveraging the order-invariance of incremental averaging operation, we design an update rule that achieves batch-order invariance, which guarantees that the contributions of early image in the sequence do not diminish as time steps increase. Experiments on IJB-B and IJB-S benchmark datasets show the superiority of the proposed two-stage paradigm in unconstrained face recognition. Code and pretrained models are available in https://github.com/mk-minchul/caface

• The paper introduces a two-stage feature fusion approach called 'Cluster and Aggregate', combining a Cluster Network and an Aggregation Network to effectively handle large probe sets.
• It decouples identity features from style cues using a simplified Style Input Maker, thereby enhancing recognition accuracy and computational efficiency.
• Empirical results on datasets like IJB-B and IJB-S demonstrate improved True Acceptance Rates and reduced memory usage compared to state-of-the-art methods.

Cluster and Aggregate: Face Recognition with Large Probe Set

The paper "Cluster and Aggregate: Face Recognition with Large Probe Set" presents a method for enhancing face recognition performance when dealing with large sets of probe images. The central motivation for this work is the challenge posed by unconstrained face recognition scenarios where input data for each identity, either in probe or gallery sets, can comprise numerous low-quality images, requiring effective feature fusion.

Methodology

The proposed approach introduces a two-stage feature fusion paradigm termed "Cluster and Aggregate." The specific challenges addressed include handling large probe sets efficiently and maintaining robust sequential inference capabilities without being affected by input ordering. The proposal integrates two main networks: the Cluster Network (CN) and the Aggregation Network (AGN).

1. Cluster Network (CN):
   • The CN serves to linearly map varying numbers of probe images $N$ onto a fixed number of cluster centers $M$. Unlike conventional attention mechanisms which suffer from quadratic complexity and sensitivity to sequence order, CN uses a learned global clustering mechanism facilitated by fixed, shared query embeddings, termed cluster centers. This aids in efficiently summarizing large probe sets into compact representations while overcoming the quadratic complexity limit inherent in traditional attention mechanisms.
2. Style Input Maker (SIM):
   • For effective feature clustering, the authors propose extracting style information using a simplified neural module, leveraging the first and second-order statistics of intermediate feature maps. This formulation decouples the identity features from other style-related cues, aiding CN in achieving improved face recognition performance.
3. Aggregation Network (AGN):
   • The AGN is tasked with fusing the clustered features into a single representative feature vector. Through an MLP-Mixer architecture, AGN capitalizes on intra-set relationships of the clustered features, integrating them into an informative aggregate representation.

Experimental Results and Observations

The experimental results demonstrate the efficacy of the Cluster and Aggregate methodology over prior state-of-the-art methods in unconstrained face recognition contexts. The following observations highlight its performance benefits:

• Recognition Accuracy: CAFace markedly improves upon existing methods like PFE, CFAN, and RSA in metrics like True Acceptance Rate (TAR) at various False Acceptance Rates (FAR) levels, across datasets such as IJB-B and IJB-S, with superior performance particularly noted in scenarios involving large probe sizes.
• Efficiency: The paper emphasizes that CAFace handles sequential input data efficiently due to its batch-order invariance capability. This is bolstered by the cluster-based representation allowing effective summarization of high-volume inputs.
• Memory Usage: Compared to attention-based counterparts like RSA, the proposed method requires significantly less memory while providing the option for sequential inference, making it suitable for large-scale, real-time applications.

Implications and Future Prospects

The Cluster and Aggregate strategy provides a significant stride in face recognition tasks involving large input sets by decoupling the clustering assignment from the input size and enabling sequential inference. This capability holds potential for broader applications, such as surveillance systems and large-scale biometric databases, where effective and efficient face recognitions are paramount.

The proposed framework paves the way for future research into hybrid models that maintain high accuracy and computational efficiency in real-time scenarios. Additionally, incorporating more sophisticated style embeddings or adaptive clustering paradigms that can adjust to the specific dataset characteristics dynamically would be promising directions for extended research and development in this field.