LenslessFace: An End-to-End Optimized Lensless System for Privacy-Preserving Face Verification (2406.04129v1)

Abstract: Lensless cameras, innovatively replacing traditional lenses for ultra-thin, flat optics, encode light directly onto sensors, producing images that are not immediately recognizable. This compact, lightweight, and cost-effective imaging solution offers inherent privacy advantages, making it attractive for privacy-sensitive applications like face verification. Typical lensless face verification adopts a two-stage process of reconstruction followed by verification, incurring privacy risks from reconstructed faces and high computational costs. This paper presents an end-to-end optimization approach for privacy-preserving face verification directly on encoded lensless captures, ensuring that the entire software pipeline remains encoded with no visible faces as intermediate results. To achieve this, we propose several techniques to address unique challenges from the lensless setup which precludes traditional face detection and alignment. Specifically, we propose a face center alignment scheme, an augmentation curriculum to build robustness against variations, and a knowledge distillation method to smooth optimization and enhance performance. Evaluations under both simulation and real environment demonstrate our method outperforms two-stage lensless verification while enhancing privacy and efficiency. Project website: \url{lenslessface.github.io}.

• The paper introduces an end-to-end optimized lensless system that verifies faces directly from encoded captures, improving privacy and efficiency over traditional methods.
• A physics-based face center alignment method handles scene shifts unique to lensless systems by leveraging translation equivalence.
• Experimental validation confirms the system's improved accuracy and efficiency for lensless face verification while preserving the inherent privacy benefits of encoded sensor captures.

The paper "An End-to-End Optimized Lensless System for Privacy-Preserving Face Verification" introduces a novel approach for performing face verification in a lensless imaging system. Such systems offer a compact, lightweight, and cost-effective solution by replacing traditional lenses with flat optics that encode light onto sensors in a non-recognizable manner. This property is particularly suited for applications where privacy is a concern, such as facial verification.

Key Contributions:

1. End-to-End Optimization: The paper proposes an end-to-end optimization method that directly performs face verification on the encoded images captured by lensless cameras, bypassing the traditional two-stage process of image reconstruction followed by verification. This approach enhances privacy by eliminating visible intermediate images and optimizing the entire pipeline for enhanced efficiency and accuracy.
2. Face Center Alignment: To address the challenges posed by lensless captures, which do not align or crop scenes like traditional systems, the paper introduces a physics-based face center alignment method. This technique centers encoded captures by utilizing the principle of translation equivalence, which is unique to lensless systems.
3. Augmentation Curriculum: A curriculum learning strategy is employed to progressively increase the model's robustness against variations such as rotation, scaling, and background changes. This gradual increase in augmentation complexity improves the model's generalization capabilities.
4. Cross-Modality Knowledge Distillation: The paper incorporates a novel cross-modality distillation approach to transfer knowledge from pre-trained RGB-based teacher models to the lensless student model. This method leverages relational knowledge distillation, focusing on structural feature relations, which aligns and enhances learning in the student model without requiring pixel-perfect correspondence.

Experimental Validation:

• The proposed system is evaluated on both simulated and real-world data, demonstrating significant improvements over state-of-the-art lensless and traditional verification methods in terms of privacy, accuracy, and efficiency.
• Real-world evaluation involved capturing scenes with a lensless prototype consisting of a CMOS sensor and an optimized binary amplitude mask. The proposed system outperformed previous methods that utilized reconstruction or directly processed lensless captures without optimization.

Privacy Assurance:

• The paper emphasizes that the inherent privacy features of lensless systems are preserved. For instance, raw sensor captures do not reveal recognizable facial features, making them inherently resistant to software attacks. Experimental results under various attack scenarios illustrate that attackers cannot efficiently reconstruct the original scenes, even with access to multiple plaintext attacks.

Challenges and Future Work:

• Although the approach significantly closes the performance gap compared to traditional RGB systems, some discrepancies still exist between simulated and real-world conditions, primarily due to noise and system imperfections.
• Future research directions may include refining mask designs, enhancing the robustness of the learning framework against broader environmental variations, and optimizing computational load for real-time applications.

Overall, by integrating and optimizing optical and electronic components within an end-to-end framework, this research significantly advances the lensless imaging field with a strong emphasis on privacy-preserving applications.