FaceLiVTv2: An Improved Hybrid Architecture for Efficient Mobile Face Recognition

Abstract: Lightweight face recognition is increasingly important for deployment on edge and mobile devices, where strict constraints on latency, memory, and energy consumption must be met alongside reliable accuracy. Although recent hybrid CNN-Transformer architectures have advanced global context modeling, striking an effective balance between recognition performance and computational efficiency remains an open challenge. In this work, we present FaceLiVTv2, an improved version of our FaceLiVT hybrid architecture designed for efficient global--local feature interaction in mobile face recognition. At its core is Lite MHLA, a lightweight global token interaction module that replaces the original multi-layer attention design with multi-head linear token projections and affine rescale transformations, reducing redundancy while preserving representational diversity across heads. We further integrate Lite MHLA into a unified RepMix block that coordinates local and global feature interactions and adopts global depthwise convolution for adaptive spatial aggregation in the embedding stage. Under our experimental setup, results on LFW, CA-LFW, CP-LFW, CFP-FP, AgeDB-30, and IJB show that FaceLiVTv2 consistently improves the accuracy-efficiency trade-off over existing lightweight methods. Notably, FaceLiVTv2 reduces mobile inference latency by 22% relative to FaceLiVTv1, achieves speedups of up to 30.8% over GhostFaceNets on mobile devices, and delivers 20-41% latency improvements over EdgeFace and KANFace across platforms while maintaining higher recognition accuracy. These results demonstrate that FaceLiVTv2 offers a practical and deployable solution for real-time face recognition. Code is available at https://github.com/novendrastywn/FaceLiVT.

• The paper introduces FaceLiVTv2, a hybrid architecture combining Lite MHLA, RepMix, and GDConv for efficient mobile face recognition.
• It achieves a 30–35% reduction in mobile inference latency and improves mean accuracy by up to 0.21% compared to similar lightweight models.
• Its innovative use of structural reparameterization and adaptive spatial aggregation delivers robust performance on low-resolution and open-set benchmarks.

FaceLiVTv2: An Enhanced Hybrid Architecture for Mobile Face Recognition

Introduction

FaceLiVTv2 represents a substantial advancement in lightweight face recognition (FR) architectures targeting strictly resource-constrained environments such as mobile and edge devices. While prior hybrid CNN-Transformer models improved the global-local trade-off, FaceLiVTv2 introduces algorithmic and architectural innovations—most notably Lite Multi-Head Linear Attention (Lite MHLA), a structurally unified RepMix block, and adaptive spatial feature aggregation via global depthwise convolution (GDConv). The result is a model that demonstrably outperforms recent state-of-the-art (SOTA) methods in both accuracy and latency, bridging the performance gap with large-scale models while preserving practical deployment characteristics (2604.09127).

Figure 1: Comparison of FaceLiVTv2 with contemporaneous lightweight face recognition models, highlighting the trade-off between mean accuracy and mobile inference latency.

Architectural Innovations

FaceLiVTv2 is built upon three critical components: Lite MHLA for efficient global token interaction, RepMix for reparameterized local feature extraction, and GDConv for channel-adaptive spatial aggregation. This triad is systematically co-designed to minimize computational and memory overhead, particularly for deployment scenarios on NPUs and edge accelerators.

Overall Network Design

The network architecture adopts a staged macro-design with a feature pyramid backbone. RepMix blocks are used exclusively in the first two stages for efficient local spatial encoding, while the final two stages combine RepMix with Lite MHLA to emphasize global dependency modeling Figure 2.

Figure 2: FaceLiVTv2 architecture detailing the flow and position of RepMix and Lite MHLA modules across encoding stages.

Lite Multi-Head Linear Attention (Lite MHLA)

Lite MHLA is a streamlined version of the original MHLA, eliminating redundant per-head projections and nonlinearities. The main operation is a single linear mapping per head, bracketed by affine feature rescaling. Since BatchNorm precedes Lite MHLA, only an additional affine operation is required, reducing both redundancy and memory-bound workload (see Figure 3). This substantially lowers both parameter and runtime cost compared to multi-layer projections and standard attention mechanisms.

Figure 3: Block comparisons showing the design simplification from FaceLiVTv1's MHLA to FaceLiVTv2's Lite MHLA, and the respective architectural blocks.

RepMix and Structural Reparameterization

The RepMix token mixer block fuses depthwise 3×3 and 1×1 convolutions with BatchNorm during training. For inference, structural reparameterization absorbs residual and normalization paths directly into convolution weights, converting the block to a single 3×3 convolution, thus further reducing inference computational cost and memory access Figure 4.

Figure 4: Illustration of RepMix block structural reparameterization, merging multiple convolutional and normalization paths into a single deployable kernel.

Adaptive GDConv Head

Replacing the typical global average pooling, FaceLiVTv2 employs a global depthwise convolution for adaptive spatial feature aggregation. This operation allows channel-wise spatial weighting, yielding more discriminative embeddings, especially advantageous for open-set and low-resolution FR benchmarks.

Efficiency and Latency Profiling

A central claim of FaceLiVTv2 is that architectural improvements not only improve accuracy but critically reduce both compute- and memory-bound runtime on real hardware. Profiling analyses show most latency bottlenecks in prior lightweight attention modules stem from normalization and elementwise operations. Lite MHLA, by aggressively reducing LayerNorm and activation layers, achieves near-optimal operation distribution for mobile NPUs Figure 5.

Figure 5: Runtime breakdown on CoreML, highlighting reductions in memory-bound operations afforded by FaceLiVTv2’s architectural streamlining.

Numerical Results

A comprehensive benchmark evaluation across multiple FR datasets (LFW, CA-LFW, CP-LFW, CFP-FP, AgeDB-30, IJB-B/C, and TinyFace) demonstrates that FaceLiVTv2 achieves consistently favorable accuracy-latency trade-offs across all operating points Figure 6. FaceLiVTv2-XS and FaceLiVTv2-S surpass comparably sized models such as GhostFaceNet and EdgeFace by 0.16% to 0.21% mean accuracy, while reducing mobile inference latency by 30–35%. The larger FaceLiVTv2-L narrows the gap with heavyweight models (e.g., ResNet200-TopoFR) to less than 0.7% absolute mean accuracy, at >10× faster inference.

Figure 6: Pareto frontier illustrating the mean accuracy versus FLOPs and versus i5-12500 CPU latency for SOTA lightweight models including FaceLiVTv2.

Low-Resolution Robustness

On the TinyFace low-resolution benchmark, FaceLiVTv2 models, even without specialized super-resolution or distillation, outperform or match classic FR baselines with orders-of-magnitude fewer FLOPs, indicating excellent generalization under challenging visual conditions.

Ablation and Analysis

Extensive ablation analyses confirm the orthogonality and necessity of each architectural improvement:

• RepMix fusion and reparameterization provide necessary efficiency gains without sacrificing local feature fidelity.
• Lite MHLA yields the largest single accuracy-latency improvement, with non-linearity and normalization ablations supporting the choice of linear, affine-only interaction.
• Multi-head Lite MHLA configurations show optimality at four heads, saturating accuracy gains on IJB-B/IJB-C at minimal cost Figure 7.

Figure 7: Evolution of identification/verification performance and ROC as a function of Lite MHLA head count on IJBB/IJBC datasets.

Theoretical and Practical Implications

The FaceLiVTv2 design validates several theoretical points:

• Purely linear token-mixing (sans MLP, nonlinearities) suffices for effective global context capture when combined with strong local feature encoders, as previously suggested by ResMLP-like findings.
• Hardware-aligned operator design—minimizing memory-bound normalization and maximizing block fusion through structural reparameterization—translates directly to increased edge device throughput.
• Adaptive spatial aggregation via GDConv, rather than uniform pooling, enhances open-set and low-resolution recognition which is increasingly critical for mobile FR under real-world constraints.

On-device evaluation further substantiates portable deployability via direct CoreML conversion and runtime profiling. As mobile hardware progresses, FaceLiVTv2’s structurally optimized blocks will map more cleanly to accelerator primitives than more fragmented CNN-Transformer hybrids.

Future Prospects

The architecture could be further adapted by integrating generative priors for extreme pose/occlusion robustness or by applying knowledge distillation from larger diffusion-based FR systems. Additional investigation on fairness, cross-ethnicity generalization, and demographic de-biasing is critical—especially as such face recognition models proliferate in public and security applications.

Conclusion

FaceLiVTv2 establishes a new standard for accuracy–efficiency trade-offs in mobile face recognition, leveraging co-designed efficient global-local feature mixers and channel-adaptive aggregation within an architecture explicitly optimized for both statistical and hardware constraints. Its efficacy across a suite of challenging benchmarks and hardware platforms positions it as a reference model for real-time FR systems on resource-limited devices.

Figure 8: ROC curves demonstrating FaceLiVTv2 identification accuracy under challenging open-set protocols on IJBB and IJBC datasets.

Figure 9: Fine-grained latency breakdowns for all FaceLiVTv2 model variants on Apple A17 silicon (iPhone 15 Pro) using CoreML.

Figure 10: Operator distribution per layer as recorded by the CoreML runtime, highlighting the low proportion of memory-heavy operations in FaceLiVTv2.

---

Reference:

"FaceLiVTv2: An Improved Hybrid Architecture for Efficient Mobile Face Recognition" (2604.09127)