EX-FIQA: Leveraging Intermediate Early eXit Representations from Vision Transformers for Face Image Quality Assessment

Abstract: Face Image Quality Assessment is crucial for reliable face recognition systems, yet existing Vision Transformer-based approaches rely exclusively on final-layer representations, ignoring quality-relevant information captured at intermediate network depths. This paper presents the first comprehensive investigation of how intermediate representations within ViTs contribute to face quality assessment through early exit mechanisms and score fusion strategies. We systematically analyze all twelve transformer blocks of ViT-FIQA architectures, demonstrating that different depths capture distinct and complementary quality-relevant information, as evidenced by varying attention patterns and performance characteristics across network layers. We propose a score fusion framework that combines quality predictions from multiple transformer blocks without architectural modifications or additional training. Our early exit analysis reveals optimal performance-efficiency trade-offs, enabling significant computational savings while maintaining competitive performance. Through extensive evaluation across eight benchmark datasets using four FR models, we demonstrate that our fusion strategy improves upon single-exit approaches. Our proposed quality fusion approach employs depth-weighted averaging that assigns progressively higher importance to deeper transformer blocks, achieving the best quality assessment performance by effectively leveraging the hierarchical nature of feature learning in ViTs. Our work challenges the conventional wisdom that only deep features matter for face analysis, revealing that intermediate representations contain valuable information for quality assessment. The proposed framework offers practical benefits for real-world biometric systems by enabling adaptive computation based on resource constraints while maintaining competitive quality assessment capabilities.

• The paper introduces EX-FIQA, leveraging intermediate early exit representations within Vision Transformers to significantly reduce computational cost while maintaining accuracy.
• The paper details two fusion strategies—uniform averaging and depth-weighted fusion—that effectively integrate predictions from multiple transformer exits.
• The paper demonstrates strong performance improvements on eight benchmarks, enabling efficient and accurate face quality assessment in real-world scenarios.

Intermediate Early eXit Representations for Robust Face Image Quality Assessment with Vision Transformers

Introduction and Problem Definition

Face Image Quality Assessment (FIQA) plays a critical role in biometric systems, directly determining which face samples are retained for downstream recognition or verification tasks. Traditional FIQA methods—both supervised and unsupervised—predominantly employ deep CNNs or Vision Transformer (ViT) architectures, but rely exclusively on the final-layer representations, discarding potentially valuable quality signals present at intermediate depths. Deploying ViTs for FIQA in real-world, resource-constrained environments (e.g., edge biometrics, live surveillance) is further hampered by their heavy computational demands.

This paper introduces EX-FIQA, a methodological framework for leveraging intermediate early exit representations within ViTs to enhance FIQA. The central thesis is that intermediate transformer blocks encode complementary, quality-relevant information, and that judicious use of early exit strategies and multi-exit score fusion can yield substantial gains in both computational efficiency and accuracy. Extensive evaluation on eight public benchmarks demonstrates that the proposed EX-FIQA system enables up to 50% reduction in computational cost while sustaining or even improving upon state-of-the-art FIQA performance (2604.22842).

EX-FIQA: Architectural Design and Mechanisms

Vision Transformer Backbone and Early Exit Implementation

EX-FIQA is built upon the ViT-FIQA architectures (CR-FIQA-style and quality-token variants), each consisting of 12 stacked transformer blocks. Unlike CNNs, ViTs maintain a constant token dimensionality across all blocks, enabling the seamless application of early exit heads at arbitrary depths without auxiliary adaptation layers.

An early exit at block $l$ corresponds to direct extraction and processing of either:

• The CR-FIQA-style patch embeddings (EX-FIQA (C)), which are concatenated and passed through a lightweight MLP and regression head;
• The quality token (EX-FIQA (T)), which is regressed to a scalar score by the pre-trained head, by design (no extra training or parameters needed).

EX-FIQA thus sidesteps the redundancy of running the full transformer for samples whose quality can be confidently assessed at shallower depths.

Figure 1: Overview of EX-FIQA (C) architecture and early exit mechanism; inference can be terminated at any intermediate block, with score fusion for robust quality prediction.

Fusion Strategies Across Depth

The paper proposes two fusion approaches that aggregate scores from all 12 exits:

• Uniform averaging (EX-FIQA-F): All layer-specific predictions are equally weighted.
• Depth-weighted fusion (EX-FIQA-FW): Predictions from deeper blocks are assigned higher weights, consistent with the empirical finding that deeper layers generally yield more discriminative quality scores.

This fusion capitalizes on complementarity between shallow and deep block predictions, consistently outperforming reliance on any single exit.

Analysis of Layer-wise Attention and Quality Prediction

A comprehensive study of quality predictions and attention map evolution across exits underpins the core claim: intermediate blocks capture distinct and semantically diverse aspects of facial quality (e.g., low-level texture in shallow blocks, occlusion patterns in mid-level blocks, and global face configuration in deeper stages).

Figure 2: EX-FIQA (T) quality token attention maps across images and exit points demonstrate progressive semantic refinement.

Figure 3: EX-FIQA (C) patch-level attention map evolution, visualizing shifting network focus and integration over block depth.

Consistent trends are observed:

• Early exits attend to image-level artifacts and local details.
• Middle blocks increasingly focus on canonical facial regions while suppressing background.
• Deepest blocks establish global, identity-relevant context.

These dynamics enable intermediate exits to function as valid, sometimes optimal, quality predictors—particularly for samples with distinctive noise types, occlusions, or challenging poses.

Performance and Efficiency Evaluation

Trade-off Analysis: Computational Cost vs. Predictive Accuracy

The most substantial practical advantage of EX-FIQA is the demonstrated computational efficiency gained via early exits, without significant degradation in FIQA performance.

Figure 4: Performance-efficiency trade-off for EX-FIQA exits; pAUC-EDC improves monotonically with exit depth, but early to mid-level exits yield most of the gains at a fraction of the FLOPs.

Middle-block exits (6–10) achieve comparable pAUC-EDC to the final layer, but at 41–50% lower FLOPs. Similarly, fusion (especially EX-FIQA-FW) requires all exits but only imposes sub-10% additional overhead when fully parallelized. The marginal benefit of propagating all the way to exit 12 is dataset-dependent and only justified under stringent accuracy constraints.

Figure 5: Joint depiction of AUC/pAUC-EDC and complexity, highlighting the sweet spot for deployment in constrained environments.

Benchmark Results versus State-of-the-Art

Quantitative evaluation on LFW, AgeDB-30, CFP-FP, CALFW, Adience, CPLFW, XQLFW, and IJB-C shows that EX-FIQA-FW achieves:

• Top-1 pAUC-EDC and AUC-EDC on IJB-C and Adience across all tested FR backbones (ArcFace, ElasticFace, MagFace, CurricularFace).
• Consistently superior trade-offs when measured at stringent security thresholds (FNMR@FMR = 1e-3, 1e-4).
• On small-scale benchmarks, EX-FIQA-FW performs competitively with, and often surpasses, final-exit and standard single-layer FIQA baselines.

  Figure 6: Error-vs-Discard Curves at FNMR/FMR=1e-2, demonstrating strong performance of EX-FIQA across all datasets.

Theoretical and Practical Implications

The results challenge the standard assumption that only final-layer transformer features encode utility for biometric quality. EX-FIQA reveals a dense distribution of task-relevant information throughout the ViT stack, suggesting potential for layer-wise, adaptive model deployment across other computer vision applications, such as object detection, medical image triage, or multimodal fusion, where inference efficiency is paramount.

In practice, EX-FIQA provides an architecture-agnostic pathway to application-specific trade-off tuning: sub-selecting optimal early exit layers according to time, hardware, or energy constraints. Adaptive inference—leveraging prediction confidence or thresholding at intermediate blocks—becomes directly feasible with this technique.

Future Directions

Further research is warranted to:

• Incorporate dynamic routing and conditional computation informed by quality score uncertainty.
• Jointly train early exit heads with auxiliary distillation/fusion losses rather than relying on single-head pre-training, closing the residual gap for very shallow exits.
• Extend EX-FIQA to other modalities or cross-modal biometric quality assessment, e.g., iris or fingerprint image utility.

Additionally, integration with lightweight ViT variants and quantization will further reduce resource footprints for on-device quality screening.

Conclusion

EX-FIQA systematically demonstrates that intermediate ViT representations can be exploited for high-fidelity, computationally efficient FIQA. Through early exit and score fusion, it achieves strong operating points, allowing practitioners to flexibly balance performance and efficiency. The findings redefine best practices in both biometrics and ViT-based assessment, opening new avenues for adaptive, scalable, and robust quality evaluation in practical face recognition pipelines (2604.22842).