Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Model Compression Techniques in Biometrics Applications: A Survey (2401.10139v1)

Published 18 Jan 2024 in cs.CV and cs.AI

Abstract: The development of deep learning algorithms has extensively empowered humanity's task automatization capacity. However, the huge improvement in the performance of these models is highly correlated with their increasing level of complexity, limiting their usefulness in human-oriented applications, which are usually deployed in resource-constrained devices. This led to the development of compression techniques that drastically reduce the computational and memory costs of deep learning models without significant performance degradation. This paper aims to systematize the current literature on this topic by presenting a comprehensive survey of model compression techniques in biometrics applications, namely quantization, knowledge distillation and pruning. We conduct a critical analysis of the comparative value of these techniques, focusing on their advantages and disadvantages and presenting suggestions for future work directions that can potentially improve the current methods. Additionally, we discuss and analyze the link between model bias and model compression, highlighting the need to direct compression research toward model fairness in future works.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (93)
  1. P. C. Neto, S. P. Oliveira, D. Montezuma, J. Fraga, A. Monteiro, L. Ribeiro, S. Gonçalves, I. M. Pinto, and J. S. Cardoso, “imil4path: A semi-supervised interpretable approach for colorectal whole-slide images,” Cancers, vol. 14, no. 10, p. 2489, 2022.
  2. T. Melo, J. Cardoso, A. Carneiro, A. Campilho, and A. M. Mendonca, “Oct image synthesis through deep generative models,” in CBMS, pp. 561–566, 2023.
  3. J. N. Kolf, F. Boutros, F. Kirchbuchner, and N. Damer, “Lightweight periocular recognition through low-bit quantization,” in IJCB, pp. 1–12, IEEE, 2022.
  4. M. Vitek, M. Bizjak, P. Peer, and V. Štruc, “Ipad: Iterative pruning with activation deviation for sclera biometrics,” Journal of King Saud University-Computer and Information Sciences, vol. 35, no. 8, p. 101630, 2023.
  5. P. C. Neto, T. Gonçalves, J. R. Pinto, W. Silva, A. F. Sequeira, A. Ross, and J. S. Cardoso, “Explainable biometrics in the age of deep learning,” arXiv preprint arXiv:2208.09500, 2022.
  6. P. Delgado-Santos, R. Tolosana, R. Guest, R. Vera-Rodriguez, and J. Fierrez, “M-gaitformer: Mobile biometric gait verification using transformers,” Engineering Applications of Artificial Intelligence, vol. 125, p. 106682, 2023.
  7. B. Kocacinar, B. Tas, F. P. Akbulut, C. Catal, and D. Mishra, “A real-time cnn-based lightweight mobile masked face recognition system,” Ieee Access, vol. 10, pp. 63496–63507, 2022.
  8. F. Boutros, N. Damer, K. Raja, R. Ramachandra, F. Kirchbuchner, and A. Kuijper, “On benchmarking iris recognition within a head-mounted display for ar/vr applications,” in IJCB, pp. 1–10, IEEE, 2020.
  9. F. Boutros, N. Damer, K. B. Raja, R. Ramachandra, F. Kirchbuchner, and A. Kuijper, “Iris and periocular biometrics for head mounted displays: Segmentation, recognition, and synthetic data generation,” Image Vis. Comput., vol. 104, p. 104007, 2020.
  10. R. Miller, N. K. Banerjee, and S. Banerjee, “Temporal effects in motion behavior for virtual reality (VR) biometrics,” in VR, pp. 563–572, IEEE, 2022.
  11. J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” Int. J. Comput. Vis., vol. 129, pp. 1789–1819, 2021.
  12. S. Ge, S. Zhao, C. Li, and J. Li, “Low-resolution face recognition in the wild via selective knowledge distillation,” IEEE Trans. Image Process., vol. 28, no. 4, pp. 2051–2062, 2018.
  13. R. Krishnamoorthi, “Quantizing deep convolutional networks for efficient inference: A whitepaper,” arXiv preprint arXiv:1806.08342, 2018.
  14. A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” in Low-Power Computer Vision, pp. 291–326, Chapman and Hall/CRC, 2022.
  15. M. Zhu and S. Gupta, “To prune, or not to prune: Exploring the efficacy of pruning for model compression,” 2018.
  16. X. Wang, “Teacher guided neural architecture search for face recognition,” in AAAI, vol. 35, pp. 2817–2825, 2021.
  17. P. C. Neto, E. Caldeira, J. S. Cardoso, and A. F. Sequeira, “Compressed models decompress race biases: What quantized models forget for fair face recognition,” in BIOSIG, pp. 1–5, IEEE, 2023.
  18. F. Boutros, N. Damer, and A. Kuijper, “Quantface: Towards lightweight face recognition by synthetic data low-bit quantization,” in ICPR, pp. 855–862, IEEE, 2022.
  19. Y. Choi, J. Choi, M. El-Khamy, and J. Lee, “Data-free network quantization with adversarial knowledge distillation,” in CVPR Workshops, pp. 710–711, 2020.
  20. A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen, “Incremental network quantization: Towards lossless CNNs with low-precision weights,” in ICLR, 2017.
  21. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in CVPR, pp. 2704–2713, 2018.
  22. H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” in ICLR, 2017.
  23. P. Luo, Z. Zhu, Z. Liu, X. Wang, and X. Tang, “Face model compression by distilling knowledge from neurons,” in AAAI, vol. 30, 2016.
  24. A. Polyak and L. Wolf, “Channel-level acceleration of deep face representations,” IEEE Access, vol. 3, pp. 2163–2175, 2015.
  25. P. C. Neto, A. F. Sequeira, and J. S. Cardoso, “Myope models-are face presentation attack detection models short-sighted?,” in WACV Workshops, pp. 390–399, 2022.
  26. F. Boutros, P. Siebke, M. Klemt, N. Damer, F. Kirchbuchner, and A. Kuijper, “Pocketnet: Extreme lightweight face recognition network using neural architecture search and multistep knowledge distillation,” IEEE Access, vol. 10, pp. 46823–46833, 2022.
  27. L. Wang, X. Dong, Y. Wang, L. Liu, W. An, and Y. Guo, “Learnable lookup table for neural network quantization,” in IEEE/CVF CVPR, pp. 12423–12433, 2022.
  28. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in NeurIPS, pp. 8024–8035, 2019.
  29. D. Miyashita, E. H. Lee, and B. Murmann, “Convolutional neural networks using logarithmic data representation,” arXiv preprint arXiv:1603.01025, 2016.
  30. D. Zhang, J. Yang, D. Ye, and G. Hua, “Lq-nets: Learned quantization for highly accurate and compact deep neural networks,” in ECCV, pp. 365–382, 2018.
  31. Y. Jeon, C. Lee, E. Cho, and Y. Ro, “Mr. biq: Post-training non-uniform quantization based on minimizing the reconstruction error,” in CVPR, pp. 12329–12338, 2022.
  32. V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in ICML, pp. 807–814, 2010.
  33. S. Bunda, L. Spreeuwers, and C. Zeinstra, “Sub-byte quantization of mobile face recognition convolutional neural networks,” in BIOSIG, pp. 1–5, IEEE, 2022.
  34. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
  35. M. H. Aslam, M. O. Zeeshan, M. Pedersoli, A. L. Koerich, S. Bacon, and E. Granger, “Privileged knowledge distillation for dimensional emotion recognition in the wild,” in CVPRW, pp. 3337–3346, 2023.
  36. R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. I. Morariu, X. Han, M. Gao, C.-Y. Lin, and L. S. Davis, “Nisp: Pruning networks using neuron importance score propagation,” in CVPR, pp. 9194–9203, 2018.
  37. M. Huber, F. Boutros, F. Kirchbuchner, and N. Damer, “Mask-invariant face recognition through template-level knowledge distillation,” in FG, pp. 1–8, IEEE, 2021.
  38. S. Ge, S. Zhao, C. Li, Y. Zhang, and J. Li, “Efficient low-resolution face recognition via bridge distillation,” IEEE Trans. Image Process., vol. 29, pp. 6898–6908, 2020.
  39. F. Boutros, O. Kaehm, M. Fang, F. Kirchbuchner, N. Damer, and A. Kuijper, “Low-resolution iris recognition via knowledge transfer,” in BIOSIG, pp. 1–5, IEEE, 2022.
  40. W. Zhao, X. Zhu, K. Guo, X. Zhang, and Z. Lei, “Grouped knowledge distillation for deep face recognition,” pp. 3615–3623, 2023.
  41. J. N. Kolf, J. Elliesen, F. Boutros, H. Proença, and N. Damer, “Syper: Synthetic periocular data for quantized light-weight recognition in the nir and visible domains,” Image Vis. Comput., vol. 135, p. 104692, 2023.
  42. L. Wang and K.-J. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 6, pp. 3048–3068, 2021.
  43. C. N. Duong, K. Luu, K. G. Quach, and N. Le, “Shrinkteanet: Million-scale lightweight face recognition via shrinking teacher-student networks,” arXiv preprint arXiv:1905.10620, 2019.
  44. X. Wu, R. He, Y. Hu, and Z. Sun, “Learning an evolutionary embedding via massive knowledge distillation,” International Journal of Computer Vision, vol. 128, pp. 2089–2106, 2020.
  45. J. Liu, H. Qin, Y. Wu, J. Guo, D. Liang, and K. Xu, “Coupleface: Relation matters for face recognition distillation,” in ECCV, pp. 683–700, Springer, 2022.
  46. J. Li, Z. Guo, H. Li, S. Han, J.-w. Baek, M. Yang, R. Yang, and S. Suh, “Rethinking feature-based knowledge distillation for face recognition,” in CVPR, pp. 20156–20165, 2023.
  47. F. Boutros, N. Damer, K. Raja, F. Kirchbuchner, and A. Kuijper, “Template-driven knowledge distillation for compact and accurate periocular biometrics deep-learning models,” Sensors, vol. 22, no. 5, p. 1921, 2022.
  48. Y. Huang, J. Wu, X. Xu, and S. Ding, “Evaluation-oriented knowledge distillation for deep face recognition,” in CVPR, pp. 18740–18749, 2022.
  49. E. Caldeira, P. C. Neto, T. Goncalves, N. Damer, A. F. Sequeira, and J. S. Cardoso, “Unveiling the two-faced truth: Disentangling morphed identities for face morphing detection,” in EUSIPCO, 2023.
  50. F. Boutros, N. Damer, M. Fang, K. Raja, F. Kirchbuchner, and A. Kuijper, “Compact models for periocular verification through knowledge distillation,” in BIOSIG, pp. 1–5, IEEE, 2020.
  51. G. Chechik, I. Meilijson, and E. Ruppin, “Synaptic pruning in development: a computational account,” Neural computation, vol. 10, no. 7, pp. 1759–1777, 1998.
  52. W. Zukerman and A. Purcell, “Brain’s synaptic pruning continues into your 20s,” 2011.
  53. L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov, “Knowledge distillation: A good teacher is patient and consistent,” in CVPR, pp. 10925–10934, 2022.
  54. N. Lee, T. Ajanthan, and P. Torr, “SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY,” in ICLR, 2019.
  55. X. Lin, S. Kim, and J. Joo, “Fairgrape: Fairness-aware gradient pruning method for face attribute classification,” in ECCV, pp. 414–432, Springer, 2022.
  56. J. Liu, B. Zhuang, Z. Zhuang, Y. Guo, J. Huang, J. Zhu, and M. Tan, “Discrimination-aware network pruning for deep model compression,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 8, pp. 4035–4051, 2021.
  57. S. Chen, Y. Liu, X. Gao, and Z. Han, “Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices,” in CCBR, vol. 10996 of Lecture Notes in Computer Science, pp. 428–438, Springer, 2018.
  58. J. N. Kolf, J. Elliesen, F. Boutros, and N. Damer, “How colorful should faces be? harmonizing color and model quantization for resource-restricted face recognition,” in IJCB, pp. 1–10, IEEE (to appear), 2023.
  59. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
  60. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NEURIPS, vol. 30, 2017.
  61. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in ICLR, 2021.
  62. M. Ji, G. Peng, S. Li, F. Cheng, Z. Chen, Z. Li, and H. Du, “A neural network compression method based on knowledge-distillation and parameter quantization for the bearing fault diagnosis,” Applied Soft Computing, vol. 127, p. 109331, 2022.
  63. B. Zoph and Q. Le, “Neural architecture search with reinforcement learning,” in ICLR, 2017.
  64. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, pp. 770–778, 2016.
  65. D. Li, G. Wen, X. Li, and X. Cai, “Graph-based dynamic ensemble pruning for facial expression recognition,” Applied Intelligence, vol. 49, pp. 3188–3206, 2019.
  66. J. Xu, S. Li, A. Deng, M. Xiong, J. Wu, J. Wu, S. Ding, and B. Hooi, “Probabilistic knowledge distillation of face ensembles,” in CVPR, pp. 3489–3498, 2023.
  67. O. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in BMVC, British Machine Vision Association, 2015.
  68. F. Alonso-Fernandez, K. Hernandez-Diaz, J. M. B. Rubio, and J. Bigün, “Squeezerfacenet: Reducing a small face recognition CNN even more via filter pruning,” in IWAIPR, vol. 14335 of Lecture Notes in Computer Science, pp. 349–361, Springer, 2023.
  69. J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in ICLR, 2019.
  70. S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh, “Improved knowledge distillation via teacher assistant,” in AAAI, vol. 34, pp. 5191–5198, 2020.
  71. C. Blakeney, N. Huish, Y. Yan, and Z. Zong, “Simon says: Evaluating and mitigating bias in pruned neural networks with knowledge distillation,” arXiv preprint arXiv:2106.07849, 2021.
  72. J. P. Robinson, G. Livitz, Y. Henon, C. Qin, Y. Fu, and S. Timoner, “Face recognition: too bias, or not too bias?,” in CVPRW, pp. 0–1, 2020.
  73. V. Albiero and K. W. Bowyer, “Is face recognition sexist? no, gendered hairstyles and biology are,” in BMVC, 2020.
  74. V. Albiero, K. Zhang, and K. W. Bowyer, “How does gender balance in training data affect face recognition accuracy?,” in IJCB, pp. 1–10, IEEE, 2020.
  75. B. Fu and N. Damer, “Towards explaining demographic bias through the eyes of face recognition models,” in IJCB, pp. 1–10, IEEE, 2022.
  76. D. Deb, N. Nain, and A. K. Jain, “Longitudinal study of child face recognition,” in ICB, pp. 225–232, IEEE, 2018.
  77. C. Huang, Y. Li, C. C. Loy, and X. Tang, “Deep imbalanced learning for face recognition and attribute prediction,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 11, pp. 2781–2794, 2020.
  78. P. Terhörst, J. N. Kolf, M. Huber, F. Kirchbuchner, N. Damer, A. M. Moreno, J. Fierrez, and A. Kuijper, “A comprehensive study on face recognition biases beyond demographics,” IEEE Transactions on Technology and Society, vol. 3, no. 1, pp. 16–30, 2022.
  79. M. Fang, W. Yang, A. Kuijper, V. Struc, and N. Damer, “Fairness in face presentation attack detection,” Pattern Recognition, vol. 147, p. 110002, 2024.
  80. M. Fang, N. Damer, F. Kirchbuchner, and A. Kuijper, “Demographic bias in presentation attack detection of iris recognition systems,” in EUSIPCO, pp. 835–839, IEEE, 2020.
  81. P. Terhörst, J. N. Kolf, N. Damer, F. Kirchbuchner, and A. Kuijper, “Face quality estimation and its correlation to demographic and non-demographic bias in face recognition,” in IJCB, pp. 1–11, IEEE, 2020.
  82. S. Mittal, K. Thakral, P. Majumdar, M. Vatsa, and R. Singh, “Are face detection models biased?,” in FG, pp. 1–7, IEEE, 2023.
  83. M. Huber, M. Fang, F. Boutros, and N. Damer, “Are explainability tools gender biased? A case study on face presentation attack detection,” in EUSIPCO, pp. 945–949, IEEE, 2023.
  84. M. Wang, W. Deng, J. Hu, X. Tao, and Y. Huang, “Racial faces in the wild: Reducing racial bias by information maximization adaptation network,” in ICCV, pp. 692–702, 2019.
  85. K. Karkkainen and J. Joo, “Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation,” in WACV, pp. 1548–1558, 2021.
  86. M. Wang, Y. Zhang, and W. Deng, “Meta balanced network for fair face recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 11, pp. 8433–8448, 2021.
  87. T. Xu, J. White, S. Kalkan, and H. Gunes, “Investigating bias and fairness in facial expression recognition,” in ECCVW, pp. 506–523, Springer, 2020.
  88. S. Stoychev and H. Gunes, “The effect of model compression on fairness in facial expression recognition,” vol. 13646, pp. 121–138, 2022.
  89. B. Liu, S. Zhang, G. Song, H. You, and Y. Liu, “Rectifying the data bias in knowledge distillation,” in ICCVW, pp. 1477–1486, IEEE, 2021.
  90. E. Iofinova, A. Peste, and D. Alistarh, “Bias in pruned vision models: In-depth analysis and countermeasures,” in CVPR, pp. 24364–24373, IEEE, 2023.
  91. J. Ahn, H. Lee, J. Kim, and A. Oh, “Why knowledge distillation amplifies gender bias and how to mitigate from the perspective of DistilBERT,” in Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 266–272, Association for Computational Linguistics, 2022.
  92. G. Gonçalves and E. Strubell, “Understanding the effect of model compression on social bias in large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2663–2675, Association for Computational Linguistics, 2023.
  93. M. Paganini, “Prune responsibly,” arXiv preprint arXiv:2009.09936, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Eduarda Caldeira (7 papers)
  2. Pedro C. Neto (21 papers)
  3. Marco Huber (25 papers)
  4. Naser Damer (96 papers)
  5. Ana F. Sequeira (19 papers)
Citations (7)
View on Semantic Scholar

Summary

Overview of Model Compression Techniques in Biometrics

Introduction to Model Compression in Biometrics

Modern deep learning models have significantly advanced automation capabilities, but their increasing complexity has raised several issues, including high memory demands and computational costs. In biometrics applications such as facial or iris recognition, these limitations are more pronounced because these systems are often deployed on resource-constrained edge devices. This has driven research towards model compression techniques, which aim to reduce the resource footprint of deep learning models without substantially sacrificing performance. Model compression is of particular interest in biometric applications due to their widespread use in resource-constrained environments.

Advantages and Challenges

Within the field of model compression, three primary techniques are often utilized: quantization, knowledge distillation (KD), and pruning. Each technique has its advantages and peculiarities:

  • Quantization involves decreasing the precision of the numerical variables in the model, allowing for faster and more memory-efficient operations.
  • Knowledge Distillation is the process where a smaller "student" model learns from a larger "teacher" model to perform similarly without the bulk of the original model.
  • Pruning refers to the technique of eliminating unnecessary weights or neurons in a neural network, resulting in a sparser but still effective model.

However, while these techniques can yield more compact models with little loss in accuracy, their individual application and results are nuanced. They each carry some challenges, especially when the goal is to maintain fairness and prevent increased bias against certain sub-groups in the processed data.

Fairer Compression Research Directions

The paper highlights that these compression methods sometimes inadvertently magnify biases present in the original models. For instance, after model compression, biometric systems can yield even more biased results against specific demographic groups. This situation underscores the need for research focused on understanding and mitigating these unintended consequences.

To tackle this, the paper suggests several approaches, including the development of fairness-aware compression techniques, the use of balanced datasets, and enhancing algorithms with the capability to recognize and minimize bias during the compression process.

Summary and Future Work

The survey conducted in the paper stands as the first comprehensive review of compression techniques in biometric applications, analyzing not only the quantitative effects of compression on model performance but also highlighting the qualitative impacts on model bias and fairness. The paper underscores the importance of future research that aims to improve current methods, suggesting a pathway towards compression techniques that enhance performance while also considering the crucial aspect of fairness.

By calling for more nuanced studies and technically mindful developments in the field, this paper contributes to paving the way for advanced and equitable biometric systems that are both resource-efficient and socially responsible.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown