Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Meta-Auxiliary Learning for Micro-Expression Recognition (2404.12024v1)

Published 18 Apr 2024 in cs.CV

Abstract: Micro-expressions (MEs) are involuntary movements revealing people's hidden feelings, which has attracted numerous interests for its objectivity in emotion detection. However, despite its wide applications in various scenarios, micro-expression recognition (MER) remains a challenging problem in real life due to three reasons, including (i) data-level: lack of data and imbalanced classes, (ii) feature-level: subtle, rapid changing, and complex features of MEs, and (iii) decision-making-level: impact of individual differences. To address these issues, we propose a dual-branch meta-auxiliary learning method, called LightmanNet, for fast and robust micro-expression recognition. Specifically, LightmanNet learns general MER knowledge from limited data through a dual-branch bi-level optimization process: (i) In the first level, it obtains task-specific MER knowledge by learning in two branches, where the first branch is for learning MER features via primary MER tasks, while the other branch is for guiding the model obtain discriminative features via auxiliary tasks, i.e., image alignment between micro-expressions and macro-expressions since their resemblance in both spatial and temporal behavioral patterns. The two branches of learning jointly constrain the model of learning meaningful task-specific MER knowledge while avoiding learning noise or superficial connections between MEs and emotions that may damage its generalization ability. (ii) In the second level, LightmanNet further refines the learned task-specific knowledge, improving model generalization and efficiency. Extensive experiments on various benchmark datasets demonstrate the superior robustness and efficiency of LightmanNet.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. A survey of textual emotion detection. In 2018 8th International Conference on Computer Science and Information Technology (CSIT). IEEE, 136–142.
  2. A new meta-baseline for few-shot learning. arXiv preprint arXiv:2003.04390 1, 2 (2020), 3.
  3. Towards unbiased visual emotion recognition via causal intervention. In Proceedings of the 30th ACM International Conference on Multimedia. 60–69.
  4. Yi Dai and Ling Feng. 2021. Cross-domain few-shot micro-expression recognition incorporating action units. IEEE Access 9 (2021), 142071–142083.
  5. Interpretation and working through contemptuous facial micro-expressions benefits the patient-therapist relationship. International Journal of Environmental Research and Public Health 16, 24 (2019), 4901.
  6. Samm: A spontaneous micro-facial movement dataset. IEEE transactions on affective computing 9, 1 (2016), 116–129.
  7. A tutorial on the cross-entropy method. Annals of operations research 134 (2005), 19–67.
  8. Paul Ekman. 2009. Telling lies: Clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & Company.
  9. SelfME: Self-Supervised Motion Learning for Micro-Expression Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13834–13843.
  10. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning. PMLR, 1126–1135.
  11. OFF-ApexNet on micro-expression recognition system. Signal Processing: Image Communication 74 (2019), 129–139.
  12. Spyros Gidaris and Nikos Komodakis. 2018. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4367–4375.
  13. Meta-MMFNet: Meta-learning-based multi-model fusion network for micro-expression recognition. ACM Transactions on Multimedia Computing, Communications and Applications 20, 2 (2023), 1–20.
  14. SL Happy and Aurobinda Routray. 2017. Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Transactions on Affective Computing 10, 3 (2017), 394–406.
  15. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  16. Learning to learn using gradient descent. In Artificial Neural Networks—ICANN 2001: International Conference Vienna, Austria, August 21–25, 2001 Proceedings 11. Springer, 87–94.
  17. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence 44, 9 (2021), 5149–5169.
  18. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4700–4708.
  19. Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In Proceedings of the IEEE international conference on computer vision workshops. 1–9.
  20. Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing (Jan 2016), 564–578. https://doi.org/10.1016/j.neucom.2015.10.096
  21. Dual-stream shallow networks for facial micro-expression recognition. In 2019 IEEE international conference on image processing (ICIP). IEEE, 36–40.
  22. Sentiment Analysis and Emotion Detection with Healthcare Perspective. In Augmented Intelligence in Healthcare: A Pragmatic and Integrated Analysis. Springer, 189–204.
  23. Sparsity in dynamics of spontaneous subtle emotions: analysis and application. IEEE Transactions on Affective Computing 8, 3 (2016), 396–411.
  24. A novel graph-tcn with a graph structured representation for micro-expression recognition. In Proceedings of the 28th ACM International Conference on Multimedia. 2237–2245.
  25. Dilated context integrated network with cross-modal consensus for temporal emotion localization in videos. In Proceedings of the 30th ACM International Conference on Multimedia. 5083–5092.
  26. A multi-view spectral-spatial-temporal masked autoencoder for decoding emotions with self-supervised learning. In Proceedings of the 30th ACM International Conference on Multimedia. 6–14.
  27. Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods. IEEE transactions on affective computing 9, 4 (2017), 563–577.
  28. A spontaneous micro-expression database: Inducement, collection and baseline. In 2013 10th IEEE International Conference and Workshops on Automatic face and gesture recognition (fg). IEEE, 1–6.
  29. Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019). IEEE, 1–5.
  30. Less is more: Micro-expression recognition from video using apex frame. Signal Processing: Image Communication 62 (2018), 82–92.
  31. Meta-Auxiliary Learning for Future Depth Prediction in Videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 5756–5765.
  32. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition. IEEE Transactions on Affective Computing (Oct 2016), 299–310. https://doi.org/10.1109/taffc.2015.2485205
  33. Motion descriptors for micro-expression recognition. Signal Processing: Image Communication 67 (2018), 108–117.
  34. A review on facial micro-expressions analysis: datasets, features and metrics. arXiv preprint arXiv:1805.02397 (2018).
  35. Emo-Sensory Communication, Emo-Sensory Intelligence and Gender. Journal of Business, Communication & Technology 1, 2 (2022), 54–66.
  36. FacialCueNet: unmasking deception-an interpretable model for criminal interrogation using facial expressions. Applied Intelligence 53, 22 (2023), 27413–27427.
  37. Auxiliary learning by implicit differentiation. arXiv preprint arXiv:2007.02693 (2020).
  38. Bernhard Nebel. 2000. On the compilability and expressive power of propositional planning formalisms. Journal of Artificial Intelligence Research 12 (2000), 271–315.
  39. Micron-BERT: BERT-based Facial Micro-Expression Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1482–1492.
  40. Alex Nichol and John Schulman. 2018. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 2, 3 (2018), 4.
  41. Jicai Pan and Shangfei Wang. 2023. Progressive Visual Content Understanding Network for Image Emotion Classification. In Proceedings of the 31st ACM International Conference on Multimedia. 6034–6044.
  42. A boost in revealing subtle facial expressions: A consolidated eulerian framework. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 1–5.
  43. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7229–7238.
  44. CAS (ME)-: A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition. IEEE Transactions on Affective Computing 9, 04 (2018), 424–436.
  45. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157 (2019).
  46. Sachin Ravi and Hugo Larochelle. 2016. Optimization as a model for few-shot learning. In International conference on learning representations.
  47. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 234–241.
  48. Meta-learning with memory-augmented neural networks. In International conference on machine learning. PMLR, 1842–1850.
  49. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
  50. Prototypical networks for few-shot learning. Advances in neural information processing systems 30 (2017).
  51. Dynamic micro-expression recognition using knowledge distillation. IEEE Transactions on Affective Computing 13, 2 (2020), 1037–1043.
  52. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1199–1208.
  53. Richard Szeliski et al. 2007. Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision 2, 1 (2007), 1–104.
  54. Matching networks for one shot learning. Advances in neural information processing systems 29 (2016).
  55. Micro-Expression Recognition Based on MAML Meta-Learning Algorithm. In 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta). IEEE, 1322–1328.
  56. AMSA: adaptive multimodal learning for sentiment analysis. ACM Transactions on Multimedia Computing, Communications and Applications 19, 3s (2023), 1–21.
  57. Ease: Robust facial expression recognition via emotion ambiguity-sensitive cooperative networks. In Proceedings of the 30th ACM International Conference on Multimedia. 218–227.
  58. Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In Computer Vision–ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part I 12. Springer, 525–537.
  59. A novel micro-expression recognition approach using attention-based magnification-adaptive networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2420–2424.
  60. Bin Xia and Shangfei Wang. 2021. Micro-Expression Recognition Enhanced by Macro-Expression from Spatial-Temporal Domain.. In IJCAI. 1186–1193.
  61. Learning from macro-expression: a micro-expression recognition framework. In Proceedings of the 28th ACM International Conference on Multimedia. 2936–2944.
  62. Au-assisted graph attention convolutional network for micro-expression recognition. In Proceedings of the 28th ACM International Conference on Multimedia. 2871–2880.
  63. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PloS one 9, 1 (2014), e86041.
  64. CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, 1–7.
  65. Disentangled representation learning for multimodal emotion recognition. In Proceedings of the 30th ACM International Conference on Multimedia. 1642–1651.
  66. Feature Representation Learning with Adaptive Displacement Generation and Transformer Fusion for Micro-Expression Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22086–22095.
  67. Micro Expression Recognition by Machine Learning Based Profit Function Analysis in Intelligent Marketing of Financial Industry. In 2023 IEEE 6th Eurasian Conference on Educational Innovation (ECEI). IEEE, 188–192.
  68. Learning spatiotemporal features using 3dcnn and convolutional lstm for gesture recognition. In Proceedings of the IEEE international conference on computer vision workshops. 3120–3128.
  69. Guoying Zhao and Matti Pietikainen. 2007. Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence (Jun 2007), 915–928. https://doi.org/10.1109/tpami.2007.1110
  70. Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition. Pattern Recognition 122 (2022), 108275.
  71. A survey of micro-expression recognition. Image and Vision Computing 105 (2021), 104043.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets