2000 character limit reached
Enhancing Apparent Personality Trait Analysis with Cross-Modal Embeddings (2405.03846v1)
Published 6 May 2024 in cs.CV and cs.HC
Abstract: Automatic personality trait assessment is essential for high-quality human-machine interactions. Systems capable of human behavior analysis could be used for self-driving cars, medical research, and surveillance, among many others. We present a multimodal deep neural network with a Siamese extension for apparent personality trait prediction trained on short video recordings and exploiting modality invariant embeddings. Acoustic, visual, and textual information are utilized to reach high-performance solutions in this task. Due to the highly centralized target distribution of the analyzed dataset, the changes in the third digit are relevant. Our proposed method addresses the challenge of under-represented extreme values, achieves 0.0033 MAE average improvement, and shows a clear advantage over the baseline multimodal DNN without the introduced module.
- Analysis of emotion recognition using facial expressions, speech and multimodal information. In Proceedings of the 6th International Conference on Multimodal Interfaces, pages 205–211, 2014.
- J. M. Digman. Personality structure: Emergence of the five-factor model. Annual Review of Psychology, 41(1):417–440, 1990.
- Explaining first impressions: Modeling, recognizing, and explaining apparent personality from videos. IEEE Transactions on Affective Computing, pages 1–18, 2018.
- Modeling, recognizing, and explaining apparent personality from videos. IEEE Transactions on Affective Computing, 2020.
- The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2):190–202, 2015.
- Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462, 2010.
- Emotion recognition in speech with latent discriminative representations learning. Acta Acustica united with Acustica, 104(5):737–740, 2018.
- Emobed: Strengthening monomodal emotion recognition via training with crossmodal emotion embeddings. IEEE Transactions on Affective Computing, 2019.
- E. Hoffer and N. Ailon. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition, pages 84–92, 2015.
- First impressions: A survey on vision-based apparent personality trait analysis. IEEE Transactions on Affective Computing, pages 1–20, 2019.
- First impressions: A survey on vision-based apparent personality trait analysis. arXiv: Computer Vision and Pattern Recognition, 2018.
- Investigating audio, visual, and text fusion methods for end-to-end automatic personality prediction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 606–611, 2018.
- Learning consistent feature representation for cross-modal multimedia retrieval. IEEE Transactions on Multimedia, 17(3):370–381, 2015.
- Multimodal score fusion and decision trees for explainable automatic job candidate screening from video cvs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1–9, 2017.
- D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint, arXiv:1412.6980, 2014.
- Cr-net: A deep classification-regression network for multimodal apparent personality analysis. International Journal of Computer Vision, pages 1–18, 2020.
- Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML), pages 689–696, 2011.
- Chalearn lap 2016: First round challenge on first impressions - dataset and results. In Computer Vision – ECCV 2016 Workshops, Lecture Notes in Computer Science, pages 400–418, 2016.
- C. Raffel and D. P. W. Ellis. Feed-forward networks with attention can solve some long-term memory problems. arXiv preprint, arXiv:1512.08756, 2016.
- Learning factorized multimodal representations. In International Conference on Learning Representations (ICLR), pages 1–20, 2019.
- Adversarial cross-modal retrieval. In Proceedings of the 25th ACM international conference on Multimedia, pages 154–162, 2017.
- Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5022–5030, 2019.
- J. Willis and A. Todorov. First impressions making up your mind after a 100-ms exposure to a face. Psychological Science, 17(7):592–598, 2006.
- Low-level fusion of audio, video feature for multi-modal emotion recognition. In Proceedings of the Third International Conference on Computer Vision Theory and Applications (VISAPP), pages 145–151, 2008.
- Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional lstm modeling. In 11th Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 1–4, 2010.
- Recovering realistic texture in image super-resolution by deep spatial feature transform. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
- Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114, 2017.
- A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE transactions on pattern analysis and machine intelligence, 31(1):39–58, 2008.
- PersEmoN: A deep network for joint analysis of apparent personality, emotion and their relationship. IEEE Transactions on Affective Computing, pages 1–10, 2019.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.