2000 character limit reached
The AffectToolbox: Affect Analysis for Everyone (2402.15195v1)
Published 23 Feb 2024 in cs.HC, cs.AI, and cs.LG
Abstract: In the field of affective computing, where research continually advances at a rapid pace, the demand for user-friendly tools has become increasingly apparent. In this paper, we present the AffectToolbox, a novel software system that aims to support researchers in developing affect-sensitive studies and prototypes. The proposed system addresses the challenges posed by existing frameworks, which often require profound programming knowledge and cater primarily to power-users or skilled developers. Aiming to facilitate ease of use, the AffectToolbox requires no programming knowledge and offers its functionality to reliably analyze the affective state of users through an accessible graphical user interface. The architecture encompasses a variety of models for emotion recognition on multiple affective channels and modalities, as well as an elaborate fusion system to merge multi-modal assessments into a unified result. The entire system is open-sourced and will be publicly available to ensure easy integration into more complex applications through a well-structured, Python-based code base - therefore marking a substantial contribution toward advancing affective computing research and fostering a more collaborative and inclusive environment within this interdisciplinary field.
- James A Russell and Albert Mehrabian “Evidence for a three-factor theory of emotions” In Journal of research in Personality 11.3 Elsevier, 1977, pp. 273–294
- Paul Ekman “An argument for basic emotions” In Cognition & Emotion 6.3 Psychology Press, 1992, pp. 169–200
- A. Mehrabian “Framework for a comprehensive description and measurement of emotional states.” In Genetic, social, and general psychology monographs 121.3, 1995, pp. 339–361
- Peter Lang, Margaret Bradley and B. Cuthbert “Motivated attention: Affect, activation, and action” In Attention and orienting: Sensory and motivational processes Psychology Press, 1997, pp. 97–135
- “Synthesizing gesture expressivity based on real sequences” In LREC 2006 Conference, 2006
- Jinni Harrigan, Robert Rosenthal and Klaus Scherer “New handbook of methods in nonverbal behavior research” Oxford University Press, 2008
- P. Ekman “The Philosophy of Deception: Lie Catching and Micro Expressions” Ed. Clancy Martin, Oxford University Press, 2009
- “PAD-based multimodal affective fusion” In International Conference on Affective Computing and Intelligent Interaction (ACII), 2009 DOI: 10.1109/ACII.2009.5349552
- “A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions” In IEEE Trans. Pattern Anal. Mach. Intell. 31.1, 2009, pp. 39–58
- “A multimodal corpus for gesture expressivity analysis” In International Conference on Language Resources and Evaluation (LREC), Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, 2010
- Nele Dael, Marcello Mortillaro and KlausR. Scherer “The Body Action and Posture Coding System (BAP): Development and Reliability” In Nonverbal Behavior 36.2 Springer US, 2012, pp. 97–121 DOI: 10.1007/s10919-012-0130-0
- “The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time” In Proceedings of the 21st ACM international conference on Multimedia, 2013, pp. 831–834
- “An Event Driven Fusion Approach for Enjoyment Recognition in Real-time” In Proceedings of the ACM International Conference on Multimedia, MM’14, Orlando, FL, USA, November 03 - 07, 2014, 2014, pp. 377–386
- “Apache flink: Stream and batch processing in a single engine” In The Bulletin of the Technical Committee on Data Engineering 38.4 Institute of ElectricalElectronics Engineers (IEEE), 2015
- Robert Neßelrath “SiAM-dp: An open development platform for massively multimodal dialogue systems in cyber-physical environments”, 2015
- Johannes Wagner, Florian Lingenfelser and Elisabeth André “Building a Robust System for Multimodal Emotion Recognition” Wiley, 2015, pp. 379–410
- Tobias Baur, Dominik Schiller and Elisabeth André “Modeling Users Social Attitude in a Conversational System” In Emotions and Personality in Personalized Services Springer, 2016, pp. 181–199
- “Asynchronous and Event-based Fusion Systems for Affect Recognition on Naturalistic Data in Comparison to Conventional Approaches” In IEEE Transactions on Affective Computing IEEE, 2016
- “Attention is all you need” In Advances in neural information processing systems 30, 2017
- Ionut Damian, Michael Dietz and Elisabeth André “The SSJ framework: Augmenting social interactions using mobile signal processing and live feedback” In Frontiers in ICT 5 Frontiers Media SA, 2018, pp. 13
- “Bert: Pre-training of deep bidirectional transformers for language understanding” In arXiv preprint arXiv:1810.04805, 2018
- “MobileNetV2: Inverted Residuals and Linear Bottlenecks” In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 4510–4520 DOI: 10.1109/CVPR.2018.00474
- “Deep learning in paralinguistic recognition tasks: Are hand-crafted features still relevant?”, 2018
- “Roberta: A robustly optimized bert pretraining approach” In arXiv preprint arXiv:1907.11692, 2019
- “Mediapipe: A framework for building perception pipelines” In arXiv preprint arXiv:1906.08172, 2019
- A. Mollahosseini, B. Hasani and M.H. Mahoor “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild” In IEEE Transactions on Affective Computing 10.1, 2019, pp. 18–31 DOI: 10.1109/TAFFC.2017.2740923
- “wav2vec 2.0: A framework for self-supervised learning of speech representations” In Advances in neural information processing systems 33, 2020, pp. 12449–12460
- “BlazePose: On-device Real-time Body Pose tracking”, 2020 arXiv:2006.10204 [cs.CV]
- “Dominance or prestige: A review of the effects of power poses and other body postures” In Social and Personality Psychology Compass 14, 2020 DOI: 10.1111/spc3.12559
- “Relevance-based data masking: a model-agnostic transfer learning approach for facial expression recognition” In Frontiers in Computer Science 2 Frontiers Media SA, 2020, pp. 6
- “Multisensor-pipeline: a lightweight, flexible, and extensible framework for building multimodal-multisensor interfaces” In Companion Publication of the 2021 International Conference on Multimodal Interaction, 2021, pp. 13–18
- “Platform for situated intelligence” In arXiv preprint arXiv:2103.15975, 2021
- Francesco Barbieri, Luis Espinosa Anke and Jose Camacho-Collados “XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond” In Proceedings of the Thirteenth Language Resources and Evaluation Conference Marseille, France: European Language Resources Association, 2022, pp. 258–266 URL: https://aclanthology.org/2022.lrec-1.27
- Nastaran Saffaryazdi, Aidin Gharibnavaz and Mark Billinghurst “Octopus Sensing: A Python library for human behavior studies” In Journal of Open Source Software 7.71, 2022, pp. 4045
- “Probing Speech Emotion Recognition Transformers for Linguistic Knowledge”, 2022
- “Robust speech recognition via large-scale weak supervision” In International Conference on Machine Learning, 2023, pp. 28492–28518 PMLR
- “Dawn of the transformer era in speech emotion recognition: closing the valence gap” In IEEE Transactions on Pattern Analysis and Machine Intelligence IEEE, 2023
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.