Crowd Emotion Recognition (CER)

• Crowd Emotion Recognition (CER) is the study of analyzing and inferring collective emotional states from diverse signals such as video, audio, and social media.
• It encompasses multiple formulations including veracity detection, group-level classification in images/videos, and audio-based emotion recognition with distinct label spaces.
• CER supports operational applications in emergency management, surveillance, and consumer insights while addressing challenges like data heterogeneity and privacy concerns.

Crowd Emotion Recognition (CER) denotes the recognition, quantification, and interpretation of emotional states expressed by individuals in a crowd, or the inference of a collective affective state from multimodal evidence. In the current literature, CER is not a single canonical task. It has been operationalized as veracity detection of emotional expressions from aggregated human judgments, emotion-driven crowd-type inference in mass gatherings from social media, group-level affect classification in images and videos, emotion-as-attribute representations for abnormality detection, rule-based personality and emotion inference from pedestrian trajectories, and audio-based classification of collective affect in public events (Qin et al., 2018, Ngo et al., 2016, Quach et al., 2018, Rabiee et al., 2016, Favaretto et al., 2021, Phukan et al., 19 Sep 2025). Accordingly, CER has no single universal label space: published formulations include genuine versus acted, Positive/Neutral/Negative, Approval/Disapproval/Neutral, four-emotion and six-emotion taxonomies, and mappings from emotion distributions to crowd-type ontologies.

1. Conceptual scope and problem formulations

A first major formulation treats CER as veracity detection of emotional expressions. In that setting, the input is a short video of a person displaying an emotion, and the task is to decide whether the expression is genuine or acted/posed. The reported emotions are anger, smile, fear, and happiness; genuine clips come from reality TV shows and acted clips from movies, with binary labels defined by source (Qin et al., 2018).

A second formulation treats CER as crowd monitoring in mass gatherings. Here, CER is the recognition, quantification, and interpretation of emotional states expressed by individuals in a crowd via social media, followed by aggregation of those signals to infer the crowd’s prevailing emotional distribution and the crowd type or types present. The operational target is emergency management, particularly timely response and effective resource allocation, and the framework couples four basic emotions—anger, fear, happiness, and sadness—to Berlonghi’s eleven crowd types (Ngo et al., 2016).

A third formulation centers on group-level emotion recognition in visual media. In crowd videos, the task can be defined simultaneously at three levels: individual faces with eight discrete categories, group regions within a frame with positive/negative/neutral labels, and whole frame or video with the same three coarse labels. In group images, the task is typically a single global image-level classification into Positive, Neutral, or Negative (Quach et al., 2018, Garg, 2019).

A fourth formulation uses CER as an intermediate semantic representation rather than as the final prediction target. In abnormality detection, crowd emotions such as Angry, Happy, Excited, Scared, Sad, and Neutral serve as mid-level attributes that bridge low-level motion descriptors and high-level behavior classes such as Panic, Fight, Obstacle, Congestion, and Neutral (Rabiee et al., 2016). A related but distinct formulation infers OCEAN personality traits from motion and social-interaction cues, then maps those traits to OCC emotions—fear, happiness, sadness, and anger—at individual and group levels (Favaretto et al., 2021).

A fifth formulation is audio-based collective emotion recognition. In this setting, CER is defined over short audio segments extracted from crowd recordings at stadiums, concerts, rallies, and public gatherings, with three clip-level classes: Approval, Disapproval, and Neutral (Phukan et al., 19 Sep 2025).

These formulations make clear that CER spans both perceptual recognition and operational inference. A common misconception is that CER is synonymous with facial-expression recognition. The literature contradicts that reduction by including social media text, trajectory and proxemic cues, crowd judgments, scene context, and audio (Ngo et al., 2016, Favaretto et al., 2021, Phukan et al., 19 Sep 2025).

2. Data modalities, annotations, and benchmark structures

CER datasets differ sharply in signal type, annotation strategy, and output semantics. Some works rely on human binary judgments over short videos, some on public tweets segmented into short intervals, some on dense face detections in crowd videos, and some on audio blocks extracted from longer recordings. This heterogeneity is one of the defining characteristics of the field.

Formulation	Primary signal	Labels or outputs
Veracity detection	80 YouTube videos, 117 participant judgments	genuine/acted for anger, smile, fear, happiness
Mass-gathering monitoring	Twitter data in 15-minute intervals	anger/fear/happiness/sadness rates mapped to crowd types
Crowd video recognition	GECV with 627 videos, 438k group regions, 900k faces	8 face emotions; group/video positive/neutral/negative
Abnormality detection	31 outdoor video sequences, about 44,000 clips	6 crowd emotions plus 5 behavior classes
Trajectory-based inference	pedestrian video sequences from public scenes and controlled experiments	OCEAN traits and OCC emotions
Group image emotion	GAF 3.0 images with face crops and scene descriptors	Positive/Neutral/Negative
Audio CER	69 clips, 9,515 1-second blocks	Approval/Disapproval/Neutral

In the veracity-detection setting, stimuli are videos sourced from YouTube, with contextual backgrounds maintained rather than cropped to faces. The dataset comprises 80 videos, organized as 4 emotions × 20 videos per emotion, and 117 participants each provide a binary judgment per video. The experimental protocol uses leave-one-video-out cross-validation per emotion and in a combined-emotion setting (Qin et al., 2018).

In social-media CER for emergency management, Twitter functions as a “soft sensor.” Tweets are collected by event hashtags and geofencing, tokenized into uni-grams, and analyzed in fixed intervals $t$, which are 15 minutes in the evaluation. In the reported boxing-match case study, 33,935 tweets were collected over one week, and 28,698 tweets were labeled with one of four emotions using the NRC Hashtag Emotion Lexicon (Ngo et al., 2016).

For visual deep learning on crowd videos, the GECV dataset provides three levels of labels. GEVC-SingleImg contains 900k aligned face images for eight emotion categories; GEVC-GroupImg contains 438k group regions from about 140k frames with positive/neutral/negative labels; GEVC-GroupVid contains 627 videos, each about 10–20 seconds, with positive, negative, or neutral labels at the video level (Quach et al., 2018). Group-image CER in GAF 3.0 uses 9,815 training images and 4,346 validation images, each annotated with a single global label in $$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$ (Garg, 2019).

In abnormality detection, the reported dataset contains 31 outdoor video sequences and about 44,000 normal and abnormal video clips, with frame-level emotion annotations and behavior labels. Inter-annotator agreement for the six-emotion labeling protocol is reported as 92%, with Cohen’s $\kappa = 0.81$ (Rabiee et al., 2016). In the trajectory-based rule system, videos come from spontaneous public scenes in Brazil and Germany and from controlled “Fundamental Diagram” experiments with $N = 15, 25, 34$ individuals walking single-file in a 17.3 m corridor of width 0.8 m (Favaretto et al., 2021).

Audio CER uses the only openly accessible audio CER dataset cited in the study, originally curated from sports, concerts, political rallies, and public gatherings. Audio is resampled to 16 kHz and segmented into 1-second blocks with 0.25-second overlap, yielding 9,515 labeled blocks from 69 clips; the 1-second blocks are then further split into 500 ms and 250 ms segments after silence removal (Phukan et al., 19 Sep 2025).

3. Methodological families

The literature contains several largely disjoint methodological families.

Crowd-answer aggregation learns how to combine judgments from multiple people. The simplest estimator is majority vote,

$$\hat{y} = \mathbb{1}\left[\sum_{i=1}^{N} x_i \ge N/2\right],$$

where $x_i \in \{0,1\}$. More expressive variants learn participant-specific weights through logistic aggregation,

$$p(y=1|x) = \sigma\left(\sum_{i=1}^{N} w_i x_i + b\right),$$

or through a one-hidden-layer MLP with architecture $117 \rightarrow 10 \rightarrow 1$, sigmoid activations, cross-entropy loss, SGD with learning rate 0.01, and 5000 epochs per cross-validation fold. A distinctive property of this approach is that the learned weights may be positive for reliable participants and negative for systematically unreliable participants, effectively flipping consistent error into useful signal (Qin et al., 2018).

Lexicon-driven and rule-based CER from social media represents each tweet as a Bag-of-Words document and assigns an emotion weight vector using association scoring from the NRC Hashtag Emotion Lexicon. Each tweet is labeled with its dominant emotion, and crowd-level emotion is represented as emotion rates over short time windows. These rates are thresholded into low/high levels and mapped by explicit logic rules to Berlonghi crowd types, for example a high fear level implying Escaping or Dense/Suffocating crowd types (Ngo et al., 2016).

Attribute-based emotion representations for behavior understanding treat emotion as a semantic intermediate layer between motion features and crowd-behavior labels. Low-level descriptors are Dense Trajectories with HOG, HOF, MBH, and Trajectories on 32×32 pixel, 15-frame space-time patches, followed by Bag-of-Words with $d = 1000$ visual words per descriptor. Per-emotion binary linear SVMs produce confidence scores, combined as

$\phi(x) = [s_1(x), s_2(x), \dots, s_K(x)],$

which then feed a multi-class linear SVM for behavior classification. A latent-emotion variant augments raw features with per-emotion and pairwise emotion-co-occurrence terms and is optimized by coordinate descent with belief propagation for latent inference (Rabiee et al., 2016).

Scene-level visual motion methods deliberately avoid identity- and part-specific cues. The edge-based grid super-imposition method applies Canny edge detection, overlays a 20×20 grid, tracks 200 static and 200 velocity values across windows of $$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$0 frames, reduces the 400-dimensional representation to 23 features by Best-First Search, and trains a multi-class RBF-SVM with $$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$1 (Patwardhan, 2016). This design is explicitly view and occlusion independent because features are tied to grid cells and edge activity rather than to faces or body parts.

Deep fusion for group-level visual CER combines per-face encoding, grouping, and temporal modeling. RetinaFace detects faces and 5 landmarks; aligned 112×112 crops are passed to EmoNet, a compact CNN with depthwise separable convolutions and MobileNetV2-style bottlenecks. Per-face features are fused by Non-Volume Preserving Fusion (NVPF), a RealNVP-inspired masked-coupling model with class-conditional Gaussian priors,

$$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$2

Temporal NVPF (TNVPF) then propagates frame-level fused features with a GRU-like recurrence and a hidden size of 4096 (Quach et al., 2018).

Hybrid face-plus-scene pipelines integrate bottom-up and top-down evidence. One representative system detects faces with Dlib HOG and an MTCNN fallback, classifies cropped 64×64 faces with an ensemble of CNNs, averages per-face probabilities to produce a group-level facial estimate, extracts 809 scene descriptors using Google Cloud Vision Label Detection, and performs final inference with a Bayesian network whose posterior factorization is

$$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$3

Here $$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$4 is global emotion, $$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$5 are descriptor nodes, and $$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$6 is the CNN evidence node calibrated from the CNN confusion matrix (Garg, 2019).

Training-free trajectory-based CER uses tracked pedestrian motion and proxemics. Individuals are rectified to world coordinates via planar homography, then described by speed, heading, isolation, socialization, and collectivity. These features are mapped to 25 NEO PI-R items, aggregated to OCEAN traits, thresholded at 0.5 for positive/negative polarity, and converted to OCC emotion scores with weights in $$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$7 for fear, happiness, sadness, and anger (Favaretto et al., 2021).

Audio CER with frozen speech foundation models extracts mean-pooled representations from multilingual or monolingual SFMs and trains downstream classifiers such as SVM, Random Forest, FCN, or 1D-CNN. Polyglot encoders in the study include XLS-R, Whisper, and MMS; downstream CNNs use three 1D convolution layers with 32/64/128 filters, kernel size 3, ReLU, batch normalization, max-pooling, Dense(512), Dense(128), dropout $$\{ \text{Positive}, \text{Neutral}, \text{Negative} \}$$8, and a final three-way softmax, trained with Adam at learning rate 0.001 for 50 epochs with early stopping (Phukan et al., 19 Sep 2025).

4. Empirical findings and reported performance

Reported CER numbers are highly task-specific. Accuracy on binary veracity detection, mean accuracy on group-video recognition, macro-F1 on imbalanced audio classification, and incident-aligned threshold crossings in emergency monitoring should not be interpreted as directly comparable indicators of a single underlying difficulty.

Setting	Reported result	Metric or note
Veracity detection from crowd judgments	63% individual; about 80% majority vote; about 92–94% elite majority; 99.69% neural aggregation; 100% with combined-emotion training in repeated runs	accuracy
Social-media emergency monitoring	fear exceeded the threshold at 10:45 PM and 11:00 PM; inferred Group 4 crowd type	aligned with stampede onset
Deep crowd-video fusion	FeC: 76.12% mAC on EmotiW 2018 validation; 77.02% mAC on GECV-GroupImg validation; TNVPF: 70.97% mAC on GECV-GroupVid test	mAC, with UAR and F1 also reported
Emotion attributes for abnormality detection	38.71% low-level Dense Trajectory baseline; 43.64% emotion-based; 43.90% latent-emotion; 83.79% emotion-aware with GT emotions	average accuracy
Edge-grid CER	70.9% overall; happy 76.6%, disgust 72.9%, surprise 72.0%, sad 70.4%, neutral 69.1%, anger 67.8%, fear 67.2%	overall accuracy and per-class recall
Hybrid group-image CER	64.26% CNN-only; 61.04% BN-only; 65.27% fusion	validation accuracy
Audio CER	MMS+CNN: 99.11% accuracy / 96.85% macro-F1 at 1 s; 99.20% / 96.65% at 250 ms	5-fold averages

In the crowd-veracity study, a central empirical result is that neural aggregation outperforms both unweighted majority voting and elite-only voting. The paper also reports sample efficiency: accuracy increases with the number of participants, plateau-like behavior starts around 20 participants, and about 30 randomly selected participants are sufficient for the neural network aggregator to reach about 99% accuracy across emotions (Qin et al., 2018).

The emergency-management framework is evaluated on the Floyd Mayweather versus Marcos Maidana boxing event at MGM Grand, where a stampede occurred around 10:45 PM. During the one-hour window encompassing the incident, only fear exceeded the threshold at 10:45 PM and 11:00 PM, and rule-based inference placed the crowd under Group 4—Escaping and Dense/Suffocating. Detection in the 10:45 PM segment coincided with the time the police received the emergency call (Ngo et al., 2016).

The deep visual fusion literature reports consistent improvements for learned fusion over averaging or concatenation. On EmotiW 2018 validation, FeA achieves 73.7% mAC, FeB 72.88%, and FeC with NVPF 76.12%. On GECV-GroupVid test, FeC + RNN yields 59.68% mAC, FeC + LSTM 69.35%, and TNVPF 70.97%, with particular improvement on the negative class (Quach et al., 2018).

In abnormality detection, the attribute-based argument is supported by the gap between low-level visual features and emotion-mediated representations. With Dense Trajectories, the low-level baseline reaches 38.71% average accuracy, the emotion-based representation reaches 43.64%, and the latent-emotion model reaches 43.90%. The upper bound with ground-truth emotions is 83.79%, while direct emotion recognition from Dense Trajectories is only 34.13% average accuracy (Rabiee et al., 2016). This indicates that the semantic idea is strong, but the bottleneck lies in emotion prediction quality.

For spontaneous group reactions to sports events, the edge-grid approach reports 70.9% overall accuracy, with the best average performance at edge threshold 0.4 and grid size 20×20. Happy versus surprise is a recurrent confusion, and anger degrades when participants leave the frame or mask their faces with caps or pillows (Patwardhan, 2016). In static crowd images, the hybrid CNN-plus-Bayesian-network pipeline reaches 65.27% validation accuracy, compared with 64.26% for the facial ensemble alone and 61.04% for the Bayesian scene module alone (Garg, 2019).

In audio CER, the headline result is the consistent superiority of polyglot SFMs over monolingual and speaker-recognition baselines. With the CNN downstream classifier, MMS reaches 99.11% accuracy and 96.85% macro-F1 at 1 second, 99.19% and 96.66% at 500 ms, and 99.20% and 96.65% at 250 ms. Confusion matrices show that Neutral is easiest, while residual ambiguity remains between Approval and Disapproval (Phukan et al., 19 Sep 2025).

5. Operational uses and application domains

Emergency management is the most explicitly operational CER application in the surveyed literature. Social-media CER is proposed for rapid detection of hazardous crowd states and informed resource allocation during mass gatherings. When fear or anger rises above threshold, responders can prioritize medical teams, security resources, evacuation management, and crowd-control measures; the framework is designed for the “during the event” phase, where real-time situational awareness is critical (Ngo et al., 2016).

Security, forensics, and veracity assessment form another application cluster. The crowd-veracity paradigm is proposed for lie detection, border/control interviews, and other high-stakes interactions in which the genuineness of emotional display matters. The same paper also suggests extension to fake news prevention and broader human–AI collaboration, where neural aggregation of human judgments can elevate group performance and support personalized feedback to annotators (Qin et al., 2018).

Surveillance and abnormality detection constitute a third cluster. Emotion attributes have been used to improve prediction of panic, fight, congestion, and obstacle-related abnormalities in outdoor scenes, while rule-based trajectory systems target surveillance, public spaces, and controlled experiments where faces may be unavailable or unreliable (Rabiee et al., 2016, Favaretto et al., 2021). Visual crowd-video systems further support mixed-affect scene analysis by allowing multiple group regions within a frame, each with its own prediction (Quach et al., 2018).

Consumer and media-facing applications also appear repeatedly. Group-image and group-video CER are motivated by social networking sites, audience understanding, advertising, customer interaction, retail experiences, and HCI. Audio CER extends this operational space to live entertainment analytics and public-event monitoring, where short-latency recognition of approval, disapproval, or neutral background state is useful (Garg, 2019, Phukan et al., 19 Sep 2025). The edge-grid approach specifically targets spontaneous reactions while watching sports events and is framed as suitable for near real-time processing on commodity CPUs (Patwardhan, 2016).

Taken together, these use cases show that CER frequently functions less as an abstract emotion benchmark and more as a decision-support layer. This suggests that practical CER systems are usually evaluated by their usefulness inside a downstream workflow—incident detection, crowd-type inference, anomaly recognition, or engagement monitoring—rather than by emotion recognition alone.

6. Limitations, controversies, and research directions

A first limitation is problem fragmentation. CER results are often numerically impressive within a given benchmark, but the benchmarks themselves differ in modality, granularity, and semantics. A binary genuine/acted task on 80 videos, a positive/neutral/negative group-video task, a four-emotion crowd-type inference problem, and a three-class audio problem do not measure the same capability (Qin et al., 2018, Ngo et al., 2016, Quach et al., 2018, Phukan et al., 19 Sep 2025). This suggests that CER should be understood as a family of related problems rather than as a single leaderboard.

A second limitation is dataset scope and labeling uncertainty. The veracity-detection study explicitly notes that 80 videos are substantial for a human study but small for machine learning, and that labels are inferred by source—reality TV for genuine, movies for acted—which can introduce bias or labeling noise (Qin et al., 2018). The audio study uses only 69 source clips and a highly imbalanced class distribution dominated by Neutral (Phukan et al., 19 Sep 2025). The group-image pipeline trains face classifiers on crops whose labels are inherited from image-level annotations, thereby introducing label noise (Garg, 2019). In abnormality detection, the strong gap between 83.79% with ground-truth emotions and 43.64% with predicted emotions shows that the value of emotion attributes depends heavily on the quality of the CER front end (Rabiee et al., 2016).

A third limitation is modality dependence and domain shift. Social-media CER depends on language resources and threshold calibration; the reported evaluation is restricted to English tweets because the lexicon is English, and thresholds may vary across emotions, contexts, countries, and time periods (Ngo et al., 2016). Trajectory-based CER depends on tracking quality, homography correctness, and empirical mappings from motion to OCEAN and from OCEAN to OCC; the paper emphasizes that no direct in-situ emotion labels are available (Favaretto et al., 2021). Visual CER remains vulnerable to missed face detections, extreme poses, very small faces, occlusion, and order sensitivity in feature stacking (Quach et al., 2018). Edge-based scene methods are sensitive to contrast, scene clutter, and camera motion, even though they are view and occlusion independent under stationary-camera conditions (Patwardhan, 2016).

A fourth issue is ethics and privacy. Participant-specific weighting in crowd-judgment aggregation requires tracking reliability over time, and the authors note that participants may object to being labeled as “poor performers” (Qin et al., 2018). Public-space CER raises privacy, consent, and demographic-bias concerns in visual systems (Quach et al., 2018). Audio CER raises surveillance and chilling-effect risks, even when sourced from public events (Phukan et al., 19 Sep 2025). Social-media CER avoids certain physical-sensor deployment burdens, but representativeness bias, coordinated misinformation, bot activity, and geolocation inaccuracies remain pertinent concerns (Ngo et al., 2016).

The main research directions are correspondingly diverse. Reported proposals include adaptive weighting and active selection of annotators, multimodal fusion with audio or physiology, online learning, and robustness to adversarial annotators (Qin et al., 2018); integration of mobile activity recognition with social-media analysis for better distinction among crowd types (Ngo et al., 2016); graph-based modeling, permutation-invariant set functions, uncertainty estimation, person tracking, and multi-camera fusion in crowd videos (Quach et al., 2018); fine-tuning or adapter-based specialization of polyglot SFMs, clip-level temporal aggregation, fairness audits with language and region metadata, and profiling of real-time latency for audio CER (Phukan et al., 19 Sep 2025); cleaner supervision, weighted face aggregation, and stronger context modeling in group-image CER (Garg, 2019); and zero-shot abnormal behavior recognition by manually defining the emotion-to-behavior mapping function (Rabiee et al., 2016).

The cumulative picture is that CER has advanced along multiple methodological lines, but its central open problem is not merely higher benchmark accuracy. It is the construction of systems that remain reliable across modalities, cultures, event types, densities, and deployment constraints while preserving interpretability, interoperability, and acceptable privacy safeguards.