FedMultiEmo: Federated Multimodal Emotion Recognition

• FedMultiEmo is a federated multimodal emotion recognition framework that integrates visual facial analysis with physiological sensing for in-vehicle use.
• It combines a federated CNN processing face images and a local Random Forest classifying physiological signals, using decision-level fusion to enhance accuracy.
• Empirical results show that multimodal fusion boosts recognition accuracy significantly while maintaining raw data privacy on resource-constrained edge devices.

Searching arXiv for FedMultiEmo and closely related federated emotion-recognition papers to ground the article in current literature. FedMultiEmo is a multimodal federated emotion-recognition framework for real-time automotive settings that combines visual facial analysis and physiological sensing while keeping raw data local to edge devices. It was introduced as a privacy-preserving in-vehicle system in which facial images are processed by a federated Convolutional Neural Network and physiological signals are classified by a local Random Forest, with decision-level fusion used to produce the final emotion label (Gül et al., 21 Jul 2025). The framework is positioned around three deployment constraints: modality fragility in vision-based cabin monitoring, inter-individual variability in physiological responses, and the privacy risk of centralized training on facial and bodily data. In its reported prototype, FedMultiEmo is implemented on Raspberry Pi clients coordinated by a Flower server and achieves multimodal performance that is substantially stronger than either unimodal branch alone (Gül et al., 21 Jul 2025).

1. Conceptual scope and problem setting

FedMultiEmo addresses emotion recognition inside vehicle cabins, where sensing conditions are unstable and the input streams themselves are sensitive. The target application is adaptive driver-assistance and occupant-safety support, but the methodological contribution is broader: it is a concrete instance of multimodal federated affect modeling in which each client holds synchronized local observations of face images, physiological signals, and emotion labels. The client-side dataset for participant $n$ is defined as

$D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$

where $I_i$ is the facial image, $s_i$ the physiological signal tuple, and $y_i$ the emotion label (Gül et al., 21 Jul 2025).

The federated learning objective is defined as

$$\mathcal{L}_{\mathrm{global}}(w)=\sum_{n=1}^{N}\frac{m_n}{M}\,\mathcal{L}_n(w), \qquad \mathcal{L}_n(w)=\frac{1}{m_n}\sum_{(I,s,y)\in D_n}\ell(f_w(I,s),y),$$

with $M=\sum_{n=1}^{N} m_n$ (Gül et al., 21 Jul 2025). This formulation makes the global model explicitly sample-size weighted across clients. For the visual branch, the local loss is cross-entropy; for the physiological Random Forest branch, the paper states that training relies on Gini impurity rather than gradient-based optimization (Gül et al., 21 Jul 2025).

A recurring terminological point in the FedMultiEmo description is the use of the label “personalized Federated Averaging.” The aggregation rule is weighted by local data volume, but mathematically this is the standard sample-size-weighted FedAvg form rather than a more elaborate personalization mechanism (Gül et al., 21 Jul 2025). This matters because the framework is privacy-preserving in the data-locality sense, yet it is not presented as a formally private or strongly personalized federated method.

2. Multimodal architecture and local processing pipeline

FedMultiEmo is organized as a multimodal federated pipeline with six client-side stages—data acquisition, preprocessing, feature extraction, local model training, local inference, and decision-level fusion—plus a server-side aggregation stage (Gül et al., 21 Jul 2025). Its multimodality is therefore operational rather than latent: each modality is processed by a separate branch, and predictions are fused only at the decision level.

The visual branch operates on facial images captured at $640\times 480$ resolution and resized to $48\times 48$ grayscale to match FER2013. The resizing is written as

$$I'(x', y') = I\left(\left\lfloor x' \frac{h}{48} \right\rfloor, \left\lfloor y' \frac{w}{48} \right\rfloor \right),$$

with $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$0 and $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$1 (Gül et al., 21 Jul 2025). In the prototype description, images are face-cropped to $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$2, rescaled by $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$3, and augmented using random horizontal flips and random rotations of $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$4 (Gül et al., 21 Jul 2025). Feature extraction uses a CNN with three convolutional blocks, each described as $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$5, with filter depths $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$6, dropout $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$7 after each block, and a dense layer with 1024 units, ReLU activation, and dropout $D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$8 before softmax classification over seven emotions (Gül et al., 21 Jul 2025). The layerwise extraction rule is

$D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},$9

The physiological branch is intentionally low-dimensional and uses heart rate, electrodermal activity, and skin temperature, with the implementation section also mentioning HRV in the collected streams (Gül et al., 21 Jul 2025). Acquisition is carried out with a Fitbit Versa 3 via BLE and an $I_i$0 EDA sensor, and the conceptual description gives a sampling rate of 1 Hz (Gül et al., 21 Jul 2025). Preprocessing consists of a 4th-order Butterworth low-pass filter with 0.5 Hz cutoff, 5-point moving-average smoothing, and z-score normalization (Gül et al., 21 Jul 2025). Features are computed over 5-second windows and include HRV, maximum EDA or “EDA peak rate” depending on the section, and temperature fluctuation, yielding a three-dimensional handcrafted feature vector $I_i$1 (Gül et al., 21 Jul 2025). Classification is performed by a Random Forest with 200 trees and Gini criterion (Gül et al., 21 Jul 2025).

The fusion rule is majority voting over the predicted class labels of the two branches: $I_i$2 Although the text elsewhere refers to “highest aggregated probability,” the equation is a hard-vote rule (Gül et al., 21 Jul 2025). The paper does not specify a tie-breaking mechanism, which is a notable omission because two-modality disagreement creates a $I_i$3 tie.

Branch	Inputs and model	Reported test performance
Visual	$I_i$4 grayscale facial images; CNN with 3 convolutional blocks and 1024-unit dense layer	$I_i$5 in the abstract and often $I_i$6; $I_i$7 in the results section (Gül et al., 21 Jul 2025)
Physiological	Heart rate, EDA, skin temperature; 3 handcrafted features over 5-second windows; Random Forest with 200 trees	About $I_i$8 accuracy, precision $I_i$9, recall $s_i$0 (Gül et al., 21 Jul 2025)
Fused system	Majority-vote decision-level fusion of the two branches	$s_i$1 accuracy; average precision, recall, and F1-score all $s_i$2 (Gül et al., 21 Jul 2025)

3. Federated training protocol and edge deployment

The federated protocol is implemented with Flower and follows a central-server topology with three clients (Gül et al., 21 Jul 2025). The deployment is described in two phases. Phase I is a laptop-based prototype with one high-performance laptop running the Flower server and three Docker-isolated client instances for debugging and end-to-end validation. Phase II is the edge-to-cloud configuration: three Raspberry Pi 4 devices serve as clients, each equipped with a USB camera, a Fitbit Versa 3 via BLE, and an $s_i$3-connected EDA sensor, while the server runs on a central laptop (Gül et al., 21 Jul 2025).

At each round, the server sends the current global model to the clients. Each client performs four local epochs and returns model updates and validation metrics (Gül et al., 21 Jul 2025). The local CNN update rule is given as

$s_i$4

with Adam optimization, learning rate $s_i$5, and exponential decay (Gül et al., 21 Jul 2025). Aggregation uses sample-size-weighted FedAvg: $s_i$6 The paper describes this as “personalized” because the weights depend on client data volume, but the formula itself is standard weighted FedAvg (Gül et al., 21 Jul 2025).

The physiological branch is more weakly specified from a federated perspective. The text clearly states that the Random Forest is trained locally and used locally for probability estimation, but it does not rigorously define tree-level federated aggregation or explain how the forest participates in the global optimization scheme (Gül et al., 21 Jul 2025). The strongest supported interpretation is that the federated mathematics maps cleanly onto the CNN branch, while the Random Forest is trained on-device within the same local multimodal pipeline.

FedMultiEmo’s privacy claim is limited to raw-data locality. Facial images and physiological signals remain on client devices; only model weight updates and validation metrics are transmitted (Gül et al., 21 Jul 2025). The framework does not include differential privacy, secure aggregation, homomorphic encryption, or adversarial-robust aggregation. As a result, “privacy-preserving” in this context means that raw data are not centrally pooled, not that the system provides formal privacy guarantees (Gül et al., 21 Jul 2025).

4. Datasets, sensing infrastructure, and evaluation setup

FedMultiEmo splits its evaluation by modality. For the visual branch, the framework uses FER2013, described as 35,887 grayscale $s_i$7 facial-expression images over seven emotion classes, with FERPlus relabeling adopted so that crowdworker consensus labels replace the original annotations (Gül et al., 21 Jul 2025). For the physiological branch, the paper uses a custom lab-collected dataset gathered while subjects wear a Fitbit Versa 3 and an EDA sensor and watch emotion-eliciting video clips, with labels synchronized to the video timeline (Gül et al., 21 Jul 2025). The participant count, exact class balance, split sizes, and explicit IID or non-IID partition strategy for the physiological data are not provided (Gül et al., 21 Jul 2025).

The client hardware is central to the paper’s practical framing. Each Raspberry Pi client performs local sensing, preprocessing, feature extraction, model training, local inference, and fusion (Gül et al., 21 Jul 2025). The onboard camera supplies facial frames, while the wearable and EDA sensors provide physiological streams. Communication is over WiFi, and the paper reports a per-client memory footprint below 200 MB, supporting the claim of feasibility on resource-constrained edge devices (Gül et al., 21 Jul 2025).

The efficiency reporting is partly inconsistent across sections. The abstract and conclusion state that the system converges in 18 rounds, with average round time 120 seconds and memory below 200 MB (Gül et al., 21 Jul 2025). The introduction refers to 18–20 rounds, the FL setup section states that 20 global rounds were executed, and the training-time table describes a federated setup of 10 rounds $s_i$8 4 epochs with total time 4300 seconds and 430 seconds per round (Gül et al., 21 Jul 2025). A faithful summary is therefore that the paper reports practical convergence in roughly 18–20 rounds but does not present fully consistent runtime figures across all sections.

5. Empirical performance and reported robustness

The core quantitative result is the complementarity of the two modalities. The physiological Random Forest reaches about $s_i$9 test accuracy, with precision $y_i$0 and recall $y_i$1 (Gül et al., 21 Jul 2025). The visual CNN reaches $y_i$2 in the results section, although the abstract and several other passages report $y_i$3, so the safest reading is that visual performance is approximately $y_i$4 (Gül et al., 21 Jul 2025). The multimodal fusion reaches $y_i$5 accuracy, and average precision, recall, and F1-score are all reported as $y_i$6 (Gül et al., 21 Jul 2025).

The paper interprets the gain from $y_i$7 and $y_i$8 unimodal performance to $y_i$9 fused performance as evidence that visual and physiological cues compensate for one another under difficult sensing conditions (Gül et al., 21 Jul 2025). Confusion matrices are used to support this reading. The physiological model shows uneven class-wise behavior due to imbalance; the image model is more balanced but still struggles with subtle expressions, particularly disgust; and the fused model improves difficult classes, especially disgust (Gül et al., 21 Jul 2025). This suggests that the benefit is not only average-accuracy aggregation but also an increase in resilience to modality-specific failure modes.

The centralized comparison is less rigorously quantified than the multimodal result itself. The abstract states that the fusion result matches a centralized baseline while keeping raw data local (Gül et al., 21 Jul 2025). Elsewhere, the paper says that the multimodal classifier matches or surpasses centralized baselines, but it does not provide a separate complete centralized multimodal accuracy table (Gül et al., 21 Jul 2025). The claim is therefore present, but its numerical basis is underreported.

The framework also makes a robustness-oriented argument without presenting a dedicated controlled ablation suite. It repeatedly asserts that multimodal fusion improves reliability under poor lighting, occlusions, and physiological noise (Gül et al., 21 Jul 2025). However, the experiments do not systematically inject lighting degradation, occlusion, modality dropout, or client heterogeneity in a controlled fashion. The reported evidence is therefore empirical but indirect: improved confusion matrices and stronger aggregate performance under a multimodal design.

6. Position within federated affective computing

FedMultiEmo occupies a distinct place within federated affective computing because it combines multimodality, edge deployment, and automotive use in a single system (Gül et al., 21 Jul 2025). Earlier federated emotion-recognition work had already established that federated learning is feasible for speech-only emotion recognition under label scarcity, including semi-supervised pseudo-labeling with FedAvg (Tsouvalas et al., 2022) and multiview pseudo-labeling with SCAFFOLD in speech emotion recognition (Feng et al., 2022). Those works are directly relevant to client-level data scarcity and non-IID speaker distributions, but they are explicitly single-modal and do not address multimodal fusion (Tsouvalas et al., 2022).

Relative to more recent federated affect modeling, FedMultiEmo is also narrower in some dimensions and broader in others. FedFAP studies cross-country mood inference from smartphone sensing with explicit feature heterogeneity and a shared/local decomposition of client representations, which is highly relevant when available sensing modalities differ across clients (Kalpande et al., 17 Feb 2026). FedDISC addresses federated multimodal emotion recognition under systematically missing modalities by training modality-specific diffusion recovery models across clients, which goes beyond FedMultiEmo’s simple decision-level fusion and directly tackles incomplete-modality settings (Qiu et al., 1 Nov 2025). Hardware-aware federated SER, by contrast, focuses on client selection and resource-aware orchestration rather than multimodal recognition quality (Yuksel et al., 23 May 2026). Emoji-prediction papers in Hindi and multilingual settings are affect-adjacent and privacy-relevant, but they study single-label emoji classification rather than direct multimodal emotion recognition (Gandhi et al., 2022, Gamal et al., 2023).

This comparison clarifies a common misconception. FedMultiEmo is genuinely multimodal, unlike speech-only federated SER and text-only emoji prediction, but its multimodality is relatively simple: two branches, local training, and hard decision-level fusion (Gül et al., 21 Jul 2025). It is not a framework for missing-modality recovery, feature-aware personalization, or formal privacy guarantees. A plausible implication is that it is best understood as an edge-deployable multimodal federated baseline rather than a complete solution to every systems and modeling challenge in federated affective computing.

7. Limitations, open issues, and extensions

The framework’s main limitations are explicitly acknowledged in the paper and reinforced by the reporting gaps. The physiological dataset is only lightly described; client partitioning is unspecified; the federated treatment of the Random Forest is not rigorously formalized; privacy is justified through data locality alone; and runtime statistics are inconsistent across sections (Gül et al., 21 Jul 2025). The decision-level fusion mechanism is also deliberately simple and does not specify how ties are resolved, which is consequential in a two-modality system (Gül et al., 21 Jul 2025).

The authors identify several extensions. On the physiological side, they propose replacing handcrafted features with learned temporal models such as LSTMs or Transformers (Gül et al., 21 Jul 2025). On the fusion side, they point to adaptive confidence-weighted fusion rather than simple majority voting (Gül et al., 21 Jul 2025). On the federated systems side, missing modalities, asynchronous updates, and heterogeneous hardware are presented as future work rather than evaluated capabilities (Gül et al., 21 Jul 2025). These directions are consistent with the broader trajectory of related work: feature-aware personalization under heterogeneous sensing (Kalpande et al., 17 Feb 2026), federated recovery for incomplete modalities (Qiu et al., 1 Nov 2025), and hardware-aware client scheduling (Yuksel et al., 23 May 2026).

The central implementation lesson remains comparatively narrow but technically consequential. FedMultiEmo demonstrates that a multimodal federated architecture built from a lightweight CNN, a local Random Forest, sample-size-weighted FedAvg, and edge-to-cloud orchestration in Flower can achieve strong reported multimodal performance while leaving raw facial and physiological data on device (Gül et al., 21 Jul 2025). Its principal contribution is therefore the integration of multimodal emotion recognition with practical federated deployment, not the introduction of a radically new federated optimizer or a formally private training protocol.