---
title: 'FedMultiEmo: Federated Multimodal Emotion Recognition'
url: https://www.emergentmind.com/topics/fedmultiemo
type: topic
---

# FedMultiEmo: Federated Multimodal Emotion Recognition

Searching arXiv for FedMultiEmo and closely related federated emotion-recognition papers to ground the article in current literature.
FedMultiEmo is a multimodal federated emotion-recognition framework for real-time automotive settings that combines visual facial analysis and physiological sensing while keeping raw data local to edge devices. It was introduced as a privacy-preserving in-vehicle system in which facial images are processed by a federated Convolutional Neural Network and physiological signals are classified by a local Random Forest, with decision-level fusion used to produce the final emotion label [2507.15470]. The framework is positioned around three deployment constraints: modality fragility in vision-based cabin monitoring, inter-individual variability in physiological responses, and the privacy risk of centralized training on facial and bodily data. In its reported prototype, FedMultiEmo is implemented on Raspberry Pi clients coordinated by a Flower server and achieves multimodal performance that is substantially stronger than either unimodal branch alone [2507.15470].

## 1. Conceptual scope and problem setting

FedMultiEmo addresses emotion recognition inside vehicle cabins, where sensing conditions are unstable and the input streams themselves are sensitive. The target application is adaptive driver-assistance and occupant-safety support, but the methodological contribution is broader: it is a concrete instance of multimodal federated affect modeling in which each client holds synchronized local observations of face images, physiological signals, and emotion labels. The client-side dataset for participant \(n\) is defined as
\[
D_n=\{(I_i, s_i, y_i)\}_{i=1}^{m_n},
\]
where \(I_i\) is the facial image, \(s_i\) the physiological signal tuple, and \(y_i\) the emotion label [2507.15470].

The federated learning objective is defined as
\[
\mathcal{L}_{\mathrm{global}}(w)=\sum_{n=1}^{N}\frac{m_n}{M}\,\mathcal{L}_n(w), \qquad
\mathcal{L}_n(w)=\frac{1}{m_n}\sum_{(I,s,y)\in D_n}\ell(f_w(I,s),y),
\]
with \(M=\sum_{n=1}^{N} m_n\) [2507.15470]. This formulation makes the global model explicitly sample-size weighted across clients. For the visual branch, the local loss is cross-entropy; for the physiological Random Forest branch, the paper states that training relies on Gini impurity rather than gradient-based optimization [2507.15470].

A recurring terminological point in the FedMultiEmo description is the use of the label “personalized Federated Averaging.” The aggregation rule is weighted by local data volume, but mathematically this is the standard sample-size-weighted FedAvg form rather than a more elaborate personalization mechanism [2507.15470]. This matters because the framework is privacy-preserving in the data-locality sense, yet it is not presented as a formally private or strongly personalized federated method.

## 2. Multimodal architecture and local processing pipeline

FedMultiEmo is organized as a multimodal federated pipeline with six client-side stages—data acquisition, preprocessing, feature extraction, local model training, local inference, and decision-level fusion—plus a server-side aggregation stage [2507.15470]. Its multimodality is therefore operational rather than latent: each modality is processed by a separate branch, and predictions are fused only at the decision level.

The visual branch operates on facial images captured at \(640\times 480\) resolution and resized to \(48\times 48\) grayscale to match FER2013. The resizing is written as
\[
I'(x', y') = I\left(\left\lfloor x' \frac{h}{48} \right\rfloor, \left\lfloor y' \frac{w}{48} \right\rfloor \right),
\]
with \(h=640\) and \(w=480\) [2507.15470]. In the prototype description, images are face-cropped to \(48\times48\), rescaled by \(1/255\), and augmented using random horizontal flips and random rotations of \(\pm 10^\circ\) [2507.15470]. Feature extraction uses a CNN with three convolutional blocks, each described as \((\mathrm{Conv}+\mathrm{ReLU}) \rightarrow \mathrm{MaxPool}\), with filter depths \(\{32,64,128\}\), dropout \(p=0.25\) after each block, and a dense layer with 1024 units, ReLU activation, and dropout \(p=0.5\) before softmax classification over seven emotions [2507.15470]. The layerwise extraction rule is
\[
F_l=\mathrm{ReLU}(W_l * F_{l-1}+b_l), \quad F_0=I'.
\]

The physiological branch is intentionally low-dimensional and uses heart rate, electrodermal activity, and skin temperature, with the implementation section also mentioning HRV in the collected streams [2507.15470]. Acquisition is carried out with a Fitbit Versa 3 via BLE and an \(I^2C\) EDA sensor, and the conceptual description gives a sampling rate of 1 Hz [2507.15470]. Preprocessing consists of a 4th-order Butterworth low-pass filter with 0.5 Hz cutoff, 5-point moving-average smoothing, and z-score normalization [2507.15470]. Features are computed over 5-second windows and include HRV, maximum EDA or “EDA peak rate” depending on the section, and temperature fluctuation, yielding a three-dimensional handcrafted feature vector \(\mathbf f_{\mathrm{physio}}\in\mathbb R^3\) [2507.15470]. Classification is performed by a Random Forest with 200 trees and Gini criterion [2507.15470].

The fusion rule is majority voting over the predicted class labels of the two branches:
\[
\hat{y}_{\mathrm{final}} = \arg\max_{c \in \mathcal{C}}
\left[\mathbbm{1}(\hat{y}_{\mathrm{visual}} = c) + \mathbbm{1}(\hat{y}_{\mathrm{physio}} = c)\right].
\]
Although the text elsewhere refers to “highest aggregated probability,” the equation is a hard-vote rule [2507.15470]. The paper does not specify a tie-breaking mechanism, which is a notable omission because two-modality disagreement creates a \(1\!-\!1\) tie.

| Branch | Inputs and model | Reported test performance |
|---|---|---|
| Visual | \(48\times48\) grayscale facial images; CNN with 3 convolutional blocks and 1024-unit dense layer | \(77\%\) in the abstract and often \(77\%\); \(78\%\) in the results section [2507.15470] |
| Physiological | Heart rate, EDA, skin temperature; 3 handcrafted features over 5-second windows; Random Forest with 200 trees | About \(74\%\) accuracy, precision \(67\%\), recall \(73\%\) [2507.15470] |
| Fused system | Majority-vote decision-level fusion of the two branches | \(87\%\) accuracy; average precision, recall, and F1-score all \(87\%\) [2507.15470] |

## 3. Federated training protocol and edge deployment

The federated protocol is implemented with Flower and follows a central-server topology with three clients [2507.15470]. The deployment is described in two phases. Phase I is a laptop-based prototype with one high-performance laptop running the Flower server and three Docker-isolated client instances for debugging and end-to-end validation. Phase II is the edge-to-cloud configuration: three Raspberry Pi 4 devices serve as clients, each equipped with a USB camera, a Fitbit Versa 3 via BLE, and an \(I^2C\)-connected EDA sensor, while the server runs on a central laptop [2507.15470].

At each round, the server sends the current global model to the clients. Each client performs four local epochs and returns model updates and validation metrics [2507.15470]. The local CNN update rule is given as
\[
w_n^{t+1}=w^t-\eta \nabla \mathcal{L}_n(w^t),
\]
with Adam optimization, learning rate \(\eta=10^{-4}\), and exponential decay [2507.15470]. Aggregation uses sample-size-weighted FedAvg:
\[
w^{t+1}=\sum_{n=1}^{N}\frac{m_n}{M}w_n^{t+1}.
\]
The paper describes this as “personalized” because the weights depend on client data volume, but the formula itself is standard weighted FedAvg [2507.15470].

The physiological branch is more weakly specified from a federated perspective. The text clearly states that the Random Forest is trained locally and used locally for probability estimation, but it does not rigorously define tree-level federated aggregation or explain how the forest participates in the global optimization scheme [2507.15470]. The strongest supported interpretation is that the federated mathematics maps cleanly onto the CNN branch, while the Random Forest is trained on-device within the same local multimodal pipeline.

FedMultiEmo’s privacy claim is limited to raw-data locality. Facial images and physiological signals remain on client devices; only model weight updates and validation metrics are transmitted [2507.15470]. The framework does not include differential privacy, secure aggregation, homomorphic encryption, or adversarial-robust aggregation. As a result, “privacy-preserving” in this context means that raw data are not centrally pooled, not that the system provides formal privacy guarantees [2507.15470].

## 4. Datasets, sensing infrastructure, and evaluation setup

FedMultiEmo splits its evaluation by modality. For the visual branch, the framework uses FER2013, described as 35,887 grayscale \(48\times48\) facial-expression images over seven emotion classes, with FERPlus relabeling adopted so that crowdworker consensus labels replace the original annotations [2507.15470]. For the physiological branch, the paper uses a custom lab-collected dataset gathered while subjects wear a Fitbit Versa 3 and an EDA sensor and watch emotion-eliciting video clips, with labels synchronized to the video timeline [2507.15470]. The participant count, exact class balance, split sizes, and explicit IID or non-IID partition strategy for the physiological data are not provided [2507.15470].

The client hardware is central to the paper’s practical framing. Each Raspberry Pi client performs local sensing, preprocessing, feature extraction, model training, local inference, and fusion [2507.15470]. The onboard camera supplies facial frames, while the wearable and EDA sensors provide physiological streams. Communication is over WiFi, and the paper reports a per-client memory footprint below 200 MB, supporting the claim of feasibility on resource-constrained edge devices [2507.15470].

The efficiency reporting is partly inconsistent across sections. The abstract and conclusion state that the system converges in 18 rounds, with average round time 120 seconds and memory below 200 MB [2507.15470]. The introduction refers to 18–20 rounds, the FL setup section states that 20 global rounds were executed, and the training-time table describes a federated setup of 10 rounds \(\times\) 4 epochs with total time 4300 seconds and 430 seconds per round [2507.15470]. A faithful summary is therefore that the paper reports practical convergence in roughly 18–20 rounds but does not present fully consistent runtime figures across all sections.

## 5. Empirical performance and reported robustness

The core quantitative result is the complementarity of the two modalities. The physiological Random Forest reaches about \(74\%\) test accuracy, with precision \(67\%\) and recall \(73\%\) [2507.15470]. The visual CNN reaches \(78\%\) in the results section, although the abstract and several other passages report \(77\%\), so the safest reading is that visual performance is approximately \(77\%-78\%\) [2507.15470]. The multimodal fusion reaches \(87\%\) accuracy, and average precision, recall, and F1-score are all reported as \(87\%\) [2507.15470].

The paper interprets the gain from \(77\%-78\%\) and \(74\%\) unimodal performance to \(87\%\) fused performance as evidence that visual and physiological cues compensate for one another under difficult sensing conditions [2507.15470]. Confusion matrices are used to support this reading. The physiological model shows uneven class-wise behavior due to imbalance; the image model is more balanced but still struggles with subtle expressions, particularly disgust; and the fused model improves difficult classes, especially disgust [2507.15470]. This suggests that the benefit is not only average-accuracy aggregation but also an increase in resilience to modality-specific failure modes.

The centralized comparison is less rigorously quantified than the multimodal result itself. The abstract states that the fusion result matches a centralized baseline while keeping raw data local [2507.15470]. Elsewhere, the paper says that the multimodal classifier matches or surpasses centralized baselines, but it does not provide a separate complete centralized multimodal accuracy table [2507.15470]. The claim is therefore present, but its numerical basis is underreported.

The framework also makes a robustness-oriented argument without presenting a dedicated controlled ablation suite. It repeatedly asserts that multimodal fusion improves reliability under poor lighting, occlusions, and physiological noise [2507.15470]. However, the experiments do not systematically inject lighting degradation, occlusion, modality dropout, or client heterogeneity in a controlled fashion. The reported evidence is therefore empirical but indirect: improved confusion matrices and stronger aggregate performance under a multimodal design.

## 6. Position within federated affective computing

FedMultiEmo occupies a distinct place within federated affective computing because it combines multimodality, edge deployment, and automotive use in a single system [2507.15470]. Earlier federated emotion-recognition work had already established that federated learning is feasible for speech-only emotion recognition under label scarcity, including semi-supervised pseudo-labeling with FedAvg [2202.02611] and multiview pseudo-labeling with SCAFFOLD in speech emotion recognition [2203.08810]. Those works are directly relevant to client-level data scarcity and non-IID speaker distributions, but they are explicitly single-modal and do not address multimodal fusion [2202.02611].

Relative to more recent federated affect modeling, FedMultiEmo is also narrower in some dimensions and broader in others. FedFAP studies cross-country mood inference from smartphone sensing with explicit feature heterogeneity and a shared/local decomposition of client representations, which is highly relevant when available sensing modalities differ across clients [2602.15478]. FedDISC addresses federated multimodal emotion recognition under systematically missing modalities by training modality-specific diffusion recovery models across clients, which goes beyond FedMultiEmo’s simple decision-level fusion and directly tackles incomplete-modality settings [2511.00344]. Hardware-aware federated SER, by contrast, focuses on client selection and resource-aware orchestration rather than multimodal recognition quality [2605.24712]. Emoji-prediction papers in Hindi and multilingual settings are affect-adjacent and privacy-relevant, but they study single-label emoji classification rather than direct multimodal emotion recognition [2211.06401] [2304.01005].

This comparison clarifies a common misconception. FedMultiEmo is genuinely multimodal, unlike speech-only federated SER and text-only emoji prediction, but its multimodality is relatively simple: two branches, local training, and hard decision-level fusion [2507.15470]. It is not a framework for missing-modality recovery, feature-aware personalization, or formal privacy guarantees. A plausible implication is that it is best understood as an edge-deployable multimodal federated baseline rather than a complete solution to every systems and modeling challenge in federated affective computing.

## 7. Limitations, open issues, and extensions

The framework’s main limitations are explicitly acknowledged in the paper and reinforced by the reporting gaps. The physiological dataset is only lightly described; client partitioning is unspecified; the federated treatment of the Random Forest is not rigorously formalized; privacy is justified through data locality alone; and runtime statistics are inconsistent across sections [2507.15470]. The decision-level fusion mechanism is also deliberately simple and does not specify how ties are resolved, which is consequential in a two-modality system [2507.15470].

The authors identify several extensions. On the physiological side, they propose replacing handcrafted features with learned temporal models such as LSTMs or Transformers [2507.15470]. On the fusion side, they point to adaptive confidence-weighted fusion rather than simple majority voting [2507.15470]. On the federated systems side, missing modalities, asynchronous updates, and heterogeneous hardware are presented as future work rather than evaluated capabilities [2507.15470]. These directions are consistent with the broader trajectory of related work: feature-aware personalization under heterogeneous sensing [2602.15478], federated recovery for incomplete modalities [2511.00344], and hardware-aware client scheduling [2605.24712].

The central implementation lesson remains comparatively narrow but technically consequential. FedMultiEmo demonstrates that a multimodal federated architecture built from a lightweight CNN, a local Random Forest, sample-size-weighted FedAvg, and edge-to-cloud orchestration in Flower can achieve strong reported multimodal performance while leaving raw facial and physiological data on device [2507.15470]. Its principal contribution is therefore the integration of multimodal emotion recognition with practical federated deployment, not the introduction of a radically new federated optimizer or a formally private training protocol.

Source: https://www.emergentmind.com/topics/fedmultiemo