TACTFL: Semi-Supervised Federated Learning

• TACTFL is a unified framework for semi-supervised multi-modal federated learning, enabling robust training on heterogeneous unlabelled client data with a small labelled proxy dataset.
• It employs modality-agnostic temporal contrastive training that segments inputs to align overlapping segments via temporal IoU, enhancing cross-modal representation learning.
• The server-side similarity-guided aggregation mitigates semantic drift and accelerates convergence, outperforming traditional federated methods in benchmarks.

Searching arXiv for the primary TACTFL paper and closely related MMFL references. TACTFL is a unified framework for semi-supervised multi-modal federated learning in heterogeneous settings, designed for scenarios in which clients hold locally collected, potentially heterogeneous multi-modal data while the central server has access only to a small labelled proxy dataset (Sun et al., 22 Sep 2025). It combines modality-agnostic temporal contrastive training on clients with similarity-guided model aggregation on the server, with the stated objective of learning a global model for downstream tasks such as action recognition and emotion classification without transferring raw data. The framework is motivated by two recurrent difficulties in real-world federated learning: limited access to labelled data and the presence of heterogeneous multi-modal inputs, together with semantic drift induced by local self-supervised training and the need to exploit cross-modal temporal alignment without labels (Sun et al., 22 Sep 2025).

1. Problem formulation

TACTFL is formulated in a federated setting with $C$ clients, each holding locally collected, potentially heterogeneous multi-modal data from sources such as smart cameras, wearable sensors, and mobile devices (Sun et al., 22 Sep 2025). Each client is assumed to possess at least two synchronised modalities, denoted $x^A$ and $x^B$, such as RGB video frames and audio signals or accelerometer and gyroscope time series. The local unlabelled dataset at client $c$ is written as

$D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$

while the central server alone holds a small labelled proxy dataset

$D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$

This formulation is explicitly semi-supervised. All client datasets are fully unlabelled, and the labelled server-side dataset is used to train a task head and to support similarity-based aggregation (Sun et al., 22 Sep 2025). The framework therefore separates unsupervised representation learning on clients from supervised task adaptation on the server.

The main challenges identified for this setting are client and data heterogeneity, including non-IID distributions via Dirichlet partitioning, semantic drift caused by self-supervised local training, and the need to recover cross-modal temporal alignment without labels (Sun et al., 22 Sep 2025). TACTFL addresses these issues by pairing a local temporal contrastive objective with a server-side aggregation rule that upweights clients whose learned feature spaces remain more representationally consistent with the population.

2. Temporal contrastive training

The local learning mechanism in TACTFL is modality-agnostic temporal contrastive training. Modality-specific encoders $E^m$ are used, such as $E^A$ for RGB/video and $E^B$ for audio, while LSTMs are used for wearable signals (Sun et al., 22 Sep 2025). These encoders are trained locally on unlabelled data, and the downstream task head $H$ is frozen during this stage. No separate projection heads are introduced; contrastive learning is performed directly on encoder embeddings.

For each multi-modal sample, the input sequence is segmented into two temporal chunks per modality, yielding sets

$x^A$0

over a mini-batch, where $x^A$1 is the total number of segments (Sun et al., 22 Sep 2025). Segments may overlap, and the window size is defined as a proportion of the full input, with examples in the range $x^A$2–$x^A$3 and ablations over $x^A$4–$x^A$5.

TACTFL derives soft supervision from temporal intersection-over-union. For each pair $x^A$6, the temporal intersection-over-union score $x^A$7 is computed and each row is normalised to produce a soft target distribution,

$x^A$8

Cross-modal similarity is measured by cosine similarity between encoder outputs,

$x^A$9

and a temperature $x^B$0 scales the similarities (Sun et al., 22 Sep 2025).

The loss for a pair $x^B$1 is given as

$x^B$2

with symmetric aggregation over both modalities, aggregating all pairs $x^B$3 and $x^B$4 within the mini-batch (Sun et al., 22 Sep 2025). This design encourages embeddings of temporally overlapping segments across modalities to be close while pushing apart embeddings from different time steps. Positives and negatives are therefore not defined by labels but by temporal overlap, with low-tIoU pairs implicitly acting as negatives.

The method is described as modality-agnostic because the training signal is derived from temporal alignment rather than modality-specific priors. When a modality is missing, the same modality can be duplicated to form pseudo-pairs for training (Sun et al., 22 Sep 2025). This is a core design choice because it permits the same training mechanism to operate across RGB/video, audio, text, and wearable sensors.

3. Semi-supervised federated protocol

TACTFL interleaves local self-supervised learning and server-side supervised training in a round-based federated procedure (Sun et al., 22 Sep 2025). The server initializes global encoders $x^B$5 and the downstream head $x^B$6, then distributes them to clients. Each client performs local training on unlabelled data by temporally segmenting its multi-modal sequences, computing the tIoU matrix $x^B$7, normalising rows to obtain soft targets $x^B$8, computing cosine similarities $x^B$9 between modalities, and updating encoder weights while keeping $c$0 frozen.

Only updated encoder weights are uploaded to the server. No intermediate representations or prototypes are communicated, and only model parameters are exchanged (Sun et al., 22 Sep 2025). The paper characterises this as communication-light.

After aggregation, the task head $c$1 is trained on $c$2 with encoders frozen. This is described as “standard supervised central training” with a conventional cross-entropy objective over labels $c$3, although no explicit LaTeX formula for the supervised loss is provided in the main text (Sun et al., 22 Sep 2025). The head is updated after aggregation each round, and the updated encoders and head are then redistributed to clients.

The training schedule follows FedMM’s setup of 200 global rounds with 1 local epoch per round and batch size 16 (Sun et al., 22 Sep 2025). Client sampling specifics and the number of clients per dataset are deferred to supplementary material. This suggests that the paper’s principal reproducibility anchors are the encoder choices, the round structure, the temporal segmentation procedure, and the similarity-guided aggregation rule, rather than extensive scheduling heuristics.

4. Similarity-guided model aggregation

TACTFL’s server-side aggregation mechanism is similarity-guided model aggregation, introduced to address semantic divergence among locally trained self-supervised models (Sun et al., 22 Sep 2025). The stated motivation is that standard aggregation methods such as FedAvg and FedOpt become unreliable when clients train on heterogeneous data and their learned feature spaces drift apart semantically.

To measure representational consistency, the server passes the labelled set $c$4 through each client encoder model $c$5, computes embeddings, and averages across samples to obtain summary vectors $c$6 (Sun et al., 22 Sep 2025). It then constructs the pairwise cosine similarity matrix

$c$7

From this matrix, the similarity-based weight vector $c$8 is computed as

$c$9

and the updated global encoder model is

$D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$0

Clients whose representations are more consistent with the majority receive greater weight (Sun et al., 22 Sep 2025).

The aggregation mechanism introduces no additional thresholds or temperatures beyond cosine similarity; the weighting is produced by normalisation alone (Sun et al., 22 Sep 2025). The intended effect is to mitigate semantic drift and accelerate convergence relative to uniform weighting or purely optimizer-driven aggregation. The paper provides empirical evidence for improved performance and convergence but does not present a formal theoretical analysis.

A plausible implication is that the server-side labelled proxy dataset plays a dual role: it is not only the source for task-head supervision, but also the anchor that makes representational consistency measurable across clients. This dual use is explicit in the training protocol and is central to the framework’s semi-supervised design.

5. Heterogeneity, missing modalities, and implementation

TACTFL is explicitly designed for heterogeneous multi-modal federated learning. It assumes synchronised modalities at each client, and uses temporal overlap as a soft supervisory signal across modalities (Sun et al., 22 Sep 2025). Moderate window sizes and overlap are reported to improve robustness to small timing differences. The framework also supports clients with missing modalities: when a modality is absent, the same modality can be duplicated to form pseudo-pairs for within-modality alignment.

Missing-modality experiments are simulated by dropping entire modalities per client with rate $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$1, and robustness analyses also report results over $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$2 (Sun et al., 22 Sep 2025). The label fraction parameter $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$3 controls how much labelled data resides on the server; for example, $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$4 implies 10% labelled data on the server and 90% unlabelled data at clients.

The implementation follows prior benchmark conventions for encoders. Following FedMM, MobileNetV2 is used for videos and images, MFCC-based features for audio, and MobileBERT for text; following FedMEKT, two LSTM layers are used for wearable data such as accelerometer, gyroscope, and magnetometer streams (Sun et al., 22 Sep 2025). Contrastive learning is applied directly on encoder outputs, without a projection head. Two temporal chunks are extracted per sample, with window size proportions varied from $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$5 to $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$6.

The paper reports a temporal window ablation on UCF101 in which performance peaks at moderate window sizes: $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$7 gives $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$8, $D_c = \{(x_i^A, x_i^B)\}_{i=1}^{N_c},$9 gives $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$0, $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$1 gives $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$2, $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$3 gives $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$4, and $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$5 gives $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$6 (Sun et al., 22 Sep 2025). This suggests that neither very large nor very small windows are optimal for the temporal-overlap signal used by tIoU-based supervision.

6. Empirical performance and ablations

TACTFL is evaluated across diverse benchmarks and modalities, including video, audio, text, and wearable sensors (Sun et al., 22 Sep 2025). The reported datasets include UCF101 and MiT-10 for audio-video clients, UCI-HAR for accelerometer and gyroscope data, Hateful Memes for image-plus-text, MELD for audio-plus-text, and FedMEKT-style benchmarks such as mHealth, UR-Fall, and Opportunity.

On UCF101 with non-IID partitioning at $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$7 and $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$8, the reported accuracies are FedAvg $D_S = \{(x_j^A, x_j^B, y_j)\}_{j=1}^{N_s}.$9, FedOpt $E^m$0, and TACTFL $E^m$1; with $E^m$2, they are FedAvg $E^m$3, FedOpt $E^m$4, and TACTFL $E^m$5 (Sun et al., 22 Sep 2025). On MiT-10 at $E^m$6, the corresponding results are FedAvg $E^m$7, FedOpt $E^m$8, and TACTFL $E^m$9 for $E^A$0, and FedAvg $E^A$1, FedOpt $E^A$2, and TACTFL $E^A$3 for $E^A$4. On UCI-HAR, the reported F1 scores are FedAvg $E^A$5, FedOpt $E^A$6, and TACTFL $E^A$7 for $E^A$8, and FedAvg $E^A$9, FedOpt $E^B$0, and TACTFL $E^B$1 for $E^B$2.

The missing-modality results also favour TACTFL. For UCI-HAR, the reported F1 scores are $E^B$3 at $E^B$4, $E^B$5 at $E^B$6, and $E^B$7 at $E^B$8; for Hateful Memes, the reported AUC values are $E^B$9 at $H$0, $H$1 at $H$2, and $H$3 at $H$4; for MELD, the reported UAR values are $H$5, $H$6, and $H$7 at the same missing-modality rates (Sun et al., 22 Sep 2025).

Ablations isolate the contributions of temporal contrastive training and similarity-guided aggregation. On UCF101, temporal contrastive training alone substantially improves over the semi-supervised federated learning baseline: under FedAvg, accuracy rises from $H$8 to $H$9, and under FedOpt, from $x^A$00 to $x^A$01 (Sun et al., 22 Sep 2025). Adding similarity-guided model aggregation yields further gains, with reported accuracies of $x^A$02 under FedAvg and $x^A$03 under FedOpt, together with faster convergence toward fully supervised performance.

These results underpin the paper’s claim of state-of-the-art performance across video, audio, and wearable sensor benchmarks (Sun et al., 22 Sep 2025). Since the paper does not include formal theory, the empirical case is the principal basis for the framework’s significance.

7. Limitations, interpretation, and future directions

TACTFL assumes synchronised modalities at the client, and the paper notes that severe temporal misalignment or asynchronous signals may weaken tIoU-based supervision (Sun et al., 22 Sep 2025). Fixed segmentation windows are also identified as a limitation, since a single windowing strategy may not optimally capture semantic dynamics for all modalities or activities. Dynamic windowing and alignment are identified as future directions.

The framework is reported to be robust at high label sparsity and missing-modality rates, but extremely non-IID distributions or very sparse labels can still challenge aggregation and head training (Sun et al., 22 Sep 2025). This identifies an important boundary condition: similarity-guided aggregation mitigates semantic drift, but does not eliminate all pathologies induced by extreme heterogeneity.

The paper also raises fairness and privacy considerations. Although TACTFL preserves privacy by not sharing raw data or intermediate features, similarity-guided aggregation may penalise clients from underrepresented populations, and proxy datasets may contain demographic or cultural biases (Sun et al., 22 Sep 2025). Fairness-aware aggregation and personalised adaptation are recommended as mitigation strategies. This suggests that representational consensus, while beneficial for convergence, may also privilege dominant client subpopulations when the proxy dataset is not representative.

Planned extensions include dynamic temporal alignment, asynchronous modality support, and personalised model adaptation for highly heterogeneous clients (Sun et al., 22 Sep 2025). Code is stated to be released upon publication, although no repository link is provided in the main text. The implementation details singled out as critical for reproduction are the encoder choices, the 200-round schedule with 1 local epoch per round and batch size 16, the two-segment tIoU-based contrastive pipeline with cosine similarity and temperature $x^A$04, and the cosine-similarity-based server aggregation scheme (Sun et al., 22 Sep 2025).