---
title: 'Social-MAE: Transformer Autoencoders for Social Data'
url: https://www.emergentmind.com/topics/social-mae
type: topic
---

# Social-MAE: Transformer Autoencoders for Social Data

Social-MAE refers to a family of transformer-based masked autoencoder frameworks designed for learning representations of social behaviors in both multimodal (face and voice) and multi-person motion contexts. These frameworks leverage self-supervised masked modeling to address the challenge of limited annotated data in higher-level social tasks such as emotion recognition, laughter detection, apparent personality estimation, multi-person pose forecasting, social grouping, and social action understanding. Notably, the name "Social-MAE" is used independently in two prominent streams: audiovisual modeling for social perception [2508.17502] and motion-based social representation learning [2404.05578]. Both approaches share the underlying principle of learning robust social representations by reconstructing masked input signals, but are instantiated for different data modalities and downstream objectives.

## 1. Architectural Foundations and Modalities

### Audiovisual Multimodal Social-MAE

The Social-MAE model for synchronized face and voice input [2508.17502] is a three-stage transformer-based autoencoder based on an extension of the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE). Its architecture employs a mid-fusion scheme:

- **Modality encoders**: Separate stacks (11 transformer layers each) process audio (log-Mel spectrogram) and video (face-cropped frames) independently.
- **Tokenization**: Audio is converted to log-Mel spectrograms (128×1024) and patchified into 512 16×16 tokens; video is segmented into eight 224×224 RGB frames, patchified into spatiotemporal cubes (1568 tokens).
- **Joint encoder and decoder**: A joint encoder (1 transformer layer) fuses modality outputs; an 8-layer joint decoder reconstructs both modalities from the pooled latent state and mask tokens.
- **Embedding details**: Each token receives a 768-dimensional linear projection, learned positional embedding, and a learned modality embedding. All transformer blocks employ 768-dim embedding, 12 self-attention heads (64 dim/head), and MLPs with inner dimension 3072.

Masking is independently applied to each modality at a high random ratio (75%), with only 25% of tokens visible, encouraging inter-modality inference.

### Multi-person Motion-based Social-MAE

For multi-person motion, Social-MAE [2404.05578] processes joint trajectories over time:

- **Input**: A sequence of T frames for N individuals, each with J body joints. Joint trajectories are normalized to remove global translation by referencing the pelvis joint at the final frame.
- **Frequency domain encoding**: Each trajectory is projected into the frequency domain using a discrete cosine transform (DCT), retaining K coefficients for each trajectory.
- **Embedding**: Tokens are enhanced with joint-type, person-identity, and global-position embeddings (all 1024-dim).
- **Masking (“tube masking”)**: For a randomly selected 50% of (person, joint) pairs, the entire DCT trajectory is masked, enforcing cross-person and cross-joint dependency.
- **Transformer encoder/decoder**: The encoder (6 layers, 8 heads, 1024-dim) operates only on unmasked tokens. The decoder (3 layers, 4 heads, 1024-dim) processes encoder output plus learnable mask tokens to reconstruct the full trajectory set.

## 2. Self-Supervised Pre-training Objectives and Losses

### Audiovisual Social-MAE

Pre-training combines two objectives [2508.17502]:

- **Masked reconstruction loss**: For masked tokens, the mean squared error (MSE) between reconstructed and original embeddings:
  $$
  L_\text{rec} = \frac{1}{|Ω_{\text{mask}}|} \sum_{i\in Ω_{\text{mask}}} \lVert \hat M_i - x_i \rVert^2
  $$
- **InfoNCE-style contrastive loss**: Enforces alignment of average-pooled unimodal latent representations between audio ($c^a$) and video ($c^v$):
  $$
  L_\text{contra} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp\left(\text{sim}(c^v_i, c^a_i)/\tau\right)}{\sum_{j=1}^N \exp\left(\text{sim}(c^v_i, c^a_j)/\tau\right)}
  $$
  with τ = 0.1.
- **Unified loss**: Total pre-training objective is $L = L_\text{rec} + λ_c L_\text{contra}$, with $λ_c = 1.0$.

### Motion-based Social-MAE

The only pre-training loss is a reconstruction MSE on masked DCT trajectories [2404.05578]:
$$
L_\text{rec} = \frac{1}{|M|} \sum_{(n,j)\in M} \lVert X_{1:T}^{n,j} - \hat X_{1:T}^{n,j}\rVert^2
$$
No contrastive or adversarial losses are included.

## 3. Pre-training Data and Protocols

### Audiovisual Social-MAE

- **Dataset**: VoxCeleb2—1M utterances from over 6,000 speakers and 145 nationalities.
- **Augmentations**: Random horizontal flips, random cropping/scaling for videos. No specAugment due to MAE token masking.
- **Optimization**: 25 epochs, AdamW (β₁=0.9, β₂=0.999, ϵ=1e-8), batch size 128, learning rate 1e-4 decaying by 0.5 every 5 epochs, weight decay 0.05.
- **Masking rationale**: High (75%) random token masking is justified to foster cross-token and inter-modal structural learning.

### Motion-based Social-MAE

- **Datasets**: Pre-trained on combined 3DPW, BMLmovi, BMLrub, and CMU MoCap, covering both 2D and 3D pose.
- **Optimization**: Pre-training for 800 epochs, Adam (lr = 1e-4 decaying to 1e-5).
- **Efficiency**: Tube masking reduces encoder attention cost by half; the shallow decoder and DCT reduction of token count support efficient end-to-end learning. S-MAE remains more data-efficient and robust to limited labeled data during fine-tuning, especially for long-term prediction tasks.

## 4. Fine-tuning Protocols and Downstream Task Configurations

### Audiovisual Social-MAE

For all downstream tasks, the 8-layer decoder is removed. The encoders and joint encoder are retained, with a task-specific linear head added. No masking is applied during fine-tuning.

#### Downstream tasks and metrics:

| Task                                 | Dataset           | Epochs | Batch | Metrics     | Best Social-MAE Score                                  |
|-------------------------------------- |------------------ |--------|-------|-------------|--------------------------------------------------------|
| Emotion Recognition                  | CREMA-D           | 20     | 8     | F1 (Mi/Ma)  | AV Mi 0.837*, AV Ma 0.842*                            |
| Apparent Personality Estimation      | ChaLearn FI       | 10     | 8     | Acc.(1–MAE) | Avg. 0.903*                                            |
| Smile & Laughter Detection           | NDC-ME            | 10     | 8     | F1          | Audio+Video: 0.776*, Audio: 0.546*, Video: 0.728*      |

*: Statistically significant (p < 1e-5) improvement over all baselines for emotion and laughter tasks.

### Motion-based Social-MAE

After pre-training, the decoder is replaced by a downstream-specific head and the encoder is fine-tuned end-to-end for 256 epochs with Adam (lr=1e-3 → 1e-4).

#### Downstream tasks:

1. **Pose forecasting**: Predict future DCT coefficients for all joints of all people, using a token-wise linear regressor. Aggregate auxiliary losses from all encoder layers.
2. **Social grouping**: Predicts group assignment via binary affinity matrices and estimates the number of social groups using pooled person embeddings and MLPs, supervised via BCE and Laplacian eigenvalue count penalties.
3. **Social action understanding**: Multi-label per-person action labels (“pose-based”, Softmax; and “interaction”, Sigmoid).

## 5. Empirical Results and State-of-the-Art Performance

### Audiovisual Social-MAE [2508.17502]

- **Emotion Recognition (CREMA-D)**: Social-MAE achieves multimodal (AV) F1-scores of 0.837 (micro) and 0.842 (macro), surpassing all baselines and previous SOTA, with large gains attributed to the in-domain pre-training and multi-frame video encoding.
- **Apparent Personality Estimation (ChaLearn FI)**: Social-MAE achieves an average accuracy of 90.3%, marginally below the SOTA of 91.3% (NJU-LAMDA), but with tenfold fewer epochs; multi-frame input provides substantive improvement for 4 of 5 Big-5 traits.
- **Smile & Laughter Detection (NDC-ME)**: Outperforms supervised and CAV-MAE baselines in all settings; AV F1-score increases by 18 points over the baseline.
- **Ablations** demonstrate that the use of multiple video frames and re-pre-training on social video data are crucial for both loss reduction and downstream performance improvements.
- **Zero-shot reconstruction** on unseen datasets yields plausible masked face reconstructions, indicating robust learned representations (see Fig. 2 of original).

### Motion-based Social-MAE [2404.05578]

- **Pose forecasting** (SoMoF, CMU-Mocap, MuPoTS-3D): Achieves new SOTA results (e.g., overall VIM 48.6 vs. prior 49.4; reductions in MPJPE at all time horizons). Pre-training improves robustness and generalization, particularly at long horizons and limited data.
- **Social grouping** (JRDB-Act): Best mean AP (32.4) on test, outperforming prior methods and scratch-trained baselines.
- **Social action detection** (JRDB-Act): Outperforms prior work in mAP (11.2 test), with a +1.3 mAP advantage over scratch.
- **Ablations** confirm the tube-masking optimality at 50%, decoder depth of 3, and the positive effect of comprehensive pre-training data coverage.
- **Data efficiency**: Outperforms scratch with substantially fewer fine-tuning samples.

## 6. Distinctive Characteristics, Insights, and Practical Considerations

- **Representation learning**: Both frameworks demonstrate that masked modeling—whether on audiovisual tokens or DCT-encoded multi-person trajectories—yields representations with strong transferability to high-level, socially relevant downstream tasks.
- **Masking and fusion strategy**: High random mask ratios and mid-level fusion architectures (for multimodal data) or tube masking (for multi-person motion) are central for enforcing the exploitation of inter-token and cross-modality/cross-individual dependencies.
- **Data efficiency**: Pre-training with Social-MAE configurations markedly improves performance in low-supervision and long-horizon scenarios.
- **Practical deployment**: Efficient encoder-decoder splits, frequency-domain reductions, and relatively shallow transformer stacks keep training and inference costs competitive with corresponding baseline architectures.
- **Generalization**: Representations learned via Social-MAE are robust as evidenced by zero-shot reconstructions and transfer performance across diverse social perception and behavioral analysis tasks.

*Social-MAE*, in both multimodal and multi-person instantiations, provides a unified strategy for social perception, achieving state-of-the-art results with minimal task-specific architectural modifications during fine-tuning and establishing the value of masked autoencoding for social representation learning in complex human-centered scenarios [2508.17502], [2404.05578].

Source: https://www.emergentmind.com/topics/social-mae