---
title: 'OmniSapiens-7B SFT: Multimodal Behavior Benchmark'
url: https://www.emergentmind.com/topics/omnisapiens-7b-sft
type: topic
---

# OmniSapiens-7B SFT: Multimodal Behavior Benchmark

OmniSapiens-7B SFT is a 7B-parameter multimodal model for unified psychological and social behavior understanding, introduced as the principal supervised baseline in "Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understanding" [2510.04899]. In that work, **SFT** denotes **supervised fine-tuning**: a pretrained multimodal LLM backbone is fine-tuned on a unified benchmark of behavioral tasks using a standardized prompt–target interface. The model is intended less as a novel architectural proposal than as a benchmarked, "vanilla" instance for testing whether psychological and social behavior understanding can be trained as a single multimodal multitask problem spanning affective states, cognitive states, pathology, and social processes [2510.04899].

## 1. Position within the OmniSapiens family

Within the Human Behavior Atlas study, OmniSapiens-7B SFT is the foundational member of a three-model family. The paper distinguishes **OmniSapiens-7B SFT**, **OmniSapiens-7B BAM**, and **OmniSapiens-7B RL**. SFT is the base supervised fine-tuned model; BAM is the SFT model augmented after training with a **Behavioral Adapter Module** that injects behavioral descriptors through a residual adapter while keeping the SFT backbone frozen; RL is a decoder-only variant optimized with **GRPO-style reinforcement learning** over unified free-text outputs [2510.04899].

The authors explicitly state that these variants are “meant as vanilla instances rather than new methods, intended primarily to investigate Human Behavior Atlas.” This framing is important. OmniSapiens-7B SFT is not presented as a behavior-specific architecture in the narrow sense; rather, it is an empirical test of whether multimodal instruction-style fine-tuning on a unified behavioral benchmark yields a broadly useful model for sentiment, emotion, pathology-related detection, social reasoning, intent recognition, humor, sarcasm, and non-verbal communication [2510.04899].

The practical division of labor across the three variants is also explicit. SFT is described as strongest and most stable on **structured classification tasks**. BAM selectively improves SFT on tasks benefiting from explicit descriptors. RL is relatively better on **open-ended generation and reasoning tasks** such as intent and social reasoning. This division makes OmniSapiens-7B SFT the canonical reference point for the family’s supervised behavior-understanding capabilities [2510.04899].

## 2. Architectural basis and multimodal formulation

OmniSapiens-7B SFT is initialized from **Qwen2.5-Omni-7B**, described as “a pretrained multimodal LLM ... which includes an audio and image encoder, alongside an LLM” [2510.04899]. At a high level, the architecture comprises a pretrained audio encoder, a pretrained image encoder, an LLM backbone, a projection stage that maps raw multimodal inputs into a **shared embedding space**, task-specific classifier heads for classification tasks, and a decoder head for open-ended text response tasks.

The paper summarizes the fusion strategy in a single sentence: “We project raw audio, visual, and textual inputs into a shared embedding space, enabling the LLM backbone to process multimodal benchmark samples.” The supported raw modalities for OmniSapiens-7B SFT are therefore **text**, **audio**, and **visual/video**, with visual information handled through the backbone’s image encoder. Although Human Behavior Atlas also includes extracted behavioral descriptors as auxiliary features, those descriptors are not integrated directly into the SFT model’s core path [2510.04899].

Task unification is implemented through a standardized prompt–target schema aligned with multimodal LLMs such as Qwen2.5-Omni. Prompts explicitly reference available modalities with tokens such as `<video>` and `<audio>`, incorporate transcript text when available, and pose a behavior-specific question in natural language. Targets are standardized either as discrete label sets for classification or as free-text answers for open-ended tasks. The paper’s MELD sentiment example follows exactly this pattern, combining video, audio, transcript text, and a natural-language instruction asking for a sentiment label [2510.04899].

The output layer is hybrid rather than purely generative. For categorical tasks, the model uses classifier heads fed by pooled hidden representations taken from $h_{\text{penult}}$, “the penultimate layer immediately before the final hidden states.” For free-text inference, “a separate decoder head generates outputs directly from the final hidden states after $h_{\text{penult}}$.” The losses are correspondingly split between **cross-entropy loss** for classifier heads and **teacher forcing** for decoder training. The main text does not provide a full layer-by-layer diagram for OmniSapiens-7B SFT, nor does it formalize tokenization, multimodal fusion, or the supervised objective beyond these descriptive statements [2510.04899].

## 3. Human Behavior Atlas and benchmark unification

Human Behavior Atlas is the benchmark on which OmniSapiens-7B SFT is trained and evaluated. It was created to address fragmentation in prior work, which the paper characterizes as dataset-specific, task-specific, and inconsistent in input format, labels, and evaluation. The benchmark contains **101,964 unified and standardized samples** spanning **35,046 videos**, **10,287 audio clips**, and **25,385 transcripts**, and is built from **13 publicly available multimodal datasets**. The paper also reports that **83.6% of samples contain video data** and that **29.2% of video/audio clips last more than 20 seconds** [2510.04899].

The benchmark organizes behavioral understanding into four high-level dimensions—**Affective states (Aff)**, **Cognitive states (Cog)**, **Pathology (Path)**, and **Social processes (Soc)**—and covers ten behavioral tasks.

| Task | Representative datasets |
|---|---|
| SEN | CMU-MOSEI, MELD, CH-SIMSv2 |
| EMO | CMU-MOSEI, MELD, TESS, CREMA-D |
| SOC | Social-IQ 2.0 |
| INT | IntentQA |
| NVC | MimeQA |
| HUM | UR-FUNNYv2 |
| SAR | MUStARD |
| ANX | MMPsy |
| DEP | DAIC-WOZ, MMPsy |
| PTSD | PTSD-in-the-Wild |

The underlying datasets are explicitly enumerated in the paper: **CMU-MOSEI** (31,454 samples), **MELD** (27,412), **TESS** (2,800), **CREMA-D** (7,442), **CH-SIMSv2** (4,403), **Social-IQ 2.0** (6,437), **IntentQA** (16,297), **MimeQA** (806), **UR-FUNNYv2** (2,125), **MUStARD** (690), **DAIC-WOZ** (189), **MMPsy** (1,275), and **PTSD-in-the-Wild** (634). The benchmark’s scale and heterogeneity are central to the rationale for a unified model [2510.04899].

A substantial part of the contribution lies in preprocessing and harmonization. Continuous labels are discretized where required; the example given is that **PHQ-9 scores in MMPsy are discretized into categories based on original guidelines**. Emotion labels are merged when they represent the same underlying emotion, such as “joy” and “happiness,” and split when the original label is intrinsically heterogeneous, such as dividing “surprise” into **“positive surprise”** and **“negative surprise.”** For sentiment, evaluation is normalized to **binary weighted F1 over positive and negative labels**. When transcripts are missing, text is extracted from audio using the **Whisper v3 Large model** [2510.04899].

Human Behavior Atlas also includes behavioral descriptors extracted from **MediaPipe** for visual signals—facial landmarks and body pose keypoints—and from **OpenSMILE**, **ComParE 2016** for audio signals, including pitch, energy, spectral properties, and voice quality. For OmniSapiens-7B SFT, however, these descriptors are auxiliary benchmark assets rather than part of the model’s primary input path. This suggests that the benchmark is deliberately designed to support multiple regimes of behavioral modeling, including but not limited to the baseline supervised model [2510.04899].

## 4. Supervised training regime

OmniSapiens-7B SFT is trained in a **joint multitask setup** across all behavioral tasks in Human Behavior Atlas. The paper states that “a single instance of the model is trained jointly across all behavioral tasks in the benchmark.” In the standard setting, SFT therefore uses the benchmark’s multitask training split across all included datasets and tasks, rather than separate per-task fine-tuning [2510.04899].

The training objective mixes classification and generation supervision. Classification tasks are handled by task-specific classifier heads trained with **cross-entropy loss**, while open-ended tasks are trained through a decoder head with **teacher forcing**. The behavioral understanding problem is framed through prompt unification rather than through a bespoke architecture or a formal multi-objective derivation. The paper is explicit that, for OmniSapiens-7B SFT specifically, there are **no explicit LaTeX equations** defining the supervised fine-tuning objective, multimodal tokenization or encoding, task-unification loss, or descriptor conditioning [2510.04899].

Several implementation details that would normally be expected in a training recipe are not given in the supplied main text. These include learning rate, optimizer, batch size, sequence length, number of epochs or steps, sampling or mixing ratios, modality balancing strategy, hardware count or type, and curriculum schedule. The only hardware-related acknowledgment is that Nvidia provided GPU and compute resources. As a result, the published description of OmniSapiens-7B SFT is richer in benchmark construction and empirical findings than in optimization internals [2510.04899].

This omission matters for interpretation. The model is presented primarily as an empirical baseline demonstrating the consequences of benchmark unification. A plausible implication is that the benchmark’s prompt normalization and task harmonization are treated as the key controlled intervention, while fine-grained optimization design is relegated to appendices rather than foregrounded as a methodological novelty [2510.04899].

## 5. Empirical performance profile

On Human Behavior Atlas, OmniSapiens-7B SFT is evaluated against **Gemma-3-4B**, **HumanOmniV2-7B**, **Qwen 2.5-Omni-7B**, and **Qwen-2.5-VL-7B**. The paper states that **OmniSapiens-7B SFT** and **OmniSapiens-7B BAM** each outperform the general multimodal LLMs on **8 of 10 behavioral tasks**, while **OmniSapiens-7B RL** does so on **7 of 10**. This establishes SFT as the main evidence that unified training on Human Behavior Atlas yields broad improvements over generic multimodal backbones [2510.04899].

| Dataset / task | OmniSapiens-7B SFT score |
|---|---:|
| CREMA-D (EMO) | 0.542 |
| MELD (E) | 0.709 |
| MOSEI (E) | 0.614 |
| TESS (EMO) | 0.658 |
| UR-FUNNY (HUM) | 0.532 |
| IntentQA (INT) | 0.256 |
| PTSD_WILD (PTSD) | 1.00 |
| MMPSY (A) | 0.909 |
| MMPSY (D) | 0.839 |
| DAIC-WOZ (DEP) | 0.626 |
| MELD (S) | 0.746 |
| CH-SIMSv2 (SEN) | 0.813 |
| MOSEI (S) | 0.744 |
| MUStARD (SAR) | 0.624 |
| Social-IQ 2.0 (SOC) | 0.257 |
| MimeQA (NVC) | 0.121 |

The strongest parts of the SFT profile are **emotion recognition**, **sentiment**, and several forms of **pathology detection**. The model improves over Qwen 2.5-Omni-7B on all four emotion datasets listed in the table, attains **1.00** on **PTSD_WILD**, and reaches **0.909** on **MMPSY (A)** and **0.839** on **MMPSY (D)**. Its sentiment results—**0.746** on MELD (S), **0.813** on CH-SIMSv2, and **0.744** on MOSEI (S)—also exceed Qwen 2.5-Omni-7B on all three corresponding datasets [2510.04899].

The weaker parts of the profile are **intent recognition**, **social reasoning**, and **humor detection**, with more modest results on **non-verbal communication** and, relative to BAM, on **sarcasm detection**. The paper interprets this as evidence that supervised fine-tuning is especially effective for structured classification tasks, whereas explicit descriptor injection or RL-style reasoning may be more useful for tasks requiring subtle cue integration or open-ended inference. Quantitatively, RL is much stronger on **IntentQA** (**0.486** versus **0.256**), while BAM yields major gains on **MUStARD** (**0.795** versus **0.624**) and **MimeQA** (**0.162** versus **0.121**) [2510.04899].

These comparisons clarify the role of SFT inside the OmniSapiens family. It is not uniformly dominant, but it is the most stable supervised baseline and the principal demonstration that a unified multimodal behavior benchmark can convert a general-purpose pretrained multimodal LLM into a broadly effective behavior-understanding system [2510.04899].

## 6. Transfer, limitations, and later interpretation

The paper also reports held-out transfer experiments in which four datasets are excluded during multitask pretraining of OmniSapiens-7B SFT—**MOSEI** for held-out sentiment, **MELD** for held-out emotion, **DAIC-WOZ** for held-out depression, and **MUStARD** for a novel-task sarcasm setting—followed by fine-tuning on each held-out dataset with a **minimal epoch budget** [2510.04899].

| Held-out dataset | Transfer score |
|---|---:|
| MOSEI (SEN) | 0.724 |
| MELD (EMO) | 0.711 |
| DAIC-WOZ (DEP) | 0.749 |
| MUStARD (SAR) | 0.658 |

Relative to Qwen 2.5-Omni-7B, the paper reports gains of **+18.3% (+0.112)** on MOSEI, **+3.95% (+0.027)** on MELD, **+29.4% (+0.17)** on DAIC-WOZ, and **+39.1% (+0.185)** on MUStARD. The MUStARD result is singled out as especially notable because sarcasm is excluded during pretraining; the authors argue that this suggests transfer not only across held-out datasets but also to **novel behavioral phenomena**. Their qualitative example contrasts Qwen2.5-Omni-7B, which defaults to literal “no sarcasm,” with OmniSapiens-7B SFT, which correctly identifies sarcasm in Chandler’s “balcony lights” remark. Quantitatively, Qwen2.5-Omni-7B predicts “no sarcasm” on **93.2%** of test samples with weighted recall **0.534**, whereas OmniSapiens-7B SFT attains weighted recall **0.670** and predicts “sarcasm” on **30.1%** of samples [2510.04899].

The paper also delineates several limitations. OmniSapiens-7B SFT is **not uniformly best across all tasks**; it is weaker on open-ended reasoning tasks such as intent recognition and social reasoning. Behavioral descriptors help selectively rather than universally, real-world deployment performance is not directly validated, and the work is “primarily empirical and benchmark-oriented” rather than deeply formalized. The broader challenge of subjective labels and heterogeneity across behavior datasets remains present. On ethics, the paper states that all datasets are **publicly available**, used within intended scope and licenses, and that the models are “not expected to generate harmful or inappropriate content,” with code, model checkpoints, and benchmark intended for release [2510.04899].

Subsequent literature makes the model’s historical position clearer. A later paper introducing **OmniSapiens-7B 2.0** via **HARPO** does **not** define or report a formal checkpoint called “OmniSapiens-7B SFT”; instead, it presents an RL-centered line built from **Qwen 2.5-Omni-7B**, compares against **OmniSapiens-7B RL**, and releases the HARPO-trained model [2602.10635]. This suggests that the label **OmniSapiens-7B SFT** is specific to the Human Behavior Atlas study. In a broader post-training context, later work on SFT-to-RL pipelines argues that high SFT scores can be misleading as predictors of subsequent RL performance; that work does not discuss OmniSapiens-7B specifically, but it is relevant when interpreting SFT checkpoints as possible starting points for later RL optimization [2510.01624].

Source: https://www.emergentmind.com/topics/omnisapiens-7b-sft