OmniSapiens-7B SFT: Multimodal Behavior Benchmark
- The paper demonstrates that OmniSapiens-7B SFT is a foundational supervised model fine-tuned on the Human Behavior Atlas to unify diverse behavioral tasks.
- The model processes audio, visual, and textual inputs by mapping them into a shared embedding space for both structured classification and open-ended generation.
- Empirical results show notable gains in emotion, sentiment, and pathology detection, although open-ended tasks such as intent recognition remain challenging.
OmniSapiens-7B SFT is a 7B-parameter multimodal model for unified psychological and social behavior understanding, introduced as the principal supervised baseline in "Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understanding" (Ong et al., 6 Oct 2025). In that work, SFT denotes supervised fine-tuning: a pretrained multimodal LLM backbone is fine-tuned on a unified benchmark of behavioral tasks using a standardized prompt–target interface. The model is intended less as a novel architectural proposal than as a benchmarked, "vanilla" instance for testing whether psychological and social behavior understanding can be trained as a single multimodal multitask problem spanning affective states, cognitive states, pathology, and social processes (Ong et al., 6 Oct 2025).
1. Position within the OmniSapiens family
Within the Human Behavior Atlas study, OmniSapiens-7B SFT is the foundational member of a three-model family. The paper distinguishes OmniSapiens-7B SFT, OmniSapiens-7B BAM, and OmniSapiens-7B RL. SFT is the base supervised fine-tuned model; BAM is the SFT model augmented after training with a Behavioral Adapter Module that injects behavioral descriptors through a residual adapter while keeping the SFT backbone frozen; RL is a decoder-only variant optimized with GRPO-style reinforcement learning over unified free-text outputs (Ong et al., 6 Oct 2025).
The authors explicitly state that these variants are “meant as vanilla instances rather than new methods, intended primarily to investigate Human Behavior Atlas.” This framing is important. OmniSapiens-7B SFT is not presented as a behavior-specific architecture in the narrow sense; rather, it is an empirical test of whether multimodal instruction-style fine-tuning on a unified behavioral benchmark yields a broadly useful model for sentiment, emotion, pathology-related detection, social reasoning, intent recognition, humor, sarcasm, and non-verbal communication (Ong et al., 6 Oct 2025).
The practical division of labor across the three variants is also explicit. SFT is described as strongest and most stable on structured classification tasks. BAM selectively improves SFT on tasks benefiting from explicit descriptors. RL is relatively better on open-ended generation and reasoning tasks such as intent and social reasoning. This division makes OmniSapiens-7B SFT the canonical reference point for the family’s supervised behavior-understanding capabilities (Ong et al., 6 Oct 2025).
2. Architectural basis and multimodal formulation
OmniSapiens-7B SFT is initialized from Qwen2.5-Omni-7B, described as “a pretrained multimodal LLM ... which includes an audio and image encoder, alongside an LLM” (Ong et al., 6 Oct 2025). At a high level, the architecture comprises a pretrained audio encoder, a pretrained image encoder, an LLM backbone, a projection stage that maps raw multimodal inputs into a shared embedding space, task-specific classifier heads for classification tasks, and a decoder head for open-ended text response tasks.
The paper summarizes the fusion strategy in a single sentence: “We project raw audio, visual, and textual inputs into a shared embedding space, enabling the LLM backbone to process multimodal benchmark samples.” The supported raw modalities for OmniSapiens-7B SFT are therefore text, audio, and visual/video, with visual information handled through the backbone’s image encoder. Although Human Behavior Atlas also includes extracted behavioral descriptors as auxiliary features, those descriptors are not integrated directly into the SFT model’s core path (Ong et al., 6 Oct 2025).
Task unification is implemented through a standardized prompt–target schema aligned with multimodal LLMs such as Qwen2.5-Omni. Prompts explicitly reference available modalities with tokens such as <video> and <audio>, incorporate transcript text when available, and pose a behavior-specific question in natural language. Targets are standardized either as discrete label sets for classification or as free-text answers for open-ended tasks. The paper’s MELD sentiment example follows exactly this pattern, combining video, audio, transcript text, and a natural-language instruction asking for a sentiment label (Ong et al., 6 Oct 2025).
The output layer is hybrid rather than purely generative. For categorical tasks, the model uses classifier heads fed by pooled hidden representations taken from , “the penultimate layer immediately before the final hidden states.” For free-text inference, “a separate decoder head generates outputs directly from the final hidden states after .” The losses are correspondingly split between cross-entropy loss for classifier heads and teacher forcing for decoder training. The main text does not provide a full layer-by-layer diagram for OmniSapiens-7B SFT, nor does it formalize tokenization, multimodal fusion, or the supervised objective beyond these descriptive statements (Ong et al., 6 Oct 2025).
3. Human Behavior Atlas and benchmark unification
Human Behavior Atlas is the benchmark on which OmniSapiens-7B SFT is trained and evaluated. It was created to address fragmentation in prior work, which the paper characterizes as dataset-specific, task-specific, and inconsistent in input format, labels, and evaluation. The benchmark contains 101,964 unified and standardized samples spanning 35,046 videos, 10,287 audio clips, and 25,385 transcripts, and is built from 13 publicly available multimodal datasets. The paper also reports that 83.6% of samples contain video data and that 29.2% of video/audio clips last more than 20 seconds (Ong et al., 6 Oct 2025).
The benchmark organizes behavioral understanding into four high-level dimensions—Affective states (Aff), Cognitive states (Cog), Pathology (Path), and Social processes (Soc)—and covers ten behavioral tasks.
| Task | Representative datasets |
|---|---|
| SEN | CMU-MOSEI, MELD, CH-SIMSv2 |
| EMO | CMU-MOSEI, MELD, TESS, CREMA-D |
| SOC | Social-IQ 2.0 |
| INT | IntentQA |
| NVC | MimeQA |
| HUM | UR-FUNNYv2 |
| SAR | MUStARD |
| ANX | MMPsy |
| DEP | DAIC-WOZ, MMPsy |
| PTSD | PTSD-in-the-Wild |
The underlying datasets are explicitly enumerated in the paper: CMU-MOSEI (31,454 samples), MELD (27,412), TESS (2,800), CREMA-D (7,442), CH-SIMSv2 (4,403), Social-IQ 2.0 (6,437), IntentQA (16,297), MimeQA (806), UR-FUNNYv2 (2,125), MUStARD (690), DAIC-WOZ (189), MMPsy (1,275), and PTSD-in-the-Wild (634). The benchmark’s scale and heterogeneity are central to the rationale for a unified model (Ong et al., 6 Oct 2025).
A substantial part of the contribution lies in preprocessing and harmonization. Continuous labels are discretized where required; the example given is that PHQ-9 scores in MMPsy are discretized into categories based on original guidelines. Emotion labels are merged when they represent the same underlying emotion, such as “joy” and “happiness,” and split when the original label is intrinsically heterogeneous, such as dividing “surprise” into “positive surprise” and “negative surprise.” For sentiment, evaluation is normalized to binary weighted F1 over positive and negative labels. When transcripts are missing, text is extracted from audio using the Whisper v3 Large model (Ong et al., 6 Oct 2025).
Human Behavior Atlas also includes behavioral descriptors extracted from MediaPipe for visual signals—facial landmarks and body pose keypoints—and from OpenSMILE, ComParE 2016 for audio signals, including pitch, energy, spectral properties, and voice quality. For OmniSapiens-7B SFT, however, these descriptors are auxiliary benchmark assets rather than part of the model’s primary input path. This suggests that the benchmark is deliberately designed to support multiple regimes of behavioral modeling, including but not limited to the baseline supervised model (Ong et al., 6 Oct 2025).
4. Supervised training regime
OmniSapiens-7B SFT is trained in a joint multitask setup across all behavioral tasks in Human Behavior Atlas. The paper states that “a single instance of the model is trained jointly across all behavioral tasks in the benchmark.” In the standard setting, SFT therefore uses the benchmark’s multitask training split across all included datasets and tasks, rather than separate per-task fine-tuning (Ong et al., 6 Oct 2025).
The training objective mixes classification and generation supervision. Classification tasks are handled by task-specific classifier heads trained with cross-entropy loss, while open-ended tasks are trained through a decoder head with teacher forcing. The behavioral understanding problem is framed through prompt unification rather than through a bespoke architecture or a formal multi-objective derivation. The paper is explicit that, for OmniSapiens-7B SFT specifically, there are no explicit LaTeX equations defining the supervised fine-tuning objective, multimodal tokenization or encoding, task-unification loss, or descriptor conditioning (Ong et al., 6 Oct 2025).
Several implementation details that would normally be expected in a training recipe are not given in the supplied main text. These include learning rate, optimizer, batch size, sequence length, number of epochs or steps, sampling or mixing ratios, modality balancing strategy, hardware count or type, and curriculum schedule. The only hardware-related acknowledgment is that Nvidia provided GPU and compute resources. As a result, the published description of OmniSapiens-7B SFT is richer in benchmark construction and empirical findings than in optimization internals (Ong et al., 6 Oct 2025).
This omission matters for interpretation. The model is presented primarily as an empirical baseline demonstrating the consequences of benchmark unification. A plausible implication is that the benchmark’s prompt normalization and task harmonization are treated as the key controlled intervention, while fine-grained optimization design is relegated to appendices rather than foregrounded as a methodological novelty (Ong et al., 6 Oct 2025).
5. Empirical performance profile
On Human Behavior Atlas, OmniSapiens-7B SFT is evaluated against Gemma-3-4B, HumanOmniV2-7B, Qwen 2.5-Omni-7B, and Qwen-2.5-VL-7B. The paper states that OmniSapiens-7B SFT and OmniSapiens-7B BAM each outperform the general multimodal LLMs on 8 of 10 behavioral tasks, while OmniSapiens-7B RL does so on 7 of 10. This establishes SFT as the main evidence that unified training on Human Behavior Atlas yields broad improvements over generic multimodal backbones (Ong et al., 6 Oct 2025).
| Dataset / task | OmniSapiens-7B SFT score |
|---|---|
| CREMA-D (EMO) | 0.542 |
| MELD (E) | 0.709 |
| MOSEI (E) | 0.614 |
| TESS (EMO) | 0.658 |
| UR-FUNNY (HUM) | 0.532 |
| IntentQA (INT) | 0.256 |
| PTSD_WILD (PTSD) | 1.00 |
| MMPSY (A) | 0.909 |
| MMPSY (D) | 0.839 |
| DAIC-WOZ (DEP) | 0.626 |
| MELD (S) | 0.746 |
| CH-SIMSv2 (SEN) | 0.813 |
| MOSEI (S) | 0.744 |
| MUStARD (SAR) | 0.624 |
| Social-IQ 2.0 (SOC) | 0.257 |
| MimeQA (NVC) | 0.121 |
The strongest parts of the SFT profile are emotion recognition, sentiment, and several forms of pathology detection. The model improves over Qwen 2.5-Omni-7B on all four emotion datasets listed in the table, attains 1.00 on PTSD_WILD, and reaches 0.909 on MMPSY (A) and 0.839 on MMPSY (D). Its sentiment results—0.746 on MELD (S), 0.813 on CH-SIMSv2, and 0.744 on MOSEI (S)—also exceed Qwen 2.5-Omni-7B on all three corresponding datasets (Ong et al., 6 Oct 2025).
The weaker parts of the profile are intent recognition, social reasoning, and humor detection, with more modest results on non-verbal communication and, relative to BAM, on sarcasm detection. The paper interprets this as evidence that supervised fine-tuning is especially effective for structured classification tasks, whereas explicit descriptor injection or RL-style reasoning may be more useful for tasks requiring subtle cue integration or open-ended inference. Quantitatively, RL is much stronger on IntentQA (0.486 versus 0.256), while BAM yields major gains on MUStARD (0.795 versus 0.624) and MimeQA (0.162 versus 0.121) (Ong et al., 6 Oct 2025).
These comparisons clarify the role of SFT inside the OmniSapiens family. It is not uniformly dominant, but it is the most stable supervised baseline and the principal demonstration that a unified multimodal behavior benchmark can convert a general-purpose pretrained multimodal LLM into a broadly effective behavior-understanding system (Ong et al., 6 Oct 2025).
6. Transfer, limitations, and later interpretation
The paper also reports held-out transfer experiments in which four datasets are excluded during multitask pretraining of OmniSapiens-7B SFT—MOSEI for held-out sentiment, MELD for held-out emotion, DAIC-WOZ for held-out depression, and MUStARD for a novel-task sarcasm setting—followed by fine-tuning on each held-out dataset with a minimal epoch budget (Ong et al., 6 Oct 2025).
| Held-out dataset | Transfer score |
|---|---|
| MOSEI (SEN) | 0.724 |
| MELD (EMO) | 0.711 |
| DAIC-WOZ (DEP) | 0.749 |
| MUStARD (SAR) | 0.658 |
Relative to Qwen 2.5-Omni-7B, the paper reports gains of +18.3% (+0.112) on MOSEI, +3.95% (+0.027) on MELD, +29.4% (+0.17) on DAIC-WOZ, and +39.1% (+0.185) on MUStARD. The MUStARD result is singled out as especially notable because sarcasm is excluded during pretraining; the authors argue that this suggests transfer not only across held-out datasets but also to novel behavioral phenomena. Their qualitative example contrasts Qwen2.5-Omni-7B, which defaults to literal “no sarcasm,” with OmniSapiens-7B SFT, which correctly identifies sarcasm in Chandler’s “balcony lights” remark. Quantitatively, Qwen2.5-Omni-7B predicts “no sarcasm” on 93.2% of test samples with weighted recall 0.534, whereas OmniSapiens-7B SFT attains weighted recall 0.670 and predicts “sarcasm” on 30.1% of samples (Ong et al., 6 Oct 2025).
The paper also delineates several limitations. OmniSapiens-7B SFT is not uniformly best across all tasks; it is weaker on open-ended reasoning tasks such as intent recognition and social reasoning. Behavioral descriptors help selectively rather than universally, real-world deployment performance is not directly validated, and the work is “primarily empirical and benchmark-oriented” rather than deeply formalized. The broader challenge of subjective labels and heterogeneity across behavior datasets remains present. On ethics, the paper states that all datasets are publicly available, used within intended scope and licenses, and that the models are “not expected to generate harmful or inappropriate content,” with code, model checkpoints, and benchmark intended for release (Ong et al., 6 Oct 2025).
Subsequent literature makes the model’s historical position clearer. A later paper introducing OmniSapiens-7B 2.0 via HARPO does not define or report a formal checkpoint called “OmniSapiens-7B SFT”; instead, it presents an RL-centered line built from Qwen 2.5-Omni-7B, compares against OmniSapiens-7B RL, and releases the HARPO-trained model (Ong et al., 11 Feb 2026). This suggests that the label OmniSapiens-7B SFT is specific to the Human Behavior Atlas study. In a broader post-training context, later work on SFT-to-RL pipelines argues that high SFT scores can be misleading as predictors of subsequent RL performance; that work does not discuss OmniSapiens-7B specifically, but it is relevant when interpreting SFT checkpoints as possible starting points for later RL optimization (Kang et al., 2 Oct 2025).