MTT-Bench: Predicting Social Dominance in Mice via Multimodal Large Language Models

Abstract: Understanding social dominance in animal behavior is critical for neuroscience and behavioral studies. In this work, we explore the capability of Multimodal LLMs(MLLMs) to analyze raw behavioral video of mice and predict their dominance hierarchy. We introduce MTT-Bench, a novel benchmark comprising annotated videos of pairwise mouse interactions for Mouse Tube Test analysis. Building on existing MLLM architectures, we fine-tune these models to perform zero-shot inference on unseen behavioral sequences, predicting social dominance without explicit labels during testing. Our framework demonstrates promising results, showing high agreement with tube test rankings. This work opens a new direction for applying foundation models to ethology and social behavior analysis, without the need to design domain-specific models.

• The paper introduces MTT-Bench, a benchmark leveraging multimodal LLMs to predict social dominance in mice using zero-shot inference and k-means clustering.
• It employs annotated tube test videos with extensive data augmentation to extract behavioral features through pretrained vision-language models.
• Results indicate that zero-shot MLLMs, such as InternVL3-2B with 76.14% accuracy, significantly outperform human and unsupervised baselines in behavioral prediction.

MTT-Bench: Predicting Social Dominance in Mice via Multimodal LLMs

Introduction

The paper "MTT-Bench: Predicting Social Dominance in Mice via Multimodal LLMs" (2604.22492) establishes a new benchmark for analyzing and predicting social dominance behaviors in mice using state-of-the-art multimodal LLMs (MLLMs). Exploiting data from annotated behavioral videos in the context of the mouse tube test, the authors construct the Mouse-Tube-Test-Benchmark (MTT-Bench), facilitating rigorous assessment of MLLMs for ethological and behavioral neuroscience applications. The approach diverges from conventional supervised or hand-engineered pipelines by leveraging pretrained vision-LLMs for both feature extraction and inference, targeting zero-shot or unsupervised generalization.

Dataset and Benchmark Construction

The MTT-Bench dataset was developed in collaboration with animal behavior experts and consists of annotated videos capturing mouse-mouse interactions during the tube test. Each experiment contains multiple groups, with each group having four training videos (to learn individual behavioral traits) and one or two test videos (to assess actual dominance outcomes). Data augmentation is pivotal due to the limited scale of the original recordings. The augmentation protocol involves flipping, grayscale adjustments, and rotation, as well as their mixed combinations, resulting in an eightfold increase in dataset size and supporting robust model evaluation.

Figure 1: Overview of the process of generating the Evaluated Dataset, including data collection and augmentation.

The design ensures realistic social context, high intra-group variability, and controls for confounding variables. MTT-Bench thus constitutes a challenging multimodal video-language dataset with well-aligned behavioral ground truth.

Prediction Methods

Two primary prediction paradigms are evaluated: (1) Zero-shot reasoning using MLLMs, and (2) unsupervised inference by k-means clustering on extracted embeddings. Additionally, a human-agent baseline is included to contextualize model performance.

Zero-Shot Learning Prediction

Zero-shot inference leverages the pretrained cross-modal alignment of MLLMs, formulating dominance prediction as a compositional visual reasoning problem. Three-step textual queries are posed:

1. Score individual dominance-related behavioral traits for each mouse (personality score between 0 and 1).
2. Given both scores, infer which mouse will win the tube test.

No explicit dominance labels are provided during inference, aligning with realistic annotation-limited setups. Semantic prompts guide the model to re-use general visual-linguistic knowledge for this ethological task.

K-means Clustering on Embeddings

K-means clustering is used as a baseline unsupervised approach. Feature vectors representing visual behavioral dynamics are extracted from the MLLMs' encoders. Clustering identifies prototypical behavioral signatures for "dominant" and "submissive" mice, based solely on the structure of the visual feature space. In the testing phase, proximity of new video embeddings to cluster centroids determines the inferred dominance.

Figure 2: The operation process of the two prediction methods: Zero-shot inference using MLLMs and K-means clustering on extracted embeddings.

Human Agent Baseline

A cohort of naive human observers was tasked to predict winners from training videos. This baseline quantifies the upper bound for inference without domain-specific training.

Experimental Analysis

Model Selection

The benchmark includes comprehensive evaluation across a range of leading multimodal models: BLIP-2, InternVL3 series (1B, 2B, 8B parameters), Gemini-1.5-Pro, and GPT-4o. GPT-4-turbo is included as a blind (text-only) baseline. Consistent frame sampling and input protocols are enforced to ensure fairness.

Quantitative Results

• InternVL3-2B using zero-shot achieves the highest average accuracy (76.14%), substantially outperforming human agents (51.44%) and all other models.
• InternVL3-1B and InternVL3-8B both surpass human and k-means (67.05% and 72.27%, respectively). Larger models, while in some cases more "conservative" in ambiguous contexts, maintain superior selectivity.
• BLIP-2 and Gemini-1.5-Pro yield accuracies near chance levels (44.32% and 51.14%).
• General LLMs (GPT-4o and GPT-4-turbo) are not competitive, reflecting modality alignment limitations.
• K-means clustering achieves 54.17%—slightly above humans but significantly below zero-shot MLLM performance.

These results indicate that even without fine-tuning, high-capacity multimodal models can internalize latent behavioral constructs from raw video and few-shot prompts. Notably, MLLMs outperform both naive unsupervised pipelines and non-specialist human observers.

Qualitative Observations

Larger models (InternVL3-8B) tend toward uncertainty or abstention in difficult cases, suggesting higher sensitivity to ambiguous frame-level cues. However, when discriminable behavioral signals exist, model outputs are robust and align well with experimental ground truth. The bottleneck for model performance in smaller architectures appears to be feature abstraction capacity rather than the training procedure.

Frame and Data Pipeline Efficiency

Zero-shot inference is limited to relatively few video frames (e.g., 32–64), driven by token limitations and sequence modeling costs. In contrast, the embedding-based clustering method scales to 256 frames/video and supports broader augmentation. Despite its lower accuracy, this makes it suitable for high-throughput behavioral analysis scenarios.

Theoretical and Practical Implications

The clear dominance of zero-shot MLLM approaches over hand-crafted and unsupervised methods signals that pretrained cross-modal models can serve as universal, interpretable behavioral analysis engines. This shifts the paradigm in ethology and neuroscience away from bespoke pipelines toward foundation models that generalize across species, tasks, and experimental protocols with minimal supervision.

Practically, rapid, label-free prediction of social hierarchy and latent behavioral phenotypes in animal studies becomes feasible, supporting scalable preclinical and neuropsychiatric research. The benchmark and methods also invite further integration of behavioral neuroscience and multimodal AI, e.g., for closed-loop experiments, automated welfare monitoring, or transfer to broader animal cognition studies.

Future Directions

Possible extensions illuminated by this study include:

• Incorporation of fine-tuning or domain-adaptive pretraining for enhanced model discrimination in ambiguous social scenarios.
• Extension to richer ethological contexts (e.g., group housing, long-term interaction, aggression/modulation studies).
• Fusion with high-dimensional sensory or neural data for causal inference across the brain-behavior axis.

Technical progress in input scalability (processing longer video sequences), explainable output, and robust transfer learning are expected to further refine performance and adoption of foundation models in behavioral research.

Conclusion

MTT-Bench (2604.22492) provides a rigorous, scalable testbed for assessing MLLMs on animal social dominance prediction from behavioral video. Results demonstrate that modern vision-LLMs, when equipped with zero-shot reasoning protocols, already surpass naive human and unsupervised methods by large margins for non-trivial ethological inference. The paradigm established here is poised to accelerate integration of AI in life sciences and motivates future work in multimodal, generalizable behavioral intelligence.