SVBench: Evaluation of Video Generation Models on Social Reasoning

Abstract: Recent text-to-video generation models exhibit remarkable progress in visual realism, motion fidelity, and text-video alignment, yet they remain fundamentally limited in their ability to generate socially coherent behavior. Unlike humans, who effortlessly infer intentions, beliefs, emotions, and social norms from brief visual cues, current models tend to render literal scenes without capturing the underlying causal or psychological logic. To systematically evaluate this gap, we introduce the first benchmark for social reasoning in video generation. Grounded in findings from developmental and social psychology, our benchmark organizes thirty classic social cognition paradigms into seven core dimensions, including mental-state inference, goal-directed action, joint attention, social coordination, prosocial behavior, social norms, and multi-agent strategy. To operationalize these paradigms, we develop a fully training-free agent-based pipeline that (i) distills the reasoning mechanism of each experiment, (ii) synthesizes diverse video-ready scenarios, (iii) enforces conceptual neutrality and difficulty control through cue-based critique, and (iv) evaluates generated videos using a high-capacity VLM judge across five interpretable dimensions of social reasoning. Using this framework, we conduct the first large-scale study across seven state-of-the-art video generation systems. Our results reveal substantial performance gaps: while modern models excel in surface-level plausibility, they systematically fail in intention recognition, belief reasoning, joint attention, and prosocial inference.

• The paper demonstrates that current LVGMs excel in visual realism but falter in generating socially plausible interactions.
• The benchmark uses experiments inspired by developmental psychology to assess intention recognition, joint attention, and agency detection in generated videos.
• Results indicate that explicit training and model architecture modifications are needed to improve socio-cognitive reasoning in video synthesis.

SVBench: Evaluation of Video Generation Models on Social Reasoning

Motivation and Context

Recent advances in Large Video Generation Models (LVGMs) have demonstrated remarkable progress in producing videos exhibiting photorealism and temporal coherence. However, existing benchmark suites such as FVD, VBench, Evalcrafter, and T2V-CompBench predominantly focus on metrics for visual quality, temporal consistency, physics, or semantic alignment, without systematically probing a model’s capacity for social reasoning. Social cognition, encompassing mentalizing, theory of mind, intention recognition, and other higher-order inter-agent interpretative faculties, is a key hallmark of intelligent behavior present in humans and some non-human animals. The absence of rigorous testing in this dimension leaves a gap in evaluating LVGMs for applications in social robotics, narrative generation, and video understanding. The SVBench benchmark directly addresses this deficit.

Benchmark Design and Structure

SVBench is designed to systematically assess the social reasoning abilities of video generation models. The benchmark draws inspiration from established developmental psychology paradigms such as the Heider-Simmel animation and theory-of-mind assessment protocols [Premack & Woodruff 1978; Baron-Cohen et al. 1985], adapting them for the evaluation of generative models. Tasks are constructed around animated scenarios and curated video prompts which require the model to demonstrate nuanced social reasoning, including detection of agency, goal attribution, intention inference, social norm adherence/violation, emotional contagion, and joint attention. The dataset embodies a range of agent behaviors, both artificial (e.g., geometric agents reminiscent of Heider-Simmel tests) and naturalistic, emulating interactions found in real-world social episodes.

The SVBench evaluation protocol involves not only visual generation from textual or multimodal prompts but also behavioral analysis of the generated videos. Automatic and human-in-the-loop assessments are provided for measuring whether the model artifacts exhibit correct social cues, agent-centric reasoning, and semantically plausible social dynamics.

Comparison with Prior Benchmarks

SVBench contrasts with other video generation benchmarks that have focused on physical and commonsense reasoning (e.g., Morpheus (Zhang et al., 3 Apr 2025), VBench++ (Huang et al., 2024), and the physical coherence suite (Chen et al., 8 Feb 2025)). Those primarily evaluate models’ consistency with physical laws, object permanence, or event plausibility and do not probe the ability to represent or reason about multi-agent interactions with latent psychological states. Furthermore, LLM benchmarks for theory-of-mind reasoning, such as OpenToM (Xu et al., 2024) and evaluations of LLMs as in (Kosinski, 2023, Ullman, 2023), cannot be directly applied to visual generative models due to the fundamentally different input/output modalities and reasoning demands.

SVBench thus fills a unique niche by bringing concepts from cognitive science and social psychology to the evaluation of LVGMs, opening an avenue for principled assessment of the synthetic agent’s "social intelligence."

Experimental Evaluation and Key Results

Extensive experiments are conducted with state-of-the-art LVGMs, including but not limited to Sora 2, Kling 2.5 Turbo, Veo 3.1, Hailuo, HunyuanVideo, and LongCat-Video. The benchmark demonstrates that, while these models have converged to high levels in visual realism and physical scene understanding, there are marked deficits in their capacity for generating socially plausible and goal-directed multi-agent interactions. In several canonical social reasoning tasks, models produce outputs lacking intentionality cues, agency-consistent motion, or appropriate synchrony in social actions.

Particularly salient failure cases are highlighted in scenarios requiring implicit knowledge of social rules (e.g., turn-taking, queuing, fairness), inference of hidden intentions, and emotional state mirroring. Quantitative results show that models significantly underperform human baselines and are highly sensitive to prompt variations, indicating that current architectures lack robust, compositional social reasoning faculties. Notably, for tasks requiring joint attention or cooperative problem solving, even the best models exhibit high rates of implausible or socially incoherent outputs. These discrepancies underline a limitation in how existing diffusion-based and autoregressive architectures capture distributed agent-centric reasoning.

Theoretical and Practical Implications

The findings from SVBench have considerable implications for modeling social cognition in generative architectures. The pronounced gap in social reasoning points to the inadequacy of current training protocols and data curation, which typically emphasize visually diverse but socially shallow datasets. From a theoretical standpoint, the SVBench results suggest that mere exposure to large-scale video corpora, even with human interaction, is insufficient for imbuing models with implicit social knowledge. Dedicated pretraining or auxiliary losses targeting agent-centric representations, as well as explicit incorporation of social cognitive priors (e.g., from developmental psychology), may be necessary to close this gap.

On the practical side, these findings highlight risks in deploying LVGMs for applications where accurate modeling of human intentions, social affordances, or multi-agent behavioral synchrony is required—such as autonomous video creation, assistive robotics, or simulation-based planning in social environments.

Future Directions

The introduction of SVBench sets the stage for several research directions:

• Architectural Augmentation: Development of model architectures with explicit agent-centric state tracking and multi-agent interaction modules.
• Enhanced Pretraining Protocols: Incorporation of curated social interaction datasets, including behavioral and psychological supervision signals, during LVGM pretraining.
• Causal and Compositional Reasoning: Integration of explicit mechanisms for causal inference regarding agent beliefs, desires, and intentions (akin to "mentalizing" systems in cognitive neuroscience).
• Evaluation Metrics: Formulation of more refined automatic metrics for social coherence, possibly derived from advances in video-based VQA for social reasoning or modification of currently available frameworks such as EvalCrafter.

Conclusion

SVBench constitutes a critical advancement in the evaluation of video generative models, explicitly targeting their social reasoning abilities. The evidence it provides of current LVGMs’ deficits in this area both underscores the challenges ahead and highlights the urgent need for new approaches in model design and training. The benchmark’s adoption will steer future development towards models capable of more robust, explainable, and human-aligned video synthesis, strengthening the bridge between generative AI and the study of social intelligence.