- The paper introduces Grounded Personality Reasoning and MM-OCEAN, a benchmark of 1,104 videos and 5,320 cue-grounding questions that tests ratings, explanations, and observable evidence together.
- The paper finds a 51.3% mean Prejudice Rate and just 10.4% Holistic-Grounding Rate across 27 MLLMs, showing that correct personality predictions frequently rely on unsupported impressions.
- The paper identifies fine-grained visual grounding as the main weakness, with spatial localization averaging 30.7% accuracy, while neuroticism is the hardest Big Five trait at 37.7% accuracy.
This paper introduces Grounded Personality Reasoning (GPR), a reformulation of apparent personality recognition (APR) that requires multimodal LLMs (MLLMs) to anchor each Big Five rating in observable behavioral evidence, and presents MM-OCEAN, a benchmark of 1,104 videos and 5,320 cue-grounding multiple-choice questions built through a multi-agent, human-collaborative annotation pipeline. Benchmarking 27 MLLMs under this framework reveals what the authors term the Prejudice Gap: across the field, 51% of correct personality ratings are not supported by retrievable behavioral cues, and the mean Holistic-Grounding Rate is only 10.4%. The central claim is that conventional rating-only evaluation systematically overestimates MLLM competence at personality assessment by crediting predictions that are correct for the wrong reason.
Existing APR benchmarks such as ChaLearn First Impressions frame personality inference as numerical regression on Big Five scores. This formulation cannot distinguish genuine perception from superficial pattern matching: a model may achieve low prediction error by exploiting shallow correlations (e.g., smiling implies high agreeableness) without recovering the supporting evidence. The authors ground their argument in person-perception research showing that accurate trait judgment rests on integrating specific micro-cues rather than gestalt impressions, and note that the EU AI Act classifies personality-based hiring and education systems as high-risk and mandates explainable evidence trails — giving the grounding requirement regulatory as well as scientific motivation.
GPR formalizes this requirement as a chain of three outputs: ordinal Big Five rating on a five-level scale (T1), open-ended reasoning that must cite timestamped atomic observations by OBS-ID (T2), and structured cue-grounding probes (T3). The grounding constraint — every trait judgment must cite at least one observed cue — is precisely what separates GPR from APR.
MM-OCEAN construction
The benchmark draws videos from ChaLearn First Impressions V2 (~10K fifteen-second single-speaker clips with crowd-sourced Big Five scores). A five-stage pipeline interleaves four LLM agents with human verification:
- Stage 1 (Observer + human): an LLM drafts atomic behavioral observations across four perceptual channels (Expression, Action, Audio, Background); 24 trained annotators verify each cue and refine timestamps and bounding boxes frame-accurately. Of 45,609 drafted cues, 78.2% were accepted, 14.6% corrected, 5.9% deleted; pairwise verdict agreement on an overlap pool was 77%.
- Stage 2 (Psychologist): produces per-trait analyses mapping ground-truth scores to five ordinal levels, each citing verified observations.
- Stage 3 (Examiner): generates seven cue-grounding MCQs per video spanning two clusters — a reasoning cluster (Personality Attribution, Counterfactual, Temporal-Causal, Mixed Emotion) and a visual-grounding cluster (Micro-expression, Spatial Localization, Temporal-Spatial Joint).
- Stage 4 (Aligner): deterministic checks plus LLM semantic review.
- Stage 5: a text-leakage filter drops any MCQ that two text-only LLMs answer correctly without video access, followed by expert review.
The released benchmark contains ~13.5K human-verified observations, 5,520 trait analyses, and 5,320 MCQs (4.8 per video after filtering). Distractors are designed to cover three failure modes: text-derivable answers, plausible-but-wrong-segment, and near-miss.
Evaluation framework
Beyond per-task accuracy, the paper defines four sample-level failure-mode rates obtained by binarizing success on each of the three axes: Prejudice Rate (PR; correct rating, incorrect cue retrieval), Confabulation Rate (CR; correct rating, incoherent reasoning), Integration-failure Rate (IR; correct cues, incorrect rating), and Holistic-Grounding Rate (HR; all three correct). A population-level Rating–Grounding Misalignment (RGM) score captures within-model rank disconnect between rating and downstream grounding. A 3×3×3 threshold sweep confirms HR ranking stability (ρ≥0.92 across all combinations).
Task 2 relies on an AI-as-Judge protocol (GPT-4o-mini scoring four dimensions: Evidence Coverage, Logical Coherence, Grounding Accuracy, Directional Accuracy). Cross-judge validation with Claude Haiku 4.5 and Gemini 2.5 Flash-Lite yields Spearman ρ≥0.92, and a confidently-wrong consistency check shows the judge penalizes samples where T1 was wrong uniformly across models (σΔ=0.27), suggesting it tracks correctness rather than style. The judge does share architectural lineage with two evaluated GPT models and inflates them by roughly +1 point, though rankings are preserved.
Main results: the Prejudice Gap
The headline finding is stark. Mean PR across 27 models is 51.3%, mean HR is 10.4%, and HR spans 0.0% (LLaVA-NeXT-Video, InternVL3-8B) to 33.5% (Gemini 3 Flash). A model achieving 50–56% rating accuracy would appear competent on a traditional leaderboard, yet typically exhibits PR of 40–87%. Even at the proprietary frontier, top-3 mean PR is ~14.5%; at the open-source frontier it is ~47.0%.
| Metric |
Top-3 Closed |
Top-3 Open |
Gap |
| T1 Accuracy (%) |
59.1 |
55.8 |
−5.6% |
| T2-Avg4 (1–10) |
6.63 |
6.39 |
−3.6% |
| T3 Accuracy (%) |
67.8 |
49.8 |
−26.6% |
The ecosystem gap is narrow for rating and explanation but widens sharply for cue retrieval, indicating that personality scoring and verbal reasoning have largely democratized while behavioral grounding has not.
Per-category analysis shows a stable difficulty hierarchy: Temporal-Causal Reasoning is easiest (64.8% mean), Spatial Localization (30.7%) and Micro-expression Localization (34.6%) hardest. The closed-source advantage concentrates almost entirely on the visual-grounding cluster (+19.5 pp on Spatial Localization, +21.8 pp on Temporal-Spatial Joint versus 6–11 pp on reasoning categories). Even the strongest closed model reaches only 57% on Spatial Localization, making fine-grained spatiotemporal grounding a benchmark-wide bottleneck.
HR proves highly discriminative: its coefficient of variation (~0.93) far exceeds any single-task metric (T1 ≈ 0.13, T3 ≈ 0.36), and informative rank inversions emerge — Gemma-4-31B-it ranks fifth by task mean but 13.5th by HR because its successes do not co-occur per video. RGM partitions models into two archetypes: Confident Raters (e.g., Llama-4-Maverick-FP8, RGM +14) rate well but fail downstream, while Cautious Reasoners (e.g., Gemini 2.5 Flash, RGM −16.5) ground well but rate poorly.
Auxiliary analyses add several notable patterns. Neuroticism is universally the hardest trait (37.7% mean accuracy, MAE 0.87), suggesting internal-state inference exceeds current capabilities on short clips. Open-source scaling plateaus past ~30B parameters on T3, with data quality mattering more than size. Positional-bias σ>10 reliably predicts bottom-third T3 performance (r≈−0.68), offering a cheap diagnostic. A worked example contrasts GPT-4o and Gemini 3 Flash producing near-identical plausible rationales for Low Extraversion, yet only one correctly localizes the anchoring behavioral window — the concrete signature of prejudice the benchmark targets.
Limitations and open questions
The paper concedes several constraints. MM-OCEAN covers apparent personality from short, single-speaker English clips inherited from First Impressions V2, whose labels carry cultural and linguistic bias and should not be treated as cross-culturally objective ground truth. Task 2 depends on a single AI-as-Judge, albeit with demonstrated robustness. Most importantly, the authors acknowledge that Task 3 operationalizes grounding as MCQ retrieval over a predefined cue set, so high PR may partly reflect MCQ design choices rather than general inability to ground judgments. The reasoning-capable versus non-reasoning comparison is explicitly observational, confounded by parameter count, family, and generation; a controlled matched-size experiment is left open. Whether the Prejudice Gap persists under richer grounding operationalizations beyond MCQs, or on multilingual and multi-speaker video, remains unanswered.
Conclusion
MM-OCEAN reframes personality assessment in MLLMs as a grounded-reasoning problem and supplies the annotations, tasks, and failure-mode metrics needed to evaluate it. Its central empirical result — that half of correct ratings lack retrievable evidential support, and full rating–reasoning–grounding success is rare even at the frontier — demonstrates that rating-only benchmarks overstate competence and identifies fine-grained spatiotemporal cue retrieval as the primary deficit separating current models from trustworthy social cognition.
