WOLF Benchmark: Video, Deception & Locomotion

• WOLF Benchmark is a collection of three distinct systems that assess free-form video captioning, multi-agent deception, and instruction-driven humanoid locomotion.
• Each system leverages structured annotation protocols and precise logging to ensure reproducibility across diverse experimental regimes.
• Novel evaluation metrics, including CapScore, deception classification, and kinematic error, establish new standards for performance in their respective domains.

The designation WOLF Benchmark refers to three distinct benchmark-oriented systems in recent arXiv literature: Wolf, a WOrLd summarization Framework for dense video captioning; WOLF, Werewolf-based Observations for LLM Deception and Falsehoods; and WOLF-VLA, the Whole-Body Humanoid Optimal Locomotion Framework for Vision-Language-Action Learning (Li et al., 2024, Agarwal et al., 9 Dec 2025, Boukheddimi et al., 24 Jun 2026). Although they share an acronym, they target different problem classes—free-form video captioning, adversarial multi-agent reasoning, and instruction-conditioned humanoid locomotion—and each defines its own datasets, evaluation protocol, and benchmark semantics.

1. Nomenclature, scope, and benchmark semantics

In the video-captioning work, Wolf is presented as an automated captioning framework that adopts a mixture-of-experts approach, leveraging complementary strengths of Vision LLMs, and it establishes the first public leaderboard for free-form video captioning (Li et al., 2024). In the deception work, WOLF is a fully-specified, programmable benchmark built on the social deduction game Werewolf, with a strict night–day state-machine, per-statement deception labels, and longitudinal suspicion tracking (Agarwal et al., 9 Dec 2025). In the locomotion work, WOLF-VLA is a unified pipeline that synthesizes large-scale, dynamically feasible humanoid locomotion demonstrations via optimal control and uses those demonstrations to train Vision-Language-Action policies, together with a standardized simulation suite and metrics (Boukheddimi et al., 24 Jun 2026).

A common source of confusion is that the acronym does not denote a single benchmark family with a shared task definition. The three systems differ at the level of observables, supervision, and metric design. Wolf for captioning evaluates free-form text against a human-validated reference caption per segment; WOLF for deception evaluates statements, self-assessed honesty, peer flags, and smoothed suspicion trajectories; WOLF-VLA evaluates embodied control under natural-language instructions, ego-centric vision, and proprioceptive state. This suggests that the acronym functions as a naming collision rather than a unified research program.

2. Wolf for dense video captioning

Wolf is designed to measure progress on three interrelated video-captioning tasks: Dense Captioning, Temporal Localization, and Domain Splits (Li et al., 2024). Dense Captioning requires a long, detailed, free-form text description for a short video clip of 5–20 s, with arbitrarily long captions rather than one sentence per video. Temporal Localization is built into the benchmark by splitting each video into fixed-length temporal segments: 5 s for driving and 10–20 s for Pexels and robotics. Domain robustness is probed through three domains—autonomous driving with Interactive and Normal splits, general daily scenes in Pexels, and robot manipulation—with held-out test sets on each.

The benchmark’s human-annotated data comprise four subsets totaling approximately 25.7 h of video. The Driving–Interactive subset contains 500 clips of 5 s each, selected for complex maneuvers such as overtaking and yielding. The Driving–Normal subset contains 4,785 clips of 5 s, representing routine driving. The Pexels subset contains 473 high-quality videos of 10 s–2 min, covering 15 categories including Travel, Sports, Pets, People, News, and Music; each is split into 10–20 s clips. The Robotics subset contains 100 videos of 5 s–60 s, drawn from open-source robotics benchmarks, with captions focused on object-robot interactions such as pick/place and push (Li et al., 2024).

The driving annotation pipeline is unusually structured. It uses agent-level motion from NuScenes 3D bounding boxes and map topology, ego-centric interaction represented through homotopy $(<S,CW,CCW>)$ and lane-mode $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$, and then GPT-3.5 rewriting of heuristic summaries into fluent captions. This makes the benchmark simultaneously a captioning dataset and a data-alignment resource for scenario-rich video understanding.

Evaluation is defined per temporal segment. Interactive and Normal driving videos are sampled at 2 fps; robotics at 1 fps; Pexels at 2–5 fps depending on length. Each segment has one human-validated caption, and every predicted caption is scored against that gold reference. The benchmark is therefore dense in both the temporal and descriptive senses: it measures caption quality at segment resolution rather than only at whole-video resolution (Li et al., 2024).

3. World summarization, CapScore, and leaderboard behavior

Wolf’s core modeling idea is a mixture-of-experts world summarization framework combining image-level and video-level VLMs (Li et al., 2024). The expert sets are written as

$$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$

with example image experts $M_1=$ CogAgent and $M_2=$ GPT-4V, and example video experts $V_1=$ VILA-1.5 and $V_2=$ Gemini-Pro-1.5. For a video split into sampled frames $f_1,\dots,f_T$, each image model performs a chain-of-thought-like frame progression: $$h_{k,1}=M_k(f_1),\quad h_{k,t}=M_k\bigl(f_t,\;h_{k,t-1}\bigr)\quad (t\ge2).$$ Each video expert produces a caption $v_\ell(V)$ from the full video, and the partial captions are then aggregated by GPT-4: $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$0 The resulting caption $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$1 is the final world summary. The stated purpose of this summarization stage is to cross-check image and video expert outputs to reduce redundancy and hallucination.

Caption quality is evaluated with CapScore, an LLM-based metric that produces two scalar scores in $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$2 for each candidate caption $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$3 against ground truth $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$4: a similarity-wise score $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$5 for semantic alignment and a quality-wise score $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$6 that penalizes hallucination and factual errors. Formally,

$$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$7

For a batch of $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$8 captions, GPT-4 is prompted to score which caption is closest to the ground truth and which has fewer hallucinations, and no additional training is required because GPT-4 acts as a zero-shot evaluator (Li et al., 2024).

The benchmark ranks methods by average CapScore across test splits. Baselines are divided into image-level methods—CogAgent and GPT-4V over sampled frames—and video-level methods—VILA-1.5 and Gemini-Pro-1.5. On Highly Interactive Driving, Wolf reports Similarity 0.55 and Quality 0.56, exceeding CogAgent $$(\text{LEFT}/\text{RIGHT}/\text{AHEAD}/\text{BEHIND}/\text{NOTON})$$9, GPT-4V $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$0, VILA-1.5 $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$1, and Gemini-Pro-1.5 $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$2. On Pexels, Wolf reports 0.88 similarity and 0.89 quality; on Robotics, 0.72 similarity and 0.75 quality. On Normal Driving, Wolf (w/ CogAgent+VILA) reports 0.56 similarity and 0.60 quality, versus 0.27/0.30 for CogAgent and 0.35/0.39 for VILA-1.5 (Li et al., 2024).

The paper also states a concrete comparison against GPT-4V on challenging driving videos: Wolf improves CapScore quality-wise by 55.6% and similarity-wise by 77.4%. Its stated design goals are high-quality data alignment and a new standard for free-form caption evaluation, while open challenges include scarcity of long-form video caption data, LLM verbosity, domain-specific irrelevancies, and lack of per-token confidence. Future directions include task-aware captioning, advanced metrics, uncertainty quantification via conformal prediction, and efficiency through batched GPU inference and 4-bit quantization of constituent VLMs and VLAs (Li et al., 2024).

4. WOLF as a deception benchmark for multi-agent LLMs

The 2025 WOLF benchmark formalizes deception evaluation as a multi-agent social deduction benchmark based on Werewolf (Agarwal et al., 9 Dec 2025). It uses Players $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$3 with fixed role assignment

$$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$4

equivalently with role probabilities determined by $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$5, $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$6, $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$7, and $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$8. The game state at round $$\mathcal{M}^{\rm img}=\{M_1,\dots,M_{K}\},\qquad \mathcal{M}^{\rm vid}=\{V_1,\dots,V_{L}\},$$9 is

$M_1=$0

where $M_1=$1, $M_1=$2, memory stores past statements, votes, and assessments, and $M_1=$3 is a matrix $M_1=$4 describing observer $M_1=$5’s smoothed suspicion of target $M_1=$6.

The night–day dynamics are explicitly specified. At night, Werewolves choose a kill target $M_1=$7, the Doctor selects a protect target $M_1=$8, and the Seer inspects a player $M_1=$9. If $M_2=$0, the target is removed from the alive set; otherwise no elimination occurs. During the day, the system runs a fixed number of debate turns $M_2=$1, after which players vote. If some player receives a strict majority,

$M_2=$2

that player is exiled; otherwise no exile occurs. The system cycles until one of two win conditions is met: the Villagers win if all Werewolves are eliminated, and the Werewolves win if $M_2=$3 (Agarwal et al., 9 Dec 2025).

Debate itself is structured through bidding. On each day-turn, every alive player bids $M_2=$4; speakers are sorted by decreasing bid, with ties broken by mention priority from the prior turn; the selected speaker then produces a statement. Every public statement is an explicit analysis unit. The speaker provides a self-assessed deception type and confidence, while peers provide a binary deception flag and a continuous suspicion score. This makes the benchmark separable across deception production and deception detection, rather than collapsing both into a single label.

The benchmark uses a four-part deception taxonomy plus a non-deceptive class: $M_2=$5 The formal definitions distinguish withholding relevant facts, altering true facts, redirecting focus, and introducing content with no basis in truth. Metrics include standard classification quantities

$M_2=$6

as well as Brier Score, ROC AUC, and AUPRC from continuous suspicion scores. Longitudinal trust dynamics are modeled by exponential smoothing: $M_2=$7

Empirically, the benchmark reports 100 full LLM-driven games on an NVIDIA A100 GPU (40 GB) + 64 CPU cores, yielding approximately 7,320 total statements and an average of 32.4 statements/game with $M_2=$8. Werewolves self-reported deception in 31% of debate turns, while the overall self-labeled deception rate across all roles is reported as approximately 69.4%. Peer detection achieves 71–73% precision, 48–61% recall, approximately 0.52 overall accuracy, and $M_2=$9. Suspicion toward Werewolves rises from approximately 52% to approximately 62% by the final round, while Villagers and the Doctor remain stable at approximately 44–46%. Calibration is weak by ROC AUC (0.55–0.58) and Brier score (0.26–0.29), but AUPRC is reported as approximately 0.75. By deception type, omission has Count = 129, Avg. suspicion = 61.9%, and flagged = 57.5%, while fabrication has Count = 28, Avg. suspicion = 51.7%, and flagged = 40.1% (Agarwal et al., 9 Dec 2025).

The benchmark emphasizes reproducibility through NDJSON streams recording bids, debate turns, votes, self- and peer-annotations, and state snapshots. It also logs private scratchpads and chain-of-thought for audit without contaminating public prompts, and exposes a programmable interface through LangGraph state-machine APIs. This establishes WOLF as a benchmark not only for accuracy but also for auditability and longitudinal social inference (Agarwal et al., 9 Dec 2025).

5. WOLF-VLA for whole-body humanoid locomotion

WOLF-VLA defines the benchmark problem as whole-body humanoid locomotion from natural-language instructions and ego-centric vision, trained on demonstrations synthesized by optimal control (Boukheddimi et al., 24 Jun 2026). Its stated purpose is to address data scarcity, the lack of dynamically consistent demonstrations, and the difficulty of encoding optimality and safety in learning-based pipelines. By construction, the optimal-control trajectories are described as energy-efficient, smooth, contact-coherent, and satisfying joint position, velocity, and torque limits.

The dataset contains 15,276 episodes, corresponding to approximately 277 h of motion, with an average episode duration of 28 s recorded at 33.33 Hz. The benchmark reports six locomotion-related task families: forward walking toward a target (WF), lateral locomotion toward a target (WA), stair ascent only (W.CS.U), stair ascent + descent (W.CS.U/D), 180° turning motions, and variable-height squatting. Episode counts are reported for the first four families: 2,874 WF episodes with 13.5 s average length, 8,234 WA episodes with 43.2 s, 2,358 W.CS.U episodes with 21.6 s, and 1,810 W.CS.U/D episodes with 33.6 s. Environmental variation is introduced through target types including box, cylinder, sphere, ground marker, single 3-step staircase, and double staircase, 6 distinct colors per target, systematic 40×40 grid placements along the X–Y plane, and random non-target distractors (Boukheddimi et al., 24 Jun 2026).

The motion synthesis backbone is an optimal-control formulation under contact dynamics: $V_1=$0 The trajectory optimization problem is written as

$V_1=$1

subject to admissible sets $V_1=$2 and the contact dynamics. The multi-phase OCP uses a per-phase cost

$V_1=$3

including CoM tracking, foot position, torque minimization, and posture regularization. The implementation uses Crocoddyl with Box-FDDP and multiple shooting, together with Pinocchio for efficient dynamics and derivative computation. Torque limits are enforced via box constraints, joint limits are encoded in $V_1=$4, and each motion is manually validated to remove OCP failures (Boukheddimi et al., 24 Jun 2026).

The learning pipeline trains a VLA policy with a visual encoder $V_1=$5, a language encoder $V_1=$6, and an action decoder $V_1=$7. Language instructions include structured spatial and height tags, written as $V_1=$8 and $V_1=$9. The action head uses a diffusion-style flow-matching objective. With $V_2=$0 and $V_2=$1, and defining $V_2=$2, training minimizes

$V_2=$3

and inference proceeds through $V_2=$4 denoising steps

$V_2=$5

The backbone is initialized from GR00T-N1.5-3B; the vision and language encoders are frozen, while the action-diffusion head and projection layers are trained using 4×A100 GPUs, 200,000 gradient steps, batch size 128, AdamW with $V_2=$6, $V_2=$7, $V_2=$8, weight decay $V_2=$9, warmup 500 steps to $f_1,\dots,f_T$0, cosine decay to $f_1,\dots,f_T$1, bfloat16, and gradient clipping norm 10 (Boukheddimi et al., 24 Jun 2026).

Benchmark evaluation is carried out in a Gymnasium wrapper + MuJoCo physics environment using the RH5 humanoid with 25 actuated joints + free-flyer. Observations include proprioceptive state $f_1,\dots,f_T$2, 224×224 RGB head-mounted vision with FOV = 120°, and a natural-language instruction. The action space is $f_1,\dots,f_T$3. Metrics are Success Rate (%) over 20 rollouts per task, Soft Success Rate (SSR) for stair tasks,

$f_1,\dots,f_T$4

and Range-of-Motion error $f_1,\dots,f_T$5, defined from

$f_1,\dots,f_T$6

Quantitatively, the reported Groot-N1 model achieves 99% average success on WF, 27% on WA, 51% SSR on W.CS.U, 44% SSR on W.CS.U/D, and 55.3% average success over all tasks. $f_1,\dots,f_T$7 is approximately 1%–2% for hip, knee, and ankle on several tasks, but reaches up to 27% under disturbance for side walking. Baselines are substantially weaker: ACT-1 reports 1.4% overall success, and $f_1,\dots,f_T$8 reports 0% success. Ablations show No vision: 11% average success, No language: 38%, No spatial tags: 42%, and Instruction paraphrasing: 51%, while distractor analysis shows that simple forward walking remains above 95% success under distractors but more complex tasks drop by 10–40% (Boukheddimi et al., 24 Jun 2026).

The reported limitations are also benchmark-relevant: the dataset currently omits mixed loco-manipulation tasks, episodes for turning and squatting are not yet benchmarked in evaluation tables, and real-world sim-to-real transfer is not demonstrated. Future directions include slope walking, obstacle negotiation, loco-manipulation, richer photorealistic rendering or domain randomization, trajectory retargeting across humanoid embodiments, and integration with MPC-based residual RL (Boukheddimi et al., 24 Jun 2026).

6. Comparative interpretation and recurrent design themes

Across these three systems, the term benchmark denotes three different experimental regimes. In Wolf for video captioning, the benchmark centers on human-validated gold captions, segment-level evaluation, an LLM-based metric, and a public leaderboard (Li et al., 2024). In WOLF for deception, the benchmark is a controlled adversarial interaction protocol with a formal game state, per-statement labeling, longitudinal smoothing, and reproducible logs (Agarwal et al., 9 Dec 2025). In WOLF-VLA, the benchmark is a simulation suite plus dataset for embodied control, grounded in contact-consistent optimal-control demonstrations and evaluated through task success, stair progress, and kinematic similarity to reference motion (Boukheddimi et al., 24 Jun 2026).

Despite these differences, several design motifs recur. Each benchmark builds around structured supervision rather than weakly specified outcomes: Wolf uses one human-validated caption per segment; WOLF records self- and peer-annotations for every statement; WOLF-VLA aligns language, vision, proprioception, and demonstration trajectories. Each also emphasizes reproducibility, whether through open-sourced code, data, and leaderboard for Wolf, structured logs and fixed random seeds for WOLF, or release of the dataset, model checkpoints, and simulation suite for WOLF-VLA. This suggests a broader pattern in benchmark construction: the task definition is inseparable from the annotation protocol and logging interface.

The benchmarks also expose different unresolved issues. Wolf identifies scarcity of long-form video caption data, LLM verbosity, domain-specific irrelevancies, and lack of per-token confidence. WOLF reports weak discrimination by ROC AUC 0.55–0.58 and underconfident signals by Brier scores 0.26–0.29, even though AUPRC is actionable. WOLF-VLA reports strong results on forward walking but materially lower performance on side walking and stair tasks, and it does not demonstrate sim-to-real transfer. Taken together, the three WOLF benchmarks illustrate how modern benchmark design increasingly couples dataset curation, model scaffolding, and metric formalization, but does so under domain-specific constraints that prevent the acronym from denoting a single canonical benchmark.