ActPLD: Action Understanding Benchmark

• ActPLD is a benchmark designed to assess if multimodal large language models can interpret human actions from sparse point-light displays.
• It leverages real motion-capture data divided into single-actor and social-interaction subsets and uses both 3-AFC classification and chain-of-thought descriptions.
• Results reveal that current models perform near chance levels compared to near-ceiling human accuracy, underscoring challenges in spatiotemporal reasoning.

Searching arXiv for the benchmark and nearby acronym-confusions to ground the article. ActPLD is the first benchmark specifically designed to evaluate whether multimodal LLMs (MLLMs) can understand human actions from point-light displays (PLDs). A PLD is a sparse animation made of moving dots placed at key human joints, so that nearly all semantic information must be recovered from body kinematics alone. ActPLD is built from real human motion-capture data, organized into single-actor and social-interaction subsets, and evaluated through action classification and intention inference under both 3-AFC and free-form Chain-of-Thought description scoring. It is intended to test whether current video-capable MLLMs can infer actions and intentions when motion itself is the only source of meaning (Kadambi et al., 27 Sep 2025).

1. Biological-motion motivation and benchmark rationale

The motivation for ActPLD is grounded both in biological motion perception and in broader claims about embodied action understanding. In humans, PLDs are a classic stimulus for showing that rich social and semantic information can be inferred from minimal kinematic cues. The benchmark description emphasizes that humans can recover not just action class, but also intentionality, identity, emotions, and other attributes from these sparse dot patterns. This sensitivity appears early in development and is often attributed to a mapping between perception and action in the human brain, including a body schema and action experience that support understanding even from impoverished visual input.

ActPLD is designed to probe a core weakness of current MLLMs. The benchmark description argues that, although such systems can often perform reasonably on natural-image or context-rich video tasks, they may rely heavily on texture, objects, background cues, and broad semantic priors rather than genuinely grounded spatiotemporal understanding of human movement. PLDs remove those confounds: they strip away objects, texture, and scene context, isolate motion, and expose whether a model can form an abstract, temporally grounded representation of action. The paper also notes that PLDs mimic real-world settings where perception is noisy, partial, or low-fidelity, making them practically relevant as well as theoretically informative.

This positioning distinguishes ActPLD from prior multimodal benchmarks such as MotionBench, MVBench, Video-MME, and embodied-agent evaluations. Those benchmarks already suggest that MLLMs struggle with temporal reasoning and motion understanding, but they typically involve semantically rich scenes. ActPLD instead targets abstract generalization over sparse human movement. Figure 1, as described in the paper, contrasts human ability to reconstruct 3D motion and infer rich semantic information from sparse PLDs with MLLMs’ tendency to rely on 2D feature aggregation, which often fails to yield the necessary semantic and spatiotemporal grounding.

2. Benchmark composition and stimulus generation

ActPLD contains 228 PLD videos derived from real human motion-capture data. The single-actor subset contains 158 actions sourced from the Carnegie Mellon Motion Capture (CMU) database. The social-interaction subset contains 70 interactions total: 34 from the CMU database and 36 from a prior social-interaction PLD source cited in the paper. The benchmark size is summarized as

$$158 \text{ single-actor} + 70 \text{ social interactions} = 228 \text{ total actions/interactions.}$$

The paper does not define train/validation/test splits; ActPLD is presented as an evaluation benchmark rather than a training corpus, and all reported numbers are benchmark accuracies rather than held-out split results in the machine learning training sense.

Stimuli were generated from the motion-capture recordings using BioMotionToolbox in MATLAB R2024a together with Psychtoolbox. The paper states that the PLDs were created by “displaying and capturing the videos” with this pipeline. For model input, each video was preprocessed into eight consecutive frames, each at $128 \times 128$ pixels, extracted from the central portion of the clip. The first and last $10\%$ of frames were trimmed to avoid artifacts such as T-poses, and a middle window large enough to contain eight consecutive frames was selected. If a model gave an uninformative response, the sampling window could be shifted earlier or later, with up to three retries.

Input formatting depended on provider. For Gemini, the eight frames were combined into a $4 \times 2$ montage; for Anthropic, OpenAI, Qwen, and Replicate-based runs, frames were sent individually as base64-encoded images. The paper does not report video clips being fed as continuous video streams; instead, models were evaluated from sampled still frames representing the PLD motion. Figure 2 depicts this pipeline for both a single-actor PLD (“dancing”) and a social-interaction PLD (“high-fiving”), showing sampled frames, prompting, and sample outputs.

3. Tasks, prompting, and scoring

ActPLD measures two kinds of understanding: action classification and intention inference (Kadambi et al., 27 Sep 2025). The paper defines the two core tasks as follows: Action Classification is whether the model can identify the depicted motion, and Intention Inference is whether the model can attribute purposeful behavior to the motion pattern. These are evaluated under two output formats, giving four evaluation conditions overall: 3-AFC action label classification, 3-AFC intention classification, free-form Chain-of-Thought (CoT) description scored for action-label match, and free-form Chain-of-Thought description scored for intention match.

The 3-AFC setting is multiple choice. Around 60 action/intention labels were organized into semantic groups to avoid distractors that were too close to the ground-truth label, such as “sit down,” “squat down,” and “sit on stool” appearing together. For each video, the correct label was paired with two distractor labels sampled from all possible labels or intentions while excluding semantically overlapping groups. The true label was parsed directly from the filename, and the three answer choices were shuffled per trial. Accuracy was computed as the percentage of correct predictions over valid trials, excluding API failures and timeouts.

The free-form evaluation is called “Chain-of-Thought Description Matching.” Here, the model generates a natural-language explanation or description of the PLD, and an independent LLM scorer judges whether the description supports the ground-truth action. Gemini 2.5 Pro is identified in the main description as the scorer, while GPT-5 and Gemini 2.5 Pro are also mentioned earlier as independent evaluators. The scoring procedure is explicitly negation-aware and prioritizes positive cue matching near the true label. A response is marked as a match if any of the following hold: the true label has more positive than negative evidence; the true label has at least one positive cue while competing labels are unsupported or negated; or the response does not name the label directly but clearly describes defining features characteristic of the true action, with competing labels unsupported. Otherwise, it is marked as a non-match.

The supplementary scorer instructions make the proximity criterion explicit: positive cues include phrases such as “is,” “shows,” “depicts,” “represents,” “looks like,” “best described as,” “most consistent with,” “clearly,” “demonstrates,” “doing,” “best match(es),” “matches,” “consistent with,” “resembles,” and “most likely,” provided they occur in the same sentence or within approximately 80 characters of the label. The instructions state: “Near” means the cue appears in the same sentence or within $\sim 80$ characters of the label name (including across line breaks/bullets). This evaluation design is intended to test not only final-answer accuracy but also whether a model’s apparent reasoning genuinely tracks the visual motion.

4. Evaluated models and quantitative findings

The evaluated MLLMs include both proprietary and open-source systems (Kadambi et al., 27 Sep 2025). The main evaluation set is GPT-5, GPT-4o, Gemini 2.5 Pro, Claude Sonnet-4, Claude Opus-4.1, and Qwen-2.5-VL-7B. The abstract and introduction also mention other state-of-the-art systems more broadly, such as CogVLM2-Video and PLLaVA, but these are not part of the actual benchmark results tables.

The central quantitative result is that all models perform poorly, generally only slightly above the 3-AFC chance level of $33.33\%$, while humans perform near ceiling. For single-actor PLDs, human performance on a sampled subset ($N=2$) is reported as $93.33\%$. Model performance falls roughly in the $34\%-41\%$ overall range, depending on model and evaluation mode. The best mean single-actor performance is reported for GPT-4o at $M = 40.19\%$, while Claude Sonnet-4 is lowest at $128 \times 128$0.

For the single-actor subset, the paper reports the following metric breakdowns:

• Claude Opus-4.1: label 3AFC 41.14; label description 29.75; intention 3AFC 36.71; intention description 29.75.
• Claude Sonnet-4: label 3AFC 36.71; label description 36.08; intention 3AFC 34.81; intention description 28.48.
• GPT-4o: label 3AFC 37.34; label description 40.51; intention 3AFC 44.30; intention description 38.61.
• Gemini 2.5 Pro: label 3AFC 29.11; label description 38.61; intention 3AFC 31.01; intention description 40.51.
• Qwen-2.5-VL-7B: label 3AFC 36.71; label description 36.08; intention 3AFC 45.57; intention description 33.54.
• GPT-5: label 3AFC 40.51; label description 34.18; intention 3AFC 36.71; intention description 29.75.

For social interactions, human performance on a sampled subset is also near ceiling. The text gives $128 \times 128$1, while the figure caption reports approximately $128 \times 128$2. The detailed supplementary human tables show $128 \times 128$3 and $128 \times 128$4 for the two human participants on social interactions. Model performance on this subset ranges approximately from the high 20s to about 50 overall. Gemini 2.5 Pro is the strongest model on this subset with $128 \times 128$5, and Claude Opus-4.1 is the weakest with $128 \times 128$6.

For the social-interaction subset, the paper reports:

• Claude Opus-4.1: label 3AFC 25.71; label description 25.71; intention 3AFC 34.29; intention description 28.57.
• Claude Sonnet-4: label 3AFC 37.14; label description 28.57; intention 3AFC 27.14; intention description 25.71.
• GPT-4o: label 3AFC 32.86; label description 45.71; intention 3AFC 38.57; intention description 40.00.
• Gemini 2.5 Pro: label 3AFC 41.27; label description 53.97; intention 3AFC 48.48; intention description 56.06.
• Qwen-2.5-VL-7B: label 3AFC 27.14; label description 40.00; intention 3AFC 34.29; intention description 40.00.
• GPT-5: label 3AFC 41.43; label description 47.14; intention 3AFC 32.86; intention description 38.57.

The benchmark-level comparison across subsets is summarized as:

$128 \times 128$7

$128 \times 128$8

$128 \times 128$9

These values indicate that social interactions are not uniformly easier in every metric, but they do have slightly better overall mean performance and notably stronger description-matching performance.

5. Error patterns, dissociations, and interpretive claims

The paper’s error analysis identifies the single-actor PLD condition as the hardest setting (Kadambi et al., 27 Sep 2025). In that condition, models must infer action from sparse joint trajectories without contextual support from other actors or objects. The discussion attributes failure to several factors.

• Minimalistic stimuli: PLDs contain almost no static visual cues, so models cannot lean on texture, object affordances, or scene priors.
• Weak spatiotemporal integration: current MLLMs often sample only a small number of frames, typically around $10\%$0–$10\%$1, and may compress them aggressively, losing crucial motion information.
• Lack of body schema priors: humans infer invisible limbs, joint dependencies, and plausible skeletal structure; MLLMs see moving pixel clusters without an innate kinematic model.
• Inconsistent reasoning: especially for single actors, good verbal explanations do not reliably correspond to correct perceptual judgments.
• Overreliance on linguistic priors: a model may generate a plausible narrative that “sounds right” without being tied to the actual motion pattern.

A notable dissociation concerns Gemini 2.5 Pro. It performs poorly on single-actor 3-AFC, but relatively well on single-actor description matching and best overall on social interactions. The paper interprets this as possibly reflecting an architectural trade-off in cross-modal fusion: strong language-linked interpretation may help in richer or relational settings, but in sparse single-actor PLDs it may produce over-elaborate but incorrect hypotheses. The paper’s phrasing is that Gemini may “lean too heavily on linguistic priors” and “over-elaborate its interpretations.”

The relation between multiple-choice performance and free-form reasoning was assessed with Spearman correlation. For social interactions, the paper reports

$10\%$2

For single-actor PLDs, it reports

$10\%$3

This suggests that 3-AFC accuracy and description-matching consistency align for social interactions but not for single-actor actions. The supplementary analyses also report strong correlations between label and intention inference across actions for both single-actor actions and social interactions ($10\%$4).

Qualitative observations reinforce the quantitative findings. The discussion notes that MLLMs sometimes mistake PLDs for non-agentive patterns such as constellations or rotating lines. In that sense, the benchmark can reveal failure at the initial stage of biological-motion perception: recognizing the dots as a human body in motion.

6. Terminological disambiguation and related acronyms

The term ActPLD in (Kadambi et al., 27 Sep 2025) refers to a benchmark for evaluating action processing in MLLMs from human point-light displays. It is not synonymous with several superficially similar acronyms that appear in other arXiv papers.

In "Demonstrating (Hybrid) Active Logic Documents and the Ciao Prolog Playground, and an Application to Verification Tutorials" (Ferreiro et al., 2023), the paper does not use the term “ActPLD” explicitly. The closest and clearly intended concepts are Active Logic Documents (ALDs), Hybrid Active Logic Documents (HALDs), and the Ciao Prolog Playground. Those refer to web pages which incorporate embedded Prolog engines and to a framework for interactive logic-programming documents, not to biological-motion evaluation.

In "PLD+: Accelerating LLM inference by leveraging LLM Artifacts" (Somasundaram et al., 2024), PLD+ stands for Prompt Lookup Decoding+. That paper introduces a tuning-free inference-acceleration method for input-guided tasks and explicitly states that it does not mention ActPLD at all. The shared substring “PLD” therefore has a different meaning.

In "Procedural Refinement by LLM-driven Algorithmic Debugging for ARC-AGI-2" (Qiu et al., 20 Mar 2026), the introduced method is Abduction-Based Procedural Refinement (ABPR), grounded in Algorithmic Program Debugging (APD). The paper states that it does not mention “ActPLD” anywhere. Here again, the overlap is nominal rather than conceptual.

7. Limitations and prospective directions

ActPLD is described as a “first preliminary benchmark,” and the paper identifies several limitations (Kadambi et al., 27 Sep 2025). Human data were collected only from a very small sample ($10\%$5) and on randomly selected subsets rather than the full benchmark. The paper does not report train/val/test splits, exact prompt templates for all models, or comprehensive sensitivity analyses over frame count, camera/viewpoint changes, or alternative scoring methods. It also notes a small inconsistency in the reported Qwen model size—an earlier section mentions Qwen2.5-VL-72B, while the result tables specifically list Qwen-2.5-VL-7B—and some inconsistency in human-performance reporting across text and figure captions.

The future directions are stated in terms of model architecture, training signals, and representational priors. The paper argues for better spatiotemporal processing that preserves informative motion trajectories; explicit body-schema or skeletal priors; training on sparse or low-fidelity motion representations beyond PLDs, such as stick figures or trajectories; stronger motor or experiential grounding; and prediction-based learning objectives such as next-frame prediction or anticipating social outcomes. A broader implication proposed by the paper is that, if MLLMs are to move toward embodied, human-like intelligence, they will need mechanisms for representing action dynamics and biological structure, not just multimodal co-occurrence statistics.

ActPLD’s main empirical conclusion is that current state-of-the-art MLLMs, including leading proprietary systems, do not yet approach human-like action understanding when biological motion is presented in sparse PLD form. Performance remains close to chance across both action recognition and intention inference, for both single-actor and social-interaction stimuli. The benchmark therefore functions as a diagnostic instrument for spatiotemporal abstraction, embodied action understanding, and perceptually grounded reasoning under conditions where motion is the only cue.