StrokeVision-Bench: Multimodal Stroke Benchmark

• StrokeVision-Bench is a multimodal benchmark dataset for assessing upper extremity motor function in stroke survivors via a standardized Box and Block Test.
• The dataset comprises 1,000 one-second video clips with paired RGB frames and 2D skeletal trajectories, annotated into four clinically relevant action classes.
• Baseline evaluations indicate that models like R3D and MotionBERT achieve high accuracy, enabling precise, computer vision–based tracking of rehabilitation performance.

Searching arXiv for the benchmark paper and closely related referenced methods to ground citations. StrokeVision-Bench is a multimodal benchmark dataset and evaluation suite for objective, vision-based assessment of upper extremity motor function in stroke survivors, centered on the standardized Box and Block Test (BBT). It comprises 1,000 annotated short video clips of stroke-impaired patients performing clinically structured block transfer tasks, with each sample represented in two modalities: raw RGB video frames and 2D skeletal joint trajectories. The benchmark defines four clinically meaningful action classes—Grasping, Non-task Movement, Transport with Object, and Transport without Object—and provides baseline evaluations for both video-based and skeleton-based action recognition models, with the stated aim of enabling fine-grained assessment of the BBT beyond a simple block count (Robinson et al., 2 Sep 2025).

1. Clinical setting and motivation

StrokeVision-Bench is situated in the context of post-stroke upper extremity rehabilitation, where assessment remains dominated by therapist observation and coarse scoring systems. The benchmark is motivated by the observation that conventional Outcome Measures such as the Action Research Arm Test and the Box and Block Test are standardized and clinically entrenched, but often have low sensitivity to subtle changes in movement quality, compensatory trunk movements, movement speed, and coordination. The underlying clinical motivation is particularly strong because stroke is a leading cause of long-term disability; about 95% of stroke survivors experience upper extremity dysfunction, and 30–66% have severely impaired arm use (Robinson et al., 2 Sep 2025).

The Box and Block Test is widely used to assess gross manual dexterity. In its standard form, patients transfer as many small blocks as possible from one compartment of a box to another in a fixed time, typically 60 seconds, and the usual outcome is a single count: the number of blocks transferred. StrokeVision-Bench targets this gap between standardized assessment and coarse scoring by recasting the BBT as a computer vision problem over clinically meaningful sub-actions. In the benchmark’s framing, this supports a purely vision-based, low-cost pathway to quantitative tracking of movement over time, automatic classification of BBT sub-actions, and extraction of joint-level kinematic information without wearables.

A central point of contrast with prior rehabilitation datasets is task structure and cohort specificity. Existing datasets such as StrokeRehab are described as focusing on activities of daily living and often mixing healthy and stroke-impaired subjects. StrokeVision-Bench instead centers on a single standardized clinical test and focuses exclusively on stroke patients. This suggests a benchmark designed for higher clinical specificity, particularly when the target is recovery tracking rather than generic action recognition.

2. Dataset composition and multimodal representation

The dataset consists of 1,000 short video clips, each 1 second long and containing 30 frames. All participants are stroke survivors with upper extremity dysfunction undergoing weekly rehabilitation, and the recordings include both pre- and post-therapy sessions. The task is block transfer in a BBT-like setting, and the dataset is explicitly organized around that clinical structure rather than around unconstrained daily activities (Robinson et al., 2 Sep 2025).

The RGB modality is represented as a fixed-length video tensor

$V \in \mathbb{R}^{T \times C \times H \times W},$

where $T = 30$ frames and $C = 3$ color channels. The frame rate is 30 FPS. The recording setup is described as a single, human-centric viewpoint in a consistent clinical environment with the same apparatus and typical clinical lighting. Spatial resizing and cropping to model-specific resolutions, standard normalization, and fixed-length temporal sampling are part of the training preprocessing.

The skeletal modality is obtained by applying Sapiens to the RGB videos and assembling one pose per frame. The resulting representation is

$V_J \in \mathbb{R}^{T \times J \times 2},$

where $T = 30$ and $J = 17$ keypoints. The final dimension corresponds to image-space coordinates $(x, y)$. The benchmark notes that the 17 keypoints typically include major joints such as head, shoulders, elbows, wrists, hips, knees, and ankles, with the exact mapping following Sapiens’ keypoint set. Poses are directly taken from Sapiens, and model-specific preprocessing may normalize, center, or convert coordinates to joint-relative form following standard practice on NTU RGB+D.

The dual-modality design serves two distinct research agendas. Raw video preserves appearance cues, including object presence and hand–object contact, while 2D skeletons support privacy preservation and joint-level trajectory analysis. The latter is explicitly linked to clinically relevant quantities such as the shoulder–abdomen joint angle.

3. Action taxonomy, annotation, and data characteristics

Each 1-second clip is assigned a single clip-level label from one of four action classes. These classes partition the BBT into sub-actions that are both visually distinguishable and clinically meaningful (Robinson et al., 2 Sep 2025).

1. Grasping: the patient attempts or completes grasping a block from the starting container.
2. Non-task Movement: movements not directly contributing to block transfer, including adjusting posture, resting, or compensatory trunk movements without grasping or transporting.
3. Transport with Object: carrying the block from one compartment to another.
4. Transport without Object: unsuccessful or empty-handed transport-like movements.

The distinction between Transport with Object and Transport without Object is clinically consequential because it separates successful functional action from unsuccessful transport-like behavior. Non-task Movement isolates motion that may reflect compensatory behavior or interruption of task execution, while Grasping isolates a key dexterity component. The class design therefore supports analysis of functional success, attempts, and compensations within a single standardized task.

All clips were manually annotated into the four action classes. The description does not specify the exact annotators or inter-rater reliability metrics, but it emphasizes manual curation to ensure correctness. The dataset is class-imbalanced: there are more examples of Grasping and Non-task Movement than transport phases, because some patients struggle to successfully transfer blocks. Train–validation–test splits were created with no data leakage, although the public description does not tabulate exact counts per split and does not explicitly state whether splitting is subject-wise or clip-wise. A plausible implication is that the benchmark prioritizes careful partitioning despite limited dataset size.

The demographic and clinical metadata released in the description are limited. The benchmark does not provide a detailed table of age, sex, time since stroke, or scores on ARAT or BBT. It does state that all subjects are stroke-impaired, that they are undergoing weekly rehabilitation, and that the recordings include pre- and post-therapy sessions that can be used to study changes in movement speed and quality across weeks.

4. Tasks, evaluation protocol, and baseline performance

StrokeVision-Bench supports two explicit benchmark tasks. The first is video-based action recognition, where the input is a 1-second RGB clip $V$ and the output is one of the four action classes. The second is skeleton-based action classification, where the input is a 2D pose sequence $V_J \in \mathbb{R}^{T \times J \times 2}$ and the output is again one of the four classes. The benchmark also supports implicit tasks not explicitly benchmarked, including phase segmentation of the BBT, approximate block counting by detecting Transport with Object events, and recovery tracking through pre- versus post-session comparisons (Robinson et al., 2 Sep 2025).

Classification accuracy is the primary evaluation metric, reported as mean accuracy ± standard deviation over multiple runs. No specialized clinical scoring metric or F1 score is defined in this initial version. Training uses standard cross-entropy loss,

$$\mathcal{L}_{CE} = - \sum_{i=1}^N y_i \log(\hat{y}_i),$$

where $T = 30$0 is the one-hot ground-truth label and $T = 30$1 is the predicted probability.

The benchmark includes baselines for both frame-based and skeleton-based recognition. Video models are initialized with Kinetics-400 pretraining. Skeleton models are pretrained on NTU RGB+D unless otherwise noted. The reported mean accuracies are as follows.

Model	Modality	Accuracy (%)
R3D	Frames	87.68 ± 2.48
R2Plus1D	Frames	86.96 ± 1.20
Video MViT	Frames	77.14 ± 2.15
Video Swin Transformer	Frames	78.93 ± 2.15
MotionBERT	2D joints	84.29 ± 1.35
PoseConv3D	2D joints	68.93 ± 3.06
MS-G3D	2D joints	78.39 ± 1.72

R3D is identified as the best-performing model on StrokeVision-Bench, and MotionBERT is the highest-performing skeleton-based baseline. The benchmark’s stated observations are that 3D CNNs outperform video transformers on this small, specialized dataset, that MotionBERT nearly matches R3D while using only skeleton input, and that PoseConv3D struggles relative to the others, possibly because of limited data and clinical variability. MS-G3D is trained without pretraining because of keypoint set mismatch.

The confusion matrices indicate that Non-task Movement is generally recognized well, with many models surpassing approximately 85–90% accuracy, and that Grasping is also recognized strongly, with R3D and R2Plus1D reaching approximately 89% and MS-G3D reaching approximately 92.9% on that class. The hardest distinction is Transport with Object versus Transport without Object. Video MViT and Video Swin heavily confuse these two classes, while 3D CNNs perform reasonably, with 86–93% accuracy on transport classes. Skeleton models show mixed behavior: MotionBERT is described as balanced and relatively high on both transport classes, PoseConv3D struggles on both transports at approximately 63–66%, and MS-G3D is good on “with block” but weaker on “without block.” This pattern suggests that the presence of a small block, grip configuration, and hand–object contact remain difficult cues for standard action recognition models.

5. Technical pipeline and interpretability

The data processing pipeline begins with video acquisition of patients performing BBT-like block transfers before and after therapy sessions. Recordings are segmented into 1-second clips of 30 frames. Sapiens is then applied to each frame to produce 17 joint coordinates, yielding pose sequences of shape $T = 30$2. Each 1-second clip is manually labeled into one of the four action classes, after which train, validation, and test sets are assembled with balancing pursued as far as possible. Model preprocessing then differs by modality: video is resized, cropped, and normalized; skeleton coordinates are formatted and normalized according to model requirements (Robinson et al., 2 Sep 2025).

The reported implementation setup uses PyTorch and a single NVIDIA V100 GPU with 32 GB VRAM. Video models—R3D, R2Plus1D, Video MViT, and Video Swin—use Kinetics-400 weights. MotionBERT and PoseConv3D use NTU RGB+D weights, while MS-G3D is trained from scratch. The description states that the exact batch size, number of epochs, and learning rates are not enumerated, although conventional settings for these architectures are implied. It also states that standard spatial augmentations are likely used for video and that mild augmentations may be used for skeletons depending on the baseline code; these are inferences rather than fully specified protocol elements.

Interpretability is a major design feature of the benchmark. The model outputs map directly to clinically interpretable quantities. Counting Transport with Object events over a session can approximate the canonical BBT score. Ratios involving Transport with Object, Transport without Object, and Non-task Movement can be used to characterize movement efficiency. Time spent in Grasping versus Transport versus Non-task Movement can indicate difficulty with grasp, difficulty with transport, or excessive compensatory behavior. The skeleton modality further supports kinematic analysis through joint trajectories, including shoulder–abdomen angles, range of motion of the shoulder and elbow, and hand-trajectory velocity or smoothness. The benchmark explicitly identifies two clinically valuable outputs: total number of objects transferred before versus after each session, and joint angle between shoulder and abdomen as a proxy for compensatory trunk lean and improved body movement.

6. Position within rehabilitation vision research, limitations, and future directions

StrokeVision-Bench is presented as the first dedicated dataset of stroke patients performing clinically structured block transfer tasks modeled on the BBT, and more specifically as the first dataset of stroke patients performing the Box and Block Test with paired video and 2D skeleton data and sub-action labels. It is also described as the first benchmark to systematically evaluate multiple video and skeleton-based action recognition methods in this clinical setting (Robinson et al., 2 Sep 2025).

Its position relative to adjacent literatures follows from three contrasts. Relative to rehabilitation datasets such as StrokeRehab, it focuses on a single standardized clinical test and includes only stroke patients rather than mixed healthy and impaired cohorts. Relative to generic action datasets such as Kinetics and NTU RGB+D, it contains clinical movements with subtle pathology-specific patterns and therefore constitutes a domain-specific benchmark with different data distribution and visual characteristics. Relative to prior kinematic assessment approaches based on wearables, optical motion capture, or robotics, it offers a camera-only benchmark that bridges modern deep learning and clinically structured movement analysis.

The limitations are explicitly stated. The dataset covers only the Box and Block Test. Its size—1,000 clips—is relatively small for modern deep learning, especially for transformers. Recordings likely come from a single clinical site with a relatively consistent setup. The skeletal representation is limited to 2D pose rather than 3D pose or depth data. The released description also provides limited metadata and no long-term follow-up table. These constraints matter because they bound generalization across environments, viewpoints, patient populations, and richer clinical covariates.

The proposed future directions are correspondingly concrete. The benchmark invites extension to 3D or multi-view capture, broader coverage of rehabilitation tasks and standardized tests, movement from categorical classification to continuous severity or quality scores, predictive modeling of future recovery trajectories, robustness to real-world clinical variability such as camera angle and lighting, and further emphasis on privacy-preserving and interpretable models. Because the dataset includes pre- and post-session recordings, a plausible implication is that it can serve not only as an action classification benchmark but also as a scaffold for longitudinal modeling of recovery-sensitive visual biomarkers.