KASportsFormer: 3D Pose Estimation in Sports Video

• KASportsFormer is a transformer-based framework for monocular 3D human pose estimation in short sports video clips.
• It integrates anatomy-informed bone and limb tokens with multimodal cross-attention to robustly capture spatial and temporal dynamics.
• The approach demonstrates resilience to motion blur, occlusions, and detector noise, achieving competitive MPJPE metrics on benchmark sports datasets.

Searching arXiv for KASportsFormer and closely related entries to ground the article and disambiguate the name. {"query":"KASportsFormer arXiv", "max_results": 10} KASportsFormer is a transformer-based framework for monocular 3D human pose estimation in short sports video clips. It is designed for settings in which decisive actions unfold over only a few frames and where motion blur, self-occlusions, detector noise, and domain shift in broadcast footage degrade standard spatio-temporal lifting models. The framework takes 2D keypoints with per-joint confidence as input, constructs kinematic anatomy-informed representations at the bone and limb levels, integrates those representations through multimodal cross-attention alongside a joint stream with self-attention and GCN branches, and regresses root-centered 3D poses in a Human3.6M 17-joint layout. In the reported short-window protocol with fixed clip length $T=27$, it achieves MPJPE of $58.0\,\mathrm{mm}$ on SportsPose and $34.3\,\mathrm{mm}$ on WorldPose (Yin et al., 28 Jul 2025).

1. Problem setting and representational scope

KASportsFormer addresses single-person monocular 3D human pose estimation from short sports clips rather than long temporal sequences. The input modality is 2D keypoints and detection confidence per joint per frame; no raw RGB is used in the lifting stage. The input tensor is $X \in \mathbb{R}^{T\times J\times 3}$, where each entry contains $(x,y,c)$, $T$ is the clip length, and $J$ is the number of joints. The output is $\hat{P} \in \mathbb{R}^{T\times J\times 3}$ under a Human3.6M 17-joint layout with pelvis-root normalization, so the pelvis is fixed at the origin (Yin et al., 28 Jul 2025).

The design target is short-horizon sports analysis. The fixed clip length is $T=27$ frames for both SportsPose and WorldPose experiments, explicitly chosen to emulate instantaneous-action modeling. This short-window assumption is motivated by sports actions such as shooting, passing, jumping, and kicking, which often complete in moments of time. A plausible implication is that the method prioritizes moment sensitivity over the broader temporal context exploited by long-sequence lifting models.

The framework operates in a root-centered camera or scene space consistent with dataset conventions. Camera parameters are used only to prepare GT 2D projections for detector evaluation when available, such as on WorldPose. The method therefore belongs to the class of 2D-to-3D lifting systems rather than end-to-end image-based pose estimators.

2. Architectural organization

The pipeline begins by embedding the 2D joint inputs into joint features $$H_{\text{joint}} \in \mathbb{R}^{T\times J\times d}$$. In parallel, anatomy features are constructed from the same 2D input, yielding bone tokens $58.0\,\mathrm{mm}$0 and limb tokens $58.0\,\mathrm{mm}$1, with $58.0\,\mathrm{mm}$2 in practice. Spatial and temporal positional encodings are then added to all token streams. These streams are processed by a multistream spatio-temporal transformer encoder with $58.0\,\mathrm{mm}$3 layers, followed by an MLP regression head that maps the final blended features to 3D joint coordinates (Yin et al., 28 Jul 2025).

The backbone uses MetaFormer-style encoder blocks with token mixing, normalization, and residual connection:

$58.0\,\mathrm{mm}$4

The hidden dimension is $58.0\,\mathrm{mm}$5, the attention modules use $58.0\,\mathrm{mm}$6 heads, and the model has approximately $58.0\,\mathrm{mm}$7 parameters. The encoder has three principal components: an anatomy mixer, a joint mixer, and a token-blending mechanism.

The anatomy mixer performs multimodal cross-attention between bone and limb streams in both spatial and temporal domains. The joint mixer combines spatial and temporal MHSA with spatial and temporal GCN modules. Token blending then fuses anatomy, attention, and graph-derived features with learned soft gating at each layer. The blended feature is reused as the next-layer joint and bone input, whereas limb tokens are always re-encoded from the original limb input to preserve a consistent kinematic context anchor. This suggests that the architecture treats limb tokens as a stable anatomical prior rather than a fully free-running latent stream.

3. Kinematic anatomy-informed features

The kinematic feature construction is centered on a predefined pelvis-rooted skeleton topology over the joint set $58.0\,\mathrm{mm}$8, with skeleton edges $58.0\,\mathrm{mm}$9 defining the bone set. For 2D joint coordinate $34.3\,\mathrm{mm}$0, each bone $34.3\,\mathrm{mm}$1 is represented by a vector, a length, and a unit direction:

$34.3\,\mathrm{mm}$2

BoneExt forms per-bone features as $34.3\,\mathrm{mm}$3 and additionally computes an “ultra-bone” through the average bone direction and average bone length across the skeleton. These statistics are concatenated to produce $34.3\,\mathrm{mm}$4 bone-like slots per frame, giving $34.3\,\mathrm{mm}$5 before projection to $34.3\,\mathrm{mm}$6 (Yin et al., 28 Jul 2025).

LimbFus aggregates selected subsets of bones into limb tokens. The limb definitions include both biologically plausible groups such as arms and legs and hyper-limb connections intended to capture holistic kinematic synergy, including shoulder-hip cross-links. For limb $34.3\,\mathrm{mm}$7, the model collects the component stacks $34.3\,\mathrm{mm}$8, $34.3\,\mathrm{mm}$9, and $X \in \mathbb{R}^{T\times J\times 3}$0 from the constituent bones and composes them with three small MLPs:

$X \in \mathbb{R}^{T\times J\times 3}$1

Stacking the resulting $X \in \mathbb{R}^{T\times J\times 3}$2 gives $X \in \mathbb{R}^{T\times J\times 3}$3 and then $X \in \mathbb{R}^{T\times J\times 3}$4 after projection.

A notable design choice is that limb angles are not explicitly used. Instead, limb direction and length components are learned via MLPs, which is stated to preserve intermediate turning geometry such as elbow and knee bends better than naive vector chaining. The ablation over LimbFus depth selects hidden width $X \in \mathbb{R}^{T\times J\times 3}$5 as best; values that are too small are insufficient for complex limb transformations, whereas a larger value of $X \in \mathbb{R}^{T\times J\times 3}$6 underfits without heavy normalization.

4. Multimodal integration, graph modeling, and moment emphasis

The anatomy mixer applies spatial multi-head cross-attention with bone queries and limb keys and values:

$X \in \mathbb{R}^{T\times J\times 3}$7

For each head,

$X \in \mathbb{R}^{T\times J\times 3}$8

and the multi-head output is concatenated and projected. The same mechanism is applied temporally after transposing the streams along time, producing temporal MHCA. Each attention block is followed by MLP, residual connection, and LayerNorm in a MetaFormer-style configuration (Yin et al., 28 Jul 2025).

The joint stream uses spatio-temporal MHSA and a GCN branch. The spatial GCN uses self-connected adjacency $X \in \mathbb{R}^{T\times J\times 3}$9 with symmetric normalization:

$(x,y,c)$0

where $(x,y,c)$1 is ReLU. The temporal GCN constructs adjacency from frame-feature similarity and connects the top-$(x,y,c)$2 similar frames, with $(x,y,c)$3, to emphasize momentary actions. The moment-attentive strategy therefore arises not only from short clip length but also from an explicit graph prior over temporally salient frames.

Layerwise token blending combines anatomy, self-attention, and graph outputs. If $(x,y,c)$4, $(x,y,c)$5, and $(x,y,c)$6 denote the outputs of the anatomy, joint-attention, and GCN streams, the soft gating weights are defined by

$(x,y,c)$7

The blended feature is then

$(x,y,c)$8

The next-layer joint and bone inputs are both set to $(x,y,c)$9, while the limb stream is re-tokenized from $T$0. This fixed-anchor treatment of limb tokens is one of the model’s distinguishing structural choices.

5. Optimization, datasets, and empirical results

Training uses a pose loss and a velocity loss. The pose term is

$T$1

and the velocity term is defined from temporal differences $T$2 and $T$3 as

$T$4

The total objective is $T$5. Evaluation uses MPJPE and PA-MPJPE, where MPJPE is

$T$6

PyTorch is used for implementation, training runs on an NVIDIA A6000, the optimizer is AdamW with learning rate $T$7 and weight decay $T$8, training lasts 120 epochs with early stopping of patience 10, and the first 10 epochs use warmup from $T$9, after which the learning rate decays by $J$0 with patience 2 via scheduler (Yin et al., 28 Jul 2025).

The reported datasets are SportsPose and WorldPose. SportsPose contains $J$1 frames, 24 subjects, 7 cameras, and 5 sports activities: soccer, volleyball, tennis, jump, and throw. WorldPose comprises 8 World Cup soccer games, $J$2 frames, and approximately $J$3 SMPL player annotations, with single-person crops prepared via projected 2D keypoints using provided camera parameters. In both datasets, clip length is fixed at $J$4 for DET and GT inputs.

Dataset / input	MPJPE	P-MPJPE
SportsPose, DET	58.0 mm	44.3 mm
SportsPose, GT	30.9 mm	27.9 mm
WorldPose, DET	34.3 mm	22.0 mm
WorldPose, GT	8.5 mm	6.2 mm

On SportsPose with DET inputs, the per-action MPJPE values reported for KASportsFormer are throw $J$5, soccer $J$6, tennis $J$7, jump $J$8, and volley $J$9. On WorldPose with DET inputs, the table value is approximately $\hat{P} \in \mathbb{R}^{T\times J\times 3}$0 while the text reports $\hat{P} \in \mathbb{R}^{T\times J\times 3}$1. The paper attributes strong robustness under occlusions and motion blur to the stability of bone directions and lengths and to the higher-order interaction cues supplied by limb fusion.

Ablation results on SportsPose DET isolate the contribution of each module. The baseline is reported as $\hat{P} \in \mathbb{R}^{T\times J\times 3}$2; a limb-only MHSA variant gives $\hat{P} \in \mathbb{R}^{T\times J\times 3}$3; a bone-only MHSA variant gives $\hat{P} \in \mathbb{R}^{T\times J\times 3}$4; softmax fusion without MHCA gives $\hat{P} \in \mathbb{R}^{T\times J\times 3}$5; and the full KASportsFormer with BoneExt, LimbFus, and MHCA gives $\hat{P} \in \mathbb{R}^{T\times J\times 3}$6. These results indicate that bone tokens, limb tokens, and multimodal cross-attention each contribute meaningfully.

6. Interpretation, deployment, and terminological ambiguity

The method is positioned as a sports-specific short-clip lifting framework. Its reported robustness to detector noise and low-resolution crops suggests that the anatomy tokens reduce reliance on high-fidelity appearance cues, although the lifting stage itself uses only 2D keypoints rather than RGB. For deployment, the reported guidance is to use strong sports-robust 2D detectors such as HRNet or ViTPose, maintain the 17-joint layout and pelvis normalization, run the model on sliding windows of 27 frames, and stitch longer sequences through overlapping windows. Horizontal flip augmentation is part of the preprocessing setup, and HRNet without fine-tuning is used to generate DET inputs in the experiments (Yin et al., 28 Jul 2025).

The stated limitations are equally specific. Soccer kicking remains difficult because rapid limb acceleration and contact-induced deformations are not explicitly modeled, and the gains are reported to be smaller there. The method depends on the quality of 2D detections, and extreme occlusions or multi-person interactions are not addressed. A plausible implication is that further progress would require either stronger detection robustness or explicit modeling of inter-person dynamics and contact events.

The name “KASportsFormer” also requires disambiguation. In a distinct 2021 sports game summarization paper, the term was used only as a mapping to the paper’s knowledge-enhanced summarizer, abbreviated KES; that work concerns generation of sports news from live commentaries and explicitly states that there is no separate model named “KASportsFormer” in that paper (Wang et al., 2021). In current usage, however, KASportsFormer refers to the 2025 “Kinematic Anatomy Enhanced Transformer” for 3D human pose estimation on short sports scene video (Yin et al., 28 Jul 2025). The overlap is purely nominal; the two systems address different tasks, modalities, and architectures.