HMR-ViT: Video Mesh Recovery with ViT

• The paper introduces HMR-ViT, which replaces recurrent temporal aggregation with a Vision Transformer and a learnable Channel Rearranging Matrix to jointly model temporal and kinematic cues.
• It constructs a temporal-kinematic feature image from CNN-extracted frame features and processes patches via ViT followed by a SPIN-style iterative regressor for SMPL parameter estimation.
• Benchmark results on 3DPW show competitive performance with reduced parameters, highlighting improved PVE and MPJPE metrics compared to methods like VIBE.

HMR-ViT, short for "Video Inference for Human Mesh Recovery with Vision Transformer," is a video-based human mesh recovery method that estimates a person’s 3D body mesh and associated SMPL parameters for a target frame from a short RGB video clip of a single person. Its defining claim is that human mesh recovery benefits from modeling both temporal information and kinematic information jointly, rather than emphasizing only one of these cues. The method follows a VIBE-style video pipeline, but replaces recurrent temporal aggregation with a Vision Transformer operating on a specially constructed "Temporal-kinematic Feature Image" (Cho et al., 11 Jul 2025).

1. Problem formulation and output representation

HMR-ViT addresses Human Mesh Recovery from video under the setting in which a short sequence

$V=\{I_t\}_{t=1}^{T}$

is given and the objective is to infer the 3D human body mesh and SMPL parameters for the target frame. The paper identifies the task as inherently ambiguous because image evidence alone is often insufficient due to depth ambiguity, occlusion, self-similar body parts, and motion ambiguity across frames (Cho et al., 11 Jul 2025).

The method predicts the SMPL body model parameters

$$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$

where $\boldsymbol{\theta}\in\mathbb{R}^{72}$ denotes pose parameters, specified as 23 joint rotations plus global orientation in axis-angle form; $\boldsymbol{\beta}\in\mathbb{R}^{10}$ denotes shape parameters; and $\boldsymbol{\pi}=[s,t]$ denotes weak-perspective camera parameters, consisting of scale and translation. The corresponding SMPL body model is

$$B = \mathcal{M}(\boldsymbol{\theta},\boldsymbol{\beta}) \in \mathbb{R}^{6890\times 3},$$

with 3D joints regressed from the mesh by

$J = WB,$

where $W$ is a fixed linear joint regressor. The 2D joints are obtained through weak-perspective projection,

$K=\Pi(J).$

This representation places HMR-ViT within the standard SMPL-based HMR paradigm. A common misconception is that the method directly predicts vertices from video without an intermediate body model; in fact, the prediction target is the SMPL parameter tuple $\Theta$, from which the mesh and joints are derived.

2. End-to-end pipeline

HMR-ViT uses a five-stage pipeline. For each video frame, a frozen pretrained CNN encoder $$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$0, specifically ResNet-50 pretrained for single-image HMR, extracts a feature vector

$$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$1

with $$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$2. The sequence of features is

$$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$3

The encoder is frozen, and precomputed features are used (Cho et al., 11 Jul 2025).

Instead of passing the sequence to an RNN or GRU, HMR-ViT stacks frame features into a 2D tensor

$$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$4

This tensor is the Temporal-kinematic Feature Image. Its height corresponds to time and its width to feature channels. The method interprets the channel dimension as carrying kinematic information because the channels are learned feature components associated with body configuration.

A learnable Channel Rearranging Matrix,

$$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$5

is then applied to refine the feature image: $$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$6 where $$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$7 is matrix multiplication. The refined feature image is treated as a 2D image and divided into patches of temporal height $$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$8 and channel width $$\Theta=\{\boldsymbol{\theta},\boldsymbol{\beta},\boldsymbol{\pi}\}$$9. Each patch is flattened as

$\boldsymbol{\theta}\in\mathbb{R}^{72}$0

with

$\boldsymbol{\theta}\in\mathbb{R}^{72}$1

These patches are processed using standard ViT components: linear projection, positional embedding, and transformer encoder. If $\boldsymbol{\theta}\in\mathbb{R}^{72}$2 denotes the ViT without the classification head, the encoded representation is

$\boldsymbol{\theta}\in\mathbb{R}^{72}$3

Finally, the encoded vector is passed to a regression network $\boldsymbol{\theta}\in\mathbb{R}^{72}$4 to infer

$\boldsymbol{\theta}\in\mathbb{R}^{72}$5

The regressor is the same iterative regressor used in SPIN, with maximum 3 iterations, and the method predicts the SMPL parameters for the middle frame of the input video sequence. This clarifies another possible misunderstanding: the transformer does not directly emit mesh vertices; it produces a latent representation that is decoded by a SPIN-style iterative regressor (Cho et al., 11 Jul 2025).

3. Temporal-kinematic feature image and Channel Rearranging Matrix

The conceptual core of HMR-ViT is the conversion of a video sequence into a 2D feature image whose axes correspond to time and channel-wise kinematic feature components. The paper’s mathematical interpretation is that patching this feature image enables self-attention to model temporal continuity across nearby frames, kinematic dependencies across feature channels, and cross-time and cross-kinematic relationships between patches (Cho et al., 11 Jul 2025).

The Channel Rearranging Matrix is introduced because raw CNN channels are not ordered in a kinematically meaningful way. The CRM is intended to sort or rearrange channels so that spatially close channels correspond to kinematically similar information. Its structure is constrained to behave like a permutation-like sorting matrix: one value of 1 per row and column, and zeros elsewhere. Implementation-wise, it is learned by random initialization and sequential softmax with temperature scaling along rows and columns. A regularization term, $\boldsymbol{\theta}\in\mathbb{R}^{72}$6, enforces the row- and column-sum constraints so that the matrix becomes an appropriate sorting matrix.

This design implies a particular view of the channel axis. HMR-ViT does not assume that convolutional feature channels are inherently aligned with the human kinematic tree; rather, it imposes a learned reordering so that local ViT patches cover more coherent body-related semantics. The paper reports that CRM converges from a random matrix to a near-sorting matrix. This suggests that the benefit is not only architectural but also organizational: the transformer sees a tokenization whose locality is made more semantically meaningful.

A useful contrast appears in neighboring transformer-based HMR work. "DeforHMR: Vision Transformer with Deformable Cross-Attention for 3D Human Mesh Recovery" uses a frozen ViT encoder and a transformer decoder with query-agnostic deformable cross-attention for single-image HMR, emphasizing spatially nuanced decoding rather than video-based temporal modeling (Heo et al., 2024). The comparison highlights that HMR-ViT’s novelty lies specifically in a video representation that couples time and channel-wise kinematic structure.

4. Training objective and optimization

HMR-ViT uses the standard HMR training losses. The 2D keypoint loss is

$\boldsymbol{\theta}\in\mathbb{R}^{72}$7

the 3D joint loss is

$\boldsymbol{\theta}\in\mathbb{R}^{72}$8

the pose loss is

$\boldsymbol{\theta}\in\mathbb{R}^{72}$9

and the shape loss is

$\boldsymbol{\beta}\in\mathbb{R}^{10}$0

The CRM regularization term $\boldsymbol{\beta}\in\mathbb{R}^{10}$1 encourages each row and column of the CRM to sum to 1. The total loss is

$\boldsymbol{\beta}\in\mathbb{R}^{10}$2

with weights $\boldsymbol{\beta}\in\mathbb{R}^{10}$3, $\boldsymbol{\beta}\in\mathbb{R}^{10}$4, $\boldsymbol{\beta}\in\mathbb{R}^{10}$5, $\boldsymbol{\beta}\in\mathbb{R}^{10}$6, and $\boldsymbol{\beta}\in\mathbb{R}^{10}$7. Losses are used when corresponding supervision is available (Cho et al., 11 Jul 2025).

Optimization uses Adam with batch size 32, training for 300 epochs, and learning rate $\boldsymbol{\beta}\in\mathbb{R}^{10}$8. The image encoder is ResNet-50 pretrained on HMR/SPIN-style single-image mesh recovery; the ViT follows the standard Dosovitskiy et al. architecture except that the classification head is removed and the number of transformer encoder layers is adjusted; the regressor is the SPIN regressor initialized from SPIN pretrained weights; and the regressor performs 3 iterations.

For fair comparison with VIBE, the paper also reports results with an AMASS-based motion compensation constraint using the same discriminator as VIBE. This is explicitly described as an optional training addition rather than a required component of the core method. That distinction matters when interpreting reported numbers: the base HMR-ViT model is presented as not needing the same extra AMASS-based setup used in the VIBE comparison.

5. Quantitative performance and ablation evidence

The principal benchmark evaluation is on 3DPW using PVE, MPJPE, and PA-MPJPE. Reported results are as follows (Cho et al., 11 Jul 2025).

Method	PVE	MPJPE	PA-MPJPE
HMR	-	130.0	76.7
SPIN	116.4	96.9	59.2
HMMR	139.3	116.5	72.6
VIBE	113.4	93.5	56.5
HMR-ViT	112.0	93.2	59.3
HMR-ViT + motion disc.	110.8	91.5	58.4
HMR-ViT trained with 3DPW	103.8	85.4	55.9

The paper’s stated takeaways are that HMR-ViT improves over HMMR and SPIN, and slightly improves over VIBE in PVE and MPJPE. With motion discriminator training, HMR-ViT improves further and is reported as achieving better performance than VIBE under the same AMASS-based setting. The paper also emphasizes that this is achieved with fewer parameters and without needing the same extra AMASS-based setup in the base model.

An ablation on the role of CRM and the feature-image formulation is reported on 3DPW:

Method	PVE	MPJPE	PA-MPJPE
Baseline	114.2	94.2	58.0
HMR-ViT w/o CRM	113.3	93.6	59.2
HMR-ViT	112.0	93.2	59.3

Here, "Our baseline" is defined as a naive transformer applied directly to frame features as tokens. The paper interprets these results as showing that the naive transformer baseline is already competitive, that the feature-image formulation is better, and that adding CRM gives further improvement. The numerical pattern is mixed on PA-MPJPE, where the full model is not the lowest entry in the table. A plausible implication is that the paper’s strongest empirical support for CRM lies in PVE and MPJPE, together with the reported convergence of CRM to a near-sorting matrix.

The paper also studies patch size on Human3.6M using MPJPE. For $\boldsymbol{\beta}\in\mathbb{R}^{10}$9, $\boldsymbol{\pi}=[s,t]$0 yields 64.7 and $\boldsymbol{\pi}=[s,t]$1 yields 65.2. For $\boldsymbol{\pi}=[s,t]$2, $\boldsymbol{\pi}=[s,t]$3 yields 65.4 and $\boldsymbol{\pi}=[s,t]$4 yields 66.1. The reported conclusion is that smaller patches perform better, indicating that finer temporal and kinematic tokenization helps the transformer model these structures.

6. Efficiency, interpretation, and position within HMR research

HMR-ViT is reported to have 44M parameters, compared with 69M for VIBE (Cho et al., 11 Jul 2025). In the paper’s framing, this supports two simultaneous claims: competitive or better performance, and improved parameter efficiency. Since the image encoder is frozen and precomputed features are used, the architecture can also be understood as separating per-frame visual encoding from sequence-level temporal-kinematic reasoning.

Its main conceptual contribution is summarized by a five-step design: CNN encodes each frame into a feature vector; the vectors are arranged into a temporal-kinematic image; a learnable CRM makes channel positions more semantically coherent; ViT models patchwise interactions across time and kinematics; and a regression head predicts SMPL pose, shape, and camera parameters. The stated rationale is that temporal context reduces uncertainty from single-frame depth ambiguity, kinematic organization makes body-related dependencies easier to model, and transformer attention captures long-range dependencies across both dimensions.

Within the broader HMR literature, prior video HMR methods cited by the paper mostly exploit temporal information, including VIBE, HMMR, and TCMR, while transformer-based mesh methods often emphasize kinematic relationships among body-related features. HMR-ViT positions itself between these lines of work by asserting that both temporal and kinematic cues should be modeled jointly. This does not mean that HMR-ViT is a generic ViT replacement for all HMR pipelines. More precisely, it is a specific video-HMR formulation in which a Vision Transformer operates not on raw RGB tokens but on a reorganized feature image built from frozen ResNet-50 frame features.

No extensive limitation section is reported in the HMR-ViT summary. However, the motivating ambiguities identified by the paper—depth ambiguity, occlusion, self-similar body parts, and motion ambiguity across frames—remain the operative challenges. The method is therefore best understood not as eliminating these ambiguities, but as providing a representation and inference mechanism intended to resolve them more effectively by coupling temporal context with kinematic organization.