HaMeR: Hand Mesh Recovery Framework

• HaMeR is a monocular 3D hand reconstruction framework that uses a Vision Transformer backbone and a transformer-decoder to map cropped RGB hand images to MANO pose, shape, and camera parameters.
• It employs a composite supervision regime with parameter, 2D reprojection, and adversarial losses, trained on 2.7M images to ensure accuracy and robustness across diverse conditions.
• Its latent features are repurposed for downstream tasks such as continuous sign segmentation, gloss-free sign language translation, and multimodal sign generation, showcasing practical applicability.

HaMeR, short for Hand Mesh Recovery, is a monocular 3D hand reconstruction framework that maps a cropped RGB hand image to MANO pose, shape, and camera parameters through a fully transformer-based architecture. In its original formulation, HaMeR combines a large Vision Transformer backbone with a transformer-decoder regressor head and a differentiable MANO layer, and it was introduced as a high-capacity system for accurate and robust hand analysis in both controlled and in-the-wild settings (Pavlakos et al., 2023). Subsequent work has also reused HaMeR as a hand-centric representation layer: instead of reconstructing meshes explicitly, downstream systems extract and freeze internal hand-pose and orientation parameters for continuous sign segmentation, gloss-free sign language translation, and multimodal sign generation (He et al., 11 Apr 2025).

1. Original formulation and representational target

HaMeR was introduced as an approach for reconstructing hands in 3D from monocular input. The pipeline begins with a hand crop from an RGB image, resized to $224\times224$, which is split into $14\times14$ patches. These patches are linearly embedded, augmented with absolute positional embeddings, and processed by a ViT-H encoder. The encoder handles $N=(224/14)^2=256$ patch tokens plus one class token; the class token is dropped downstream, and the remaining 256 patch tokens are consumed by a transformer-decoder regressor with a single learned query token (Pavlakos et al., 2023).

The final decoder output is projected into the parameter vector

$\Theta=\{\theta,\beta,\pi\},$

where $\theta\in\mathbb{R}^{48}$ represents MANO hand pose, $\beta\in\mathbb{R}^{10}$ represents low-dimensional PCA shape coefficients, and $\pi=\{t\in\mathbb{R}^{3}\}$ represents weak-perspective translation. Given $\theta$ and $\beta$, a differentiable MANO model produces $M(\theta,\beta)$, yielding $14\times14$0 mesh vertices and $14\times14$1 joints; under known intrinsics $14\times14$2, 2D keypoints are obtained by projection as $14\times14$3 (Pavlakos et al., 2023).

The architectural rationale given for HaMeR is twofold: scaling up the training data and scaling up the model capacity. The backbone is the “huge” Vision Transformer variant with patch size $14\times14$4, embedding dimension $14\times14$5, encoder depth $14\times14$6, feed-forward dimension $14\times14$7, and $14\times14$8 attention heads per block. The decoder is a single-token transformer-decoder head that cross-attends to the encoder tokens and replaces the MLP bottleneck common in earlier hand mesh recovery pipelines (Pavlakos et al., 2023).

2. Supervision regime, training corpus, and benchmark performance

HaMeR is trained with a composite objective that combines direct parameter and joint supervision, 2D reprojection supervision, and adversarial priors. The reported losses are

$14\times14$9

$N=(224/14)^2=256$0

$N=(224/14)^2=256$1

and

$N=(224/14)^2=256$2

The paper states that equal weighting is used by default and omits exact $N=(224/14)^2=256$3 values. The adversarial term is implemented with discriminators on overall pose, shape, and per-joint angle subsets in an LSGAN setup (Pavlakos et al., 2023).

The training corpus comprises 2.7 M hand images, approximately 4× more than FrankMocap, drawn from controlled 3D-supervised datasets and in-the-wild 2D-supervised datasets. The summary specifies a split of roughly 95% studio/3D and 5% in-the-wild/2D. This scale is central to the method’s stated robustness under occlusion and domain shift (Pavlakos et al., 2023).

Benchmark	Reported results	Comparison context
FreiHAND	PA-MPJPE 6.0 mm, PA-MPVPE 5.7 mm, F@5mm 0.785, F@15mm 0.990	Ties or outscores prior baselines on mesh metrics
HO3Dv2	AUC$N=(224/14)^2=256$4 0.846, PA-MPJPE 7.7 mm, AUC$N=(224/14)^2=256$5 0.841, PA-MPVPE 7.9 mm, F@5mm 0.635, F@15mm 0.980	Outperforms HandOccNet and AMVUR on PA-MPJPE
HInt	PCK@0.05, all joints: 48.0% on New Days, 43.0% on VISOR, 38.9% on Ego4D	Large gains over prior 3D methods on in-the-wild data

On HInt, the reported visible-joint PCK ranges from 60–88%, whereas occluded-joint PCK is 23–27%. The paper also reports runtime of approximately 60 ms per frame on a single modern GPU, framing HaMeR as near-real-time and suitable for temporally smooth video results (Pavlakos et al., 2023).

3. Hand-centric latent representation beyond full mesh recovery

A major development after the original reconstruction paper is the reuse of HaMeR as a frozen feature extractor rather than a full mesh regressor. In continuous sign segmentation, the open-sourced HaMeR model is used for 3D hand mesh recovery, but the downstream system specifically “repurpose[s] the transformer head to extract hand pose and global orientation parameters.” These outputs are organized as

$N=(224/14)^2=256$6

where each $N=(224/14)^2=256$7 block in $N=(224/14)^2=256$8 is a joint-local rotation matrix $N=(224/14)^2=256$9 for hand $\Theta=\{\theta,\beta,\pi\},$0 and joint $\Theta=\{\theta,\beta,\pi\},$1, while $\Theta=\{\theta,\beta,\pi\},$2 contains two global hand-orientation matrices $\Theta=\{\theta,\beta,\pi\},$3 (He et al., 11 Apr 2025).

For downstream computation, these tensors are flattened: $\Theta=\{\theta,\beta,\pi\},$4 The resulting HaMeR feature is therefore a 288-dimensional real vector per frame. The summary explicitly notes that no further normalization, such as mean–std standardization, is reported. Frames are processed at the original frame rate, stated as 50 fps for DGS, and then temporally downsampled by a factor of 2 before feature fusion. The HaMeR transformer head is kept frozen, and during segmentation training and inference these features remain fixed (He et al., 11 Apr 2025).

This reuse clarifies an important point about HaMeR’s practical scope. In downstream sign-language systems, the mesh itself is often not the computational object of interest. Instead, the salient product is a compact rotation-matrix representation of hand articulation and global orientation. A plausible implication is that HaMeR’s value in these settings lies less in explicit vertex recovery than in the structure of its latent geometric parameterization.

4. Continuous sign segmentation with HaMeR features

In “Hands-On: Segmenting Individual Signs from Continuous Sequences,” HaMeR is fused with a 3D skeleton-angle stream to support BIO sequence labeling for continuous sign segmentation. Alongside the HaMeR feature $\Theta=\{\theta,\beta,\pi\},$5, the system extracts a 3D-angle feature $\Theta=\{\theta,\beta,\pi\},$6. Two feature-specific auxiliary MLPs, each with 3 layers, project the inputs to

$\Theta=\{\theta,\beta,\pi\},$7

After temporal downsampling, the features are concatenated,

$\Theta=\{\theta,\beta,\pi\},$8

refined by a 3-layer mixer MLP into $\Theta=\{\theta,\beta,\pi\},$9, and then passed to a standard transformer encoder for per-frame BIO tagging. The summary emphasizes that no architectural changes are made to the transformer itself; only the input embedding is modified (He et al., 11 Apr 2025).

The reported DGS ablation isolates the effect of HaMeR and feature fusion under frame-level $\theta\in\mathbb{R}^{48}$0 in the BIO scheme. Using 3D angles alone gives $\theta\in\mathbb{R}^{48}$1; HaMeR alone gives $\theta\in\mathbb{R}^{48}$2; both fused without MLPs give $\theta\in\mathbb{R}^{48}$3; adding auxiliary MLPs gives $\theta\in\mathbb{R}^{48}$4; and adding the mixer MLP gives $\theta\in\mathbb{R}^{48}$5. The same study reports state-of-the-art results on the DGS Corpus and states that the HaMeR-based features surpass prior benchmarks on BSLCorpus (He et al., 11 Apr 2025).

The BSLCorpus comparison, under an MS-TCN baseline, reports that replacing I3D with HaMeR raises mF1B from 68.68 to 76.22 and mF1S from 47.71 to 49.56; adding 3D angles on top of HaMeR yields mF1S = 50.18. No formal $\theta\in\mathbb{R}^{48}$6-values or significance tests are reported, but the summary characterizes the gains as substantial. It also records two practical observations: feature-specific MLP adapters are crucial, raising $\theta\in\mathbb{R}^{48}$7 by approximately $\theta\in\mathbb{R}^{48}$8 percentage points on DGS, and a lightweight MLP mixer is more effective than a more complex cross-attention mixer, which degrades performance. The reported limitation is that HaMeR currently provides only rotation matrices, without joint translations or velocities; future extensions are suggested to include fingertip trajectories or temporal-derivative features (He et al., 11 Apr 2025).

5. Translation and generation uses in sign-language research

HaMeR has also been incorporated into gloss-free sign language translation and sign-language generation systems as a hand-specific modality. In BeyondGloss, each raw RGB frame $\theta\in\mathbb{R}^{48}$9 is processed through the HaMeR pipeline up to its transformer head, yielding the same two groups of per-frame parameters used in continuous segmentation: $\beta\in\mathbb{R}^{10}$0 These are flattened into $\beta\in\mathbb{R}^{10}$1 and $\beta\in\mathbb{R}^{10}$2, concatenated into

$\beta\in\mathbb{R}^{10}$3

and used as a teacher target for distilling hand-aware structure into DINO V2 video features. The paper introduces a mapper $\beta\in\mathbb{R}^{10}$4 from DINO features to HaMeR space, supervises it with

$\beta\in\mathbb{R}^{10}$5

and then maps the distilled features back into DINO space with $\beta\in\mathbb{R}^{10}$6, concatenating them with the original visual stream before temporal encoding. The reported ablations state that adding $\beta\in\mathbb{R}^{10}$7 on top of video–description and video–target contrastive pre-training yields a +2–3 point improvement in BLEU-4 and ROUGE-L (Asasi et al., 31 Jul 2025).

In SignAligner, HaMeR is elevated from a latent teacher signal to a full pose modality. The paper defines, at each frame $\beta\in\mathbb{R}^{10}$8, a gesture parameter $\beta\in\mathbb{R}^{10}$9, a shape parameter $\pi=\{t\in\mathbb{R}^{3}\}$0, and a high-resolution mesh $\pi=\{t\in\mathbb{R}^{3}\}$1 with $\pi=\{t\in\mathbb{R}^{3}\}$2, obtained by a differentiable function $\pi=\{t\in\mathbb{R}^{3}\}$3. This yields either a compact feature

$\pi=\{t\in\mathbb{R}^{3}\}$4

or a mesh representation. The coordinate convention places the wrist at the origin, with axes aligned to camera image axes. Within the model, the Hamer stream is linearly embedded, decoded by a stack of transformer decoder blocks with text cross-attention, and then refined by a triple cross-modal attention mechanism with Pose and Smplerx. Its correction loss is

$\pi=\{t\in\mathbb{R}^{3}\}$5

and the overall multimodal correction loss is dynamically weighted by learned scalars $\pi=\{t\in\mathbb{R}^{3}\}$6 passed through a softmax (Wang et al., 13 Jun 2025).

The reported Text-to-Hamer results in SignAligner are: BLEU-1 29.94% versus 13.26% for baseline PTSLP, ROUGE 29.12% versus 13.03%, SSIM 0.96 versus 0.95, PSNR 21.31 dB versus 19.06 dB, and FID 4.43 versus 25.65. The ablation reports a +16.68% BLEU-1 gain for the Hamer stream when HaMeR co-generation and online collaborative correction are added (Wang et al., 13 Jun 2025). Taken together, these two works show that HaMeR can function either as a distillation target for hand-aware video encoding or as an explicit generative modality coupled to text.

6. Efficiency variants, deployment trade-offs, and nomenclature

The main limitation of original HaMeR in deployment settings is computational cost. Fast-HaMeR addresses this by keeping the decoder and MANO head while replacing the original backbone with lighter alternatives: MobileNet-L, MobileViT-S, ResNet-50 / -101, and ConvNeXt-L. In this formulation, the student network predicts $\pi=\{t\in\mathbb{R}^{3}\}$7 and is trained with ground-truth losses plus teacher-based knowledge distillation from a frozen HaMeR model. Three distillation strategies are defined: output-level distillation,

$\pi=\{t\in\mathbb{R}^{3}\}$8

feature-level distillation across selected layers,

$\pi=\{t\in\mathbb{R}^{3}\}$9

and a hybrid objective that combines both terms (Jillani et al., 17 Mar 2026).

On HO3D-v2 and RTX-4060Ti, the teacher HaMeR is reported at 671 M parameters, 167.3 GFLOPs, and 27 FPS. By comparison, ConvNeXt-L has 240 M parameters, 34.4 GFLOPs, and 40 FPS; ResNet-50 has 69.3 M, 4.1 GFLOPs, and 50 FPS; MobileNet-L has 42.0 M, 0.2 GFLOPs, and 48 FPS. In accuracy terms, HaMeR reports 7.7 mm PA-MPJPE and 7.9 mm PA-MPVPE, while ConvNeXt-L + feature KD reports 8.1 mm and 8.3 mm with F@5/F@15 = 0.599 / 0.979. The paper summarizes the central trade-off as a 1.5× faster inference speed with only a 0.4 mm accuracy difference for lightweight backbones that are 35% the size of the original (Jillani et al., 17 Mar 2026).

The same study reports that output-level distillation is most helpful for mid-sized students, feature-level distillation benefits only high-capacity students such as ConvNeXt-L, and hybrid distillation rarely surpasses the best single strategy. This suggests that HaMeR’s decoder and MANO parameterization remain portable across backbones, but the usefulness of teacher signals depends strongly on student capacity.

A recurring misconception is that “HaMeR” always denotes explicit mesh reconstruction during inference. The downstream literature shows otherwise: in segmentation and translation, systems frequently use only frozen transformer-head outputs such as joint rotation matrices and global orientations, with no mesh reconstruction step in the operational pipeline (He et al., 11 Apr 2025). A separate nomenclature issue is that Hamer-type system in the PDE literature refers to an unrelated parabolic–elliptic model in radiation hydrodynamics, not to Hand Mesh Recovery; for example, “Propagating Fronts for a Viscous Hamer-Type system” studies traveling fronts for a $\theta$0 viscous conservation-law system coupled to an elliptic equation (Carnevale et al., 2021).