Fast-HaMeR: Accelerated 3D Hand Mesh Reconstruction

• Fast-HaMeR is a knowledge-distilled approach for monocular 3D hand mesh reconstruction that replaces a heavy ViT-H backbone with lighter alternatives.
• It employs both output-level and feature-level distillation to mimic the teacher network, achieving faster inference while incurring minimal accuracy loss.
• The method demonstrates near-teacher performance with up to 1.5x faster speed and 36% reduced model size on challenging hand-object interaction datasets.

Fast-HaMeR is a knowledge-distilled variant of HaMeR for monocular 3D hand mesh reconstruction. Introduced as a distillation-driven acceleration of the original HaMeR pipeline, it replaces HaMeR’s ViT-H backbone with lightweight backbones while training the student model to mimic the teacher’s predictions and, in some configurations, its intermediate features. The resulting models are smaller and faster than the teacher while preserving nearly the same reconstruction quality, with the central empirical claim being that backbones at roughly $35\%$ of the original size can achieve $1.5\times$ faster inference with only about $0.4$ mm degradation in PA-MPVPE on HO3D-v2 (Jillani et al., 17 Mar 2026).

1. Conceptual position within 3D hand mesh recovery

Fast-HaMeR is built directly on HaMeR, a fully transformer-based hand mesh recovery system for reconstructing a full 3D hand from monocular RGB input. In the HaMeR formulation, the model regresses MANO hand parameters, 3D joints, a 3D mesh, and camera parameters from a cropped hand image using a ViT-H backbone and a transformer decoder head (Pavlakos et al., 2023). HaMeR’s central design principle is scaling: it combines a high-capacity Vision Transformer with a large multi-dataset supervision pool.

Within that lineage, Fast-HaMeR does not introduce a new hand parameterization or a new decoder topology. Instead, it preserves the overall HaMeR structure—backbone, transformer head, and MANO-based output layer—but substitutes the heavy ViT-H encoder with lighter alternatives and compensates for the corresponding capacity loss through knowledge distillation. This places Fast-HaMeR within the class of acceleration methods that preserve a strong teacher architecture as the reference system and optimize the student primarily through representation transfer rather than architectural redesign (Jillani et al., 17 Mar 2026).

A useful way to characterize the method is as a deployment-oriented reformulation of HaMeR. The original HaMeR paper reports strong benchmark performance, but its ViT-H backbone implies high computational cost and makes real-time operation challenging without optimization (Pavlakos et al., 2023). Fast-HaMeR addresses exactly that bottleneck.

2. Model structure and student backbones

The teacher model is HaMeR. In the formulation used by Fast-HaMeR, HaMeR is a fully transformer-based hand mesh regressor whose ViT-H backbone produces a token sequence consumed by a transformer decoder head. That head regresses MANO pose parameters $\theta \in \mathbb{R}^{48}$, MANO shape parameters $\beta \in \mathbb{R}^{10}$, and camera parameters $\pi$, and produces 3D hand vertices and 21 joints. The decoder uses self-attention over a single query token with cross-attention to backbone tokens (Jillani et al., 17 Mar 2026).

The underlying hand model remains MANO. Fast-HaMeR retains the same parametric mapping from pose and shape to geometry:

$V = V(\beta, \theta), \qquad J = J(V).$

The regressed joints can be obtained via a learned linear regressor from vertices. For projection, the paper presents the commonly used weak-perspective form

$x = s R X + t,$

while also noting that HaMeR regresses a 3D translation $t \in \mathbb{R}^3$, whose image-plane projection yields 2D joints (Jillani et al., 17 Mar 2026).

The student models preserve the HaMeR topology but replace the original backbone with one of five lighter families:

• MobileNet-Large
• MobileViT-Small
• ResNet-50
• ResNet-101
• ConvNeXt-Large

For feature-level distillation, Fast-HaMeR introduces a learnable $1\times1$ convolution $1.5\times$0 to align teacher and student feature dimensionality, with spatial mismatches handled by bilinear interpolation. This is an auxiliary training component rather than an inference-time architectural block (Jillani et al., 17 Mar 2026).

The design implication is straightforward: the decoder head and MANO output path are left largely intact, so the speed–accuracy trade-off is controlled primarily through encoder replacement and the choice of distillation signal.

3. Distillation formulation

Fast-HaMeR studies three knowledge-distillation strategies: output-level distillation, feature-level distillation, and a hybrid combination of both. The teacher $1.5\times$1 is frozen during student training. The student $1.5\times$2 predicts features $1.5\times$3 and outputs $1.5\times$4, where

$1.5\times$5

The base supervision follows the HaMeR training structure. The main task loss is

$1.5\times$6

with

$1.5\times$7

$1.5\times$8

and

$1.5\times$9

For output-level distillation, the student is trained to match the teacher’s predicted joints and parameter outputs:

$0.4$0

The corresponding total loss is

$0.4$1

For feature-level distillation, the student is trained to match the teacher’s intermediate representation after channel projection:

$0.4$2

with

$0.4$3

The hybrid objective combines both signals:

$0.4$4

The paper studies $0.4$5 and $0.4$6. The teacher is used only during training; inference uses only the student. This separation is central to Fast-HaMeR’s deployment logic, because the computational cost of the distillation apparatus vanishes at test time (Jillani et al., 17 Mar 2026).

4. Training data, protocol, and evaluation

Fast-HaMeR trains students on the same multi-dataset pool used by HaMeR, approximately $0.4$7 million samples. The supervised sources include FreiHAND, HO3D/H2O3D, InterHand2.6M, MTC, DexYCB, and RHD for 3D or MANO supervision, together with COCO WholeBody, AlphaPose WholeBody, MPII, and NZSL for 2D joint supervision (Jillani et al., 17 Mar 2026). This continuity is important: the acceleration claims are not achieved by changing the data regime.

Students are initialized from ImageNet-1K pretrained weights for the selected backbone. The teacher is the frozen ViT-H HaMeR model. Quantitative comparison is performed primarily on HO3D-v2, which the paper describes as challenging because of realistic hand–object interactions (Jillani et al., 17 Mar 2026).

The evaluation protocol reports the following quantities:

• PA-MPJPE ($0.4$8)
• PA-MPVPE ($0.4$9)
• $\theta \in \mathbb{R}^{48}$0 mm
• $\theta \in \mathbb{R}^{48}$1 mm
• inference speed in FPS on an RTX 4060 Ti
• model size in parameters
• GFLOPs from torchvision

This combination makes the paper’s comparison explicitly multi-objective. Fast-HaMeR is not evaluated only as a compression method or only as a regression method; it is evaluated as a speed–accuracy trade-off mechanism.

5. Empirical performance and speed–accuracy trade-offs

The teacher HaMeR baseline on HO3D-v2 is reported as $\theta \in \mathbb{R}^{48}$2, $\theta \in \mathbb{R}^{48}$3, $\theta \in \mathbb{R}^{48}$4, $\theta \in \mathbb{R}^{48}$5, with $\theta \in \mathbb{R}^{48}$6M parameters and $\theta \in \mathbb{R}^{48}$7 FPS. Student models without distillation already yield substantial speed gains, but at a clear accuracy cost. Distillation narrows that gap, with the best overall balance obtained by ConvNeXt-L under feature-level distillation (Jillani et al., 17 Mar 2026).

Configuration	HO3D-v2 accuracy	Size / speed
HaMeR teacher (ViT-H)	$\theta \in \mathbb{R}^{48}$8, $\theta \in \mathbb{R}^{48}$9, $\beta \in \mathbb{R}^{10}$0, $\beta \in \mathbb{R}^{10}$1	671M, 27 FPS
ConvNeXt-L, no KD	$\beta \in \mathbb{R}^{10}$2, $\beta \in \mathbb{R}^{10}$3, $\beta \in \mathbb{R}^{10}$4, $\beta \in \mathbb{R}^{10}$5	240M, 40 FPS
ConvNeXt-L, feature KD	$\beta \in \mathbb{R}^{10}$6, $\beta \in \mathbb{R}^{10}$7, $\beta \in \mathbb{R}^{10}$8, $\beta \in \mathbb{R}^{10}$9	240M, 40 FPS
MobileViT-S, output KD	$\pi$0, $\pi$1, $\pi$2, $\pi$3	42.0M, 42 FPS
ResNet-101, output KD	$\pi$4, $\pi$5, $\pi$6, $\pi$7	88.3M, 36 FPS

The key reported outcome is the ConvNeXt-L configuration with feature-level distillation at $\pi$8 and $\pi$9, which reaches $V = V(\beta, \theta), \qquad J = J(V).$0, $V = V(\beta, \theta), \qquad J = J(V).$1, $V = V(\beta, \theta), \qquad J = J(V).$2, and $V = V(\beta, \theta), \qquad J = J(V).$3. The paper identifies this as near-teacher accuracy at $V = V(\beta, \theta), \qquad J = J(V).$4 FPS and about $V = V(\beta, \theta), \qquad J = J(V).$5 of the teacher’s size (Jillani et al., 17 Mar 2026).

The ablation pattern is structurally informative. Output-level KD improves MobileViT-S from $V = V(\beta, \theta), \qquad J = J(V).$6 to $V = V(\beta, \theta), \qquad J = J(V).$7, which the paper attributes to architectural affinity with the transformer teacher. ResNet-101 also benefits from output-level KD, improving to $V = V(\beta, \theta), \qquad J = J(V).$8. By contrast, feature-level KD is ineffective or slightly harmful for MobileNet-L, MobileViT-S, ResNet-50, and ResNet-101, but distinctly effective for ConvNeXt-L. Hybrid KD generally lands between output-only and feature-only performance and does not surpass feature-only KD for ConvNeXt-L (Jillani et al., 17 Mar 2026).

This suggests that the usefulness of a distillation signal is capacity-dependent and architecture-dependent rather than universal.

6. Interpretation, limitations, and deployment significance

The paper’s interpretation is explicit. Output-level distillation helps smaller students because it supervises directly on regressands—joints, 2D projections, and parameters—without requiring the student to emulate a complex high-dimensional internal feature geometry. Feature-level distillation is more effective for higher-capacity students because those models can absorb and use a richer internal representation; for ConvNeXt-L, feature alignment acts as a strong regularizer on the student feature space (Jillani et al., 17 Mar 2026).

Several limitations are also stated. Domain shifts, severe occlusions, multi-hand interactions, and annotation noise remain challenges. Fast-HaMeR inherits HaMeR’s reliance on large-scale training data. Robust generalization beyond HO3D-v2 is suggested by qualitative results on in-the-wild images but is not quantitatively measured. The paper also notes that combined KD can dilute the stronger signal of feature KD for high-capacity students (Jillani et al., 17 Mar 2026).

From a deployment standpoint, the method is explicitly intended for resource-constrained devices. The recommended backbone choice depends on the device budget: MobileNet or MobileViT for tight latency and thermal constraints, ConvNeXt-L for desktop or premium mobile SoCs. The students are described as quantization-friendly, especially the CNN-heavy variants, and the paper recommends export through ONNX to TensorRT for FP16 or INT8 acceleration. The feature-alignment layer $V = V(\beta, \theta), \qquad J = J(V).$9 is used only during training and is not required at inference. Code and models are publicly available at the project repository listed in the paper (Jillani et al., 17 Mar 2026).

In the broader trajectory of 3D hand reconstruction, Fast-HaMeR can therefore be understood as a systems-oriented refinement of HaMeR rather than a replacement for it. It preserves the teacher’s MANO-based regression formulation and large-scale supervision regime, but rebalances the architecture around lighter backbones and selective distillation. The principal empirical conclusion is not merely that smaller backbones are viable, but that different students require different distillation signals: output-level KD for smaller or transformer-like students, and feature-level KD for higher-capacity backbones such as ConvNeXt-L.