---
title: 'XFormer: Real-Time Monocular 3D Body Capture'
url: https://www.emergentmind.com/topics/xformer
type: topic
---

# XFormer: Real-Time Monocular 3D Body Capture

XFormer most commonly denotes a two-branch, cross-modal Transformer–based system for real-time monocular 3D human body capture that estimates 3D human mesh vertices, 3D joints, and a weak-perspective camera from monocular images [2305.11101]. Introduced in "XFormer: Fast and Accurate Monocular 3D Body Capture" [2305.11101], the method couples a keypoint branch, driven by 2D joint coordinates, with an image branch, driven by CNN image features, and exchanges information between them through cross-modal attention. Its design is notable for accommodating heterogeneous supervision, including images with 2D or 3D annotations, images with pseudo 3D labels, and motion-capture data without associated images, while maintaining real-time inference on consumer CPUs [2305.11101].

## 1. Problem formulation and system scope

XFormer addresses monocular 3D body capture under a deployment constraint that is explicit in the original work: real-time performance on consumer CPUs from monocular RGB input alone [2305.11101]. The system is formulated as a direct predictor of 3D body joints, dense mesh vertices, and a weak-perspective camera. In the reported architecture, these predictions are produced independently by two parallel branches and then refined through cross-modal interaction.

The keypoint branch takes 2D keypoint inputs \(C\in\mathbb R^{K\times2}\), either predicted from the image or projected from motion-capture data, and regresses \(J^{3D}_{kp}\), \(V^{3D}_{kp}\), and \(\Pi_{kp}\). The image branch takes grid feature maps and a global pooled feature from a CNN backbone—MobileNetV3 for the small model and HRNet-W64 for the large model—tokenizes these together with template joint and vertex coordinates, and regresses \(J^{3D}_{img}\), \(V^{3D}_{img}\), and \(\Pi_{img}\) [2305.11101].

A common misconception is to read XFormer as a purely image-only transformer for 3D pose estimation. In this usage, that is inaccurate: the method is explicitly bimodal, with one modality built from 2D keypoint coordinates and the other from RGB image features. A plausible implication is that the architecture treats geometric evidence from sparse keypoints and appearance evidence from dense image features as complementary error-correcting signals rather than interchangeable representations.

## 2. Cross-modal Transformer architecture

At the center of the model are XFormer blocks that mediate information exchange between the two branches. Each block contains four stages: front self-attention on each modality separately, cross-modal multi-head attention, back self-attention on each modality, and a modality switch for cases where one modality is missing [2305.11101]. This sequencing is architecturally important because cross-modal fusion is inserted between modality-specific processing stages rather than replacing them.

Cross-modal attention is implemented by swapping keys and values between modalities. For image tokens \(F_{img}\in\mathbb R^{N_{img}\times C_t}\) and keypoint tokens \(F_{kp}\in\mathbb R^{K'\times C_t}\), the model computes
\[
Q_{img}=F_{img}W^Q_{img},\qquad K_{img}=F_{img}W^K_{img},\qquad V_{img}=F_{img}W^V_{img},
\]
\[
Q_{kp}=F_{kp}W^Q_{kp},\qquad K_{kp}=F_{kp}W^K_{kp},\qquad V_{kp}=F_{kp}W^V_{kp}.
\]
The cross-modal outputs are then
\[
F_{img}^{MHA}
=\mathrm{softmax}\Bigl(\tfrac{Q_{img}K_{kp}^T}{\sqrt{C_t}}\Bigr)V_{kp},\qquad
F_{kp}^{MHA}
=\mathrm{softmax}\Bigl(\tfrac{Q_{kp}K_{img}^T}{\sqrt{C_t}}\Bigr)V_{img},
\]
followed by residual addition and layer normalization,
\[
F_{img}^{att}=\mathrm{LN}\bigl(F_{img}+F_{img}^{MHA}\bigr),\qquad
F_{kp}^{att}=\mathrm{LN}\bigl(F_{kp}+F_{kp}^{MHA}\bigr)
\]
[2305.11101].

The use of swapped keys and values gives the architecture a specific interpretation: image queries attend over keypoint-derived structure, and keypoint queries attend over image-derived appearance. This suggests that XFormer is less a generic fusion block than a directed mechanism for exchanging geometric and photometric context under a shared tokenization framework.

## 3. Input representations and missing-modality handling

The 2D keypoint pathway begins with a heatmap-based decoder operating on shared backbone features. For each of \(K\) joints, the network predicts a confidence heatmap \(\mathcal H_k\) and a small offset map \(\mathcal O_k\), and the final coordinate is obtained as
\[
C_k = \arg\max_{(u,v)}\mathcal H_k(u,v) \;+\;\mathcal O_k\bigl(\arg\max\mathcal H_k\bigr).
\]
These \(K\) coordinate vectors are embedded by a small Graph Convolutional Network to produce per-joint features, concatenated with their 2D coordinates, mean-pooled into a global keypoint feature, and appended to template joint and vertex tokens to form \(F_{kp}\) [2305.11101].

The image pathway takes an input crop \(I\in\mathbb R^{H\times W\times3}\) and processes it through MobileNetV3 or HRNet. The model extracts a spatial grid of size \(\tfrac H4\times\tfrac W4\times C\) and a global pooled feature. Token embeddings are then created by concatenating each spatial location and the global vector with the fixed 3D coordinates of the \(M=431\) coarse mesh vertices and \(K\) joints used for positional encoding, yielding \(F_{img}\) [2305.11101].

A distinctive component is the modality switch. When image features are unavailable—for example, during training on AMASS, which provides motion capture without associated images—the block replaces \(F_{kp}^{MHA}\) with an MLP prediction \(F_{kp}^{MLP}\). The corresponding consistency term later distills the cross-modal representation into this fallback pathway. This mechanism is central to the claim that XFormer can absorb purely kinematic motion sources without requiring paired imagery [2305.11101].

## 4. Supervision, datasets, and optimization

XFormer is trained under a heterogeneous supervision regime spanning four dataset types: images with 3D annotations (Human3.6M, 3DPW, UP-3D, MuCo-3DHP); images with 2D keypoint annotations only (COCO, MPII); images with pseudo 3D labels from SPIN or Pose2Mesh fits; and pure motion-capture data without images (AMASS, projected to 2D by random orthographic cameras) [2305.11101]. This multi-source setup is integral rather than auxiliary, because the architecture is explicitly constructed to tolerate missing modalities.

The total loss is
\[
L_{total}=L_{map}+L_{kp}+L_{img}+L_{cons}.
\]
The keypoint detector loss combines per-joint heatmap and offset supervision,
\[
L_{map}
=\frac1K\sum_{k=1}^Kw_k\bigl\|\bar{\mathcal H}_k-\mathcal H_k\bigr\|_1
+\frac1K\sum_{k=1}^Kw_k\bigl\|\bar{\mathcal O}_k-\mathcal O_k\bigr\|_1.
\]
For each reconstruction branch, the training objective includes vertex, joint, joint-regression, and projected-joint losses:
\[
L^V=\frac1M\sum_{i=1}^M\bigl\|V_i^{3D}-\bar V_i^{3D}\bigr\|_1,\qquad
L^J=\frac1K\sum_{i=1}^K\bigl\|J_i^{3D}-\bar J_i^{3D}\bigr\|_1,
\]
\[
L^{J}_{reg}
=\frac1K\sum_{i=1}^K\bigl\|\mathcal W\,V^{3D}-\bar J^{3D}\bigr\|_1,\qquad
L^{J}_{proj}
=\frac1K\sum_{i=1}^K\bigl\|\Pi\,J^{3D}-\bar J^{2D}\bigr\|_1.
\]
The modality-switch distillation term is
\[
L_{cons} = \bigl\|F_{kp}^{MHA}-F_{kp}^{MLP}\bigr\|_2
\]
[2305.11101].

Optimization uses Adam, with training for 200 epochs. The reported training cost is 3 days on 4 V100 GPUs for the small model and 5 days on 8 V100s for the large model. The learning rate and batch size are described as chosen per-standard practice and are not explicitly reported. The small model does not require COCO pre-training [2305.11101]. This combination of losses and data sources indicates that the model is trained as a joint estimator rather than as a late-stage fusion of separately pretrained subsystems.

## 5. Accuracy, runtime, and ablation findings

On Human3.6M under Protocol 1, XFormer-Large with HRNet reports MPJPE \(=52.6\) mm and PA-MPJPE \(=35.2\) mm, while XFormer-Small with MobileNetV3 reports MPJPE \(=68.2\) mm and PA-MPJPE \(=44.2\) mm [2305.11101]. On 3DPW, the large model reports MPJPE \(=75.0\) mm, PA-MPJPE \(=45.7\) mm, and PVE \(=87.1\) mm, whereas the small model reports MPJPE \(=84.7\) mm, PA-MPJPE \(=55.1\) mm, and PVE \(=102.6\) mm [2305.11101]. The original paper characterizes the HRNet-based variant as state of the art on Human3.6M and 3DPW.

The runtime results are equally central to the identity of the method. On a GTX1660, XFormer-Small runs at 154 fps on GPU and 37.6 fps on CPU using a single thread. The corresponding inference cost is approximately 7 ms per frame on GPU and approximately 30 ms per frame on CPU. The small model, consisting of MobileNetV3 plus one cross-modal block, is therefore reported to run at more than 30 fps on a commodity CPU, while XFormer-Large with HRNet and three blocks remains real-time on GPU [2305.11101].

The ablation study attributes measurable gains to the defining architectural choices. Cross-modal attention reduces PA-MPJPE by 4–6 mm relative to single-branch or naive fusion baselines. Adding AMASS through the modality switch yields an approximately 1.5 mm gain, and pseudo-labelled data adds an approximately 0.7 mm gain [2305.11101]. These numbers support a specific reading of the method’s generalization strategy: accuracy improvements are not attributed solely to backbone capacity, but also to the ability to consume mixed-supervision and image-free motion data within a shared training graph.

The practical applications listed for the method are telepresence, VR/AR avatars, Virtual YouTubers, interactive gaming, and real-time AR body stickers [2305.11101]. In context, these examples are consistent with the paper’s emphasis on CPU viability and low-latency deployment rather than on offline reconstruction quality alone.

## 6. Terminological ambiguity and other uses of the name

The term “XFormer” is not unique in the arXiv literature, and disambiguation is often necessary. Besides the monocular 3D body-capture system [2305.11101], the name appears in at least four unrelated technical contexts.

In computer architecture, "X-Former: In-Memory Acceleration of Transformers" describes a hybrid in-memory accelerator combining NVM and CMOS processing elements, together with a sequence-blocking dataflow; it reports up to \(85\times\) and \(7.5\times\) improvements in latency and energy over a NVIDIA GeForce GTX 1060 GPU [2303.07470]. In low-level vision, "Xformer: Hybrid X-Shaped Transformer for Image Denoising" denotes an X-shaped dual-branch denoising network built from spatial-wise and channel-wise Transformer blocks coupled by Bidirectional Connection Units [2303.06440]. In multimodal large language models, "X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs" denotes a lightweight transformer module between frozen CLIP-ViT, MAE-ViT, and a frozen LLM, designed to combine contrastive and masked-image-modeling representations [2407.13851]. In mobile vision, "Lightweight Vision Transformer with Cross Feature Attention" introduces an efficient CNN–ViT hybrid backbone called XFormer, centered on cross feature attention and reporting 78.5% top-1 accuracy on ImageNet1K with 5.5 million parameters [2207.07268].

Related naming variants extend further. "Extreme Adaptive Transformer" uses the shorthand Exformer or XFormer for hydrologic forecasting with an extreme-adaptive sparse attention mechanism [2607.02437], while MXFormer and XCTFormer denote still other architectures for compute-in-memory Transformer acceleration and multivariate time-series analysis, respectively [2602.12480; 2605.18534]. A common misconception is therefore that XFormer denotes a single coherent architecture family. The publication record instead shows a recurrent naming pattern attached to otherwise unrelated systems across computer vision, hardware, multimodal modeling, and time-series forecasting.

Source: https://www.emergentmind.com/topics/xformer