ArmFormer: Dual Transformer Frameworks

• ArmFormer is a dual-transformer framework integrating spatial-temporal attention for precise arm-hand dynamic estimation and real-time segmentation.
• The PAHMT variant uses parallel spatial and temporal transformer branches to fuse arm-hand correlations, yielding over 20% improvements in MPJPE and MPJRE.
• The segmentation model employs CBAM-enhanced MixVisionTransformer to achieve 80.64% mIoU and 82.26 FPS, ideal for edge deployment in security systems.

ArmFormer refers to two distinct transformer-based frameworks sharing the same name but targeting separate problem domains and research objectives. The first, originating from "Spatial-Temporal Parallel Transformer for Arm-Hand Dynamic Estimation" (Liu et al., 2022), addresses joint arm-hand pose dynamics from monocular video, driven by spatial-temporal transformer mechanisms. The second, as introduced in "ArmFormer: Lightweight Transformer Architecture for Real-Time Multi-Class Weapon Segmentation and Classification" (Kambhatla et al., 19 Oct 2025), is a compact segmentation network for real-time, multi-class weapon and human parsing, constructed for deployment in edge security systems. Each system is considered separately below to preserve terminological and methodological precision.

1. Spatial-Temporal Parallel Transforming for Arm-Hand Dynamics

The initial ArmFormer framework (termed PAHMT) was designed to estimate the dynamics of arm and hand rotations from monocular video, leveraging the strong correlation between macro limb poses and micro-articulated hand gestures (Liu et al., 2022). It employs a parallel transformer architecture with spatial and temporal processing branches.

Architecture

• Two-Stage Pipeline: Initial 2D hand key-point extraction (MobileNetv3 variant) and 3D arm key-point estimation (e.g., VPose3D). These features are processed by a rotation estimation module predicting 3D axis-angle rotations for both arm and hand joints.
• Temporal Transformer Branch (AHMT): Sequences of key-point embeddings are transformed via standard transformer encoders with multi-head self-attention (MSA), MLP blocks, layer normalization, and residuals. Tokens are generated as $$z^{0}_t = [x^{1}_t E; x^{2}_t E; ... ; x^{f}_t E ] + E_{t(\textrm{pos})}$$. Successive encoder layers operate as $$z'^{l}_t = \textrm{MSA}(\textrm{LN}(z^{l-1}_t)) + z^{l-1}_t$$, $$z^l_t = \textrm{MLP}(\textrm{LN}(z'^{l}_t)) + z'^{l}_t$$.
• Spatial Transformer Branch: Arm and hand joint inputs are reshaped (hand key-points padded to 3D) as tensors of shape $f \times J \times 3$, embedding a learnable regression token. Tokens are $$z^0_s = [z_{\textrm{regress}}; x^1_s E; x^2_s E; ...; x^N_s E] + E_{s(\textrm{pos})}$$, processed as in the temporal branch.
• Fusion and Prediction: Final predictions are made by element-wise adding terminal outputs of spatial and temporal branches and passing them through a 3-layer convolutional regression head: $y = \textrm{RegressionHead}(z^{L_t}_t + z^{L_s}_s)$.

Exploitation of Arm-Hand Correlations

The core innovation is the explicit modeling of interconnected dynamics via input fusion ("ah2ah," arm–hand to arm–hand) rather than the independent or hand-to-hand ("h2h") strategies of prior work. Spatial transformers capture both joint-level articulation synchrony and high-level co-variation (e.g., raised arm correlates with open hand pose).

Quantitative ablation demonstrates superiority over architectures estimating hand gestures in isolation and subsequently inferring arm dynamics via inverse kinematics. The integrated representation yields more plausible, physically consistent predictions.

Objective Functions

Composite training objectives enforce accuracy, temporal smoothness, and plausible kinematic structure:

• Reconstruction Loss: Comprises L1 loss ($L_\textrm{L1}$), temporal smoothness ($L_\textrm{smooth}$), and forward kinematics loss ($L_\textrm{FK}$), weighted by empirical parameters $\gamma$ and $\beta$:

$$L_{\textrm{recon}} = L_{\textrm{L1}} + \gamma L_{\textrm{smooth}} + \beta L_{\textrm{FK}}$$

• Adversarial (GAN) Loss: Enforced by a discriminator $D$:

$$L_{\textrm{GAN}}(R, D) = \mathbb{E}[\log D(Y)] + \mathbb{E}[\log(1 - D(R(K)))]$$

• Combined Minimax Training:

$$\min_{R}\max_{D} \left\{ L_{\textrm{recon}} + \lambda L_{\textrm{GAN}} \right\}$$

Each loss term is organized for accurate, temporally coherent, and anatomically plausible dynamic estimation.

Dataset and Training Paradigm

A motion capture corpus of 200K frames, primarily sports and dance sequences, is used. 3D joint positions (source) are retargeted to a standard character model, with 2D hand projections synthesized for network input. Training proceeds with Adam optimizer (batch size 128, learning rate $1\times 10^{-3}$ with 50% decay each 50 epochs), the loss weights specifically set as $\lambda=0.05$, $\beta=\gamma=1.0$, over 300 epochs.

Benchmarking and Results

• Metrics: Mean Per Joint Position Error (MPJPE) for hands and body, Mean Per Joint Rotation Error (MPJRE) for arms.
• Ablation: Transition from hand-only to integrated arm–hand inputs yields major improvements. Using a temporal transformer reduces MPJPE/MPJRE by >20% versus CNN; adding the spatial transformer achieves further 13% MPJPE and 16% MPJRE improvements.
• Comparative SOTA: On the BH dataset, ArmFormer achieves hand MPJPE of 0.0281 (Body2Hands $\geq$ 0.0346). On rendered motion datasets, overall MPJPE is 0.1375 and arm MPJRE 0.1614, outperforming FrankMocap and ExPose.
• Qualitative Analysis: Outputs are robust to occlusion and motion blur, showing superior smoothness and realism relative to pipelines based on sequential estimation or inverse kinematics post hoc optimization.

Real-World and System Integration

ArmFormer's PAHMT design is positioned for applications including:

• Motion capture for animation, VR/AR, and digital character synthesis.
• Human–machine and intent-aware gesture interfaces.
• As a module in monocular video pipelines—leveraging existing 2D/3D pose estimators to produce plausible full arm–hand motion in unconstrained scenarios.

The spatial-temporal transformer structure enables resilience to adverse visual conditions (blur, occlusion) and reduces the computational cost of per-frame optimization.

2. Lightweight ArmFormer for Real-Time Weapon Segmentation

The second ArmFormer system (Kambhatla et al., 19 Oct 2025) is a transformer-driven semantic segmentation network expressly optimized for the real-time identification of multiple weapon classes and humans in high-risk environments, with strong emphasis on efficient inference for edge hardware.

Architectural Overview

• Encoder: MixVisionTransformer backbone augmented with Convolutional Block Attention Module (CBAM) at each of four hierarchically deeper stages (channel width growth: 32 $\rightarrow$ 256).
• Overlapped Patch Embedding: Input split via stride 4 and kernel size 7, enabling local context retention.
• CBAM Integration: Channel and spatial attention modules sequentially recalibrate feature representations—formally, for $F\in\mathbb{R}^{C\times H\times W}$:
  • Channel: $$M_c = \sigma(\text{MLP}(\text{AvgPool}(F)) + \text{MLP}(\text{MaxPool}(F)))$$, $F' = M_c \odot F$.
  • Spatial: $$M_s = \sigma(\text{Conv}^{7\times7}([\text{AvgPool}_c(F'); \text{MaxPool}_c(F')]))$$, $F_\text{out} = M_s \odot F'$.
• Decoder: A hamburger module—employing matrix decomposition for global context aggregation—followed by additional CBAMs. Outputs are refined multi-scale features, culminating in a classification convolution yielding final segmentation masks.

Performance Evaluation

• Metrics: Achieves 80.64% mIoU, 89.13% mean F-score, 82.26 FPS throughput, requiring only 4.886 GFLOPs and 3.66M parameters.
• Comparative Efficiency: Outperforms conventional models such as EncNet and Uppernet_swin (with up to 48$\times$ less computational cost).
• Per-Class Analysis: Consistently higher IoU across all five classes (handgun, rifle, knife, revolver, human).

Deployment and Use Cases

ArmFormer is purpose-built for:

• Embedded AI on portable security cameras, drones, and distributed surveillance nodes.
• Scenarios where pixel-level segmentation of multiple weapon types and human bodies is paramount, facilitating nuanced threat analysis over coarse bounding box detection prevalent in legacy detectors (e.g., YOLO).

The system’s runtime and power efficiency make it suitable for continuous, on-device inference under edge constraints.

Experimental Protocol and Dataset

• Dataset: 8,097 images, sourced from Google Open Images and IMFDB, annotated at the pixel level using a semi-automated pipeline incorporating SAM2.
• Baselines: Benchmarked versus both light-weight (CGNet, HrNet) and heavy-weight (EncNet, ICNet) segmentation models.

Quantitative evidence and qualitative visualizations show that ArmFormer produces smooth, accurate object boundaries and retains segmentation quality under small object scale and occlusion.

Future Directions

Planned research emphasizes:

• Quantization and pruning for ultra-low-power deployment.
• Multi-modal sensor fusion (e.g., thermal, depth) for enhanced robustness under adverse imaging conditions.
• Temporal consistency refinement in video streams.
• Application of federated learning for privacy-preserving, distributed model training.

The above suggests a trajectory toward increasingly robust and privacy-preserving threat detection platforms at scale.

3. Comparison Table

ArmFormer Variant	Primary Task	Core Innovations
PAHMT / ArmFormer (Liu et al., 2022)	Arm-hand dynamic estimation	Spatial-temporal parallel transformer; arm–hand input fusion; composite losses
ArmFormer (Kambhatla et al., 19 Oct 2025)	Real-time weapon+human segmentation	CBAM-enhanced MixVisionTransformer; hamburger decoder; resource-optimized design

This table summarizes the primary domain, distinguishing architectural features, and intended research contribution of each system.

4. Impact and Significance

Both ArmFormer variants advance their respective domains through deep architectural integration of attention mechanisms for extractive modeling of structured data. The PAHMT demonstrates state-of-the-art arm–hand motion capture in unconstrained, monocular video, improving the quality of down-stream motion synthesis and interaction modeling. The segmentation-focused ArmFormer advances edge-deployable security infrastructure, reconciling high-precision pixel-level discrimination with strict hardware constraints.

A plausible implication is that these approaches offer blueprints for further transformer-based architectures balancing performance, efficiency, and real-world deployability in specialty vision tasks.

5. Limitations and Research Directions

Challenges specific to each design persist. In PAHMT, ensuring transferability to highly diverse motion styles or unseen environments remains reliant on the available mocap diversity. For the segmentation ArmFormer, robustness under extreme lighting or traumatic occlusion is proposed to be addressable via multi-modal fusion and temporal modeling.

Quantization, pruning, and federation remain open optimization avenues—particularly for ultra-low-power or privacy-sensitive deployments.

6. Synthesis

ArmFormer, as described in (Liu et al., 2022) and (Kambhatla et al., 19 Oct 2025), embodies two discrete, transformer-based frameworks that synthesize cross-domain attention modeling with computational efficiency—one for fine-grained dynamic estimation in pose analysis, the other for real-time multi-class segmentation in high-stakes security contexts. Their respective advances demonstrate the efficacy of strategically layered attention architectures and empirically tuned loss landscapes in high-precision, low-latency computer vision applications.