Alternating Dual-Hand Vision Transformer

• The paper introduces ADH‑ViT, which uses an alternating training schedule to balance learning between left-hand and right-hand action representations.
• The methodology leverages a shared ViT backbone and independent hand-specific classifiers to mitigate gradient domination and improve recognition accuracy.
• Key results show significant performance gains in dual-hand segmentation, with measurable improvements in LH and RH accuracies through optimized alternation.

The Alternating Dual-Hand Vision Transformer (ADH‑ViT) is a spatio-temporal transformer formulation for dual-hand video understanding in which a single shared encoder is trained for both left-hand and right-hand streams, while hand-specific classifiers and an alternating optimization schedule are used to balance learning across the two hands. In "Polyphony: Diffusion-based Dual-Hand Action Segmentation with Alternating Vision Transformer and Semantic Conditioning" (Zheng et al., 29 May 2026), ADH‑ViT is the first stage of a three-stage framework for dual-hand action segmentation from untrimmed videos. It is motivated by four difficulties specific to bimanual activity analysis: complex inter-hand dependencies, visual asymmetry between hands, representation conflict in which the dominant hand monopolizes gradients, and semantic ambiguity in fine-grained actions.

1. Formal task definition and scope

Dual-hand action segmentation takes an untrimmed video

$\mathcal{V} = \{v_t\}_{t=1}^T$

and predicts two dense label sequences, one per hand: $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$ Within this setting, ADH‑ViT is designed to learn shared spatio-temporal video features useful for both hands, maintain separate hand-specific classifiers on top of that shared encoder, and use an alternating training schedule so that gradients from LH and RH mini-batches influence the shared encoder in a balanced way (Zheng et al., 29 May 2026).

The role of ADH‑ViT in Polyphony is deliberately modular. Stage 1 performs clip-level representation learning and classification; Stage 2 aligns the resulting dense features with structured language descriptions through Semantic Feature Conditioning; Stage 3 performs diffusion-based dual-hand action segmentation with cross-hand feature fusion and adaptive loss weighting. ADH‑ViT can also act as a standalone dual-hand action recognition model. This modularity is important for interpreting the term: in the cited formulation, ADH‑ViT is not the entire segmentation system, but the foundational visual encoder and alternating training mechanism on which the later stages depend.

The principal conceptual distinction is that ADH‑ViT addresses balance in shared representation learning rather than explicit hand-to-hand message passing inside its Stage 1 transformer blocks. This suggests that the adjective “alternating” refers first to optimization schedule and task balancing, not to an alternating left–right cross-attention topology.

2. Architectural formulation

ADH‑ViT uses a VideoMAE V2 ViT‑Base backbone as a shared spatio-temporal encoder $\mathcal{E}_\phi$. For each hand $h\in\{\text{LH},\text{RH}\}$, the untrimmed video is converted into a trimmed dataset

$$\mathcal{D}_{trim}^h = \{(x_i^h, y_i^h)\}_{i=1}^{N_h},$$

where clips are formed from two sources: contiguous labeled segments and random clip sampling. The full clip set for hand $h$ is

$\mathcal{X}^h = \{x_{seg}^h\} \cup \{x_{clip}^h\}.$

Each clip $x^h\in\mathbb{R}^{3\times L\times H\times W}$ is a small video clip centered on one hand’s action; cropping or hand localization is handled upstream, while ADH‑ViT itself sees full RGB clips (Zheng et al., 29 May 2026).

The encoder begins with tubelet embedding. The clip is partitioned into non-overlapping 3D patches of size $l\times p\times p$, implemented as a 3D convolution: $$z^h = \mathrm{Tubelet}(x^h; l, p) + \mathbf{E}_{\text{pos}},$$ where

$$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$0

is the number of tubelet tokens and $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$1 is a sinusoidal positional embedding. Transformer encoding then produces

$$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$2

The reported ViT‑Base dimensions are 12 transformer layers, 12 heads, and embedding dimension $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$3.

On top of the shared embedding, ADH‑ViT uses two independent linear classifiers, one per hand: $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$4 with $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$5 and $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$6. The resulting pattern is shared representation with hand-specific decision layers. Both hands pass through the same encoder weights, but their logits are produced by distinct heads.

The forward pass is therefore: tubelet embedding plus positional encoding, shared ViT encoding, and hand-specific linear classification. During Stage 1, only classification supervision is applied.

3. Alternating optimization and representation balancing

The defining mechanism of ADH‑ViT is its alternating training strategy. Instead of mixing LH and RH clips in the same mini-batch or optimizing a joint loss at every step, training alternates the active hand every $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$7 steps: $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$8 At training step $$Y^h = \{y_t^h\}_{t=1}^T,\quad h\in\{\text{LH},\text{RH}\},\quad y_t^h\in\{1,\dots,C\}.$$9, the mini-batch is sampled only from the active hand’s trimmed dataset. The per-step loss is

$\mathcal{E}_\phi$0

where $\mathcal{E}_\phi$1 is standard cross-entropy (Zheng et al., 29 May 2026).

Backpropagation updates the shared encoder parameters $\mathcal{E}_\phi$2 and only the active hand’s head parameters $\mathcal{E}_\phi$3; the inactive head is untouched at that step. In the reported implementation, the Stage 1 optimizer is AdamW, the learning rate is $\mathcal{E}_\phi$4 with cosine annealing, weight decay is 0.1, training lasts 50 epochs, and the alternation period is $\mathcal{E}_\phi$5 steps.

The rationale is explicitly tied to representation conflict. In naïve joint training, the dominant hand, typically the right, may have richer or more varied actions and can dominate gradient updates to the shared encoder. Alternation forces the encoder to spend contiguous blocks of optimization steps on LH alone and then RH alone. The authors interpret the resulting improvements as direct evidence that alternating training resolves representation conflict and balances learning between hands.

The ablation on HA-ViD reports the following joint-versus-alternating comparison:

Strategy	Hand	Top‑1 Rec. / Seg. Acc / F1@10
Joint	LH	44.5 / 53.0 / 58.6
Joint	RH	45.2 / 55.4 / 56.0
Alternating	LH	47.0 / 56.9 / 59.3
Alternating	RH	46.2 / 55.7 / 59.3

These values show that performance improves for both hands under alternation, with larger gains for LH. With joint training, RH is consistently better than LH; with alternating training, the gap closes or reverses. A plausible implication is that the alternating schedule functions as an optimization-level balancing prior without introducing an explicit balancing term into the loss.

4. Inter-hand dependency modeling and its limits

At Stage 1, ADH‑ViT has no explicit cross-hand attention or feature fusion inside the transformer backbone. Inter-hand dependency is modeled only implicitly because the encoder weights are shared across both LH and RH clip distributions, and alternating training forces that shared encoder to remain useful for both (Zheng et al., 29 May 2026).

Direct modeling of LH–RH coordination is deferred to later stages. After Stage 1, the trained encoder is applied over the full video with a sliding window. For each frame $\mathcal{E}_\phi$6, it produces a shared backbone feature $\mathcal{E}_\phi$7 and hand-specific logits $\mathcal{E}_\phi$8. Stage 2 augments these with semantic embeddings derived from structured action descriptions of the form

$\mathcal{E}_\phi$9

and combines motion, action, and semantic information into the Motion–Action–Semantic feature

$h\in\{\text{LH},\text{RH}\}$0

Stage 3 then performs dual-hand segmentation using a shared temporal encoder, cross-hand feature fusion, and diffusion-based denoising decoders. Cross-hand interaction is made explicit there through

$h\in\{\text{LH},\text{RH}\}$1

Adaptive loss weighting further balances hand performance at the segmentation level.

A common misconception is to treat ADH‑ViT itself as a cross-hand transformer. In the reported formulation, that is not the case. Another common misconception is that the balancing effect is produced by an explicit regularizer; in fact, Stage 1 uses pure per-clip classification loss, and the balancing mechanism is the alternation schedule rather than an additional loss term.

5. Relation to adjacent transformer formulations

The expression “Alternating Dual-Hand Vision Transformer” is explicit in Polyphony, but its interpretation benefits from comparison with two adjacent lines of work: dual-hand transformer pose estimation and alternating attention design patterns.

"Transformer-based Global 3D Hand Pose Estimation in Two Hands Manipulating Objects Scenarios" (Cho et al., 2022) addresses global 3D pose estimation of two hands manipulating an object from a single egocentric RGB frame in the H2O dataset. Its model is DETR-based, with a ResNet-50 backbone, a 6-layer encoder, a 6-layer decoder, shared object queries, a hand-type classification head, and a UVD regression head. Both hands are processed jointly in a single transformer, and the left/right distinction is learned at the prediction stage. However, the report states that it “does not introduce an alternating refinement loop between hands,” and there is “no explicit designed cross-hand attention module.” The dual-hand interaction is implicit through global image attention and query self-attention. This makes it related in spirit to joint bimanual modeling, but not an instance of ADH‑ViT as defined in Polyphony.

"Dual Aggregation Transformer for Image Super-Resolution" (Chen et al., 2023) provides a different kind of alternation: consecutive transformer blocks alternately apply spatial self-attention and channel self-attention. The alternation is an architectural schedule over block types rather than an optimization schedule over hand-specific mini-batches. The paper’s explicit formulation of inter-block dual alternation and intra-block dual aggregation via AIM and SGFN is not about hands, but it supplies a generic blueprint for alternating complementary processing modes. This suggests a broader design space in which “alternation” can refer either to training dynamics, as in ADH‑ViT, or to blockwise attention specialization, as in DAT.

Aspect	ADH‑ViT in Polyphony	Related but distinct formulations
Alternation mechanism	LH/RH mini-batch alternation	Blockwise spatial/channel alternation in DAT
Shared backbone	Single shared ViT encoder	DETR shared encoder-decoder in dual-hand pose
Explicit cross-hand attention in core stage	No	No explicit cross-hand attention in (Cho et al., 2022)

From this comparison, a useful clarification emerges: ADH‑ViT is best understood as an alternating optimization strategy built around a shared spatio-temporal ViT for dual-hand tasks, rather than as a generic name for any transformer that processes both hands.

6. Empirical profile, limitations, and extensions

As a standalone action recognition model on HA-ViD, ADH‑ViT uses one shared backbone to recognize both hands simultaneously and, with segment and random clip sampling, achieves LH Top‑1 60.1% and Top‑5 90.4%, and RH Top‑1 61.4% and Top‑5 88.9% (Zheng et al., 29 May 2026). The cited interpretation is that a shared encoder with alternating training can support both hands without catastrophic interference, while using a single model rather than separate per-hand models.

In the full Polyphony pipeline, which depends on ADH‑ViT outputs, state-of-the-art performance is reported on both dual-hand datasets. On HA-ViD, LH accuracy is 57.1% versus the best baseline FACT at 45.1, and RH accuracy is 60.6% versus FACT at 43.8, corresponding to an improvement of approximately 16.8 points for RH. On ATTACH, LH accuracy is 52.8% versus DiffAct at 47.5, and RH accuracy is 47.3% versus DiffAct at 42.5. On Breakfast, Polyphony reaches 82.5% accuracy and surpasses EAST despite EAST using a 12× larger ViT-G backbone. These numbers should be attributed to the complete Polyphony system rather than to Stage 1 alone.

The reported limitations are also specific. ADH‑ViT has no explicit cross-hand learning in Stage 1; in extremely asymmetric scenarios, alternating training may still face imbalance; and the method relies on high-quality hand-centric clips, so poor upstream localization can degrade representation quality. Qualitative analysis reports over-prediction of hand coordination, confusion between tool-mediated and bare-handed actions, and difficulty with short, long-tail actions.

Several extensions are proposed. These include extending the framework from two hands to multiple limbs or multiple agents with multiple task-specific heads and a generalized alternating schedule; introducing explicit cross-hand attention in Stage 1 by processing LH and RH clips jointly; incorporating hand pose or object-centric cues as transformer tokens; and end-to-end fine-tuning together with the semantic and diffusion stages. A plausible synthesis with the DETR-based dual-hand pose literature and the DAT design pattern is that future “alternating dual-hand” transformers may combine balanced optimization, explicit cross-hand attention, and structured alternation across specialized block types, but that broader formulation goes beyond the specific ADH‑ViT instantiation reported in Polyphony.