Dual-Surrogate Contrastive Learning (DuoCLR)

• DuoCLR is a contrastive learning framework that pretrains on trimmed skeleton sequences using dual surrogate tasks to capture both local action details and global sequence order.
• It incorporates a multi-action permutation augmentation strategy called 'Shuffle and Warp' to enhance contextual diversity and ensure scale, position, and orientation consistency.
• Empirical results show up to 87.1% mAP and a 63% reduction in labeled data requirements, demonstrating superior transferability for untrimmed human action segmentation.

Dual-Surrogate Contrastive Learning (DuoCLR) is a contrastive representation learning framework for skeleton-based human action segmentation that uses pre-training on trimmed, single-action skeleton sequences to improve transfer to untrimmed, multi-action segmentation settings. The method is defined by two complementary surrogate tasks—Cross Permutation Contrasting (CPC) and Relative Order Reasoning (ROR)—together with a multi-action permutation augmentation strategy, “Shuffle and Warp,” and a hierarchical feature extractor that combines a graph convolutional visual encoder with a multi-stage temporal convolutional encoder. In this formulation, DuoCLR is not a generic label for any dual contrastive objective; it denotes a specific network designed to learn multi-scale feature representations optimized for action segmentation by coupling local action-segment discrimination with global action-order reasoning (Tian et al., 5 Sep 2025).

1. Problem setting and conceptual motivation

DuoCLR is motivated by the observation that skeleton-based action segmentation in untrimmed videos containing multiple actions is difficult for prior contrastive representation learning methods because those methods are mostly tailored to trimmed, single-action data and focus on sequence-level instance discrimination. In the reported formulation, such models lack contextual awareness and do not exploit temporal dependencies across sequences, which leads to representations that transfer poorly to segmentation tasks on untrimmed videos (Tian et al., 5 Sep 2025).

The framework therefore shifts the unit of pre-training away from isolated sequence-wise representations toward multi-scale representations in conjunction with cross-sequence variations. Its stated innovations are fourfold: a multi-action permutation augmentation called “Shuffle and Warp”; two complementary surrogate contrastive pretext tasks, CPC and ROR; a hierarchical feature extraction network composed of a Visual Encoder and a Temporal Encoder; and a dual-surrogate learning objective that operates at both the local, action-segment level and the global, action-permutation level. The resulting design is intended to leverage only trimmed skeleton videos for pretraining while producing representations that are highly transferable to untrimmed multi-action action segmentation tasks (Tian et al., 5 Sep 2025).

A central conceptual distinction in DuoCLR is that the two surrogate tasks target different statistical structure. CPC promotes permutation-invariant intra-class similarity across changing action contexts, whereas ROR reasons about inter-class context by predicting the relative mapping between two permutations. This division of labor makes the method explicitly context-aware rather than purely instance-discriminative. A plausible implication is that DuoCLR treats temporal context not as nuisance variation to be suppressed, but as structured information that can be supervised indirectly through permutation design.

2. Multi-action permutation augmentation: “Shuffle and Warp”

The augmentation pipeline begins from a minibatch of trimmed action sequences,

$$\mathcal{X} = \{X^{(a)}, X^{(b)}, X^{(c)}, \ldots\},$$

and constructs new multi-action sequences by applying a random permutation $\mathcal{P}$: $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$ The stated goal of the Shuffle stage is to introduce diverse action contexts by constructing new sequences comprised of segments in different permutations. The resulting synthetic multi-action sequence exposes each action to varying preceding and following actions, approximating the contextual structure of untrimmed videos (Tian et al., 5 Sep 2025).

The Warp stage is introduced to remove unnecessary variation—specifically scale, position, and orientation differences—that can arise when concatenating skeletons recorded under different setups. For two consecutive segments $X_i$ and $X_j$, the paper defines

$$\text{Warp}(X_i, X_j) = \text{concat}\big(X_i,\ s(X_j * \mathcal{R} + \mathcal{T})\big),$$

where $\mathcal{R}$, $\mathcal{T}$, and $s$ denote rotation, translation, and scaling. The process is recursive across the whole permutation so that view and scale consistency are preserved during concatenation. The combined augmentation is written as

$$\mathcal{H} = \text{Warp}(\text{Shuffle}(\mathcal{X}, \mathcal{P})).$$

This augmentation is a structural component rather than a peripheral preprocessing step. The reported outcome is that training with these multi-action permutations exposes the model to the temporal complexity and contextual variety found in real untrimmed videos. In ablation, “Shuffle” alone reaches $\mathcal{P}$0, traditional augmentations (“Shear, Crop”) reach $\mathcal{P}$1, “Shuffle + Warp” reaches $\mathcal{P}$2, and “All augmentations (DuoCLR-III)” reaches $\mathcal{P}$3, which is presented as evidence that Warp is critical in addition to Shuffle and that its combination with traditional augmentations is synergistic (Tian et al., 5 Sep 2025).

3. Dual surrogate tasks: CPC and ROR

The dual-surrogate designation in DuoCLR refers specifically to Cross Permutation Contrasting and Relative Order Reasoning. CPC is optimized to learn intra-class similarities by contrasting representations of the same action class across different permutations, while ROR reasons about inter-class contexts by predicting the relative mapping between two permutations (Tian et al., 5 Sep 2025).

Surrogate task	Representation scale	Primary role
CPC	local action and global permutation	permutation-invariant intra-class similarity
ROR	permutation-to-permutation relation	inter-class context and relative order reasoning

In CPC, two permutations $\mathcal{P}$4 and $\mathcal{P}$5 are generated from the same action set, for example $\mathcal{P}$6 and $\mathcal{P}$7. Action-wise features $\mathcal{P}$8 and $\mathcal{P}$9 are extracted via an MLP projection after temporal pooling. Positive pairs are the representations of the same action in the two permutations, and negatives are gathered through two memory banks: $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$0, which stores negative samples for each action class for local, action-wise contrast, and $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$1, which stores negative samples for permutations for global, permutation-wise contrast. The action-level and permutation-level losses are combined as

$$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$2

with $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$3 if $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$4, else $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$5 (Tian et al., 5 Sep 2025).

ROR is formulated as a temporal reasoning problem over the mapping from one permutation to another. For two permutations of the same action set, the mapping from $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$6 to $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$7 is treated as a classification problem with $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$8 classes if there are $$\text{Shuffle}(\mathcal{X}, \mathcal{P}) = \text{concat}(X^{\mathcal{P}[1]}, X^{\mathcal{P}[2]}, \ldots).$$9 actions. The method compares all local feature pairs between permutations, computes absolute differences

$X_i$0

and concatenates them into a positional encoding

$X_i$1

A fully connected layer maps this encoding to a probability distribution over all possible orderings, supervised by cross-entropy: $X_i$2

The final DuoCLR objective is

$X_i$3

where $X_i$4 balances the two terms. The ablation results reported for the surrogate tasks show $X_i$5 for CPC alone, $X_i$6 for ROR alone, and $X_i$7 for DuoCLR with CPC+ROR, which is used to support the claim that both tasks contribute and are complementary, with CPC generally stronger because of its intra-class focus (Tian et al., 5 Sep 2025).

4. Hierarchical architecture and transfer protocol

DuoCLR employs a hierarchical feature extraction network composed of three parts. The Visual Encoder is a graph convolutional network without temporal pooling and therefore retains per-frame feature resolution. The Temporal Encoder is a multi-stage temporal convolutional network (MS-TCN), introduced to capture long-range temporal dependencies. A projection head implemented as a multi-layer perceptron produces both local, per-action and global, sequence feature vectors. The paper characterizes this hierarchy as enabling fine-grained, sliding-window-free transfer to segmentation tasks (Tian et al., 5 Sep 2025).

The pretraining and evaluation protocol is explicitly cross-dataset. Pretraining uses trimmed, single-action skeleton datasets: NTU RGB+D (NTU), Toyota Smarthome (TS), and Kinetics-400 (K4). Evaluation is then performed on untrimmed, multi-action datasets: PKU Multi-Modality Dataset (PKU1 and PKU2), Toyota Smarthome Untrimmed (TSU), and Charades (CR). Two evaluation regimes are defined. In linear evaluation, only the segmentation classifier is trained while the feature extractor remains fixed. In fine-tuning, both the feature extractor and the classifier are trained on the target task. The reported metrics are mAP@0.1, [email protected], Accuracy, mIoU, and per-frame mAP (Tian et al., 5 Sep 2025).

This training protocol is central to the claim of transferability. Because the method is pretrained on trimmed skeleton sequences and evaluated on untrimmed segmentation tasks, its effectiveness is tied to whether the learned representations encode context at multiple temporal scales rather than merely recognizing isolated actions. The architecture is therefore aligned with the two surrogate tasks: the graph-based visual encoder preserves frame-level detail for local reasoning, while MS-TCN supplies longer-range temporal structure needed for permutation-level reasoning.

5. Empirical performance, label efficiency, and ablations

The main empirical claim is that DuoCLR demonstrates a significant boost over state-of-the-art comparatives in both multi-class and multi-label action segmentation tasks. On PKU1 under linear evaluation, the reported [email protected] is $X_i$8 for DuoCLR, compared with $X_i$9 for LAC and $X_j$0 for Baseline-I. Under fine-tuning on PKU1, DuoCLR reaches [email protected] $X_j$1, exceeding LAC at $X_j$2 and Baseline-II at $X_j$3 (Tian et al., 5 Sep 2025).

The framework is also reported to improve label efficiency. The paper states that DuoCLR can reduce the required labeled data by up to $X_j$4 to achieve parity with fully-supervised methods. It further reports that the method is effective with less target-domain data, including the case of $X_j$5 of PKU1, and that it maintains an advantage even with unsupervised, label-free pretraining. These findings are presented together with claims of transferability and generalization to unseen datasets and composite labels under both linear and fine-tuned settings (Tian et al., 5 Sep 2025).

Ablation studies isolate the sources of these gains. For surrogate-task design, CPC alone outperforms ROR alone, while the combination performs best, indicating complementarity rather than redundancy. For augmentation, the contrast between “Shuffle” and “Shuffle + Warp” is large enough to support the claim that alignment of scale, position, and orientation is not incidental. The paper also studies the number of actions per permutation, denoted $X_j$6, and reports that increasing the number of actions per permutation improves context diversity and representation quality, with diminishing returns and best performance around four actions per permutation because of the computational and optimization complexity of ROR (Tian et al., 5 Sep 2025).

These results delimit the method’s empirical profile. The strongest gains are attributed not to a single contrastive loss alone, but to the interaction between multi-action permutation construction, permutation-invariant intra-class contrast, and explicit order reasoning over the same action set.

6. Theoretical context, related dual-contrastive frameworks, and interpretive boundaries

The term “surrogate” in DuoCLR concerns surrogate tasks for representation learning, but it also intersects with a broader theoretical literature on contrastive objectives as surrogates for downstream supervised loss. In “On the Surrogate Gap between Contrastive and Supervised Losses,” the mean supervised loss is bounded above and below by shifted versions of the InfoNCE loss for all negative sample sizes $X_j$7, and the surrogate gap

$X_j$8

shrinks as $X_j$9 increases. That analysis suggests that larger negative sample sizes improve downstream classification because the surrogate gap between contrastive and supervised losses decays, and it explicitly rejects a collision-coverage tradeoff in that setting (Bao et al., 2021). This suggests a conceptual bridge to DuoCLR: insofar as DuoCLR relies on contrastive optimization, contrastive-surrogate theory helps clarify why tighter alignment between pretext loss and downstream segmentation objectives can be expected from better-designed contrastive regimes. At the same time, DuoCLR itself is centered on multi-action permutations and dual surrogate tasks, not on a theorem about negative sample size.

DuoCLR also belongs to a broader family of dual or multi-branch contrastive methods, but its mechanism is distinct. RecDCL combines batch-wise contrastive learning and feature-wise contrastive learning for recommendation, with the claim that the combination helps eliminate redundant solutions but never misses an optimal solution (Zhang et al., 2024). DCC for object re-identification maintains individual and centroid cluster memory banks coupled by a cross-view consistency constraint (Yao et al., 2021). DANCE for dense retrieval adds a dual query-retrieval objective to address anisotropy in query embeddings (Li et al., 2021). DCLGAN uses two encoders and dual PatchNCE-style objectives for unpaired image-to-image translation, with SimDCL adding a similarity loss to address mode collapse (Han et al., 2021). SIB-CL combines abundant unlabeled data, surrogate data, and known invariances to reduce label requirements in scientific tasks (Loh et al., 2021). Across these systems, “dual” may refer to dual directions, dual views, dual memories, or dual learning scales; in DuoCLR, it refers specifically to the pairing of CPC and ROR over synthetic multi-action permutations.

A common misconception is therefore to treat DuoCLR as a generic synonym for any two-loss contrastive framework. The available evidence supports a narrower definition: DuoCLR is the skeleton-based human action segmentation framework in which “Shuffle and Warp” augmentation supports CPC and ROR so that local action-segment and global action-permutation structure are learned jointly from trimmed skeleton data and transferred to untrimmed segmentation tasks (Tian et al., 5 Sep 2025).