CPC: Permutation Contrast for Action Segmentation

• CPC is a surrogate task in DuoCLR that learns permutation-invariant action embeddings by contrasting action segments across shuffled temporal contexts.
• It employs both local and global contrastive objectives using action-wise segmentation and memory banks to enforce intra-class similarity and mitigate context bias.
• Empirical results show substantial mAP improvements, highlighting CPC's role in transferring learned representations from trimmed clips to untrimmed video sequences.

Searching arXiv for the cited paper and directly related frameworks to ground the article with arXiv IDs. Cross Permutation Contrasting (CPC) is a surrogate task introduced within DuoCLR, a dual-surrogate contrastive learning framework for skeleton-based human action segmentation. Its stated purpose is to force the network to learn permutation-invariant, action-specific embeddings even when those actions appear in different temporal contexts. In DuoCLR, CPC operates on multi-action sequences synthesized from trimmed skeleton clips by a “Shuffle and Warp” augmentation, then contrasts embeddings of the same action across two randomly permuted views while pushing apart embeddings of different actions. The mechanism is designed to address a failure mode in which a model pretrained only on single-action clips acquires context-dependent shortcuts and transfers poorly to long, untrimmed videos (Tian et al., 5 Sep 2025).

1. Position within DuoCLR

CPC is one of the two surrogate tasks in DuoCLR, the other being Relative Order Reasoning (ROR). The abstract describes CPC as learning intra-class similarities by contrasting representations of the same action class across different permutations, whereas ROR reasons about inter-class contexts by predicting relative mapping between two permutations. Together, these tasks are intended to enable DuoCLR to learn multi-scale feature representations optimized for action segmentation (Tian et al., 5 Sep 2025).

The formulation is specific to the setting of skeleton-based action segmentation. The source problem is that pretraining on trimmed, single-action sequences does not by itself ensure robustness to the compositional and contextual variability of untrimmed sequences. In the formulation accompanying DuoCLR, the representation for an action such as “Put on a hat” may differ systematically depending on the surrounding action sequence; CPC is introduced to counteract that dependence by making embeddings action-centric rather than context-bound (Tian et al., 5 Sep 2025).

This suggests that CPC is not merely a generic contrastive objective applied to skeleton data. Its role is tied to the mismatch between trimmed pretraining data and untrimmed downstream segmentation, and to the need for invariance across altered temporal contexts rather than only across conventional augmentation views.

2. Shuffle and Warp augmentation

CPC depends on a data augmentation procedure termed “Shuffle and Warp.” Let $\mathcal{X}=\{X^{(a)},X^{(b)},X^{(c)},\dots\}$ be a mini-batch of $N$ trimmed skeleton clips, each $X^{(m)}\in\mathbb{R}^{T\times V\times C}$. Two random permutations $P_i$ and $P_j$ are drawn over the indices $\{a,b,c,\dots\}$ (Tian et al., 5 Sep 2025).

The shuffling operation is defined as

$$\text{Shuffle}(\mathcal{X},P):=\text{concat}(X^{P[1]},X^{P[2]},\dots,X^{P[G]}).$$

Warp then concatenates two sequences in a view-consistent coordinate system. Given $X_i$ and $X_j$,

$$\text{Warp}(X_i,X_j)=\text{concat}(X_i,\;s\cdot(X_j*\mathcal{R}+\mathcal{T})),$$

where $N$0 is a rigid rotation, $N$1 is a translation vector, and $N$2 is a scale. Warp is applied iteratively along the shuffled list to obtain two $N$3-action sequences,

$N$4

The augmentation serves two purposes stated in the source description. First, it composes multi-action sequences from trimmed clips. Second, it creates diverse multi-action permutations that support CPC and ROR. Within CPC specifically, the two permuted views provide the basis for positive pairs defined over identical actions appearing under different global temporal arrangements (Tian et al., 5 Sep 2025).

A plausible implication is that “Shuffle and Warp” functions as the bridge between single-action training data and the multi-action structure required for segmentation-oriented pretraining.

3. Encoder and representation hierarchy

Each synthesized sequence $N$5 is passed through a two-stage encoder $N$6, yielding

$N$7

The GCN maps $N$8 and produces per-frame visual embeddings, while the TCN aggregates them into a long-range temporal feature of width $N$9 (Tian et al., 5 Sep 2025).

The representation is explicitly multi-scale. The full temporal feature $X^{(m)}\in\mathbb{R}^{T\times V\times C}$0 is split into $X^{(m)}\in\mathbb{R}^{T\times V\times C}$1 contiguous segments $X^{(m)}\in\mathbb{R}^{T\times V\times C}$2, with each segment corresponding to one action. A shared MLP projection head with temporal average-pooling $X^{(m)}\in\mathbb{R}^{T\times V\times C}$3 yields both action-wise and sequence-wise embeddings:

$X^{(m)}\in\mathbb{R}^{T\times V\times C}$4

The architecture details reported for DuoCLR are specific. The visual encoder is a variant of ST-GCN with its temporal pooling removed, producing per-frame features of size $X^{(m)}\in\mathbb{R}^{T\times V\times C}$5. The temporal encoder is MS-TCN with dilated convolutions to aggregate up to $X^{(m)}\in\mathbb{R}^{T\times V\times C}$6 frames into $X^{(m)}\in\mathbb{R}^{T\times V\times C}$7. The projection heads are two-layer MLPs mapping from pooled $X^{(m)}\in\mathbb{R}^{T\times V\times C}$8 to $X^{(m)}\in\mathbb{R}^{T\times V\times C}$9 (Tian et al., 5 Sep 2025).

This organization is consequential for CPC because the objective can operate at both local and global scales. The action-wise segments support local matching across permutations, while the full-sequence embedding supports a separate global contrastive term when the two permutations coincide.

4. Positive and negative sampling in CPC

CPC distinguishes between local, action-wise matching and global, permutation-wise matching. For local positives, each segment pair $P_i$0 is treated as a positive pair because the two embeddings derive from the same action class in two different permutations. For local negatives, a class-specific memory bank $P_i$1 is maintained and filled with previous $P_i$2 from other mini-batches, which are described as mostly other actions by construction (Tian et al., 5 Sep 2025).

For global positives, the full-sequence embeddings $P_i$3 are treated as positive only if $P_i$4; otherwise this term is deactivated. A global memory bank $P_i$5 holds previous full-sequence embeddings $P_i$6 (Tian et al., 5 Sep 2025).

The resulting sampling structure creates two distinct supervisory regimes:

Component	Positive condition	Negative source
Local CPC	$P_i$7 for each action slot $P_i$8	Class-specific memory bank $P_i$9
Global CPC	$P_j$0 only if $P_j$1	Global memory bank $P_j$2

The design makes the action slot $P_j$3 a central unit of correspondence. Because matching is performed between identical action classes under different permutations, CPC is formulated to preserve action identity while suppressing dependence on the surrounding sequence arrangement. The conditional activation of the global term further separates full-sequence invariance under identical permutations from action-level invariance across distinct permutations (Tian et al., 5 Sep 2025).

5. Objective function and optimization

CPC uses cosine similarity with temperature $P_j$4,

$P_j$5

For each action slot $P_j$6, the local InfoNCE objective is

$P_j$7

When $P_j$8, the global InfoNCE term is

$P_j$9

The switching variable is defined as

$\{a,b,c,\dots\}$0

and the final CPC loss is

$\{a,b,c,\dots\}$1

The accompanying pseudocode specifies that, for each batch, two permutations are sampled; $\{a,b,c,\dots\}$2 and $\{a,b,c,\dots\}$3 are built with Shuffle and Warp; the features are encoded as $\{a,b,c,\dots\}$4, where $\{a,b,c,\dots\}$5 is a momentum-averaged copy; local losses are computed for each segment; embeddings are enqueued into the corresponding memory banks; and the network is updated by backpropagating $\{a,b,c,\dots\}$6 (Tian et al., 5 Sep 2025).

A notable property of the loss is that it does not sum local and global terms simultaneously. Instead, it selects the local branch when permutations differ and the global branch when they match. This suggests a deliberate separation between cross-permutation action discrimination and within-permutation sequence-level alignment.

6. Empirical behavior and transfer significance

The quantitative impact reported for CPC is given through ablation on NTU$\{a,b,c,\dots\}$7PKU1 with $\{a,b,c,\dots\}$8. Under linear evaluation, $\{a,b,c,\dots\}$9 increases from $$\text{Shuffle}(\mathcal{X},P):=\text{concat}(X^{P[1]},X^{P[2]},\dots,X^{P[G]}).$$0 for baseline supervised pretraining to $$\text{Shuffle}(\mathcal{X},P):=\text{concat}(X^{P[1]},X^{P[2]},\dots,X^{P[G]}).$$1 when using CPC alone, and to $$\text{Shuffle}(\mathcal{X},P):=\text{concat}(X^{P[1]},X^{P[2]},\dots,X^{P[G]}).$$2 when combining CPC and ROR. The source also states that, even when fine-tuned, CPC-only pretraining boosts $$\text{Shuffle}(\mathcal{X},P):=\text{concat}(X^{P[1]},X^{P[2]},\dots,X^{P[G]}).$$3 by over $$\text{Shuffle}(\mathcal{X},P):=\text{concat}(X^{P[1]},X^{P[2]},\dots,X^{P[G]}).$$4 points compared to baselines (Tian et al., 5 Sep 2025).

These gains are interpreted in the source as confirming that Cross Permutation Contrasting is highly effective at forcing the network to learn action-centric, context-invariant embeddings that transfer to long, untrimmed skeleton videos. Within the scope of the reported experiments, the evidence is therefore tied directly to transfer from trimmed skeleton pretraining to untrimmed action segmentation (Tian et al., 5 Sep 2025).

The broader significance follows from the task mismatch that motivates DuoCLR. CPC is built to reduce the discrepancy between isolated single-action supervision and downstream sequences containing multiple actions in varying orders. A plausible implication is that the method is especially relevant when the downstream task depends on preserving action identity across heterogeneous surrounding contexts rather than only recognizing actions in isolation.

7. Relationship to adjacent methods and likely misconceptions

DuoCLR is explicitly described as differing from previous representation learning works tailored for action recognition and built upon isolated sequence-wise representations. Its focus instead is on exploiting multi-scale representations in conjunction with cross-sequence variations (Tian et al., 5 Sep 2025). In that sense, CPC should not be reduced to a standard sequence-level contrastive loss. Its local action-wise formulation, permutation-based view generation, and conditional global branch define a more specialized objective.

A common misunderstanding would be to treat CPC as a method for learning relative order directly. In DuoCLR, that function belongs to ROR, which predicts relative mapping between two permutations and is described as reasoning about inter-class contexts. CPC, by contrast, is the component that learns intra-class similarities across different permutations (Tian et al., 5 Sep 2025).

Another potential misconception is that CPC operates on naturally occurring untrimmed sequences during pretraining. The source instead states that DuoCLR is pretrained using trimmed skeleton sequences and that Shuffle and Warp composes multi-action sequences from those clips (Tian et al., 5 Sep 2025). The generated permutations are therefore synthetic training constructions, not merely re-orderings extracted from native untrimmed videos.

Finally, the use of both local and global positives can be misread as always active. The specification is narrower: global positives are used only if $$\text{Shuffle}(\mathcal{X},P):=\text{concat}(X^{P[1]},X^{P[2]},\dots,X^{P[G]}).$$5; otherwise the global term is deactivated (Tian et al., 5 Sep 2025). This conditionality is central to the formal definition of $$\text{Shuffle}(\mathcal{X},P):=\text{concat}(X^{P[1]},X^{P[2]},\dots,X^{P[G]}).$$6 and to the distinction between permutation-sensitive and permutation-invariant aspects of the training signal.