Contextual Latent Composition

• Contextual latent composition is defined as context-conditioned latent operations that compose motion primitives via arithmetic in a learned latent space for action segmentation.
• It employs an autoencoder with Linear Action Decomposition to disentangle dynamic motion and static factors using orthogonal latent bases, enabling robust synthesis of composed actions.
• Dual-scale training leverages both sequence-level and frame-level contrastive losses to improve segmentation performance in untrimmed videos, as evidenced on TSU, Charades, and PKU-MMD.

Contextual latent composition denotes a family of context-conditioned latent operations in which structured factors are composed in a latent space rather than directly in the original observation space. In the formulation introduced by "LAC: Latent Action Composition for Skeleton-based Action Segmentation" (Yang et al., 2023), the term refers specifically to composing high-level actions by arithmetic in a learned linear latent space while exploiting contextual signals at two temporal granularities—sequence level and frame level—to train a transfer-ready skeleton encoder for action segmentation. The central premise is that many activities in untrimmed videos are compositional: multiple short-duration actions co-occur or overlap, such as walking while drinking, and these co-occurrences are more naturally modeled through latent motion codes than through direct manipulation of joint coordinates (Yang et al., 2023).

1. Conceptual basis in skeleton-based action segmentation

In LAC, contextual latent composition is built on three linked assumptions. First, skeleton-based action segmentation in untrimmed video requires representations that can encode co-occurring and overlapping actions rather than isolated trimmed gestures. Second, many atomic motion primitives, such as raising a hand, bending over, or squatting, are approximately independent and additive at moderate amplitudes. Third, contextual information relevant to segmentation exists at both the whole-sequence scale and the per-frame scale (Yang et al., 2023).

The method therefore distinguishes between a dynamic component, called “Motion,” and a static component, called “Static.” Motion captures time-varying action content. Static captures viewpoint and body size. Rather than combining raw joint trajectories, LAC learns an orthogonal latent basis in which motion directions are linearly combinable and static factors can be set independently. New composed motions are then synthesized by linear arithmetic on motion codes, often by simple averaging of two motions, while preserving or swapping the associated static code (Yang et al., 2023).

This formulation addresses a limitation of conventional two-step pipelines for skeleton segmentation. Traditional systems typically extract local skeleton features using a pretrained encoder and then pass those features to a temporal model such as a TCN or Transformer. LAC argues that this decoupling struggles with composable actions in untrimmed videos because trimmed pretraining lacks co-occurrence diversity and because the resulting local features are insufficiently expressive for subtle transitions and overlapping actions (Yang et al., 2023).

2. Latent factorization and composition mechanism

LAC implements contextual latent composition through a composable action generator: an autoencoder with Linear Action Decomposition (LAD). A skeleton sequence with motion $m$ and static factor $c$ is represented as

$p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$

where $T$ is the number of frames, $V$ the number of joints, and $C_{in} \in \{2,3\}$ for 2D or 3D skeletons. An encoder $E_{LAC}$ maps the sequence to a latent representation

$r_{m,c} \in \mathbb{R}^{T' \times C_{out}}.$

LAD then decomposes this latent into orthogonal motion and static subspaces using a learnable dictionary

$D_v \in \mathbb{R}^{C_{out} \times C_{out}},$

with motion directions $d_m^i$ and static directions $c$0. Orthogonality is enforced by Gram–Schmidt normalization at each forward pass (Yang et al., 2023).

The dynamic magnitudes are computed by projection: $c$1 and similarly the static magnitudes $c$2 are computed from the temporally averaged latent. The reconstructed motion and static components are

$c$3

A decoder $c$4 reconstructs the sequence from $c$5 (Yang et al., 2023).

Composition is then performed in latent space. Given two sequences $c$6 and $c$7, LAC synthesizes a composed motion under either static condition: $c$8 The more general form is a convex combination across $c$9 motions with coefficients $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$0, $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$1. In practice, LAC primarily uses $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$2 with $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$3 during inference (Yang et al., 2023).

The generator is trained on Mixamo through reconstruction and motion-retargeting losses: $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$4 with

$p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$5

The paper explicitly notes that no KL term is used; the generator is not VAE-based (Yang et al., 2023).

3. Dual-scale contextual training

The contextual aspect of LAC lies in how the synthesized sequences are used to pretrain a visual encoder. A topology-free skeleton encoder $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$6 based on UNIK extracts both frame-level features $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$7 and sequence-level features $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$8. A projection head $p_{m,c} \in \mathbb{R}^{T \times V \times C_{in}},$9 maps these features into contrastive space, and a memory queue supplies negatives at frame granularity (Yang et al., 2023).

Sequence-level context is defined as the global representation of an entire clip, summarizing the set and temporal evolution of primitives. Frame-level context is defined as per-frame embeddings that encode instantaneous pose and subtle dynamics. LAC trains both simultaneously through two InfoNCE-style losses, one on clip representations and one on aligned frame representations. Positives share the same composed motion but differ only in static factors; negatives are other sequences with different motions stored in the queue. The sequence-level term is

$T$0

and the frame-level term sums similarities over time: $T$1 Similarity is cosine similarity in projection space with temperature scaling. The reported setup uses $T$2 positives and approximately $T$3 negatives. Training is staged: LAD is first optimized on Mixamo with $T$4, then the visual encoder is pretrained on Posetics with $T$5 (Yang et al., 2023).

This two-scale design is central to the method’s interpretation of context. Sequence-level contrastive learning encourages invariance to static changes such as viewpoint and body size. Frame-level contrastive learning sharpens sensitivity to local dynamics and action boundaries. The reported ablations further indicate that subsampling frames every 4 steps balances signal and computational cost for the frame-level term (Yang et al., 2023).

4. Transfer to frame-wise segmentation

After self-supervised pretraining, LAC transfers the encoder directly to action segmentation without an additional temporal model. A linear classifier and Softmax are attached to the frame-level features $T$6, producing per-frame action probabilities. Binary Cross Entropy is used for multi-label frames, which are common in TSU and Charades. Long videos are processed with sliding windows, and predictions from overlapping windows are merged online (Yang et al., 2023).

This transfer protocol differs from the dominant skeleton-based segmentation pipeline. Rather than combining a local visual encoder with a separate TCN or Transformer head, LAC performs end-to-end fine-tuning of the pretrained visual encoder and classifier alone. The paper presents this as a reduction in optimization mismatch between local feature learning and temporal classification, and as a consequence of the encoder already learning globally coherent and locally precise representations during pretraining (Yang et al., 2023).

The architecture details reinforce this design. The generator encoder $T$7 consists of three stacked temporal 1D convolution blocks with kernel size 8 and channels $T$8, yielding a latent of shape $T$9. The decoder upsamples temporally and applies 1D convolutions with kernel size 7. The empirical best dictionary split is $V$0 motion directions and $V$1 static directions out of $V$2. The UNIK visual encoder uses repeated 2D convolutions over time-by-joint grids and produces 256-dimensional per-frame embeddings. Linear evaluation trains only the classifier, with approximately $V$3 parameters depending on dataset, whereas full fine-tuning updates the $V$4-parameter backbone (Yang et al., 2023).

5. Empirical profile

LAC is pretrained on Posetics for self-supervised contrastive learning and on Mixamo for motion retargeting, then evaluated on TSU, Charades, and PKU-MMD (Yang et al., 2023).

Dataset	Metric	Reported LAC results
TSU	per-frame mAP	unsupervised: 34.1% CS, 22.8% CV; supervised: 36.8% CS, 23.1% CV
Charades	per-frame mAP	unsupervised: 22.3%; supervised: 25.6%
PKU-MMD	event mAP@IoU	unsupervised: 91.8%/90.2%/88.5% at 0.1/0.3/0.5; supervised: 92.6%/91.4%/90.6%

On TSU, the paper reports that prior skeleton-based methods are substantially lower, citing SD-TCN at $V$5 CS and $V$6 CV, and TGM at $V$7 CS and $V$8 CV. The TSU ablation isolates the effect of contextual latent composition: a baseline UNIK encoder without LAC reaches $V$9 CS and $C_{in} \in \{2,3\}$0 CV; adding motion composition with two motions raises performance to $C_{in} \in \{2,3\}$1 CS and $C_{in} \in \{2,3\}$2 CV; adding frame-level contrast with sampling rate 4 yields the full $C_{in} \in \{2,3\}$3 CS and $C_{in} \in \{2,3\}$4 CV result (Yang et al., 2023).

The transfer-learning study with reduced supervision is equally prominent. With only $C_{in} \in \{2,3\}$5 labeled data, LAC pretraining lifts TSU CS from $C_{in} \in \{2,3\}$6 when trained from scratch to $C_{in} \in \{2,3\}$7, and raises Charades mAP from $C_{in} \in \{2,3\}$8 to $C_{in} \in \{2,3\}$9. With $E_{LAC}$0 labels, it raises TSU CS from $E_{LAC}$1 to $E_{LAC}$2, and Charades mAP from $E_{LAC}$3 to $E_{LAC}$4 (Yang et al., 2023).

These results are reported using per-frame mAP for TSU and Charades and event mAP@IoU for PKU-MMD. The paper notes that some segmentation work instead reports $E_{LAC}$5 and edit score, whereas LAC emphasizes metrics that directly reflect detection accuracy and localization quality (Yang et al., 2023).

6. Limits, extensions, and broader meanings of the term

The LAC formulation assumes that motion primitives are nearly linearly composable in an orthogonal latent basis. The paper explicitly notes that large or conflicting motions may produce unrealistic syntheses, that static-motion disentanglement may drift without sufficient diversity even with Gram–Schmidt normalization, and that skeleton-only representations remain sensitive to 2D pose errors, missing joints, and missing object cues such as the cup implied by a drinking action (Yang et al., 2023).

Several extensions are proposed within the same framework: nonlinear composition through attention-weighted or gated mixing of latent directions; context-aware coefficients $E_{LAC}$6 that vary over time; integration with lightweight TCN or Transformer modules for boundary refinement; multimodal expansion to RGB or optical flow; and adaptive reference poses to reduce static-decomposition bias. These are presented as potential directions rather than as implemented components (Yang et al., 2023).

Outside skeleton-based action segmentation, the phrase “contextual latent composition” has been used in distinct but related senses. In sentiment analysis, "Interpreting Sentiment Composition with Latent Semantic Tree" models composition through a latent CFG whose rule applications are conditioned on contextual span representations and marginalized by the inside algorithm (Jiang et al., 2023). In conversational agents, "Context Retrieval via Normalized Contextual Latent Interaction for Conversational Agent" uses normalized token-level late interaction and gated aggregation over persona, history, and knowledge to compose retrieval context (Liu et al., 2023). In offline meta-reinforcement learning, "Contextual Latent World Models for Offline Meta Reinforcement Learning" composes a latent state with an inferred task representation inside a task-conditioned world model (Nakheai et al., 3 Mar 2026). This suggests that the expression does not denote a single standardized formalism across machine learning; rather, it names a broader design pattern in which contextual signals are encoded, recombined, or filtered in latent space before downstream prediction.

Within that broader pattern, LAC is distinctive for making contextual latent composition operational through arithmetic over disentangled motion codes and through contrastive training at sequence and frame scales. Its contribution is therefore both architectural and epistemic: it treats co-occurrence context not as a property to be modeled only after feature extraction, but as structure that can be synthesized directly in latent motion space and then used to shape a transfer-ready visual encoder for dense action understanding (Yang et al., 2023).