MS-TCN: Multi-Stage Temporal ConvNets

• MS-TCN is a deep learning architecture that employs a multi-stage design and dilated convolutions to capture long-range temporal dependencies in video.
• It refines frame-wise action probabilities through successive predictor stages, reducing over-segmentation and ensuring label consistency.
• Empirical results on datasets like 50Salads and GTEA highlight its superior performance in action segmentation and real-time applications.

A Multi-Stage Temporal Convolutional Network (MS-TCN) is a deep neural architecture introduced for fine-grained temporal segmentation and classification of long video sequences, emphasizing the mitigation of spurious segment boundaries (over-segmentation) and leveraging large receptive fields to capture action dependencies. MS-TCN architectures have become foundational for action segmentation in computer vision, time-series human activity recognition, and surgical workflow analysis due to their compositional multi-stage design, use of dilated temporal convolutions, and state-of-the-art boundary smoothness and label consistency across diverse datasets (Farha et al., 2019, Li et al., 2020, Czempiel et al., 2020, Goldbraikh et al., 2022, Shang et al., 2024).

1. Architectural Principles

The canonical MS-TCN design is formed by stacking $S$ predictor stages, each of which is a single-stage temporal convolutional network (SS-TCN) built from a sequence of dilated 1D convolutions. The first stage operates on per-frame input features $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$, typically extracted by a CNN (e.g., I3D, ResNet50). Each subsequent stage $s>1$ refines the frame-wise softmax probabilities $Y^{s-1}\in\mathbb{R}^{T\times C}$ output by the previous stage: $$Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).$$ Within a stage, the input is first projected to a hidden dimension $D$ via a $1\times1$ convolution, followed by $L$ residual dilated convolutional layers with exponentially increasing dilation factors $d_i=2^{i-1}$ and kernel size $k=3$. The $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$0th residual layer has the form: $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$1 where $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$2 is the $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$3 projected input. The stage output is generated via a final $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$4 convolution and softmax.

The receptive field $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$5 grows exponentially with the number of layers for kernel size $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$6 and dilations $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$7: $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$8 A typical $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$9 yields $s>1$0 frames, enabling recall of minute-long contexts at standard video frame rates (Farha et al., 2019, Li et al., 2020, Czempiel et al., 2020).

2. Loss Functions and Boundary Smoothing

At each stage $s>1$1, a composite loss combines cross-entropy classification with a temporal smoothing penalty: $s>1$2 where

$s>1$3

and the truncated MSE loss on temporal log-probability differences penalizes over-segmentation: $s>1$4

$s>1$5

Summation over all stages yields the total loss $s>1$6 with empirically optimal $s>1$7 and $s>1$8. This combination reduces label fragmentation at action boundaries, providing smooth, coherent segmentations (Farha et al., 2019, Li et al., 2020, Shang et al., 2024).

3. Notable MS-TCN Family Extensions

MS-TCN++ (Li et al., 2020) introduces two chief architectural novelties:

• Dual Dilated Layers (DDL) in the first stage: For each layer $s>1$9 in the prediction-generation stage, two parallel convolutions with dilations $Y^{s-1}\in\mathbb{R}^{T\times C}$0 and $Y^{s-1}\in\mathbb{R}^{T\times C}$1 capture both local and global context. The resulting features are concatenated and projected via a $Y^{s-1}\in\mathbb{R}^{T\times C}$2 conv.
• Stage decoupling: The first stage employs $Y^{s-1}\in\mathbb{R}^{T\times C}$3 DDL blocks, while each refiner stage uses $Y^{s-1}\in\mathbb{R}^{T\times C}$4 simple dilated-residual blocks. Optionally, refinement stages may share weights ("MS-TCN++(sh)") to reduce parameter count.

$Y^{s-1}\in\mathbb{R}^{T\times C}$5

Causal variants, as in TeCNO (Czempiel et al., 2020), employ only past and present frames for online inference. Bounded-future (BF)-MS-TCN++ (Goldbraikh et al., 2022) parameterizes the lookahead at each dilated convolution with a user-specified bound $Y^{s-1}\in\mathbb{R}^{T\times C}$6, tuning the latency-accuracy trade-off without reducing network depth or total receptive field. This is achieved via asymmetric padding and future window clipping per layer.

Dual-scale MS-TCN (DS-MS-TCN) (Shang et al., 2024) applies the multi-stage refinement paradigm to two-level sequence labeling: the initial stage classifies individual action repetitions (micro labels), which subsequent stages aggregate and refine to produce robust episode-scale (macro label) segmentations.

4. Empirical Results and Benchmarks

MS-TCN and its successors achieve state-of-the-art segmentation performance across standard datasets:

Dataset	Model (features)	F1@10	F1@25	F1@50	Edit	Acc
50Salads	MS-TCN (I3D)	76.3	74.0	64.5	67.9	80.7
50Salads	MS-TCN++			70.1	74.3	83.7
GTEA	MS-TCN (w/ fine-tune)	87.5	85.4	74.6	81.4	79.2
GTEA	MS-TCN++			76.0	83.5	80.1
Breakfast	MS-TCN (I3D)	52.6	48.1	37.9	61.7	66.3
Breakfast	MS-TCN++			45.9	65.6	67.6

Performance metrics include framewise accuracy, segmental Edit distance, and F1 at different IoU thresholds (10%, 25%, 50%). MS-TCN++ provides a +10–15 point F1 and +6–17 point Acc improvement over previous approaches (ED-TCN, ST-CNN, TRN, TDRN). In clinical applications (e.g., TeCNO surgical phase recognition), MS-TCN variants outperform LSTM-based baselines by 2–12 points in framewise accuracy and corresponding F1 (Farha et al., 2019, Li et al., 2020, Czempiel et al., 2020, Shang et al., 2024).

5. Training Regimen and Implementation

MS-TCN and MS-TCN++ employ Adam with learning rate $Y^{s-1}\in\mathbb{R}^{T\times C}$7, batch size of one video (entire sequence), and dropout ($Y^{s-1}\in\mathbb{R}^{T\times C}$8) after each dilated conv layer. Input features are usually I3D (video, 15 fps) or ResNet (surgical video) frame-level descriptors; no sequential shuffling, cropping, or augmentation is applied to preserve temporal structure. Most models converge in under 100 epochs. For DS-MS-TCN, IMU signals are z-score normalized, windowed at 40s (with 50% overlap), batch 32, Adam optimizer, and typical convergence within 100–200 epochs.

Parameter efficiency is also notable: MS-TCN++ achieves a complete four-stage model with $Y^{s-1}\in\mathbb{R}^{T\times C}$91M parameters, and shared-refiner variants require only $$Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).$$00.66M, while inference runs at $$Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).$$1100 fps on standard hardware (Li et al., 2020, Shang et al., 2024).

6. Ablation Analyses and Theoretical Insights

Key ablation findings and observations:

• Number of stages ($$Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).$$2): Single-stage TCN is markedly inferior (F1@10$$Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).$$327.0); multi-stage refinement gives monotonic gains to $$Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).$$4 (F1@10=76.3), with minor overfitting at $$Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).$$5.
• Depth vs. Staging: A flat TCN of same total depth (e.g., 48 layers) is outperformed by MS-TCN's multi-stage refinement, confirming the iterative smoothing effect and ensemble-like advantages.
• Loss function: The addition of truncated MSE is critical; using cross-entropy alone or a KL penalty achieves only moderate smoothing.
• Input propagation: Only class-probability vectors should enter higher stages—concatenating raw features significantly degrades performance.
• Temporal resolution: Lowering frame rate mildly reduces over-segmentation but impairs temporal precision.
• Receptive field and video length: Performance diminishes for sequences much longer than RF, indicating the necessity for matching model RF to application sequence duration (Farha et al., 2019, Li et al., 2020, Goldbraikh et al., 2022, Shang et al., 2024).

Bounded-future MS-TCN++ ablation demonstrates that limiting lookahead per layer with $$Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).$$6 can closely approach offline accuracy with minimal delay (1s lookahead yields 80% of the offline F1@50 performance), superior to naive depth-reduction strategies (Goldbraikh et al., 2022).

7. Applications, Variants, and Limitations

MS-TCN and its derivatives are now core components for:

• Fine-grained action segmentation in video (surveillance, human activity recognition, cooking demonstrations, sports analytics) (Farha et al., 2019, Li et al., 2020).
• Surgical workflow, gesture, and phase recognition, with causal and bounded-future variants supporting real-time or minimally delayed intra-operative analytics (Czempiel et al., 2020, Goldbraikh et al., 2022).
• Wearable sensor-based rehabilitation activity recognition, notably through dual-scale and multi-label cascades, integrating both repetition-scale and episode-scale prediction (Shang et al., 2024).

Practical considerations include robustness to input features, generalizability to unconstrained or out-of-distribution data (notably in clinical and home rehabilitation deployment), and the labor intensity of frame-level or repetition-level labeling (for dual-scale or micro-label approaches) (Shang et al., 2024).

A plausible implication is that continued development of semi-supervised, transfer learning, or multi-modal feature integration (video + sensor) will further broaden the applicability of MS-TCN architectures while reducing expert annotation costs.

---

References:

• (Farha et al., 2019): MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
• (Li et al., 2020): MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
• (Czempiel et al., 2020): TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks
• (Goldbraikh et al., 2022): Bounded Future MS-TCN++ for surgical gesture recognition
• (Shang et al., 2024): DS-MS-TCN: Otago Exercises Recognition with a Dual-Scale Multi-Stage Temporal Convolutional Network