---
title: 'MS-TCN: Multi-Stage Temporal ConvNets'
url: https://www.emergentmind.com/topics/multi-stage-temporal-convolutional-network-ms-tcn
type: topic
---

# MS-TCN: Multi-Stage Temporal ConvNets

A Multi-Stage Temporal Convolutional Network (MS-TCN) is a deep neural architecture introduced for fine-grained temporal segmentation and classification of long video sequences, emphasizing the mitigation of spurious segment boundaries (over-segmentation) and leveraging large receptive fields to capture action dependencies. MS-TCN architectures have become foundational for action segmentation in computer vision, time-series human activity recognition, and surgical workflow analysis due to their compositional multi-stage design, use of dilated temporal convolutions, and state-of-the-art boundary smoothness and label consistency across diverse datasets [1903.01945][2006.09220][2003.10751][2209.14647][2402.02910].

## 1. Architectural Principles

The canonical MS-TCN design is formed by stacking $S$ predictor stages, each of which is a single-stage temporal convolutional network (SS-TCN) built from a sequence of dilated 1D convolutions. The first stage operates on per-frame input features $x_{1:T}\in\mathbb{R}^{T\times D_{\rm in}}$, typically extracted by a CNN (e.g., I3D, ResNet50). Each subsequent stage $s>1$ refines the frame-wise softmax probabilities $Y^{s-1}\in\mathbb{R}^{T\times C}$ output by the previous stage:
\[
Y^{0} = x_{1:T},\qquad Y^{s} = \mathcal{F}\bigl(Y^{s-1}\bigr)\quad(s=1\ldots S).
\]
Within a stage, the input is first projected to a hidden dimension $D$ via a $1\times1$ convolution, followed by $L$ residual dilated convolutional layers with exponentially increasing dilation factors $d_i=2^{i-1}$ and kernel size $k=3$. The $i$th residual layer has the form:
\[
\begin{aligned}
\hat H_i &= \mathrm{ReLU}\bigl(W^{(i)}_{\rm dil}\,*\,H_{i-1}+b^{(i)}\bigr),\\
H_i &= H_{i-1} + (W^{(i)}_{1\times1} * \hat H_i + c^{(i)}),
\end{aligned}
\]
where $H_0$ is the $1\times1$ projected input. The stage output is generated via a final $1\times1$ convolution and softmax.

The receptive field $R$ grows exponentially with the number of layers for kernel size $k$ and dilations $d_i$:
\[
R = 1 + \sum_{i=1}^L (k-1)d_i, \qquad \textrm{(e.g., for $k=3$, $d_i=2^{i-1}$: $R=2^{L+1}-1$).}
\]
A typical $L=10$ yields $R=2047$ frames, enabling recall of minute-long contexts at standard video frame rates [1903.01945][2006.09220][2003.10751].

## 2. Loss Functions and Boundary Smoothing

At each stage $s$, a composite loss combines cross-entropy classification with a temporal smoothing penalty:
\[
\mathcal{L}_s = \mathcal{L}_{\rm cls} + \lambda \mathcal{L}_{\rm T\!-\!MSE},
\]
where
\[
\mathcal{L}_{\rm cls} = -\frac{1}{T}\sum_{t=1}^T\sum_{c=1}^C 1[c_t=c]\,\log y_{t,c},
\]
and the truncated MSE loss on temporal log-probability differences penalizes over-segmentation:
\[
\Delta_{t,c} = |\log y_{t,c} - \log y_{t-1,c}|,\qquad \tilde{\Delta}_{t,c} = \min(\Delta_{t,c},\tau),
\]
\[
\mathcal{L}_{\rm T\!-\!MSE} = \frac{1}{T\,C}\sum_{t=2}^T\sum_{c=1}^C (\tilde{\Delta}_{t,c})^2.
\]
Summation over all stages yields the total loss $\mathcal{L} = \sum_{s=1}^S \mathcal{L}_s$ with empirically optimal $\lambda=0.15$ and $\tau=4$. This combination reduces label fragmentation at action boundaries, providing smooth, coherent segmentations [1903.01945][2006.09220][2402.02910].

## 3. Notable MS-TCN Family Extensions

**MS-TCN++** [2006.09220] introduces two chief architectural novelties:
- Dual Dilated Layers (DDL) in the first stage: For each layer $l$ in the prediction-generation stage, two parallel convolutions with dilations $d_1(l) = 2^{l-1}$ and $d_2(l) = 2^{L-l}$ capture both local and global context. The resulting features are concatenated and projected via a $1\times1$ conv.
- Stage decoupling: The first stage employs $L_1$ DDL blocks, while each refiner stage uses $L_r$ simple dilated-residual blocks. Optionally, refinement stages may share weights ("MS-TCN++(sh)") to reduce parameter count.

\[
\begin{aligned}
\hat H_{l,1} &= W^{(d_1)}_l *_{d_1(l)} H_{l-1} + b^{(d_1)}_l,\\
\hat H_{l,2} &= W^{(d_2)}_l *_{d_2(l)} H_{l-1} + b^{(d_2)}_l,\\
\hat H_l &= \mathrm{ReLU}([\hat H_{l,1}, \hat H_{l,2}]),\\
H_l &= H_{l-1} + (W^{(1)}_l *_{1} \hat H_l + b^{(1)}_l).
\end{aligned}
\]

**Causal variants**, as in TeCNO [2003.10751], employ only past and present frames for online inference. **Bounded-future (BF)-MS-TCN++** [2209.14647] parameterizes the lookahead at each dilated convolution with a user-specified bound $w_{\rm max}$, tuning the latency-accuracy trade-off without reducing network depth or total receptive field. This is achieved via asymmetric padding and future window clipping per layer.

**Dual-scale MS-TCN (DS-MS-TCN)** [2402.02910] applies the multi-stage refinement paradigm to two-level sequence labeling: the initial stage classifies individual action repetitions (micro labels), which subsequent stages aggregate and refine to produce robust episode-scale (macro label) segmentations.

## 4. Empirical Results and Benchmarks

MS-TCN and its successors achieve state-of-the-art segmentation performance across standard datasets:

| Dataset    | Model (features)     | F1@10 | F1@25 | F1@50 | Edit | Acc           |
|------------|---------------------|-------|-------|-------|------|---------------|
| 50Salads   | MS-TCN (I3D)        | 76.3  | 74.0  | 64.5  | 67.9 | 80.7          |
| 50Salads   | MS-TCN++            |       |       | 70.1  | 74.3 | 83.7          |
| GTEA       | MS-TCN (w/ fine-tune)| 87.5 | 85.4  | 74.6  | 81.4 | 79.2          |
| GTEA       | MS-TCN++            |       |       | 76.0  | 83.5 | 80.1          |
| Breakfast  | MS-TCN (I3D)        | 52.6  | 48.1  | 37.9  | 61.7 | 66.3          |
| Breakfast  | MS-TCN++            |       |       | 45.9  | 65.6 | 67.6          |

Performance metrics include framewise accuracy, segmental Edit distance, and F1 at different IoU thresholds (10%, 25%, 50%). MS-TCN++ provides a +10–15 point F1 and +6–17 point Acc improvement over previous approaches (ED-TCN, ST-CNN, TRN, TDRN). In clinical applications (e.g., TeCNO surgical phase recognition), MS-TCN variants outperform LSTM-based baselines by 2–12 points in framewise accuracy and corresponding F1 [1903.01945][2006.09220][2003.10751][2402.02910].

## 5. Training Regimen and Implementation

MS-TCN and MS-TCN++ employ Adam with learning rate $5 \times 10^{-4}$, batch size of one video (entire sequence), and dropout ($p=0.5$) after each dilated conv layer. Input features are usually I3D (video, 15 fps) or ResNet (surgical video) frame-level descriptors; no sequential shuffling, cropping, or augmentation is applied to preserve temporal structure. Most models converge in under 100 epochs. For DS-MS-TCN, IMU signals are z-score normalized, windowed at 40s (with 50% overlap), batch 32, Adam optimizer, and typical convergence within 100–200 epochs.

Parameter efficiency is also notable: MS-TCN++ achieves a complete four-stage model with $\approx$1M parameters, and shared-refiner variants require only $\approx$0.66M, while inference runs at $>$100 fps on standard hardware [2006.09220][2402.02910].

## 6. Ablation Analyses and Theoretical Insights

Key ablation findings and observations:

- **Number of stages ($S$):** Single-stage TCN is markedly inferior (F1@10$\approx$27.0); multi-stage refinement gives monotonic gains to $S=4$ (F1@10=76.3), with minor overfitting at $S>4$.
- **Depth vs. Staging:** A flat TCN of same total depth (e.g., 48 layers) is outperformed by MS-TCN's multi-stage refinement, confirming the iterative smoothing effect and ensemble-like advantages.
- **Loss function:** The addition of truncated MSE is critical; using cross-entropy alone or a KL penalty achieves only moderate smoothing.
- **Input propagation:** Only class-probability vectors should enter higher stages—concatenating raw features significantly degrades performance.
- **Temporal resolution:** Lowering frame rate mildly reduces over-segmentation but impairs temporal precision.
- **Receptive field and video length:** Performance diminishes for sequences much longer than RF, indicating the necessity for matching model RF to application sequence duration [1903.01945][2006.09220][2209.14647][2402.02910].

Bounded-future MS-TCN++ ablation demonstrates that limiting lookahead per layer with $w_{\rm max}$ can closely approach offline accuracy with minimal delay (1s lookahead yields 80% of the offline F1@50 performance), superior to naive depth-reduction strategies [2209.14647].

## 7. Applications, Variants, and Limitations

MS-TCN and its derivatives are now core components for:
- Fine-grained action segmentation in video (surveillance, human activity recognition, cooking demonstrations, sports analytics) [1903.01945][2006.09220].
- Surgical workflow, gesture, and phase recognition, with causal and bounded-future variants supporting real-time or minimally delayed intra-operative analytics [2003.10751][2209.14647].
- Wearable sensor-based rehabilitation activity recognition, notably through dual-scale and multi-label cascades, integrating both repetition-scale and episode-scale prediction [2402.02910].

Practical considerations include robustness to input features, generalizability to unconstrained or out-of-distribution data (notably in clinical and home rehabilitation deployment), and the labor intensity of frame-level or repetition-level labeling (for dual-scale or micro-label approaches) [2402.02910].

A plausible implication is that continued development of semi-supervised, transfer learning, or multi-modal feature integration (video + sensor) will further broaden the applicability of MS-TCN architectures while reducing expert annotation costs.

---

**References:**  
- [1903.01945]: MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation  
- [2006.09220]: MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation  
- [2003.10751]: TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks  
- [2209.14647]: Bounded Future MS-TCN++ for surgical gesture recognition  
- [2402.02910]: DS-MS-TCN: Otago Exercises Recognition with a Dual-Scale Multi-Stage Temporal Convolutional Network

Source: https://www.emergentmind.com/topics/multi-stage-temporal-convolutional-network-ms-tcn