---
title: Information Fused Temporal Transformation Network
url: https://www.emergentmind.com/topics/information-fused-temporal-transformation-network-if-ttn
type: topic
---

# Information Fused Temporal Transformation Network

The Information Fused Temporal Transformation Network (IF-TTN) is a spatiotemporal neural architecture for video-based action recognition that extends the Temporal Segment Network (TSN) paradigm through multi-level fusion of appearance and motion cues and explicit modeling of mid-term temporal transformations. By integrating the Information Fusion Module (IFM) and the Temporal Transformation Network (TTN), IF-TTN improves the discriminability and robustness of action representations, achieving state-of-the-art performance across prominent benchmarks [1902.09928].

## 1. Architectural Overview

IF-TTN inherits the TSN strategy of sparsely sampling $K$ temporal segments from a video, with each segment providing a single frame or stack. Within this pipeline:

- A video $V$ is divided into $K$ segments $S_1, ..., S_K$, with a snippet $T_k$ sampled from each.
- Each $T_k$ is processed by a two-stream CNN based on ResNet-50 backbones: a spatial stream $\varphi_s(\cdot; W_s)$ on RGB and a temporal stream $\varphi_t(\cdot; W_t)$ on motion fields.
- Features $a_k^l$ and $b_k^l$ from each stream and each network stage $l$ are fused via per-stage IFM to produce fused feature maps $f_k^l$.
- For each snippet, the set $\{f_k^l\}$ forms a short-term descriptor; these descriptors across all snippets are input to the TTN, which models sequential pairwise transformations.
- Three prediction sources are linearly combined: (i) spatial TSN consensus (average over softmax scores of snippets' spatial stream), (ii) temporal TSN consensus, and (iii) TTN’s classification head.

This architecture enables concurrent modeling of short-term, mid-term, and long-term temporal dynamics. The overall prediction is the sum of the three sources:
$$
\text{Final score} = \text{AvgSpatial} + \text{AvgTemporal} + \text{TTN}
$$

## 2. Information Fusion Module (IFM)

The IFM generates short-term snippet descriptors reflecting both appearance and motion. For each stage $l$, features $a_k^l \in \mathbb{R}^{C \times H \times W}$ (spatial) and $b_k^l \in \mathbb{R}^{C \times H \times W}$ (temporal) are fused via one of two variants:

- **Attention-based fusion:** 
  $$
  f_k^l = a_k^l + a_k^l \odot b_k^l
  $$
  Here, $\odot$ denotes element-wise multiplication. The $a_k^l \odot b_k^l$ term serves as an attention mask, enhancing spatial features in regions of high motion magnitude.

- **Adaptive (learned) fusion:** 
  $$
  f_k^l = \alpha_1 a_k^l + \alpha_2 b_k^l + \alpha_3 (a_k^l \odot b_k^l)
  $$
  with scalar weights $\alpha_1, \alpha_2, \alpha_3$ learned jointly. The residual-attention form arises when $\alpha_1=1, \alpha_2=0, \alpha_3=1$.

Fusion at each of stages $l_s$ through $L$ provides multilevel fused descriptor sets for input to TTN.

## 3. Temporal Transformation Network (TTN)

TTN explicitly encodes mid-term inter-snippet temporal transformations. Given the per-snippet fused descriptor $f_k = \text{concat}_l \{ f_k^l \}$, TTN constructs relational features for adjacent snippet pairs:
$$
T(V) = \sum_{i=1}^{K-1} R(f_i, f_{i+1}; W_{\text{ttn}})
$$
The relational function $R(\cdot, \cdot)$ is implemented as a truncated ResNet-50 over stages $\{3,4,5\}$, with Temporal Transformation Modules (TTMs) inserted at each transition. At stage $l$, for snippet pair $(i, j)$, TTN output and input propagate as:
$$
r_{\text{in}}^{l+1, i \rightarrow j} = (f_j^l - f_i^l) + r_{\text{out}}^{l, i \rightarrow j}
$$
This formulation encodes ordered change via feature differencing, while residual connections maintain stagewise context. TTN outputs are pooled and fed to a classification head yielding logits per video.

## 4. Role of TSN Consensus and Multi-Granularity Modeling

TSN’s segment-level averaging captures long-term consensus but ignores snippet order. IF-TTN’s multi-head design addresses the following granularities:

- **Short-term:** IFM fuses per-snippet features over a typical 0.2s window (5-frame flow stack).
- **Middle-term:** TTN models sequential order across $\sim1$ second (7 segments).
- **Long-term:** TSN global segment consensus integrates broader context.

At inference, separate softmax scores from each head (spatial, temporal, TTN) are summed to produce the final prediction.

## 5. Training Procedure and Optimization

The cross-entropy loss is computed over the aggregated class logits:
$$
L = -\sum_c y_c \log p_c,\quad \text{where } p_c = \text{softmax}_c\{ \text{score}_{\text{spatial}} + \text{score}_{\text{temporal}} + \text{score}_{\text{TTN}} \}
$$
- No auxiliary losses within IFM or TTN.
- Progressive multi-stage training: (1) TSN backbone independently trained; (2) TTN trained on frozen TSN features; (3) joint end-to-end fine-tuning.
- Learning rates: $1\text{e}{-3}$, stepwise reduction by $0.1$.
- Data augmentation: scale jitter, random cropping, horizontal flip, location jitter.
- Batch size: 64. Optimizer: SGD, momentum 0.9. Dropout: 0.8 (spatial, TTN), 0.7 (temporal).

## 6. Input Modalities and Motion Encoding

Two input representations are supported for temporal features:

- **Optical flow:** Stacked 5-frame TV-L1 flow fields as temporal stream input.
- **Motion vectors (real-time):** Directly from compressed video streams, replacing optical flow.

Fine-tuning on motion vectors after pretraining on flow preserves most of the temporal stream’s discriminative capability. On UCF-101 split 1, accuracies are:

| Method                              | Temporal Stream   | Full IF-TTN |
|--------------------------------------|------------------|-------------|
| Optical flow                        | 86.8%            | 95.0%       |
| Motion vector                       | 82.5%            | 94.4%       |

This demonstrates high robustness to the fidelity of the motion representation.

## 7. Evaluation Protocol and Benchmark Results

Experiments used UCF-101 (13,320 clips, 101 classes, 3 splits) and HMDB-51 (6,766 clips, 51 classes, 3 splits) datasets. Canonical setup:

- Backbone: ResNet-50 for all streams.
- Segments: $K=7$ (1s interval).
- IFM input from stage $l_s=3$.

Ablation studies (UCF-101 split 1):

| Method/Variant                | Accuracy (%) |
|-------------------------------|--------------|
| Two-stream TTN, no IFM        | 94.0         |
| Attention/Adaptive IFM + TTN  | 95.0         |
| Additive fusion (Feichtenhofer’16) [w/ TTN] | 93.8 |
| Multiplicative fusion (Feichtenhofer’17) [w/ TTN] | 94.0 |
| TTN alone (fused)             | 92.3         |
| TSN Two-stream                | 93.1         |
| Full IF-TTN                   | 95.0         |

Real-time variant (MV-IF-TTN) achieves 94.5% at 142 fps, outperforming OFF [Sun’17] at 93.3%/206 fps and CoViAR [Wu’17] at 90.4%/240 fps.

State-of-the-art comparison:
- UCF-101: Full IF-TTN 96.2%, Full OFF 96.0%
- HMDB-51: Full IF-TTN 74.8%, Full OFF 74.2%

## 8. Robustness and Design Rationale

Multi-scale fusion in IFM enables the network to exploit spatial cues in scenes where motion input is noisy or degraded (e.g., from motion vectors). The residual attention mechanism $a_k + a_k \odot b_k$ allows spatial activations to be adaptively gated by motion confidence. TTN, by exploiting pairwise feature differences, encodes mid-term ordering that is robust to inaccuracies in low-level optical flow. Long-term consensus via TSN averaging ensures stability across poorly modeled segments. Empirically, this three-fold granularity produces high tolerance to low-quality motion input, with accuracy drop kept to approximately 0.6% when using motion vectors instead of dense optical flow on UCF-101 [1902.09928].

Source: https://www.emergentmind.com/topics/information-fused-temporal-transformation-network-if-ttn