Information Fused Temporal Transformation Network

• The paper introduces IF-TTN, which extends TSN by fusing spatial and temporal features through an Information Fusion Module and modeling inter-snippet transformations with a Temporal Transformation Network.
• The architecture concurrently captures short-, mid-, and long-term dynamics by integrating two-stream CNNs with adaptive residual attention and sequential feature differencing.
• Empirical results demonstrate that IF-TTN achieves state-of-the-art accuracy on benchmarks like UCF-101 and HMDB-51 while maintaining robustness with both optical flow and motion vector inputs.

The Information Fused Temporal Transformation Network (IF-TTN) is a spatiotemporal neural architecture for video-based action recognition that extends the Temporal Segment Network (TSN) paradigm through multi-level fusion of appearance and motion cues and explicit modeling of mid-term temporal transformations. By integrating the Information Fusion Module (IFM) and the Temporal Transformation Network (TTN), IF-TTN improves the discriminability and robustness of action representations, achieving state-of-the-art performance across prominent benchmarks (Yang et al., 2019).

1. Architectural Overview

IF-TTN inherits the TSN strategy of sparsely sampling $K$ temporal segments from a video, with each segment providing a single frame or stack. Within this pipeline:

• A video $V$ is divided into $K$ segments $S_1, ..., S_K$, with a snippet $T_k$ sampled from each.
• Each $T_k$ is processed by a two-stream CNN based on ResNet-50 backbones: a spatial stream $\varphi_s(\cdot; W_s)$ on RGB and a temporal stream $\varphi_t(\cdot; W_t)$ on motion fields.
• Features $a_k^l$ and $b_k^l$ from each stream and each network stage $V$0 are fused via per-stage IFM to produce fused feature maps $V$1.
• For each snippet, the set $V$2 forms a short-term descriptor; these descriptors across all snippets are input to the TTN, which models sequential pairwise transformations.
• Three prediction sources are linearly combined: (i) spatial TSN consensus (average over softmax scores of snippets' spatial stream), (ii) temporal TSN consensus, and (iii) TTN’s classification head.

This architecture enables concurrent modeling of short-term, mid-term, and long-term temporal dynamics. The overall prediction is the sum of the three sources:

$V$3

2. Information Fusion Module (IFM)

The IFM generates short-term snippet descriptors reflecting both appearance and motion. For each stage $V$4, features $V$5 (spatial) and $V$6 (temporal) are fused via one of two variants:

• Attention-based fusion:

$V$7

Here, $V$8 denotes element-wise multiplication. The $V$9 term serves as an attention mask, enhancing spatial features in regions of high motion magnitude.

• Adaptive (learned) fusion:

$K$0

with scalar weights $K$1 learned jointly. The residual-attention form arises when $K$2.

Fusion at each of stages $K$3 through $K$4 provides multilevel fused descriptor sets for input to TTN.

3. Temporal Transformation Network (TTN)

TTN explicitly encodes mid-term inter-snippet temporal transformations. Given the per-snippet fused descriptor $K$5, TTN constructs relational features for adjacent snippet pairs:

$K$6

The relational function $K$7 is implemented as a truncated ResNet-50 over stages $K$8, with Temporal Transformation Modules (TTMs) inserted at each transition. At stage $K$9, for snippet pair $S_1, ..., S_K$0, TTN output and input propagate as:

$S_1, ..., S_K$1

This formulation encodes ordered change via feature differencing, while residual connections maintain stagewise context. TTN outputs are pooled and fed to a classification head yielding logits per video.

4. Role of TSN Consensus and Multi-Granularity Modeling

TSN’s segment-level averaging captures long-term consensus but ignores snippet order. IF-TTN’s multi-head design addresses the following granularities:

• Short-term: IFM fuses per-snippet features over a typical 0.2s window (5-frame flow stack).
• Middle-term: TTN models sequential order across $S_1, ..., S_K$2 second (7 segments).
• Long-term: TSN global segment consensus integrates broader context.

At inference, separate softmax scores from each head (spatial, temporal, TTN) are summed to produce the final prediction.

5. Training Procedure and Optimization

The cross-entropy loss is computed over the aggregated class logits:

$S_1, ..., S_K$3

• No auxiliary losses within IFM or TTN.
• Progressive multi-stage training: (1) TSN backbone independently trained; (2) TTN trained on frozen TSN features; (3) joint end-to-end fine-tuning.
• Learning rates: $S_1, ..., S_K$4, stepwise reduction by $S_1, ..., S_K$5.
• Data augmentation: scale jitter, random cropping, horizontal flip, location jitter.
• Batch size: 64. Optimizer: SGD, momentum 0.9. Dropout: 0.8 (spatial, TTN), 0.7 (temporal).

6. Input Modalities and Motion Encoding

Two input representations are supported for temporal features:

• Optical flow: Stacked 5-frame TV-L1 flow fields as temporal stream input.
• Motion vectors (real-time): Directly from compressed video streams, replacing optical flow.

Fine-tuning on motion vectors after pretraining on flow preserves most of the temporal stream’s discriminative capability. On UCF-101 split 1, accuracies are:

Method	Temporal Stream	Full IF-TTN
Optical flow	86.8%	95.0%
Motion vector	82.5%	94.4%

This demonstrates high robustness to the fidelity of the motion representation.

7. Evaluation Protocol and Benchmark Results

Experiments used UCF-101 (13,320 clips, 101 classes, 3 splits) and HMDB-51 (6,766 clips, 51 classes, 3 splits) datasets. Canonical setup:

• Backbone: ResNet-50 for all streams.
• Segments: $S_1, ..., S_K$6 (1s interval).
• IFM input from stage $S_1, ..., S_K$7.

Ablation studies (UCF-101 split 1):

Method/Variant	Accuracy (%)
Two-stream TTN, no IFM	94.0
Attention/Adaptive IFM + TTN	95.0
Additive fusion (Feichtenhofer’16) [w/ TTN]	93.8
Multiplicative fusion (Feichtenhofer’17) [w/ TTN]	94.0
TTN alone (fused)	92.3
TSN Two-stream	93.1
Full IF-TTN	95.0

Real-time variant (MV-IF-TTN) achieves 94.5% at 142 fps, outperforming OFF [Sun’17] at 93.3%/206 fps and CoViAR [Wu’17] at 90.4%/240 fps.

State-of-the-art comparison:

• UCF-101: Full IF-TTN 96.2%, Full OFF 96.0%
• HMDB-51: Full IF-TTN 74.8%, Full OFF 74.2%

8. Robustness and Design Rationale

Multi-scale fusion in IFM enables the network to exploit spatial cues in scenes where motion input is noisy or degraded (e.g., from motion vectors). The residual attention mechanism $S_1, ..., S_K$8 allows spatial activations to be adaptively gated by motion confidence. TTN, by exploiting pairwise feature differences, encodes mid-term ordering that is robust to inaccuracies in low-level optical flow. Long-term consensus via TSN averaging ensures stability across poorly modeled segments. Empirically, this three-fold granularity produces high tolerance to low-quality motion input, with accuracy drop kept to approximately 0.6% when using motion vectors instead of dense optical flow on UCF-101 (Yang et al., 2019).