---
title: Temporal Feature-Wise Linear Modulation (TFiLM)
url: https://www.emergentmind.com/topics/temporal-feature-wise-linear-modulation-tfilm
type: topic
---

# Temporal Feature-Wise Linear Modulation (TFiLM)

Temporal Feature-Wise Linear Modulation (TFiLM) is an architectural technique for learning long-range sequence dependencies by modulating the activations of neural network feature channels with temporally-evolving, data-dependent scale and shift parameters. Originally motivated by the constraints of convolutional models in capturing dependencies beyond the finite receptive field, TFiLM has evolved into a broadly adopted mechanism for sequence modeling in audio, speech, and text applications, often delivering significant improvements in learning efficiency, perceptual quality, and flexibility.

## 1. Concept and Mathematical Formulation

TFiLM extends the idea of Feature-Wise Linear Modulation (FiLM), which applies affine transformations to neural activations based on conditioning signals, into the temporal domain. In TFiLM, modulation parameters depend on the sequence context or external temporal control factors. The general formulation—in the case of sequential input $x_{1:T}$ processed by a stack of convolutional layers—is:

\[
h'_t(i) = \gamma_t(i)\, h_t(i) + \beta_t(i),
\]

where $h_t(i)$ is the activation of feature channel $i$ at time $t$ after convolution (and optional normalization), and modulation parameters $\gamma_t, \beta_t \in \mathbb{R}^n$ are determined by a temporally-aware controller. Across variants, the controller may be a recurrent neural network (RNN) mapping feature summaries to modulation parameters or a function of an explicit temporal control signal.

The canonical TFiLM pipeline is:
1. Convolve input sequence $x_{1:T}$ layer-wise.
2. After each convolution, pool or summarize feature activations.
3. Process the summary via an RNN or sequential encoder to generate $\gamma_t, \beta_t$.
4. Apply modulation to features and forward to subsequent layers.

Block-wise TFiLM, as employed in more recent works, partitions the sequence into $B$ non-overlapping blocks, summarizes each via max-pooling, and computes per-block modulation parameters via an RNN (typically an LSTM):

\[
[\gamma_b, \beta_b] = W H_b + b,\quad b = 1, \ldots, B,
\]
\[
Y_{t,c} = \gamma_{b(t), c} X_{t, c} + \beta_{b(t), c},
\]
where $b(t)$ maps $t$ to its block.

## 2. Architectural Integrations and Variants

TFiLM has been integrated into various neural backbones, typically interleaved with convolutional layers. Key instantiations include:

- **Convolutional Backbones with RNN Modulation**: TFiLM augments each conv layer with a small RNN (GRU or LSTM) that produces time-varying scale and shift per feature channel [1909.06628].
- **Block-wise TFiLM with Gated Convolutions**: In audio black-box modeling, TFiLM is inserted after every gated dilated convolution. Each TFiLM module modulates features over time by summarizing blocks, updating LSTM states, and applying scaling/shifting [2211.00497].
- **Encoder-Decoder with External Conditioning**: In time-scale modification, TFiLM layers receive a scalar speed factor $\alpha$ as input, and modulation parameters are computed via a small MLP, broadcast identically over the entire sequence or at various generator layers [2510.02672].
- **UNet / TUNet Extensions**: TFiLM modulates conv feature maps after each encoder/decoder block. State-carry mechanisms preserve RNN states across overlapping inference windows for block-online streaming [2110.13492].

These architectures demonstrate the flexibility of TFiLM as a plug-in module for feature-wise, temporally- or conditionally-adaptive modulation.

## 3. Role in Capturing Long-Range Dependencies

Pure convolutional or dilated convolutional models are limited by their finite, exponentially increasing but ultimately local, receptive field. TFiLM supplements these with a recurrent path that conditions the feature transformation at each time step (or block) on extended temporal context. For example:

- In [1909.06628], the RNN aggregates information from all previous time steps, enabling effectively unbounded sequence memory for the modulation.
- In block-wise TFiLM [2211.00497, 2110.13492], LSTMs ingest max-pooled summaries over broad context, generating block-level modulations that adapt to evolving long-range dynamics, such as slow-varying envelopes, attack/release behavior in audio effects, or phonetic context in speech.

A plausible implication is that TFiLM enables convolutional sequence models to handle global sequence attributes or slow-varying temporal phenomena that would otherwise require increasing model depth or dilation.

## 4. Applications and Empirical Performance

TFiLM has been employed extensively across generative and discriminative sequence tasks:

### a) Audio Black-Box Modeling ([2211.00497])
- GCNTF-3 (GatedConv + TFiLM, 3 blocks) outperforms plain GCN (GatedConvNet) by up to 45% in MR-STFT error and 4× lower MAE on challenging fuzz/compressor effects.
- Performance gains are due to TFiLM’s ability to model long-term dependencies; increasing channel width alone did not yield improvement.
- Block-size $B=128$ achieves optimal performance; both smaller and larger $B$ are suboptimal.

### b) Speech Bandwidth Extension ([2110.13492])
- TFiLM-equipped TUNet (2.9M params) achieves LSD=1.36, LSD-HF=2.54, SI-SDR=21.91 dB, surpassing TFiLM-UNet, WSRGlow, and NU-Wave baselines in both intrusive and non-intrusive metrics.
- MSM pretraining further improves metrics (LSD=1.28, SI-SDR=22.08).
- Multi-filter augmentation (randomized anti-aliasing filters) provides robustness to various narrowing codecs.

### c) Time-Scale Modification of Speech ([2510.02672])
- FiLM-based conditioning on continuous $\alpha$ (speed factor) enables fine-grained, artifact-robust time-stretching and compression of speech.
- STFT-HiFiGAN and WavLM-HiFiGAN variants demonstrate high perceptual scores (MOS ≈ 4.4), with FiLM layers stabilizing quality as $\alpha$ deviates from 1.0.
- Ablation shows PESQ improvement of ∼0.5–0.6 points and +0.02–0.03 in STOI across $\alpha$ due to FiLM.

### d) Text Sequence Modeling ([1909.06628])
- TFiLM-equipped CNNs outperform baseline CNNs and pure RNNs by 3–5% in document classification accuracy, particularly on long-sequence inputs.

## 5. Implementation Guidance and Hyperparameter Choices

Canonical settings for TFiLM integration include:

| Parameter                     | Typical Range/Value         | Comments                                |
|-------------------------------|-----------------------------|------------------------------------------|
| Convolutional layers          | 4–6, kernel size 3–5        | Optional dilation doubling               |
| TFiLM RNN hidden size $(d)$   | 32–256                      | One per TFiLM layer; usually $\simeq$ channels |
| Block size $(B)$ (block-wise) | 64, 128, 256                | $B=128$ optimal in audio tasks           |
| Modulation params ($\gamma$, $\beta$) | Dimension $n$ per layer      | Typically generated by linear layer from RNN output |
| Optimizer                     | Adam, lr = $10^{-3}$        | Weight decay $10^{-5}$                   |
| Regularization                | Dropout, layer norm, grad clip | Apply to both conv features and RNN     |

Initialization of $\gamma$ near identity (i.e., $b_\gamma \approx 1$) ensures stable early training. LSTM/GRU controller stability benefits from gradient clipping and optional layer normalization on the hidden state.

In block-online applications, carry TFiLM LSTM state across consecutive blocks to maintain temporal context for real-time or streaming deployment.

## 6. Calibration, Ablation, and Comparative Analysis

Experiments routinely demonstrate the superiority of TFiLM augmentation over deepening or widening convolutional layers:

- In [2211.00497], increasing channel width did not yield STFT error improvements, whereas adding TFiLM reduced error by up to 45%.
- Block-size ablation shows extreme $B$ choices degrade accuracy; intermediate values (e.g., 128) balance context and adaptability.
- Ablating scale or shift components individually shows each is necessary, but full affine modulation is optimal [1909.06628].

The use of TFiLM is particularly advantageous where sequence-level characteristics are critical and local convolutions are insufficient—e.g., audio effects with slow time constants, utterance-level speech prosody, and long-range textual semantics.

## 7. Connections to Related Concepts and Directions

TFiLM is conceptually related to other adaptive normalization and attention mechanisms:

- FiLM (Perez et al.): Conditional scale/shift for feature maps, initially for visual question answering.
- Squeeze-and-Excitation: Channel-wise gating as a function of global context (but usually static in time).
- Adaptive Instance Normalization: Feature-wise modulation guided by style or domain.

A notable distinction is TFiLM’s explicit temporal modeling, wherein modulation parameters evolve per time step or per block, often via recurrent networks or sequential encoders. Applications have spread from generic sequence modeling and audio effects to advanced conditional speech synthesis and bandwidth extension, with robust results across languages and domains.

Future work could investigate non-recurrent temporal controllers (e.g., attention-based modulator networks), hierarchical multi-scale TFiLM stacks, or extensions to multi-modal and multi-channel conditioning in cross-domain generative models.

Source: https://www.emergentmind.com/topics/temporal-feature-wise-linear-modulation-tfilm