Temporal Difference Attention Module (TDAM)

• Temporal Difference Attention Module (TDAM) is a domain-dependent attention mechanism that leverages temporal changes in sequential data to highlight subtle dynamic discrepancies.
• It replaces reliance on explicit boundary annotations by capturing irregular frame-to-frame variations, boosting tasks such as partial deepfake speech detection and rPPG measurement.
• Variants of TDAM demonstrate versatility across applications—from dynamic attention in video captioning to deformable aggregation in MRI segmentation—tailoring techniques to specific signal challenges.

Searching arXiv for papers using or defining TDAM variants to ground the article in the literature. Temporal Difference Attention Module (TDAM) most specifically denotes a frame-level temporal difference learning mechanism for partial deepfake speech detection that identifies unnatural temporal variations without relying on explicit boundary annotations (Li et al., 20 Jul 2025). In adjacent arXiv literature, closely related temporal-difference-guided attention appears inside PhysFormer for facial video-based physiological measurement (Yu et al., 2021). The acronym, however, is not unique: it also denotes the "Text-based Dynamic Attention Model" for video captioning (Xiao et al., 2019) and the "Temporal Deformable Aggregation Module" in 3D cardiac MRI video segmentation (Dong et al., 2020). Accordingly, TDAM is best understood as a domain-dependent label whose meaning must be read from the surrounding architecture, task, and mathematical formulation.

1. Terminological scope and disambiguation

The cited literature uses the acronym TDAM for distinct modules with different expansions, objectives, and inductive biases. That heterogeneity is central to the term’s interpretation.

Paper	Expansion of TDAM	Primary role
(Xiao et al., 2019)	Text-based Dynamic Attention Model	Video captioning model with dynamic attention over generated words
(Dong et al., 2020)	Temporal Deformable Aggregation Module	Front-end temporal fusion module in DeU-Net
(Li et al., 20 Jul 2025)	Temporal Difference Attention Module	Frame-level temporal difference learning for partial deepfake speech detection
(Yu et al., 2021)	Temporal Difference Attention Module / temporal difference guided attention	Attention mechanism inside TD-MHSA in PhysFormer

In the partial deepfake setting, TDAM is explicitly framed as a mechanism for detecting unnatural temporal dynamics in speech, rather than boundary artifacts, and is paired with adaptive average pooling in the TDAM-AvgPool model (Li et al., 20 Jul 2025). In PhysFormer, the temporal-difference component is embedded inside Temporal Difference Multi-Head Self-Attention, where temporal difference convolution is used to construct query and key representations specialized for quasi-periodic rPPG dynamics (Yu et al., 2021).

By contrast, the captioning paper uses TDAM as the name of an entire video captioning model whose core innovation is dynamic attention over previously generated words, not temporal differences (Xiao et al., 2019). The cardiac MRI segmentation paper uses TDAM for deformable temporal aggregation of adjacent frames, again without the temporal-difference attention formulation of the speech and rPPG papers (Dong et al., 2020).

2. Problem setting and design motivation

In partial deepfake speech detection, the motivation for TDAM is a shift away from boundary-centric reasoning. Earlier methods are described as depending on costly frame-level annotations and as focusing on transition artifacts between bonafide and deepfake segments. The 2025 work argues that modern generation methods increasingly smooth these transitions, making boundary artifacts harder to detect. Its alternative premise is that deepfake speech exhibits erratic directional changes and unnatural local transitions compared to bonafide speech; TDAM therefore redefines the task as detection of unnatural temporal variations using utterance-level labels only (Li et al., 20 Jul 2025).

The central empirical intuition in that setting is directional temporal evolution. Given frame-wise SSL embeddings from wav2vec2-XLSR, bona fide speech is reported to have a more consistent directional flow, while deepfakes show lower mean cosine similarity and higher variance between adjacent direction vectors. This makes temporal inconsistency, not splicing location, the discriminative signal (Li et al., 20 Jul 2025).

In PhysFormer, the motivation is different but structurally related. Remote photoplethysmography requires recovery of extremely subtle, quasi-periodic physiological cues from facial video while suppressing motion, illumination, compression, and other interference. The paper argues that vanilla self-attention is too generic for this setting and that local temporal differences are more informative than raw appearance for query-key similarity. Temporal difference guided global attention is therefore introduced to enhance quasi-periodic rPPG features and to refine local spatio-temporal representation against interference (Yu et al., 2021).

Taken together, these works suggest a shared design premise: when the target signal is subtle, nonstationary, or easily masked, attention driven by temporal variation can be more task-aligned than attention driven only by static token similarity. This is an interpretation of the common pattern rather than a claim made verbatim in any single paper.

3. Mathematical formulation and internal mechanics

In the partial deepfake model, frame embeddings are denoted

$\mathbf{x}_1,\mathbf{x}_2,\ldots,\mathbf{x}_T.$

The paper defines normalized frame-to-frame direction vectors as

$$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$

and then measures directional smoothness through cosine similarity between adjacent direction vectors. After adaptive average pooling and refinement, the model forms an embedding

$\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$

A 1D convolution adds local context, and the directional difference map is computed as

$$\boldsymbol{M}[t]=\boldsymbol{E}^{\text{conv}}[t+1]-\boldsymbol{E}[t], \quad t=1,\dots,T'-1.$$

A zero vector is appended to preserve shape consistency. TDAM then constructs a dual-level hierarchical representation: a fine-scale map $\boldsymbol{X}_1$ for local irregularities and a coarse-scale map $\boldsymbol{X}_2$ for broader temporal deviations. The attention-weighted output is

$$\boldsymbol{Y}=\sigma\!\left[l * (\boldsymbol{X}_1+\boldsymbol{X}_2)\right]\odot \boldsymbol{E},$$

where the $1 \times 1$ convolution fuses channel information, the sigmoid produces a soft attention map, and the element-wise multiplication reweights original embeddings so that frames with suspicious temporal behavior are emphasized (Li et al., 20 Jul 2025).

Adaptive average pooling is part of the same formulation for variable-length utterances. If the SSL extractor outputs

$\boldsymbol{F}\in\mathbb{R}^{T \times D},$

the timeline is partitioned into $T'$ segments and the pooled feature for segment $$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$0 is

$$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$1

The paper attributes to this step the preservation of essential temporal structure while avoiding truncation and the distortions of naive padding or concatenation (Li et al., 20 Jul 2025).

PhysFormer uses a different construction. Let the input facial video be

$$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$2

After a shallow stem and tube tokenization, temporal difference convolution is used for the query and key projections in TD-MHSA. The temporal difference convolution is given as

$$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$3

where $$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$4 controls the contribution of the temporal difference term. The paper defines

$$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$5

while $$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$6 is obtained by point-wise linear projection without batch normalization. For each attention head,

$$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$7

and the multi-head output is

$$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$8

The smaller temperature $$\Delta \mathbf{x}_t=\frac{\mathbf{x}_{t+1}-\mathbf{x}_t}{\|\mathbf{x}_{t+1}-\mathbf{x}_t\|}, \quad t=1,\ldots,T-1,$$9 is explicitly motivated by the need for sparser attention in rPPG modeling (Yu et al., 2021).

4. Integration into end-to-end systems and supervision

In the partial deepfake pipeline, TDAM is embedded in a larger system consisting of wav2vec2-XLSR feature extraction, adaptive average pooling, a small feed-forward plus ResNet refinement stage, TDAM, and frame-level scoring whose outputs are averaged to obtain the utterance-level decision. The model uses an embedding size of 1024 at the front-end, extracts frames every 20 ms, reduces the representation to 64 dimensions via two linear layers, applies dropout of 0.2, sets the fixed-length target $\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$0 to 4 seconds based on dataset statistics, optimizes weighted cross-entropy with bonafide weight 9 and partial deepfake weight 1, and is fine-tuned end-to-end with Adam using learning rate $\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$1, weight decay $\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$2, batch size 2, and 10 epochs (Li et al., 20 Jul 2025).

A defining property of that system is the absence of frame-level supervision. Frame-level predictions are produced, but training requires only utterance-level labels. The paper identifies this as a major practical advantage over methods that require segment boundaries or explicit frame annotations (Li et al., 20 Jul 2025).

In PhysFormer, TDAM is not standalone; it is the attention mechanism inside the transformer backbone. The full architecture comprises a shallow stem of three convolutional blocks, tube tokenization into non-overlapping spatio-temporal tokens, several temporal difference transformer blocks containing TD-MHSA and ST-FF, and a prediction head that temporally upsamples, spatially averages, and projects to a 1D rPPG signal. The paper further introduces label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain to provide elaborate supervisions and alleviate overfitting, while emphasizing that the model can be trained from scratch on rPPG datasets rather than requiring large-scale pretraining (Yu et al., 2021).

5. Empirical performance and ablation evidence

For partial deepfake speech detection, TDAM-AvgPool is reported to achieve state-of-the-art performance. On PartialSpoof, it records 0.19% EER on the development set and 0.59% EER on the evaluation set. On the HAD dataset, it achieves 0.03% EER and 99.99% AUC. The abstract further states that the method significantly outperforms existing methods without requiring frame-level supervision. The paper’s ablations report that removing TDAM, removing the dual-level representation, removing directional information, or replacing adaptive average pooling with trim-pad or max pooling all degrade performance (Li et al., 20 Jul 2025).

For rPPG, the clearest isolation of temporal-difference-guided attention comes from the PhysFormer ablation on TD-MHSA and ST-FF. With ST-FF only and no MHSA, the reported RMSE is 9.81. Replacing TD-MHSA with vanilla MHSA plus ST-FF yields 10.43. Using TD-MHSA plus ST-FF with $\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$3 gives 9.51, while TD-MHSA plus ST-FF with $\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$4 gives 7.56. The paper also reports good performance around $\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$5 and $\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$6, and presents qualitative attention visualizations focusing on skin regions such as forehead and cheeks and aligning with peak positions in the rPPG signal (Yu et al., 2021).

These results support two separate but related claims. First, temporal-difference-aware attention can outperform generic attention in settings where the signal is expressed through fine temporal dynamics rather than static appearance. Second, the exact form of temporal-difference guidance is highly task-specific: directional frame differences in speech and temporal-difference convolution in facial video are not interchangeable mechanisms.

6. Other TDAM usages and recurrent misconceptions

A common source of confusion is the assumption that TDAM names a standardized module. The cited literature does not support that interpretation. In video captioning, TDAM stands for Text-based Dynamic Attention Model, and its key idea is dynamic attention over all previously generated words at each decoding step. The attention output is

$\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$7

with

$\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$8

and

$\boldsymbol{E}\in\mathbb{R}^{T' \times D}.$9

The model uses two LSTM paths—LSTM1 for the last generated word and LSTM2 for the weighted combination of all previous words—and blends them through

$$\boldsymbol{M}[t]=\boldsymbol{E}^{\text{conv}}[t+1]-\boldsymbol{E}[t], \quad t=1,\dots,T'-1.$$0

The paper links this text attention to visual attention and trains the model in two stages, "starting from scratch" and "checking for gaps." On MSVD, the best variant TDAM (M-S) achieves 36.1 METEOR, 54.0 BLEU-4, 85.8 CIDEr, and 72.3 ROUGE; on MSR-VTT, TDAM (M-S) obtains 28.7 METEOR, 44.7 BLEU-4, 48.9 CIDEr, and 62.3 ROUGE (Xiao et al., 2019).

In cardiac MRI video segmentation, TDAM stands for Temporal Deformable Aggregation Module and functions as the front-end temporal fusion component of DeU-Net. It takes a clip

$$\boldsymbol{M}[t]=\boldsymbol{E}^{\text{conv}}[t+1]-\boldsymbol{E}[t], \quad t=1,\dots,T'-1.$$1

uses a U-Net-based offset prediction network to predict deformable offsets

$$\boldsymbol{M}[t]=\boldsymbol{E}^{\text{conv}}[t+1]-\boldsymbol{E}[t], \quad t=1,\dots,T'-1.$$2

updates sampling positions as

$$\boldsymbol{M}[t]=\boldsymbol{E}^{\text{conv}}[t+1]-\boldsymbol{E}[t], \quad t=1,\dots,T'-1.$$3

and applies temporal aggregation deformable convolution to produce fused feature maps for the DGPA network. In experiments, $$\boldsymbol{M}[t]=\boldsymbol{E}^{\text{conv}}[t+1]-\boldsymbol{E}[t], \quad t=1,\dots,T'-1.$$4, so the module uses 3 frames, and the deformable kernel size shown in the figure is $$\boldsymbol{M}[t]=\boldsymbol{E}^{\text{conv}}[t+1]-\boldsymbol{E}[t], \quad t=1,\dots,T'-1.$$5. Ablation results report average ASSD of 0.27 for ToFlow U-Net, 0.24 for DeU-Net(t), 0.22 for DeU-Net(d), and 0.19 for full DeU-Net; average HD of 11.19, 9.28, 7.69, and 6.80; and average Dice of 0.87, 0.86, 0.88, and 0.90, respectively (Dong et al., 2020).

A second misconception concerns partial deepfake detection specifically: the 2025 TDAM paper explicitly argues against the assumption that manipulated regions must be detected through visible boundaries. Its claim is that advanced smoothing attacks weaken boundary cues, making temporal inconsistency a more robust target (Li et al., 20 Jul 2025). In an analogous way, PhysFormer argues that vanilla self-attention is insufficiently specialized for rPPG and that temporal-difference-guided attention with a smaller temperature is more appropriate for quasi-periodic physiological dynamics (Yu et al., 2021).

A plausible implication is that the stable concept across the genuinely temporal-difference uses of TDAM is not a single architecture but a modeling stance: salient signals are sought in how representations change over time, and attention is conditioned on those changes rather than on static content alone.