Extensible Deviation Perceiving Loss

• The paper introduces a loss that exploits temporal inconsistencies between adjacent segments to differentiate forged from genuine video content.
• It leverages multimodal features by combining visual and audio modalities to localize partial video forgeries using only video-level annotations.
• Empirical results on the LAV-DF dataset show significant gains in localization accuracy, validating the auxiliary deviation loss in weakly supervised settings.

Searching arXiv for the cited papers and closely related work on deviation-perceiving losses and weakly supervised temporal forgery localization. Extensible Deviation Perceiving Loss is a weakly supervised objective used in multimodal temporal forgery localization to exploit temporal inconsistency as a supervisory signal. In the formulations introduced for weakly-supervised temporal forgery localization and weakly supervised multimodal temporal forgery localization, it aims to enlarge the deviation of adjacent segments of the forged samples and reduce that of genuine samples, using only video-level annotations rather than frame- or segment-level labels. Within these frameworks, the loss is paired with multimodal feature construction and video-level classification so that temporal partial forged segments can be localized through learned discontinuities in feature trajectories (Xu et al., 22 Jul 2025, Xu et al., 4 Aug 2025).

1. Concept and problem setting

The loss arises in a setting where temporal forgery localization must be learned from coarse supervision. In weakly-supervised temporal forgery localization, only video-level forge labels are available for training, so the model must infer forged intervals without explicit segment-level ground truth. The key observation used by the loss is that genuine videos tend to have smooth, consistent content over time, whereas forged videos with temporal partial forgeries often contain spliced or generated segments that break temporal continuity and therefore exhibit greater statistical deviation between adjacent segments (Xu et al., 22 Jul 2025).

Within the multimodal deviation perceiving framework, this idea is embedded in a system that combines visual and audio information. The framework is intended to identify temporal partial forged segments using only video-level annotations, and it couples a multimodal interaction mechanism with an extensible deviation perceiving loss to perceive multimodal deviation and achieve refined start and end timestamps localization of forged segments. In the multitask formulation, visual and audio modality detection are treated as two binary classification tasks, which are then integrated into a multimodal task; the same deviation-based principle is retained as an auxiliary supervision signal for localization (Xu et al., 4 Aug 2025).

A central feature of the method is that it does not directly supervise timestamps. Instead, it operationalizes a prior: unmanipulated videos are temporally smooth, while manipulated ones show temporal inconsistencies. This makes the loss particularly suited to partial video forgeries, where strong semantic cues may be weak or absent and classification alone provides no explicit incentive to learn temporal boundaries.

2. Mathematical formulation

In the multimodal deviation perceiving framework, each video is represented by a comprehensive multimodal feature sequence

$\mathcal{X} = \{x_t\}_{t=1}^T.$

A per-video temporal deviation score is computed as

$$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$

where $f(x_t, x_{t+1})$ measures deviation between adjacent segments and $\sigma(\cdot)$ is a normalization operation. A common choice is mean squared error,

$$f(x_t, x_{t+1}) = \frac{1}{d} \| x_t - x_{t+1} \|^2.$$

The extensible deviation perceiving loss is then defined as

$$\mathcal{L}_{dp} = -\frac{1}{N}\sum_{i=1}^N \left[ (1 - y_i)\log(1 - d_i) + y_i \log d_i \right],$$

with $y_i \in \{0,1\}$ indicating genuine or forged video labels. For genuine videos, the objective encourages $d_i$ to be small; for forged videos, it encourages $d_i$ to be large (Xu et al., 22 Jul 2025).

In the multitask formulation, the same principle is specialized to multimodal features

$\mathcal{F}_m = \{m_t\}_{t=1}^{T},$

with deviation computed as

$$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$0

and default $$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$1 selected as mean squared error after ablation: $$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$2 The corresponding loss is

$$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$3

where $$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$4 if the sample is forged in any modality and $$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$5 if both modalities are genuine. Possible labels are $$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$6: both genuine, $$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$7: both forged, $$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$8: visual forged only, and $$d = \sigma\left(\sum_{t=1}^{T-1} f(x_t, x_{t+1})\right),$$9: audio forged only (Xu et al., 4 Aug 2025).

Framework	Deviation score	Supervision target
MDP	$f(x_t, x_{t+1})$0	small for genuine, large for forged
WMMT	$f(x_t, x_{t+1})$1	small for genuine, large for forged

The term “extensible” refers to the choice of $f(x_t, x_{t+1})$2 and to the scope of application. In the earlier formulation, alternatives such as $f(x_t, x_{t+1})$3 or $f(x_t, x_{t+1})$4 distances are explicitly contemplated, while in the multitask formulation functions such as $f(x_t, x_{t+1})$5, cosine, and $f(x_t, x_{t+1})$6 are reported as trialed alongside MSE.

3. Architectural role in multimodal localization systems

The loss is not used in isolation. In the multimodal deviation perceiving framework, features are first produced through a multimodal interaction mechanism that introduces a temporal property preserving cross-modal attention to measure the relevance between the visual and audio modalities in the probabilistic embedding space. This mechanism identifies inter-modality deviation and constructs comprehensive video features for temporal forgery localization. The main classifier then produces a temporal activation sequence and a video-level prediction, while the deviation perceiving loss operates in parallel on the feature trajectory without any direct timestamp supervision (Xu et al., 22 Jul 2025).

In weakly supervised multimodal temporal forgery localization via multitask learning, the architectural role is broader. The multitask learning paradigm integrates visual and audio detection into a multimodal task. A Mixture-of-Experts structure is used to adaptively select appropriate features and localization head, and a feature enhancement module with temporal property preserving attention identifies the intra- and inter-modality feature deviation and constructs comprehensive video features. The full training objective is

$f(x_t, x_{t+1})$7

with $f(x_t, x_{t+1})$8 set as $f(x_t, x_{t+1})$9 by default. In this formulation, the deviation perceiving loss is auxiliary: it complements the weakly supervised classification heads and helps the localization heads focus on robust temporal cues of forgery (Xu et al., 4 Aug 2025).

This division of labor is important. Standard video-level classification loss can determine whether a video is fake, but it does not, by itself, incentivize the model to represent temporal boundaries. The deviation perceiving term supplies that missing pressure by shaping the temporal geometry of latent features.

4. Empirical behavior and ablation findings

Empirical results in both formulations support the utility of the loss for weakly supervised localization. In the multimodal deviation perceiving framework, ablation on the LAV-DF dataset shows that introducing $\sigma(\cdot)$0 to the standard video-level classification loss significantly improves both Average Precision and Average Recall for temporal forgery localization. The reported gain is $\sigma(\cdot)$1 AP and $\sigma(\cdot)$2 AR in the weakly-supervised regime. Extensive experiments further indicate comparable results to fully-supervised approaches in several evaluation metrics (Xu et al., 22 Jul 2025).

In the multitask framework, the reported effect is smaller but still consistent. On LAV-DF, adding $\sigma(\cdot)$3 improves average mAP by $\sigma(\cdot)$4 and AR by $\sigma(\cdot)$5. The ablation analysis of the deviation measure function reports that MSE performs best, though $\sigma(\cdot)$6, cosine, and $\sigma(\cdot)$7 were also evaluated. Additional analysis of the deviation objective shows that using deviation loss on multimodal features or on all combinations achieves the highest gains, indicating that both intra- and inter-modal inconsistencies are informative (Xu et al., 4 Aug 2025).

The empirical interpretation offered by these studies is consistent across frameworks. For forged videos, the model is encouraged to produce adjacent temporal segments that become more different where actual or likely forgeries exist, often near boundaries. For genuine videos, it is penalized for producing large deviations, which pushes features toward temporal smoothness. During inference, these learned discontinuities assist localization even though no explicit segment annotation was used during training.

5. Relation to broader deviation-sensitive loss design

The extensible deviation perceiving loss belongs to a broader family of objectives that prioritize departures from a reference pattern rather than treating all errors uniformly. In scientific regression for extreme events, adjusted output weighted loss and relative entropy-based loss are presented as deviation-perceiving loss functions because they amplify the influence of rare or extreme output values and can be tailored to regions of output space deemed most important (Rudy et al., 2021).

In LLM supervised fine-tuning, MinorSFT introduces a dynamic, sample-aware regularization based on the discrepancy between the optimized model and the reference model, and the discussion explicitly refers to a generalized deviation-perceiving loss that can be extended to other objectives where model drift is a concern (Xie et al., 2024). In generative zero-shot learning, GRaWD uses a graph-based random walk deviation signal to make generated unseen representations difficult to classify as seen classes, thereby treating deviation from seen distributions as the learning target (Jha et al., 2021). In sequential lossy compression, the self-adaptive perception loss function is described as deviation-perceiving because it detects departures from locally joint source-reconstruction behavior while allowing later correction rather than locking in earlier errors (Salehkalaibar et al., 15 Feb 2025).

These examples suggest that “deviation-perceiving” names a general design pattern rather than a single mathematical template. In the forgery-localization setting, the emphasized deviation is temporal discontinuity between adjacent multimodal segments. In other settings, the emphasized deviation may be rarity in output space, drift from a reference model, deviation from seen-class manifolds, or divergence from a target joint distribution.

6. Misconceptions, limitations, and open directions

A common misconception is that the loss directly localizes forgeries by supervising timestamps. In the reported formulations, it does not. It uses only video-level labels and operates on latent temporal deviation, so localization emerges indirectly from the interaction between classification, feature construction, and deviation shaping (Xu et al., 22 Jul 2025).

Another misconception is that the loss can replace the rest of the model. The multitask formulation explicitly treats it as an auxiliary loss within a larger objective, and the reported gains depend on the surrounding multitask structure, classification heads, feature enhancement, and localization heads. Likewise, the multimodal deviation perceiving framework pairs the loss with multimodal interaction and a temporal activation sequence rather than using the loss as a standalone criterion (Xu et al., 4 Aug 2025).

The principal limitation identified in the multitask analysis is the assumption of higher deviation in forgeries. Some sophisticated forgeries could minimize temporal artifacts, weakening this cue. The normalization $\sigma(\cdot)$8, the weighting $\sigma(\cdot)$9, and the function choice may also require tuning for each data scenario. At the same time, the formulation is explicitly described as extensible and versatile: it can be applied to unimodal features, multimodal features, or their combinations, and it is suggested as potentially useful beyond Deepfake localization in settings such as video anomaly detection, action segmentation, and other weakly labeled video or audio tasks where anomalies appear as local temporal inconsistency (Xu et al., 4 Aug 2025).

A broader theoretical implication is more tentative. Relative deviation learning bounds and relative deviation margin bounds study how deviation-sensitive or unbounded losses can still admit generalization guarantees under finite moment assumptions rather than boundedness assumptions (Cortes et al., 2013, Cortes et al., 2020). This suggests a possible analytical vocabulary for future study of extensible deviation perceiving losses when their induced objectives become heavy-tailed or otherwise difficult to analyze, although such guarantees are not established for the forgery-localization formulations themselves.