Synthesized Audiovisual Forgeries (SAVFs)
- Synthesized audiovisual forgeries are AI-generated or manipulated clips that include techniques like speech-driven talking-head generation, lip-sync, and face swapping.
- Researchers use diverse methodologies such as anomaly detection, pseudo-forgery generation, and generator-side probing to achieve high detection performance across benchmarks.
- Current advancements emphasize multimodal coherence, temporal localization, proactive watermarking, and identity verification to address limitations in detecting and recovering forged content.
Synthesized audiovisual forgeries (SAVFs) encompass fully AI-generated audiovisual clips and partially manipulated videos in which forged evidence may appear in the visual stream, the audio stream, or both, including audio-only forged intervals, visual-only forged intervals, and clips where both streams contain forged evidence (Le et al., 8 May 2026). In current research practice, the category spans speech-driven talking-head generation, lip-sync and retalking systems, face swap and reenactment pipelines, cloned or converted speech paired with authentic or manipulated video, and broader audio-video scene forgeries that extend beyond human-centric deepfakes (Feng et al., 2023, Shahzad et al., 26 Mar 2026, Xia et al., 26 Nov 2025). The field has consequently moved from coarse real-versus-fake classification toward multimodal consistency modeling, temporal localization, identity and authorization verification, and, in some settings, proactive recovery of authentic content (Liang et al., 13 Aug 2025, Kim et al., 17 Jul 2025, Gerstner, 20 Mar 2026).
1. Scope and evolution
Early end-to-end audiovisual speech synthesis already demonstrated the feasibility of generating synchronized speech audio and corresponding talking-face motion from text. AVTacotron2 maps a phoneme sequence to both 80-dimensional mel-scaled filterbank features and 51-dimensional blendshape coefficients, with WaveRNN reconstructing the waveform and a 3D face model producing the talking-face video; in subjective evaluation, the end-to-end and modular systems achieved mean opinion scores of 4.1 and 3.9, respectively, compared to 4.1 for ground truth generated from professionally recorded videos (Abdelaziz et al., 2020). Although this work is avatar-centric rather than a photorealistic deepfake pipeline, it shows that coherent joint audio-visual generation is an enabling technology rather than a purely downstream forensic problem.
More recent SAVF work assumes a substantially broader generator landscape. HAVIC’s HiFi-AVDF benchmark includes commercial systems such as Veo 3.1, Kling 2.5, Seedance 1.0, PixVerse V5, WAN 2.5, and Sora 2, under caption-only, caption + reference frame, and audio track + reference frame generation strategies (Peng et al., 25 Mar 2026). X-AVDT’s MMDF similarly covers talking-head generation, self-reenactment, and face swapping across GANs, diffusion models, diffusion transformers, and flow-matching systems (Kim et al., 9 Mar 2026). AVFakeBench pushes the scope further by treating audiovisual forgery as a general audio-video problem over both human-subject and general-subject scenes, including natural landscapes, animals, social activities, music performances, transportation, daily-life scenes, sports, industrial operations, alarm signals, and science scenarios (Xia et al., 26 Nov 2025).
This broadening has changed what counts as a relevant forensic signal. In earlier talking-head settings, lip-speech timing was often the dominant cue. In broader SAVF settings, the manipulated evidence can be semantic, temporal, identity-related, scene-level, or provenance-related, and it can arise from either or both modalities. A plausible implication is that no single cue class remains sufficient once forged content extends from face-centric deepfakes to general multimodal synthesis.
2. Threat models and taxonomies
Several papers propose explicit taxonomies of SAVFs, but they do so at different granularities. FakeAVCeleb-style evaluation emphasizes combinations of real and fake speech/video streams in speaking-face clips, while AVFakeBench formalizes per-modality states of Real, Edited, and Synthesized and combines them across audio and video (Feng et al., 2023, Xia et al., 26 Nov 2025). HumanSAM, by contrast, is visual-only and classifies human-centric forgeries by anomaly type rather than by modality (Liu et al., 26 Jul 2025).
| Source | Categories | Emphasis |
|---|---|---|
| FakeAVCeleb-style SAVFs (Feng et al., 2023) | RVFA, FVRA-WL, FVFA-WL, FVFA-FS, FVFA-GAN | Speaking-face audio/video combinations |
| AVFakeBench (Xia et al., 26 Nov 2025) | RA RV, EA RV, SA RV, RA EV, EA EV, RA SV, SA SV | Human and general-subject AV forgeries |
| HFV / HumanSAM (Liu et al., 26 Jul 2025) | spatial anomaly, appearance anomaly, motion anomaly, real | Visual human-centric anomaly typing |
A second taxonomy axis concerns the type of inconsistency being modeled. AVPF distinguishes inter-modality inconsistency, meaning mismatch between the visual and audio streams, from intra-modality inconsistency, meaning temporal irregularity within a single modality even when audio and video remain mutually aligned (Wei et al., 10 Apr 2026). This distinction is important because it separates lip-sync mismatch, dubbed audio, and independently manipulated modalities from jointly manipulated but temporally suspicious clips. SAVe organizes the same problem differently, via intra-modal visual inconsistencies such as blending boundaries and local mouth distortions, and cross-modal temporal inconsistencies such as lip-speech asynchrony and temporal misalignment (Shahzad et al., 26 Mar 2026).
Identity-centric taxonomies introduce yet another dimension. Referee treats audiovisual deepfake detection as a joint problem of identity integrity verification and audiovisual synchrony reasoning, using a target clip plus a real reference clip of the claimed same person (Boo et al., 31 Oct 2025). Audio Avatar Fingerprinting formalizes synthetic speech as either self-reenactment or cross-reenactment, depending on whether the target audio identity and driving identity coincide (Gerstner, 20 Mar 2026). This reframes part of SAVF analysis from “is it fake?” to “if it is synthetic, is it authorized?”
3. Detection paradigms
A major strand of SAVF detection treats forgery as abnormal audio-visual correspondence. In self-supervised video forensics by audio-visual anomaly detection, a pretrained synchronization network converts short video and audio windows into compatibility scores,
from which a normalized synchronization probability over temporal offsets is computed as
An autoregressive Transformer then models the sequence likelihood
so low-probability synchronization trajectories become anomaly scores for manipulated speech videos (Feng et al., 2023). SpeechForensics pushes this idea further into semantic audio-visual speech consistency: a pretrained AVHuBERT-style model is run separately on audio-only and visual-only inputs, and the detection score is average frame-wise cosine similarity between the resulting speech embeddings, with fixed-offset matching outperforming dynamic DTW-based alignment (Liang et al., 13 Aug 2025).
A second strand replaces real fakes during training with authentic-only pseudo-forgeries. AVPF generates Audio-Visual Pseudo-Fakes through Audio-Visual Self-Blending and Audio-Visual Self-Splicing, covering both inter-modality inconsistency and jointly synchronized but temporally out-of-context insertions (Wei et al., 10 Apr 2026). SAVe likewise trains entirely on authentic talking-head videos, using region-aware self-blended pseudo-forgeries in FaceBlend, LipBlend, and LowerFaceBlend branches, plus an AVSync branch trained by an InfoNCE-style temporal alignment objective, and combines the four branches with parameter-free average logit fusion (Shahzad et al., 26 Mar 2026). In both cases, the explicit goal is robustness to unseen manipulation methods without dependence on labeled fake corpora.
A third family emphasizes multimodal coherence rather than direct artifact detection. HAVIC decomposes authentic audiovisual structure into modality-specific structural coherence, inter-modal micro-coherence, and inter-modal macro-coherence, and pretrains on authentic LRS2 videos with masked reconstruction, fine-grained audio-visual contrastive learning, and cross-modal semantic reconstruction before adaptive aggregation for downstream classification (Peng et al., 25 Mar 2026). FauForensics introduces FAU-enhanced feature learning, combining CSN video features, ME-GraphAU facial action unit features, and Whisper audio features; its Temporal Attentional Pooler computes dense inter-modal and intra-modal similarity matrices rather than relying on clip-level matching (Wang et al., 13 May 2025). Referee adds reference-aware verification through an identity bottleneck and reference-conditioned identity matching, thereby fusing speaker/face identity consistency with temporal audiovisual reasoning (Boo et al., 31 Oct 2025).
A fourth strand probes the generative process itself. X-AVDT takes a generator-side view, using DDIM inversion through an audio-conditioned latent diffusion model to extract a video composite from the original clip, decoded noisy latent, reconstruction, and residual, and an AV cross-attention feature from generator-internal audio-visual cross-attention (Kim et al., 9 Mar 2026). This design targets fine-grained speech-motion alignment as represented inside the generator rather than only visible artifacts. By contrast, several methods are explicitly partial components in SAVF systems: MPF-Net is visual-only and detects high-fidelity AI-generated video via manifold deviation and micro-temporal fluctuation analysis (He et al., 29 Jan 2026); VideoFACT is a visual forgery detector/localizer that conditions forensic traces on scene context and reweights them with deep self-attention under compressed-video conditions (Nguyen et al., 2022); HumanSAM is visual-only and classifies human-centric forgeries into spatial, appearance, and motion anomalies (Liu et al., 26 Jul 2025).
4. Localization, recovery, and authorization
Temporal localization has become a first-class SAVF task because many manipulations are short and modality-asymmetric. The multi-modal framework for AI-generated video detection and localization aligns its final predictions to a 40 ms temporal grid, uses a sparse LMM branch, a sliding-window spatio-temporal visual branch, and a multi-scale PartialSpoof audio branch, and reports both clip-level detection and temporal localization; on AV-Deepfake1M++, it achieves AUC (Video) 96.66 and AUC (Seg) 98.23, while localization remains substantially harder at strict IoU thresholds, with [email protected] 52.89 but [email protected] 0.78 and [email protected] 0.06 (Le et al., 8 May 2026). In the anomaly-detection setting, framewise negative log-likelihood can also support localization: when short contiguous regions are altered with Wav2Lip, the supplementary localization test reports 92.0% top-5 localization accuracy (Feng et al., 2023).
Some work moves beyond detection toward recovery. Cross-modal watermarking introduces Authentic Audio Recovery (AAR) and Tamper Localization in Audio (TLA) by embedding authentic audio into visual frames before forgery. The watermarking pipeline uses invertible neural networks over DWT-transformed frames and STFT-based audio representations, then recovers the authentic audio from tampered visuals alone and localizes tampered regions by comparing recovered and tampered audio in a semantic feature space (Kim et al., 17 Jul 2025). On HDTF under MuseTalk visual manipulation, the method reports SNR 17.82, PESQ 3.18, IoU 97.02 / AP 99.89 / AUC 99.95 for audio swapping, and IoU 95.40 / AP 98.28 / AUC 98.83 for voice cloning (Kim et al., 17 Jul 2025). This is a proactive defense rather than a post hoc detector, but it directly targets the fully coordinated SAVF case in which forged audio and forged lip motion reinforce one another.
Authorization-aware analysis adds another extension beyond standard detection. Audio Avatar Fingerprinting argues that in synthetic-media ecosystems the key question may be whether synthetic audio is driven by the authorized identity rather than merely whether it is synthetic. It adapts an off-the-shelf TitaNet speaker verification model to distinguish self-reenactment from cross-reenactment in cloned speech and introduces the NVFAIR audio dataset because existing datasets do not support this task (Gerstner, 20 Mar 2026). A plausible implication is that full SAVF defense may need to distinguish among authentic media, synthetic but authorized media, and synthetic unauthorized media, especially in telepresence, conferencing, and avatar platforms.
5. Benchmarks and reported performance
Reported performance is heterogeneous because methods target different tasks: anomaly detection, binary detection, multi-class AV attribution, spatial localization, temporal localization, recovery, or explanation. The strongest reported numbers therefore need to be read within benchmark-specific protocols rather than as a single global ranking.
| Method | Setting | Reported result |
|---|---|---|
| Audio-visual anomaly detector (Feng et al., 2023) | FakeAVCeleb fake-video average | AVG-FV AUC 94.5 / AP 94.2 |
| AVPF (Wei et al., 10 Apr 2026) | VoxCeleb2 authentic-only FAVC, AV1M, AVLips | Average 94.9 AUC / 98.0 AP |
| Multi-modal detection/localization (Le et al., 8 May 2026) | AV-Deepfake1M++ | AUC (Video) 96.66; AUC (Seg) 98.23 |
| HAVIC (Peng et al., 25 Mar 2026) | HiFi-AVDF cross-dataset | 75.41 AP / 75.77 AUC |
| X-AVDT (Kim et al., 9 Mar 2026) | MMDF cross-generator | AUROC 95.29 / AP 94.03 / Acc 91.98 |
| FauForensics (Wang et al., 13 May 2025) | Cross-dataset binary evaluation | Average AUC 89.61 |
Several results are particularly indicative of field direction. AVPF shows that authentic-only pseudo-fake augmentation can outperform prior authentic-only baselines by 6.7% average AUC and 8.0% average AP in cross-dataset evaluation, and remains strong on diffusion-based TalkingHeadBench, where AVH-Align collapses to 28.5 AUC / 44.9 AP while AVPF reaches 77.8 AUC / 79.1 AP (Wei et al., 10 Apr 2026). HAVIC demonstrates that coherence-based pretraining matters for modern high-fidelity generators: on HiFi-AVDF, it improves over the strongest prior baseline AVFF by 9.39% AP and 9.37% AUC (Peng et al., 25 Mar 2026). X-AVDT shows that probing generator-internal AV cross-attention is useful for modern multimodal synthesis, with large gains over retrained baselines on MMDF and strong external transfer to FakeAVCeleb and FaceForensics++ (Kim et al., 9 Mar 2026). FauForensics is especially notable in the harder four-class audiovisual setting, where it reports an average 81.65 AUC in cross-dataset evaluation and a within-dataset average of 99.58 Acc / 99.96 AUC for RARV, FARV, RAFV, and FAFV classification (Wang et al., 13 May 2025).
Benchmark design is itself becoming a research topic. AVFakeBench spans 3,000 clips and 12,000 question-answer pairs across seven forgery types and four annotation levels, and its evaluation of 11 AV-LMMs plus two expert detectors shows both promise and weakness: the best binary overall result is 59.9 macro-F1 / 60.1 ACC from Gemini-2.5-pro, but on seven-way forgery classification the best overall result falls to 19.2 macro-F1 / 32.1 ACC, with edited forgeries particularly difficult (Xia et al., 26 Nov 2025). This suggests that binary authenticity judgment is no longer an adequate proxy for fine-grained SAVF understanding.
6. Limitations and research directions
A recurring limitation is domain restriction. Many of the strongest multimodal detectors assume a visible, cropable speaking face and usable speech audio; this is explicit for anomaly-detection, speech-representation, and multimodal localization methods, and it also follows from preprocessing pipelines built around lip ROIs, frontal talking-head crops, or AV-HuBERT-style speech representations (Feng et al., 2023, Liang et al., 13 Aug 2025, Le et al., 8 May 2026). Several methods are correspondingly weak on silent clips, profile views, heavy occlusion, poor audio, or non-speaking scenes. At the same time, broader-scene benchmarks such as AVFakeBench expose a different weakness: AV-LMMs remain vision-dominant and often much weaker on audio forgery perception than on video forgery perception (Xia et al., 26 Nov 2025).
A second limitation is that many methods still derive their main signal from mismatch, incomplete coherence, or residual generator regularity. Self-supervised anomaly detectors explicitly state that they are more sensitive to audio-visual mismatch than to a fully coherent end-to-end generated talking face (Feng et al., 2023). SpeechForensics is motivated by the insufficiency of local lip sync, yet it also remains fundamentally dependent on visible speech and may weaken if both audio and face are synthesized jointly with strong semantic and temporal consistency (Liang et al., 13 Aug 2025). SAVe acknowledges weaker coverage when synchronization remains plausible and visual artifacts are weak (Shahzad et al., 26 Mar 2026). This suggests that future systems capable of jointly synthesizing voice, lip motion, facial dynamics, and broader scene acoustics may erode mismatch-based signals unless detectors also exploit provenance, authorization, or internal generator signatures.
A third limitation is modality incompleteness. MPF-Net, HumanSAM, and VideoFACT are useful visual detectors, but all three are explicit subsystem methods rather than full SAVF analyzers (He et al., 29 Jan 2026, Liu et al., 26 Jul 2025, Nguyen et al., 2022). Conversely, audio avatar fingerprinting addresses authorization in synthetic speech but does not model the visual stream (Gerstner, 20 Mar 2026). The proactive watermarking line solves recovery and localization under a trusted pre-embedding assumption but is not applicable to arbitrary legacy media (Kim et al., 17 Jul 2025). A plausible implication is that complete SAVF defense will remain modular: visual authenticity analysis, audio forgery analysis, cross-modal coherence, authorization verification, and proactive provenance mechanisms solve different parts of the problem and are not presently interchangeable.
Finally, scalability and robustness remain open. HAVIC uses 243.1M parameters and approximately five days of pretraining on four NVIDIA L20 GPUs (Peng et al., 25 Mar 2026). X-AVDT requires DDIM inversion and reports about one minute end-to-end for a 16-frame clip under its 40-step inversion schedule (Kim et al., 9 Mar 2026). Localization quality also remains brittle at strict temporal boundaries, and edited forgeries in general-scene benchmarks remain substantially harder than obviously synthetic content (Le et al., 8 May 2026, Xia et al., 26 Nov 2025). Current evidence therefore points to an unsettled equilibrium: multimodal coherence, pseudo-fake generation, biological priors, reference-aware identity reasoning, generator-side probing, proactive watermarking, and authorization-aware audio forensics each improve part of the SAVF problem, but none yet eliminates the core challenge of robust detection and interpretation under rapidly improving, fully coordinated audiovisual synthesis.