VisualSwap: Probing & Generative Image Editing

• VisualSwap is a framework that tests whether Vision-Language Models re-attend to updated visual input by measuring performance drops during mid-reasoning image swaps.
• It encompasses a range of applications from real-time face substitution and one-shot megapixel synthesis to video and 3D texture swapping, ensuring key attributes remain invariant.
• The approach also drives advancements in forensic detection and governance by outlining clear protocols for traceability, consent, and adversarial risk management.

Across the cited literature, “VisualSwap” appears in two closely related senses. In the narrow sense, it is an image-swap probing framework for testing whether Vision-LLMs genuinely re-examine visual input after producing reflective statements such as “Wait, let me check the figure again” (Shi et al., 15 May 2026). In a broader, inferential sense suggested by adjacent technical summaries, it functions as an umbrella label for methods that swap visual identity, texture, objects, or bodies while preserving complementary structure such as pose, motion, geometry, background, or context. Under that broader reading, the topic spans early real-time face substitution, megapixel one-shot face swapping, temporally coherent video face, head, and body swapping, arbitrary object replacement, 3D texture transfer, forensic detection of swapped content, and adversarial biometric manipulation (Nguyen et al., 2011, Zhu et al., 2021, Luo et al., 3 Jul 2025, Zhao et al., 12 Mar 2025, Gu et al., 2024, Ding et al., 2019, Narkar et al., 2024).

1. Scope, historical development, and research domains

An early antecedent is the real-time, non-intrusive face-swapper designed for infant self-recognition studies. That system combined 3D visual tracking of subject A with pose-conditioned substitution of subject B, used a generic ellipsoid head model rather than a personalized 3D face model, and achieved an end-to-end latency of approximately 99 ms (Nguyen et al., 2011). Its emphasis was not photorealistic synthesis in the modern generative sense, but accurate and instantaneous imitation under minimal calibration.

Subsequent work expanded the problem along multiple axes. Still-image face swapping and detection matured around large public datasets and transfer-learning detectors, including a public still-image swapped-face dataset with 86 celebrities and 420,053 images (Ding et al., 2019). High-resolution generation then moved to one-shot, megapixel synthesis, with MegaFS targeting $1024 \times 1024$ outputs and introducing hierarchical inversion into an extended latent space $W^{++}$ (Zhu et al., 2021). Video-centric work diversified into face swapping, head swapping, and full-body swapping, with different architectural responses to temporal coherence, identity preservation, and background consistency (Chen et al., 2024, Luo et al., 3 Jul 2025, Wang et al., 10 Dec 2025, Zhao et al., 12 Mar 2025). Parallel lines extended swapping beyond faces to arbitrary personalized objects in images and texture/material transfer onto 3D Gaussian Splatting assets (Gu et al., 2024, Cao et al., 24 Mar 2025).

A concise cross-domain view is useful.

Domain	Representative mechanism	Representative paper
Real-time face substitution	3D tracking + pose-conditioned compositing	(Nguyen et al., 2011)
Still-image face swapping and detection	ResNet-18 detection + uncertainty proxy	(Ding et al., 2019)
One-shot megapixel face swapping	HieRFE + FTM + frozen StyleGAN2	(Zhu et al., 2021)
Video face/head/body swapping	Temporal diffusion, canonicalization, or mask-free dual-canvas generation	(Chen et al., 2024, Luo et al., 3 Jul 2025, Wang et al., 10 Dec 2025, Zhao et al., 12 Mar 2025)
Arbitrary object swapping	Targeted variable swapping + appearance adaptation	(Gu et al., 2024)
3D texture swapping	Progressive multi-view diffusion + Gaussian fine-tuning	(Cao et al., 24 Mar 2025)
Visual re-examination probing	Mid-reasoning image replacement in VLMs	(Shi et al., 15 May 2026)

This breadth suggests that VisualSwap is less a single model family than a recurring structural problem: one visual factor is replaced, while another factor must remain invariant.

2. VisualSwap as a probing framework for visual re-examination

In its explicit and named form, VisualSwap is an image-swap probing framework for diagnosing whether a VLM actually re-attends to updated visual input during reflective reasoning (Shi et al., 15 May 2026). The protocol has two stages. In standard inference, the model reasons over the original image $I_a$ and question $Q$,

$R_a = M(I_a, Q).$

Immediately after the model emits a reflective trigger, the framework replaces $I_a$ with a visually similar but semantically different image $I_b$ while preserving the textual reasoning context, and generation continues as

$R_b = M(I_b, Q, R_a + P).$

The central metric is performance degradation,

$\Delta = Acc_{base} - Acc_{probe},$

where $Acc_{base}$ measures accuracy on $W^{++}$0 in isolation and $W^{++}$1 measures accuracy after the mid-reasoning swap.

The associated benchmark, VS-Bench, contains 800 image pairs, sourced equally from MathVista, MathVerse, MathVision, and MMMU-Pro. Each pair preserves question validity while changing answer-critical details. The reported average similarity statistics are CLIP 0.95, SSIM 0.86, and LPIPS 0.14, with human study confirming that the differences remain detectable. Representative examples include a geometry angle changing from $W^{++}$2 to $W^{++}$3, a bar chart minimum category flipping, synthetic-scene object counts changing via color replacement, and a function plot changing from $W^{++}$4 to $W^{++}$5 (Shi et al., 15 May 2026).

The principal empirical finding is that current VLMs overwhelmingly miss the swap. The paper reports accuracy drops of up to 60.3%, including 79.9% $W^{++}$6 19.6% for ERNIE-4.5-VL-28B-A3B-Thinking and 88.8% $W^{++}$7 34.1% for Qwen3-VL-235B-A22B-Thinking. Thinking models are reported as nearly $W^{++}$8 more vulnerable than their Instruct counterparts, and scaling does not mitigate the effect. By contrast, explicit multi-turn user instructions restore performance near baseline; for Qwen3-VL-235B-A22B-Thinking, probe accuracy rises from 34.1% to 85.4%, close to the 88.8% base score (Shi et al., 15 May 2026).

The attention analysis is central to the interpretation. Visual attention at decoding step $W^{++}$9 and layer $I_a$0 is defined as

$I_a$1

The paper reports that self-generated reflection produces negligible increases in attention to image tokens, whereas a new user turn substantially elevates it. This supports the claim that the core failure is not lack of capability but autonomous attentional control during continuous decoding. A plausible implication is that, in this setting, “VisualSwap” is less a content-editing primitive than a causal stress test for visual grounding.

3. Generative VisualSwap in faces, heads, and bodies

Modern generative variants of the problem are dominated by architectures that disentangle identity from attributes, then recompose them under explicit temporal or geometric control. MegaFS is a canonical still-image example. It formulates one-shot face swapping as generating $I_a$2 from a single source image $I_a$3 and single target image $I_a$4, preserving identity from the source and pose, expression, lighting, and background from the target. Its encoder maps an image into a hierarchical latent representation,

$I_a$5

where $I_a$6, $I_a$7, and $I_a$8. Identity transfer is then performed by the Face Transfer Module, with per-layer blending

$I_a$9

The generator is a pretrained, frozen StyleGAN2 trained on FFHQ at $Q$0, and the reported advantages are stability, quality, and reduced memory usage (Zhu et al., 2021).

MegaFS reports that the hierarchical $Q$1 representation improves both reconstruction and swapping relative to $Q$2. In reconstruction, LPIPS/MSE/failure are 0.2335/0.0563/0.65% for $Q$3 versus 0.2486/0.0672/1.28% for $Q$4. In swapping, ID similarity, pose, and expression improve from 0.5438/4.0640/1.7467 to 0.5816/3.8179/1.6489. On FaceForensics++, MegaFS attains the best ID retrieval among one-shot baselines, with 90.83 top-1 versus 90.68 for FaceShifter, while remaining competitive on pose and expression (Zhu et al., 2021).

Video formulations introduce a stricter temporal requirement. HiFiVFS addresses this by building on Stable Video Diffusion, adding a Fine-Grained Attribute Module that extracts identity-desensitized attribute features and a Detailed Identity Tokenizer that converts less-compressed recognition features into 49 tokens for cross-attention and temporal attention (Chen et al., 2024). On VFHQ-FS, HiFiVFS reports IDs 63.69, Exp 5.09, Pose 0.0396, Shape 1.2066, VIDD 0.5041, and FVD 81.12, outperforming the listed GAN-based and diffusion baselines on identity similarity, attribute preservation, and temporal stability.

CanonSwap adopts a different decomposition: it first removes motion by warping target appearance into a canonical space,

$Q$5

then injects source identity using Partial Identity Modulation with a learned soft spatial mask,

$Q$6

The refined canonical features are warped back with $Q$7 to recover the original target dynamics (Luo et al., 3 Jul 2025). On FF++, CanonSwap reports ID similarity 0.5751, ID retrieval 98.29%, pose error 0.0119, expression error 0.7328, and FID 6.21. On the VFS benchmark it reports ID retrieval 99.78%, gaze 0.0928, LSE-D 7.938, LSE-C 6.053, TC 0.513, and FVD 125.30, with ablations showing degradation when canonicalization, masking, or refinement is removed.

DirectSwap removes the regional inpainting mask entirely and instead trains a video diffusion model on paired cross-identity sequences. Its dual-canvas conditioning arranges driving-video and reference-image latents as

$Q$8

with a binary domain indicator $Q$9 that marks motion and identity halves but does not freeze updates (Wang et al., 10 Dec 2025). Motion- and Expression-Aware Reconstruction defines a fused weight map

$R_a = M(I_a, Q).$0

where $R_a = M(I_a, Q).$1 comes from frame-difference magnitudes and $R_a = M(I_a, Q).$2 from landmark proximity. On HeadSwapBench, DirectSwap reports Sim_ID 0.880, Pose 1.859, Expr 0.075, SSIM 0.738, PSNR 21.266, LPIPS 0.129, FID 19.376, and tLPIPS 0.012, outperforming the reported baselines.

Full-body substitution is treated by SwapAnyone as diffusion-based video inpainting with reference fidelity and motion control. It uses a nine-channel inpainting input, temporal layers after spatial attention blocks, a ReferenceNet-style identity extractor, DWpose conditioning, and an EnvHarmony curriculum that randomly perturbs brightness with $R_a = M(I_a, Q).$3 so that luminance is inherited from the target rather than the reference (Zhao et al., 12 Mar 2025). On the reported comparison, SwapAnyone reaches CLIP-Sim 0.9529, OKS 0.86, MSE-B 7.51, and FVD 121.94.

Older engineering variants remain instructive because they isolate different failure modes. The 2022 FaceShifter-derived technical report adds an eye-focused loss, a super-resolution block, and Gaussian-based adaptive mask generation to reduce video jitter, increase output resolution, and mitigate border artifacts when source and target face proportions differ (Chesakov et al., 2022). The persistent design pattern across all of these systems is explicit control over what must change and what must remain invariant.

4. Arbitrary object replacement and 3D texture transfer

VisualSwap-style editing is not restricted to human identity. SwapAnything generalizes the task to arbitrary personalized object swapping in images while preserving surrounding context. Its core mechanism is targeted variable swapping over multiple internal diffusion variables, including latent features $R_a = M(I_a, Q).$4, cross-attention maps $R_a = M(I_a, Q).$5, self-attention maps $R_a = M(I_a, Q).$6, and self-attention outputs $R_a = M(I_a, Q).$7. The paper gives a compact masked blending rule,

$R_a = M(I_a, Q).$8

together with location, style, scale/shape, and content adaptation modules (Gu et al., 2024). Style adaptation uses Masked AdaIN, scale/shape adaptation augments classifier-free guidance with a shape term, and content adaptation smooths masks spatially and temporally. The reported swapping schedule is 30 steps for $R_a = M(I_a, Q).$9, 20 for cross-attention maps, 25 for self-attention maps, and 10 for self-attention outputs, across all U-Net layers. On automatic evaluation, SwapAnything reports DINO_fore 0.61, CLIP_fore 0.79, DINO_back 0.79, and CLIP_back 0.89, all listed as best among the compared methods.

The 3D counterpart is 3DSwapping, which transfers the texture or material appearance from a single 2D reference image onto a 3D Gaussian Splatting representation while maintaining multi-view consistency (Cao et al., 24 Mar 2025). The method edits views progressively, beginning from a selected reference view $I_a$0, then expanding to adjacent views in pose order. Multi-view conditioning is implemented by weighted fused cross-attention,

$I_a$1

The guided noise predictor augments text fidelity, view-consistency, and prompt-tuned texture guidance:

$I_a$2

The learned auxiliary token $I_a$3 is optimized from the CLIP-space difference between the unedited render and the selected reference, then used as guidance during denoising.

The reported results position 3DSwapping as a high-fidelity multi-view method: CLIP score 0.9333, LPIPS (Alex) 0.1166, LPIPS (VGG) 0.1247, Vision-GPT 76, and user study 4.54, all listed as best in the summary (Cao et al., 24 Mar 2025). The limitations are also structurally distinct from 2D swapping: large shape mismatch between the reference and the 3D object can destabilize editing, and undertrained 3D Gaussians or poor segmentation can lead to artifacts or model collapse. This suggests that 3D VisualSwap problems are dominated less by local blend seams than by cross-view consistency and geometry-conditioned texture transport.

5. Detection, forensics, and adversarial security

A complete account of VisualSwap must include detection and adversarial use. The still-image swapped-face detector of “Swapped Face Detection using Deep Learning and Subjective Assessment” defines swapped faces as images in which one person’s face from a source image is transferred into the target image of another person while maintaining photo-realism. The paper constructs a public dataset of 420,053 images across 86 celebrities, with 156,930 real faces and 263,123 swapped faces, generated using both a landmarking/3D-modeling/segmentation/blending pipeline and an AE-GAN method (Ding et al., 2019). Detection is performed by a ResNet-18 pretrained on ImageNet, fine-tuned for binary classification.

A distinctive contribution is the use of raw logit differences as a simple uncertainty proxy, intended as a practical alternative to Bayesian deep learning. On the full dataset, the reported performance is 96.52% true positive, 0.60% false positive, and 97.19% accuracy for Nirkin’s method, and 99.86% true positive, 0.08% false positive, and 99.88% accuracy for AE-GAN. On a manually selected 200-per-method subset, the detector reaches 98.00% accuracy for Nirkin and 100.00% for AE-GAN. Human ranking experiments were collected through 36,112 pairwise comparisons from more than 90 registered evaluators, using Hamming-LUCB approximate ranking, and yielded good but imperfect correspondence with model confidence: for Nirkin, linear correlation 0.7896 and Spearman 0.7579; for AE-GAN, linear correlation 0.8332 and Spearman 0.7576, all with $I_a$4 (Ding et al., 2019). Grad-CAM indicates that the model predominantly focuses on central facial areas, especially the nose and eyes.

The adversarial side is represented by IrisSwap, which replaces iris texture in real eye images or streams while preserving live oculomotor cues. The attack is explicitly positioned as a digital manipulation that can evade liveness detectors because the eye remains live even though the iris pattern is replaced (Narkar et al., 2024). The headline result is that both offline and online attacks deceive current state-of-the-art defense models at rates up to 58%. The security implication is precise: liveness systems that are effective against physical presentation attacks do not necessarily defend against digital visual swapping in which the manipulated region is biometric identity itself.

Taken together, these papers establish VisualSwap as a dual-use domain. The same structural operations that support benign editing and animation also create new forensic burdens and new attack surfaces.

6. Evaluation regimes, recurrent failure modes, and governance

Evaluation in VisualSwap research is strongly task-dependent. VLM probing uses $I_a$5, $I_a$6, and $I_a$7; still-image forensics emphasizes true positive, false positive, and accuracy; megapixel and image swapping use LPIPS, MSE, identity similarity, pose, expression, and FID; video swapping adds FVD, temporal consistency, tLPIPS, VIDD, lip-sync measures such as LSE-D and LSE-C, gaze error, Eye Aspect Ratio, and retrieval-style identity scores; arbitrary object and texture swapping incorporate CLIP, DINO, SSIM, user studies, and Vision-GPT (Shi et al., 15 May 2026, Ding et al., 2019, Zhu et al., 2021, Luo et al., 3 Jul 2025, Chen et al., 2024, Wang et al., 10 Dec 2025, Zhao et al., 12 Mar 2025, Gu et al., 2024, Cao et al., 24 Mar 2025). A plausible implication is that “quality” in VisualSwap is never a single scalar: identity fidelity, attribute preservation, temporal coherence, context preservation, and semantic correctness are often in tension.

The reported failure modes are similarly heterogeneous but recurrent in structure. MegaFS notes sensitivity to extreme poses, strong occlusions, unusual lighting, accessories, and bias induced by an FFHQ-trained StyleGAN2, including a tendency to generate smiles (Zhu et al., 2021). CanonSwap identifies blending-border artifacts, extreme occlusions, and very fast motion as persistent challenges, especially when motion extraction or warping degrades (Luo et al., 3 Jul 2025). 3DSwapping is sensitive to large geometry mismatch, undertrained 3D Gaussians, poor segmentation, extreme specularities, and large pose gaps (Cao et al., 24 Mar 2025). The still-image detector notes that logit margins do not explicitly account for domain shift, and that human errors cluster near ambiguous middle-ranked examples (Ding et al., 2019). VisualSwap probing of VLMs isolates a different failure mode—textual inertia—where self-generated reflection does not re-engage visual tokens even when multi-turn user instructions can (Shi et al., 15 May 2026).

Governance recommendations are unusually consistent across otherwise disjoint subfields. CanonSwap recommends consent, watermarking, provenance metadata, disclosure, and the use of deepfake detection tools where required (Luo et al., 3 Jul 2025). SwapAnyone recommends consent, watermarking, and detection under a responsible-use regime (Zhao et al., 12 Mar 2025). MegaFS explicitly releases a megapixel face-swapping database to support DeepFake detection research (Zhu et al., 2021). The VLM probing work recommends explicit multi-turn re-examination steps for safety-critical visual reasoning rather than reliance on self-reflective text (Shi et al., 15 May 2026). These recommendations do not eliminate the dual-use problem, but they establish a clear technical norm: swapping systems should be paired with detection, traceability, and interaction protocols that make the boundary between edited and unedited content operationally observable.