Dual-Use AI Face Swap Apps Are Mostly Unsafe: A Systematic Safety Audit

Abstract: AI-based image editing tools, such as face swapping algorithms, can be used to transform a clothed image of a person into a sexually explicit image of that person. These tools are made easily accessible to non-expert users through mobile apps, and have been linked to reports of image-based sexual abuse and cyberbullying involving synthetic non-consensual intimate imagery. Apple and Google have begun to remove "nudification" apps from their platforms: apps that are marketed with the capability to "undress", "nudify", or create nude face swaps from images of people. However, AI image editing apps that have the same underlying capabilities, but do not present as nudification apps could be also abused to create non-consensual explicit images. In this paper, we investigate whether AI face swap apps for iOS and Android implement safety measures to prevent the creation of SNCII. We identified and downloaded 420 face swap apps, and manually tested 155 eligible apps to see whether they would permit the user to create face swaps with nude images. Our evaluation shows that 70% of apps with face swap functionality have no technical safeguards against generation of nude images. Additionally, we investigated whether face swap apps' descriptions, terms of service, or privacy policies addressed harmful uses of the app, finding that no apps self-describe as nudification apps, but that the majority do not have specific terms of service provisions prohibiting this kind of use. Our findings suggest that to mitigate the threat of UI-bound SNCII threats, platforms and lawmakers must implement policies to mandate safety filters in dual-use AI image editing applications like face swap apps.