Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

Abstract: AI-powered Applications (AI-Apps), hosted on platforms such as Hugging Face, are democratizing access to pre-trained models through online inference and fine-tuning services. While lowering AI adoption barriers, these platforms introduce an unexplored attack surface, as AI-Apps are often developed by untrusted parties with weak isolation and misconfigured security settings. In this paper, we present the first systematic security analysis of AI-Apps across three leading platforms. To structure our investigation, we map the AI-App lifecycle to established risk taxonomies (e.g., OWASP), identifying five threat categories and ten attack vectors ranging from generic web flaws to high-impact architectural issues. Our analysis reveals critical failures including broken access control, insecure resource reuse, insufficient input validation, and sensitive data exposure. Notably, we uncover three novel architectural vulnerabilities inherent to platform design and demonstrate how traditional issues (e.g., world-readable logs) are uniquely amplified in this ecosystem. To assess real-world impact, we develop an analysis framework Insightor and apply it to over 970,000 public AI-Apps. Alarmingly, we find thousands of apps leaking credentials, hundreds containing input injection vulnerabilities that allow arbitrary code execution, and tens harboring embedded backdoors -- indicating active exploitation. We have responsibly disclosed all findings to the affected platforms and developers.

• The paper presents the first systematic security analysis of the AI-App ecosystem, revealing structural vulnerabilities and novel attack vectors.
• It employs large-scale empirical measurements over 972,546 samples to quantify risks such as broken access control, identifier reuse, and input injection.
• The study offers actionable recommendations for enhancing platform architectures, including token revocation, log masking, and strict resource isolation.

Systematic Security Risks in AI-Powered Application Platforms

Overview of the AI-Apps Ecosystem

The proliferation of AI-powered Applications (AI-Apps) on pre-trained model hubs such as Hugging Face, Replicate, and ModelScope has enabled accessible, scalable deployment and fine-tuning of large pretrained models. These platforms facilitate rapid development and democratize AI adoption by abstracting infrastructure complexities for both inference and training workloads. However, this ecosystem introduces a novel attack surface: AI-Apps are often developed by third parties with variable security expertise, subject to weak environment isolation and problematic defaults. The study presents the first systematic security analysis of this AI-App landscape, characterizing the technical architecture, lifecycle, and unique threat vectors.

Figure 1: The overall architecture of AI-App platforms, indicating key actors, resource sharing points, and exposure boundaries.

A key architectural insight is that AI-Apps are essentially user-managed web servers orchestrated by the platform, frequently embedded via iframes into platform web pages. They interface with users via browser GUIs and APIs and are deployed either as source-code-based containers (Hugging Face, ModelScope) or pre-built images (Replicate). Critical resources such as logs, credentials (“secrets”), and user data are managed by platform abstractions, but are frequently exposed via misconfigurations and propagation mechanisms. The survey identifies five broad threat categories and ten attack vectors, ranging from classic web vulnerabilities (broken access control, injection) to platform-induced architectural flaws and supply-chain propagation channels.

Taxonomy of Threats and Attack Vectors

The study demonstrates that the AI-App ecosystem is vulnerable to both generic web attacks and uniquely amplified threats stemming from its composite design. Comprehensive mapping to the OWASP Top 10 and supply chain attack frameworks is provided, highlighting the dual nature of these risks.

Architectural and Authentication Flaws

One critical finding is widespread broken access control. Hugging Face, relying on stateless JWT authentication, fails to properly revoke or scope tokens when resources are deleted or transferred—permitting “Ghost Token Attacks” where attackers retain access after identifier recycling. Similarly, over-privileged iframe usage exposes browsers to excessive permissions (e.g., camera, microphone), creating persistent lateral movement vectors across embedded AI-Apps.

Authentication mechanisms are also compromised by developer mishandling: input “password protection” via hard-coded credentials is trivially bypassed, exposing administration and user data in multiple confirmed AI-Apps.

Figure 2: Authentication workflow of AI-Apps on Hugging Face exposes the stateless token propagation and points of failure for token invalidation.

Resource Reuse and Propagation Risks

A novel “Identifier Reuse Attack” is enabled by the subdomain-mapping policy of Hugging Face: AI-App identifiers are mapped to subdomains in a way that allows newly created accounts to reclaim domains abandoned by previously deleted AI-Apps, including those embedded across the web. This creates a persistent supply-chain attack: pre-existing iframes (in trusted third-party websites) can silently begin serving malicious content controlled by the attacker.

Figure 3: Identifier Reuse Attack—attacker reclaims orphaned subdomain by exploiting inconsistent identifier mapping, injecting malicious apps into existing trusted iframes.

Propagation of malicious or backdoored code via duplication features or automated “fork” workflows (as in training-AI-App creation) is another vector. The study traces multiple instances of AI-Apps replicating backdoors throughout the ecosystem, frequently without developer awareness.

Input Injection and Log/Data Leakage

Standard input validation failures pervade the ecosystem. Both direct (developer) input injection and indirect (outdated third-party SDKs, e.g. Gradio) lead to numerous AI-Apps with confirmed arbitrary code execution vulnerabilities. The dataflow analysis uncovers 1,442 candidate injection points, 83% of samples confirmed exploitable.

Additionally, pervasive misconfiguration of logging (“log exfiltration attack”) results in substantial leakage of developer secrets (936 confirmed cases), user-submitted prompts (including medical and business data), and tokens—including those required for sensitive operations. Logs are typically world-readable for public AI-Apps on Hugging Face and Replicate, despite interface controls purportedly restricting access.

Figure 4: Web page generation and input processing workflows, where improper validation or sanitization amplifies exposure in auto-generated interfaces (e.g., Replicate).

Empirical Measurement and Impact

The presented Insightor analysis framework enables scalable, ecosystem-wide measurement. Collecting and analyzing 972,546 AI-Apps across Hugging Face, Replicate, and ModelScope, the study reveals:

• 936 AI-Apps leaking credentials, including cases where developers followed official guidelines for secret management;
• 1,442 AI-Apps with input injection vulnerabilities, with exploitation allowing arbitrary code execution, data exfiltration, or persistent installation of crypto miners and backdoors;
• 27 AI-Apps confirmed with embedded backdoors, some propagating via cloning for over a year;
• 139,475 AI-Apps using outdated and vulnerable SDKs (e.g., Gradio) with public RCE exploits;
• Significant propagation of identical tokens across multiple AI-Apps and images due to poor base image hygiene and duplication practices;
• Emergence of cryptojacking as a practical threat, with 43 AI-Apps identified as actively mining cryptocurrency using user-funded GPU resources or during platform build stages.

The screen-and-manual verification approach confirms that high-risk clusters are efficiently isolated for review, and that recall is robust for critical classes of vulnerability.

Implications for Platform Design and the AI Supply Chain

The findings demonstrate that many security issues in the AI-App ecosystem are not simply due to isolated developer negligence but are artifacts of platform design and policy. Risks are amplified by default deployment recipes, incentivized resource sharing, and weak or missing vetting/monitoring practices on the part of the platforms.

The practical implication is that current AI-App platforms, while fast-maturing, have not inherited the hardened defaults and review cycles of legacy web application platforms or software distribution channels. The threat of supply-chain compromise, credential exfiltration, and persistent backdoor propagation is systemic rather than incidental.

Theoretically, this study informs the design of future AI workflows and platforms by highlighting the need for namespacing policies that prevent identifier collisions and domain takeovers, robust log redaction and privacy-by-default, mandatory dependency scanning and provenance tracking, and runtime isolation at multi-tenancy boundaries.

Recommendations and Future Directions

For platforms, critical mitigations include robust token revocation, log masking, principle-of-least-privilege on permissions, and tight restrictions on resource (identifier) reuse. Automated vetting of containers, source code, and SDK versions must be instituted; third-party developer reliance cannot substitute for platform-level controls. Model registry systems should incorporate supply-chain risk scoring procedures analogous to those deployed in established software ecosystems.

For AI-App developers, source code and container image scanning, careful management of secrets, disabling default logging of sensitive request parameters, and explicit sanitization of all user input are non-negotiable practices.

Future work will likely expand into automation of information flow tracking in heterogeneous AI-App landscapes, formalization of supply chain security metrics for model hubs, and integration of sandboxing/privilege separation mechanisms for AI inference and training applications. As AI-Apps grow to encompass fine-tuning, retrieval-augmentation, and multimodal workflows, the attack surface and potential for systemic compromise will only increase.

Conclusion

This study establishes the critical, systematic security risks inherent to AI-Apps served on public model hubs, identifies several new architectural attack vectors, and quantifies their prevalence in real deployments. Lax authentication management, insecure resource reuse, insufficient input validation, poor logging hygiene, and unchecked code propagation together create a threat ecosystem significantly broader than in traditional ML deployment. Addressing these issues will require joint effort between platform designers, security researchers, and the broader AI community to ensure safe, trustworthy, and scalable AI service delivery.

---

Reference: "Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs" (2606.30373)