- The paper highlights that LLMs lack transparency and auditability, producing high error rates in security-critical environments.
- It demonstrates that LLM vulnerabilities—such as data leakage and hallucinations—undermine compliance with EU security norms.
- The analysis advocates for hybrid AI solutions that combine generative and symbolic approaches to meet rigorous regulatory standards.
LLMs and Security Norms: Regulatory and Practical Challenges
Overview of LLMs in Security Contexts
The paper critically examines the deployment of LLMs within cybersecurity and infrastructure, offering a comprehensive critique of their suitability as components or assistants in the design, implementation, and lifecycle management of security-critical systems. The author delineates the structural attributes of LLM-based AI—including their reliance on vast, opaque datasets and stochastic transformer architectures—and contrasts these with more deterministic symbolic AI approaches. The assessment is contextualized by the recent European Union legal framework, notably the AI Act, Cyber Resilience Act (CRA), and NIS2 Directive.
Technical and Regulatory Deficiencies of LLM-based AI
Inherent Weaknesses in LLM-driven Security
The paper highlights multiple technical deficiencies of LLMs when tasked with supporting or automating security practices:
- Data Leakage and Black-Box Risk: LLMs are vulnerable to adversarial prompt injection, model inversion, and extraction attacks. The lack of transparency in model internals and training data violates Kerchkoff’s Principle, which prescribes security not reliant on secrecy of implementation.
- Hallucination and Reliability: Empirical studies demonstrate that LLM-supported coding tools (e.g., Copilot) are inadequate for high-assurance or security-critical code generation, exhibiting substantial error rates and generating unsafe code structures [imas_is_2022, moradi_dakhel_github_2023].
- Traceability and Auditing: The non-deterministic nature and insufficient explainability of LLM outputs hinder forensic traceability and compromise post-mortem analysis, thus failing basic security and compliance mandates.
Symbolic AI, by contrast, possesses a highly deductive structure, yielding predictable and auditable outputs, and is thus more compatible with regulatory and security best practices.
Security Engineering Norms and LLM Contradictions
The author catalogues foundational norms in security engineering—open design, best practice and state-of-the-art protocols, exhaustive component comprehension, and continuous availability—and demonstrates the systemic inability of LLMs to satisfy these criteria except in highly constrained or supervised settings. The black-box character, unpredictability, and inability to guarantee output correctness render LLMs inconsistent with security lifecycle norms (development, deployment, sunsetting), especially under adversarial conditions.
Legal and Contractual Implications
EU Regulatory Alignment
Under the CRA and NIS2 Directive, regulated entities are mandated to implement state-of-the-art cybersecurity measures, and explicitly address both design and operational vulnerabilities. The AI Act reinforces these requirements (Article 15), with additional obligations for resilience against unique machine learning or LLM-based exploits. Critically, LLM-based systems currently fail to meet the formalized precision and reliability standards enshrined in Annex I of the CRA and Article 21 of NIS2, implicating both providers and end-users in potential noncompliance and liability exposure.
Contractual Dynamics
While contractual provisions may stipulate lower technical standards in certain unregulated scenarios, multi-layered compliance—merging legal, contractual, and normative obligations—predominates in critical infrastructure and regulated sectors. The lack of concrete statutory protections against proprietary data leakage via LLMs further complicates risk allocation, necessitating the explicit preclusion or heavily supervised use of such models in certain contracts.
Practical, Economic, and Policy Consequences
The systematic deployment of LLMs in real-world security contexts portends significant practical and economic downside. The inability to verify and validate model outputs, combined with the persistent risk of introducing latent vulnerabilities during code generation or incident response automation, substantially raises the prospect of catastrophic failure and associated financial liability. The paper underscores that any cost savings or perceived efficiency gains are likely offset by regulatory sanctions, product withdrawals, and reputational damage tied to security breaches enabled or exacerbated by LLMs.
On a broader political scale, the author frames the decision to tolerate or incentivize LLM deployment in security as a societal choice—one inherently entwined with unresolved copyright and data governance dilemmas. Unlike other domains, security engineering’s culture of openness and peer review exacerbates LLM contradictions, further undermining their normative fit.
Theoretical and Future Trajectories in AI Security
The analysis projects a substantive separation of roles for inductive LLM-based AI and deductive symbolic AI within regulated security environments. LLMs may retain marginal utility in low-risk, highly supervised, or knowledge-base-driven domains, but their integration into development or deployment pipelines for critical systems is proscribed by both technical and regulatory standards. The advancement of specialized Small LLMs (SLMs) and hybrid architectures—pairing generative capabilities with symbolic reasoning and real-time human oversight—may offer incremental improvements; however, these approaches will face ongoing scrutiny and likely be circumscribed by evolving juridical and sectoral norms.
Future research is urged in empirically validating the failure rates and attack surfaces of LLM-mediated security automation, quantifying regulatory non-compliance costs, and designing robust, auditable AI support systems for secure development and operational assurance.
Conclusion
The paper delivers a rigorous technical and legal indictment of the normalization of LLM-based AI as a “security norm” within regulated domains. It concludes that, by virtue of their inherent unpredictability, lack of transparency, and misalignment with codified security engineering and regulatory standards, LLMs are fundamentally incompatible with both the theoretical and practical imperatives of modern cybersecurity. Consequently, developers, operators, and regulators should adopt secondary AI systems, augment or restrict LLM involvement, or wholly preclude their use in security-critical contexts. The evolution of both legal frameworks and technical solutions will doubtless dictate the contours of AI’s future relationship with cybersecurity.
