Cryptographic Capabilities of LLMs

Updated 9 December 2025

Cryptographic Capabilities of LLMs are defined by using language models to perform tasks such as decryption, steganography, and protocol verification, as evaluated by benchmarks like AICrypto and CipherBank.
Evaluations demonstrate that LLMs excel in factual recall and formal proof construction for classical ciphers while showing significant limitations with high-entropy modern cryptosystems.
LLM-enabled cryptography presents dual-use potential by aiding in automated vulnerability detection and covert communications while posing risks of semantic errors and adversarial exploitation.

LLMs have become central not only in natural language understanding and generation, but also as computational agents in domains historically requiring algorithmic rigor, such as cryptography. The cryptographic capabilities of LLMs now span factual recall, automated secret-code breaking, protocol vulnerability analysis, steganography, and binary reverse engineering. Systematic evaluations across recent benchmarks reveal both striking competencies and fundamental shortcomings relative to human cryptographic proficiencies.

1. Benchmarks and Evaluation Paradigms

A set of rigorous frameworks—AICrypto, CipherBank, CryptoFormalEval, FoC, and task-specific cryptography–LLM integration protocols—forms the basis for empirical assessment of LLM cryptographic prowess (Wang et al., 13 Jul 2025, Li et al., 27 Apr 2025, Curaba et al., 20 Nov 2024, Shang et al., 27 Mar 2024, Maskey et al., 30 May 2025).

Task Categories:

Factual recall: Multiple-choice questions on cryptographic foundations, historical cipher mechanics, and modern primitives.
Practical cryptanalysis: Capture-the-flag (CTF) tasks involving classical and modern schemes (AES, RSA, ECC, lattice-based, block ciphers), both static and dynamic (requiring environment/tool interaction).
Proofs and formal reasoning: Construction and verification of security claims, including hybrid arguments and protocol analysis.
Binary code analysis: Predicting, summarizing, and matching cryptographic routines in stripped or obfuscated binaries.
Steganographic and covert communication: Embedding ciphertexts within naturalistic text via controlled LLM token sampling.

Evaluation Metrics:

Exact-match accuracy, BLEU score, normalized Levenshtein distance, BERTScore for decrypted text (Maskey et al., 30 May 2025, Li et al., 27 Apr 2025).
Success rate for challenge-based tasks (pass@3, composite scores).
ProofRate as a normalized score for formal proof correctness.
Semantic labeling and similarity (ROUGE, BLEU, METEOR, Recall@1) in binary code summarization and retrieval (Shang et al., 27 Mar 2024).

Human Baselines: For comparison, cryptography PhDs and top CTF finishers provide reference norms (e.g., 77.5% MCQ accuracy, 81.2% CTF, 88.1% proof rate) (Wang et al., 13 Jul 2025).

2. Decryption and Reasoning Capabilities

Classical Cipher Decryption

State-of-the-art models achieve near-perfect performance on monoalphabetic substitution and simple encoding ciphers:

Model	Caesar EM	Atbash EM	Morse EM	“Hard” EM (AES/RSA)
Claude-3.5 (few-shot)	0.99	0.93	0.96	~0.00
GPT-4o (few-shot)	0.90	0.19	0.81	~0.00

Zero-shot results demonstrate wide variance, with large models outperforming compact LLMs by >0.7 EM on classical ciphers; none achieve non-negligible performance on modern ciphers regardless of setting (Maskey et al., 30 May 2025).
Chain-of-thought prompting and few-shot demonstrations lift EM by up to 0.3–0.4 on Caesar/Vigenère for top-tier models, but yield no progress on high-entropy (AES, RSA) or token-inflated ciphers.

Multi-Step and Symbolic Reasoning

New benchmarks (CipherBank) show that general-purpose LLMs (e.g. GPT-4o, DeepSeek-V3) score <10% overall on structured cryptanalytic reasoning. Reasoning-tuned models (o1, DeepSeek-R1, Claude-3.5) reach 40–45% overall, with substitution ciphers (Rot13, Atbash) best-solved, but even here, exact key inference and expansion to polyalphabetic schemes (Vigenère) remain unreliable.
Positional and pattern-based attacks (transpositions, custom ciphers such as DualAvgCode, ParityShift) remain largely unsolved, reflecting architecture constraints and limitations in positional dependency modeling (Li et al., 27 Apr 2025).

Semantic Hallucination & Partial Comprehension

LLMs often generate semantically plausible but incorrect decryptions—“partial comprehension”—where BLEU or NL is high but exact match fails. This exposes risk for side-channel leaks and adversarial jailbreak attacks, as demonstrated when decrypted plaintext preserves relevant keywords or structure absent precise reversal (Maskey et al., 30 May 2025).

3. Formal Verification and Protocol Analysis

Integrations with symbolic verifiers (Tamarin) enable automated protocol vulnerability detection:

LLMs can successfully formalize and validate protocol flaws in scenarios involving freshness, replay, or key exposure at moderate rates (Claude 3 Opus: 60% fully verified on sample protocols) (Curaba et al., 20 Nov 2024).
Major bottlenecks include syntax errors, failure to handle multi-stage proofs, and combinatorial explosion in operator-rich settings (e.g., XOR, exponentiation).
LLMs display nascent grasp of Dolev-Yao–adversary counterexample generation, but lack the planning depth for end-to-end automated verification and proof maintenance.

4. Steganography and Covert Communication

Recent frameworks establish provable, model-agnostic schemes for embedding ciphertexts in human-like LLM-generated chat streams. The core principle is aligning each ciphertext symbol to a pseudo-random target position in the output, ensuring semantic naturalness by restricting LLM sampling parameters (temperature $T \in [0.7,0.9]$ , top- $k \in [40,60]$ ) and controlling insertion offsets (e.g., $d_0=32$ ). Security is formally bounded: the probability that any token violates these sampling constraints is $<2^{-sec}$ for designated security levels. Schemes are both LLM and post-quantum agnostic and provide indistinguishability-of-covert-communication under practical assumptions (Gligoroski et al., 11 Apr 2025).

Table: Security Parameterization for Embedding

sec	max attempts $l=1$	$l=2$	$l=3$	$l=4$
32	12	11	10	10
128	25	23	21	19

Performance is fundamentally bandwidth- and latency-limited (roughly $8\times$ text expansion for $d_0=32$ ), but achieves negligible distinguishability from plain LLM output unless the embedding positions are leaked (Gligoroski et al., 11 Apr 2025).

5. Binary Code Analysis of Cryptographic Implementations

FoC-BinLLM and FoC-Sim establish that transformer-based LLMs trained on binary pseudo-code can generate high-fidelity, human-interpretable summaries of stripped cryptographic functions, outperforming generic LLMs by 14.61% in ROUGE-L and enabling real-world detection of cryptographic routines and vulnerabilities (Recall@1 = 0.91 vs. 0.38 for prior best). Multi-task architectures combining generative and contrastive objectives yield both patch-insensitive semantic summaries and patch-sensitive embeddings for reliable retrieval, supporting practical malware analysis and firmware triage (Shang et al., 27 Mar 2024).

6. Strengths, Weaknesses, and Security Implications

Strengths:

Near-perfect factual recall and routine formal proof construction in classical cryptography (Wang et al., 13 Jul 2025).
Competence at known-attack exploitation and static challenge-solving in CTF settings.
Viable stego-embedding for covert encrypted messaging using LLMs as high-entropy cover generators (Gligoroski et al., 11 Apr 2025).
Automated function and vulnerability discovery in binary cryptographic code (Shang et al., 27 Mar 2024).

Weaknesses:

Profound difficulty with high-entropy, key-rich ciphers (AES, RSA, Playfair), positional tracking, and custom/obfuscated schemes (Li et al., 27 Apr 2025, Maskey et al., 30 May 2025).
Prone to shortcut reasoning and overreliance on statistical priors, leading to semantic but incorrect decryptions (“hallucination”).
Weak numerical precision, limited deep abstract reasoning, and poor performance in dynamic, multi-step attack settings—both for cryptanalysis and protocol verification (Wang et al., 13 Jul 2025, Curaba et al., 20 Nov 2024).
Vulnerable to prompt engineering and side-channel exploits due to partial output comprehension.

Security Implications:

Dual-use risk: LLMs feasible as both cryptanalytic tools (assisting reverse engineering, pen-testing) and as adversarial agents (automated decryption of leaked secrets, concealed command-and-control).
Countermeasures include guard models (e.g., LLaMa-Guard), input mutation, prompt filters (entropy-based), rate-limiting, and safe decoding to suppress leakage or side-channel exploitation (Maskey et al., 30 May 2025).
The indistinguishability of LLM-embedded steganography is largely predicated on secrecy of embedding positions, suggesting further study is needed when these parameters are no longer hidden (Gligoroski et al., 11 Apr 2025).

7. Directions for Future Enhancement

Targeted pretraining and augmentation with cryptanalysis/cryptographic reasoning tasks (synthetic cipher pairs, protocol logs) (Li et al., 27 Apr 2025).
Integration of explicit symbolic reasoning modules, finite-state trackers, or neural-symbolic hybrids for algorithmic and reversible inference.
Incorporation of exact numerical toolkits to address arithmetic precision limitations.
Advanced agent frameworks coupling LLMs with external solvers or collaborative workflows for dynamic CTF and protocol reasoning.
Robust, automated proof evaluation for open-ended formal tasks.
Research into secure, efficient embedding methods that maintain indistinguishability even with partial system information exposure (Gligoroski et al., 11 Apr 2025).

LLMs are approaching parity with human experts in cryptographic recall and formulaic proof generation, but currently lack the compositional and symbolic rigor for generalized cryptanalysis, protocol design validation, or creative exploitation beyond known patterns. Ongoing work on models, benchmarks, and hybrid symbolic–neural architectures will determine the extent to which LLMs will become reliable agents in the cryptographic domain.