CipherBank: A Cryptographic Reasoning Benchmark

• CipherBank is a structured benchmark that assesses LLMs’ reasoning in cryptographic decryption and obfuscation tasks across diverse, high-stakes data domains.
• It systematically evaluates multiple cipher algorithms—including substitution, transposition, and custom ciphers—using a 3-shot Known-Plaintext Attack protocol with metrics such as exact-match accuracy and Levenshtein similarity.
• The toolkit features an RL-driven cipher selection module for efficient adversarial prompt obfuscation, highlighting limitations in LLMs’ abilities for precise symbolic manipulation.

CipherBank is a structured benchmark and toolkit for systematically assessing, developing, and manipulating LLMs’ (LLMs) reasoning abilities in cryptographic decryption and obfuscation tasks. Rooted in both adversarial prompt obfuscation (as a registry or cipher pool within attack frameworks) and in controlled evaluation of cryptographic reasoning (as a decryption challenge benchmark), CipherBank occupies a central position at the intersection of LLM robustness research, AI safety, and symbolic reasoning evaluation (Li et al., 27 Apr 2025, Chen et al., 27 Jun 2025).

1. Design and Construction of CipherBank Benchmark

CipherBank as a benchmark consists of 2,358 decryption problems, generated from 262 unique plaintexts. These cover five high-stakes real-world domains: Personal Privacy Data, Enterprise Sensitive Data, Public Safety Data, Financial Asset Data, and Internet Records. Each domain is further partitioned into 14 subdomains, such as Identity Information, Medical Records, and Banking Information. Every plaintext is paired with its encrypted form under nine distinct cipher algorithms, resulting in diverse and realistic scenarios with 89 unique tags (e.g., passport numbers, transaction IDs) (Li et al., 27 Apr 2025).

Cipher algorithms are drawn from three principal categories:

• Substitution ciphers: Rot13 (Caesar shift 13), Atbash (reverse-alphabet mapping), Polybius (fractionating substitution with a 6×6 grid), Vigenère (keyed polyalphabetic shift).
• Transposition ciphers: Reverse (full sequence reversal), SwapPairs (adjacent character swap).
• Custom ciphers: DualAvgCode (character expands into two such that their average encodes the plaintext), ParityShift (even/odd ASCII shifting), WordShift (intra-word rotation).

Each problem is labeled with a difficulty tier T1–T5, reflecting cryptographic complexity (with Vigenère as the most challenging).

2. CipherBank as an Obfuscation Cipher Pool

Within adversarial frameworks like MetaCipher, the CipherBank paradigm is implemented as a modular registry of cipher modules. Each cipher object is defined by:

• Identifier string (e.g., "caesar", "morse"),
• Family label: One of Substitution, Transposition, Book, Concealment,
• encrypt(·) function: A mapping from a keyword to its obfuscated form.

All ciphers $a_1, \dots, a_N$ are held in a flat array $\mathcal{A}$ and can be indexed by integer IDs. Auxiliary data structures include:

• A hashmap mapping family names to their members,
• A precomputed similarity matrix $\mathrm{Sim}\in\mathbb{R}^{N\times N}$, representing overlap of successful decryption sets and guiding RL-based credit propagation (Chen et al., 27 Jun 2025).

Cipher families supported in MetaCipher include:

• Substitution (9 ciphers): ascii, atbash, base64, caesar, grid, keyboard, leetspeak, morse, unicode,
• Transposition (8 ciphers): acrostic, anagram, letters, incomplete, insert, piglatin, reversal, wordladder,
• Book (2 ciphers): article, substitution-in-text,
• Concealment (2 ciphers): reference, riddle.

This architecture enables $O(1)$ lookup by index or family and $O(N)$ sampling from the entire pool.

3. Evaluation Methodology and Metrics

In the reasoning benchmark, CipherBank employs a 3-shot Known-Plaintext Attack protocol: the LLM receives three plaintext–ciphertext demonstration pairs and is tasked with decrypting a new ciphertext. The prompt is wrapped in a system instruction emphasizing stepwise cryptographic analysis.

Performance is measured using:

• Exact-match accuracy:

$$\text{Accuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\left[\hat{P}_i = P_i\right]$$

where $\hat{P}_i$ is the predicted and $P_i$ the gold plaintext.

• Levenshtein similarity:

$$S_{lev} = 1 - \frac{d_{lev}(\hat{P}, P)}{\max(|\hat{P}|, |P|)}$$

with $d_{lev}$ the edit distance.

For RL-controlled cipher selection in attack frameworks, the main metric is the Attack Success Rate (ASR) at query budget $\mathcal{A}$0:

$\mathcal{A}$1

where “success” indicates at least one query achieves a jailbreak.

Additionally, inter-cipher redundancy is quantified by Jaccard similarity:

$\mathcal{A}$2

with $\mathcal{A}$3 the prompt set uniquely solvable by cipher $\mathcal{A}$4.

4. Reinforcement Learning–Driven Cipher Selection

Within adaptive obfuscation frameworks, CipherBank ciphers are selected by a lightweight RL agent modeling the jailbreak as an MDP. The state space is $\mathcal{A}$5, where $\mathcal{A}$6 is the victim LLM and $\mathcal{A}$7 is the prompt category. The action space is the cipher index set $\mathcal{A}$8.

The agent maintains a tabular Q-function $\mathcal{A}$9. The policy is a softmax over Q-values with optimism and temperature parameters:

$\mathrm{Sim}\in\mathbb{R}^{N\times N}$0

Classic one-step Q-learning is performed after each trial:

$\mathrm{Sim}\in\mathbb{R}^{N\times N}$1

Rewards are also soft-shared to unused ciphers proportional to similarity:

$\mathrm{Sim}\in\mathbb{R}^{N\times N}$2

This design avoids deep policy networks and explicit $\mathrm{Sim}\in\mathbb{R}^{N\times N}$3-greedy exploration, relying on softmax-driven exploration and similarity-based reward sharing.

5. Model Performance and Error Taxonomy

State-of-the-art LLMs—including GPT-4o, DeepSeek-V3, OpenAI’s o1/o1-mini, and DeepSeek-R1—demonstrate significant performance variability on CipherBank (Li et al., 27 Apr 2025). Reasoning-focused models outperform general-purpose chat LLMs, but substantial accuracy gaps persist even for engineered reasoning architectures.

• Substitution ciphers: Top accuracy on Rot13 reaches 83.21% (Claude-3.5), Atbash/Polybius around 79% (o1), but Vigenère peaks at only 7.25% (o1).
• Transposition ciphers: Reverse up to 63.93% (Claude), SwapPairs remains difficult (max 32.14%, usually below 6%).
• Custom ciphers: DualAvgCode (50.38%, o1), ParityShift (58.21%, Claude), WordShift (39.12%, Claude).

Aggregate “Cipher Score” shows that even the best reasoning model (o1) averages <50% accuracy.

Error analysis identifies six principal categories: omission/insertion, name decryption errors, semantic inference, reorganization, reasoning failure, and other. Chat models are prone to omission and semantic errors, indicating semantic shortcutting. Reasoning failures occur, particularly over unnecessary inference on trivial ciphers. All models struggle with consistent ciphering of proper nouns.

6. Extensibility and Research Implications

CipherBank’s modular registry structure accommodates the addition of new ciphers. To extend the pool: implement an encrypt function, append the object to $\mathrm{Sim}\in\mathbb{R}^{N\times N}$4, optionally seed $\mathrm{Sim}\in\mathbb{R}^{N\times N}$5-values from validation data, and estimate the new similarity vector by comparison with existing ciphers (Chen et al., 27 Jun 2025). This facilitates zero-shot inclusion and rapid learning of cipher efficacy via RL updates.

CipherBank’s findings highlight several core limitations in LLM cryptographic reasoning: overreliance on linguistic priors, inability to generalize symbolic transformations, and brittleness to input variation. Addressing these deficits requires contrastive training objectives, explicit pattern-comparison mechanisms, and the development of verifier steps that prevent semantic overcorrection. The dataset’s structure provides a blueprint for benchmarking future model improvements, including proposed expansions to modern cryptosystems as LLM capabilities mature (Li et al., 27 Apr 2025).

7. Impact, Limitations, and Future Directions

CipherBank exposes stark challenges for contemporary LLMs in precise multi-step symbol manipulation under cryptographic constraints. These limitations have direct implications for both the safety of LLM deployments (where cipher-based obfuscation can be leveraged in adversarial attacks (Chen et al., 27 Jun 2025)) and for the future of symbolic reasoning in neural architectures. By specializing in both cryptographic obfuscation (as a cipher pool) and rigorous reasoning evaluation (as a decryption benchmark), CipherBank provides foundational methodology for safety, capability, and robustness research in generative AI.

Proposed future work includes expansion to public-key and block-cipher challenges, refinement of evaluation protocols, and systematic reduction of known reasoning pathologies. The public availability of CipherBank is intended to support reproducible research and benchmark progress in cryptographic reasoning for LLMs (Li et al., 27 Apr 2025).