Learning to Decipher from Pixels -- A Case Study of Copiale

Abstract: Historical encrypted manuscripts require both paleographic interpretation of cipher symbols and cryptanalytic recovery of plaintext. Most existing computational workflows rely on a transcription-first paradigm, in which handwritten symbols are transcribed prior to decipherment. This intermediate step is labor-intensive, error-prone, and not always aligned with the goal of direct plaintext recovery. We propose an end-to-end, transcription-free approach that directly maps handwritten cipher images to plaintext. Using the Copiale cipher as a case study, we introduce the first text-line-level dataset pairing cipher images with German plaintext. We show that pretraining on generic handwriting data followed by cipher-specific fine-tuning substantially improves decipherment accuracy. Our results demonstrate that transcription-free image-to-plaintext decipherment is both feasible and effective for historical substitution ciphers, offering a simplified and scalable alternative to traditional pipelines. https://github.com/leitro/Decipher-from-Pixels-Copiale

• The paper introduces a transcription-free, end-to-end image-to-plaintext framework that bypasses traditional transcription bottlenecks.
• It employs a two-stage training pipeline using generic handwriting pretraining followed by fine-tuning on Copiale cipher data, achieving significant reductions in CER and WER.
• Quantitative and qualitative analyses demonstrate robust spatial alignments and practical implications for scalable historical cryptanalysis.

Transcription-Free Decipherment of Historical Manuscripts: The Copiale Cipher Case Study

Motivation and Problem Formulation

Decipherment of historical handwritten ciphers presents a coupled challenge: paleographic recognition of non-standard symbols and cryptanalytic mapping from ciphertext to plaintext. Most traditional approaches follow a staged, transcription-first paradigm—manually or algorithmically transcribing cipher symbols before statistical decoding. This pipeline suffers from high annotation cost, segmentation bottlenecks, and error propagation across stages, due to the irregular symbol inventory and idiosyncratic layouts characteristic of historic manuscripts. The goal of this work is to bypass the transcription bottleneck by introducing an end-to-end, image-to-plaintext paradigm that directly maps cipher manuscript images to natural language, obviating the need for an explicit intermediate ciphertext.

Dataset Construction and Experimental Corpus

A new line-aligned dataset is introduced, pairing Copiale cipher text-line images with aligned German plaintext references, enabling direct supervision of decipherment without explicit symbol-level annotation. The Copiale cipher, chosen for its cryptanalytic significance and available parallel decoding, exemplifies a real-world, non-trivial substitution cipher. For transfer learning, 66,492 handwritten text-line images—including varied scripts, hands, and languages—are gathered from IAM, CVL, RIMES, and EU27, providing a robust initialization for visual feature extraction in the absence of cipher-specific data.

Model Architecture and Training Pipeline

A transformer-based encoder-decoder architecture, TrOCR, is adopted with minimal adaptation: only the output vocabulary is aligned to the target language (German). The system treats handwritten cipher image input as visual sequences and directly produces plaintext output, in a classical sequence-to-sequence regime. The entire pipeline is trained in two stages:

1. Handwriting pretraining: Generic handwriting data is used to obtain visually robust features, improving generalization across unseen symbol shapes and layouts.
2. Cipher-specific fine-tuning: The pretrained model is optimized over the Copiale image-to-plaintext corpus for final mapping fidelity.

The overarching training pipeline is depicted in (Figure 1).

Figure 1: Overview of a two-stage training regime: generic handwriting pretraining, followed by Copiale cipher-specific fine-tuning to facilitate transcription-free decipherment.

Quantitative Results and Performance Evaluation

Character Error Rate (CER) and Word Error Rate (WER) are reported as the primary evaluation metrics due to their appropriateness for sequence generation tasks in low-resource manuscript settings. The empirical evaluation establishes two key results:

• Direct fine-tuning on cipher data (no generic pretraining): Baseline CER/WER on Copiale test set are 46.10%/98.48%.
• Two-stage pretraining + fine-tuning pipeline: CER/WER are reduced to 11.03%/33.03%, respectively.

This demonstrates a major relative decrease in both character and word-level errors. The strong numerical reduction—over 75%—in CER and WER quantifies the value of non-cipher handwriting pretraining for visually encrypted document decoding. This result also underscores a significant claim: pretrained handwriting models, even with no direct exposure to the cryptanalytic task, transfer effectively to the image-to-plaintext mapping required for decipherment.

Qualitative Analysis and Error Modes

Attention weight visualization reveals that the model learns non-trivial spatial correspondences between cipher regions and decoded plain tokens, substantiating its ability to implicitly recover symbol alignments in the absence of supervised transcriptions. Successful and failure cases illuminate both the strengths and the remaining error patterns of the approach. The model reliably decodes diverse cipher structures (successful cases), yet exhibits failures predominantly in visually ambiguous segments or non-standard layout regions (failure cases), as illustrated in (Figure 2).

Figure 2: (a) Attention mapping between cipher regions and output tokens; (b) Example of perfect prediction; (c) Error mode with output diverging from reference.

Implications, Limitations, and Future Prospects

The paradigm shift to transcription-free decipherment has several practical and theoretical ramifications:

• Annotation Efficiency: Eliminates the costly intermediate symbol transcription phase, substantially reducing the domain expertise and time required for historical cryptanalysis workflows.
• Pipeline Simplification: Offers a single-stage mapping from manuscript to plaintext, minimizing sources of error propagation and decoupling performance from segmentation heuristics.
• Generalization Potential: Empirical transfer from generic handwriting data to cipher decipherment signals extensibility to broader classes of historical cryptographic scripts and possibly to other rare or extinct notational systems.

Remaining limitations include reduction in performance at regions with highly irregular symbol structure or severe historical manuscript degradation. The approach remains constrained by data availability for fine-tuning and the need for paired image-to-plaintext references. Future directions should target data augmentation for low-resource settings, cross-cipher transfer learning, semi-supervised adaptation to unlabeled ciphers, and integration with document layout analysis to process full-manuscript images in an end-to-end fashion.

Conclusion

This work introduces the first transcription-free, end-to-end handwritten cipher decipherment framework, demonstrated on the Copiale cipher. By incorporating large-scale handwriting pretraining and leveraging a transformer-based image-to-text model, it achieves a notable reduction in CER and WER compared to transcription-reliant baselines. The method reshapes the technical toolkit available for historical cryptology, offering both practical scalability and a foundation for further advances in visually grounded, language-informed cryptanalytic systems (2604.23683).