---
title: 'ManchuAI-OCR: Fine-Tuning VLMs for Manchu Transcription'
url: https://www.emergentmind.com/topics/manchuai-ocr
type: topic
---

# ManchuAI-OCR: Fine-Tuning VLMs for Manchu Transcription

Searching arXiv for the specified paper to ground the article in the cited source.
ManchuAI-OCR is an OCR framework for the low-resource and critically endangered Manchu language that is presented as a case study in fine-tuning open-source vision-language models for historical document transcription. It is described in “Finetuning Vision-Language Models as OCR Systems for Low-Resource Languages: A Case Study of Manchu” [2507.06761]. The system fine-tunes three open-source vision-language models—LLaMA-3.2-11B-Vision, Qwen2.5-VL-7B-Instruct, and Qwen2.5-VL-3B-Instruct—on 60,000 synthetic Manchu word images using parameter-efficient training, with the stated goal of producing effective OCR for real-world historical documents while remaining deployable on accessible infrastructure [2507.06761].

## 1. Scope and problem setting

ManchuAI-OCR is situated in the context of Manchu document transcription, where the central technical problem is the absence of effective OCR systems that can handle real-world historical documents [2507.06761]. The paper characterizes Manchu as a critically endangered language essential for understanding early modern Eastern Eurasian history, and frames OCR as a practical requirement for historians and linguists working with archival sources [2507.06761].

The framework is explicitly designed around low-resource constraints. Rather than relying on large manually annotated corpora of historical scans, it uses synthetic Manchu word images for training and evaluates transfer to real handwritten material [2507.06761]. This design choice is closely tied to the paper’s deployment argument: the approach is presented as a cost-effective solution that removes technical and financial barriers in digital humanities and can be used without specialized computing resources [2507.06761].

A plausible implication is that ManchuAI-OCR is not merely a benchmark exercise but a methodology for operating under the joint constraints of scarce labeled data, historically variable handwriting, and limited hardware budgets. The paper also presents the framework as transferable to other low-resource scripts, though that extension is stated as a framework-level possibility rather than an evaluated cross-script result [2507.06761].

## 2. Model architecture and parameter-efficient adaptation

ManchuAI-OCR fine-tunes three open-source vision-language models with Low-Rank Adaptation (LoRA) adapters: LLaMA-3.2-11B-Vision, Qwen2.5-VL-7B-Instruct, and Qwen2.5-VL-3B-Instruct [2507.06761]. In all three cases, the model processes an input image through a vision encoder, projects visual features into a joint multimodal space, and then decodes into a text sequence under a fixed “OCR” instruction prompt [2507.06761].

The parameter-efficient training scheme avoids full fine-tuning by inserting LoRA modules into both the visual projection layers and the language self-attention weights [2507.06761]. The low-rank update is given as

$$
\Delta W = AB,\qquad A\in\mathbb{R}^{d\times r},\; B\in\mathbb{R}^{r\times k},
$$

with $r=64$ and scaling $\alpha=64$ [2507.06761]. This formulation is used for any weight matrix $W_0\in\mathbb{R}^{d\times k}$.

The training objective combines standard cross-entropy loss over the target token sequence with weight-decay regularization [2507.06761]:

$$
L(\theta) = \frac{1}{N}\sum_{i=1}^{N}\left[-\sum_{t=1}^{T_i}\log p_\theta(y_i^t\mid y_i^{<t},x_i)\right] + \lambda\|\theta\|_2^2.
$$

The paper also provides the expanded form

$$
\mathcal{L}_{CE}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^{T_i}\sum_{v\in V} y_{i,t,v}\log p_\theta(v\mid y_{i,<t},x_i), \qquad
\mathcal{L}_{reg}=\lambda\|\theta\|_2^2.
$$

Within the paper’s experimental logic, this architecture is notable because it repurposes general multimodal models as OCR systems through instruction-conditioned decoding rather than through a conventional OCR-specific recognizer design alone [2507.06761]. The comparison against a CRNN baseline later in the paper makes this architectural distinction central to the interpretation of synthetic-to-real transfer.

## 3. Synthetic data construction and dataset composition

The training corpus is derived from Zhang Zhuohui’s open-source dataset of 750 K rendered word images sampled from a 130,917-entry Manchu lexicon, from which 60 K training images were randomly subsampled [2507.06761]. The synthetic generation pipeline uses 12 distinct Manchu TrueType typefaces covering a variety of historical styles [2507.06761].

Rendering is performed in grayscale on a black background, after which the images are overlaid with pixel-level noise modeled roughly as additive Gaussian noise $N(0,\sigma^2)$ with $\sigma\in[5,15]$ intensity levels and random geometric distortions consisting of elastic warping and slight rotation of $\pm 3^\circ$ [2507.06761]. Post-processing then applies color inversion, median denoising, contrast stretching, and fixed resizing to $112\times112$ pixels [2507.06761]. The paper notes that one can view each synthetic sample as

$$
\tilde x = \mathrm{Resize}\circ\mathrm{Contrast}\circ\mathrm{Invert}(x+n),
$$

where $n\sim N(0,\sigma^2 I)$ and geometric transforms are sampled from small-angle $\mathrm{Uniform}(-3^\circ, 3^\circ)$ [2507.06761].

The dataset statistics reported in the paper are as follows.

| Dataset component | Size / description |
|---|---|
| Vocabulary size | 130,917 unique Manchu words |
| Training split | 60,000 synthetic images |
| Validation split | 15,000 synthetic images |
| Synthetic test subset | 1,000 randomly sampled synthetic images |
| Real handwritten test | 218 word images extracted from *Neige Cangben Manwen Laodang* |
| Word length | 1–14 characters, mean length $\approx 4.8$ characters |

The real handwritten test set consists of 218 word images extracted from *Neige Cangben Manwen Laodang*, described as 18th c. Qing court archives [2507.06761]. This is the principal real-world evaluation set for the study. The use of synthetic training data paired with a real handwritten test set is the basis for the paper’s synthetic-to-real domain transfer analysis.

## 4. Training protocol and computational profile

All three models are fine-tuned with the Unsloth “FastVisionModel” wrapper and HuggingFace’s SFTTrainer [2507.06761]. The optimizer is AdamW (paged_adamw_8bit) with weight decay $\lambda=0.05$, a learning rate of $1\times10^{-4}$, cosine decay, and 50-step linear warm-up [2507.06761]. The training uses batch size 4 images per GPU with 2 gradient-accumulation steps, giving an effective batch size of 8, and employs bfloat16 mixed precision [2507.06761]. LoRA hyperparameters are rank $r=64$, $\alpha=64$, and dropout $0.0$ [2507.06761].

The epoch schedules differ by model, with early stopping by validation WA every 1 K steps: Qwen2.5-VL-3B for 15 epochs, Qwen2.5-VL-7B for 10 epochs, and LLaMA-3.2-11B for 5 epochs [2507.06761]. The reported hardware is an NVIDIA RTX A6000 with 48 GB VRAM, an Intel i9-13900KS, 188 GB RAM, Ubuntu 24.04, and PyTorch 2.6 [2507.06761].

Evaluation uses four metrics: Word Accuracy (WA), Character Error Rate (CER), Character F1, and Inference Time [2507.06761]. WA is defined as the number of exact matches divided by total samples, while CER uses standard Levenshtein-distance normalization [2507.06761]:

$$
\mathrm{CER} = \frac{\#\,\text{substitutions}+\#\,\text{insertions}+\#\,\text{deletions}}{\#\,\text{reference characters}}.
$$

Character F1 is described as block-based matching of character sequences, and inference time is the average wall-clock time per image including romanization decoding [2507.06761].

The deployment profile is a recurring part of the paper’s argument. LoRA adapters increase model size by less than 1% beyond base, and full model inference for the 11 B model requires approximately 20 GB GPU memory [2507.06761]. The codebase is packaged in Python and PyTorch and is described as runnable on a single A6000 or Google Colab Pro+ without special hardware [2507.06761].

## 5. Quantitative performance and baseline comparison

The paper reports quantitative results on a synthetic validation set of 1,000 samples and a real handwritten test set of 218 samples [2507.06761]. The measured values are reproduced below.

| Model | Synthetic WA (%) | CER | Real WA (%) |
|---|---:|---:|---:|
| LLaMA-3.2-11B | 98.3 | 0.0024 | 93.1 |
| Qwen2.5-VL-7B | 87.5 | 0.0264 | 43.1 |
| Qwen2.5-VL-3B | 84.4 | 0.0329 | 23.9 |
| CRNN Baseline | 99.8 | 0.0001 | 72.5 |

The full table also reports real CER values and inference times: LLaMA-3.2-11B has real CER 0.0219 and 8.9 s/image, Qwen2.5-VL-7B has real CER 0.254 and 0.9 s/image, Qwen2.5-VL-3B has real CER 0.368 and 0.7 s/image, and the CRNN baseline has real CER 0.093 and 0.05 s/image [2507.06761].

LLaMA-3.2-11B is the top-performing model on the real handwritten test, achieving 93.1% word accuracy with 0.0219 CER [2507.06761]. On synthetic data, the CRNN baseline reaches the highest reported word accuracy, 99.8%, with CER 0.0001, but its real-document performance falls to 72.5% word accuracy and 0.093 CER [2507.06761]. The paper therefore places emphasis not on synthetic-only recognition quality, but on robustness under domain shift.

The speed-accuracy trade-off is explicit. Qwen2.5-VL-3B and Qwen2.5-VL-7B are substantially faster than LLaMA-3.2-11B, but the paper states that this comes at the cost of dramatically lower real-world accuracy [2507.06761]. In this sense, ManchuAI-OCR is not a single model but a comparative framework in which the 11 B LLaMA variant is identified as the preferred operating point for historical handwritten OCR.

## 6. Synthetic-to-real transfer, error structure, and limitations

The study’s central empirical claim is effective synthetic-to-real domain transfer [2507.06761]. LLaMA-3.2-11B retains high word accuracy with only a –5.2 percentage-point drop from synthetic to real data, along with a low CER increase, which the paper interprets as strong generalization from synthetic fonts plus noise to real handwriting [2507.06761]. By contrast, the Qwen models are described as collapsing on real data, with 44–60 percentage-point WA drops that are taken to reflect overfitting to synthetic artifacts [2507.06761].

The CRNN baseline also degrades, with a –27 percentage-point WA change, but still outperforms the Qwen models on real data [2507.06761]. The paper states that this confirms that multimodal pretraining in VLMs offers superior cross-domain robustness [2507.06761]. Because this comparison is internal to the reported experiments, it is one of the article’s clearest architectural conclusions.

Error analysis identifies systematic confusion among visually similar vowels, including Manchu “A” and “E” [2507.06761]. The paper reports that approximately 55–60% of all errors in the LLaMA test set are concentrated in these confusions [2507.06761]. Morphological similarity of Manchu glyphs, especially vowels, is presented as a persistent failure mode, and isolated characters are often mis-recognized [2507.06761].

Several limitations and future improvements are explicitly enumerated. No explicit real-to-real domain adaptation was applied, and future work could incorporate unsupervised domain adaptation, adversarial style transfer, or fine-tuning on scanned archival pages [2507.06761]. The augmentation scheme could be extended with elastic distortions, blur, bleed-through modeling, and multiple scanning resolutions [2507.06761]. The concentration of errors in targeted confusion classes suggests that character-level discriminators or contrastive pretraining on similar pairs may be useful [2507.06761]. These proposals are framed as future directions rather than demonstrated components of the current system.

## 7. Reproducibility, deployment, and significance

The system is distributed through an open-source ManchuAI-OCR codebase with code, training scripts, and pretrained model weights publicly available at the project repository listed in the paper [2507.06761]. The README is described as covering environment setup for Ubuntu 24.04, CUDA 12.4, and PyTorch 2.6, along with dataset download and reproduction of training and evaluation [2507.06761]. A prebuilt Docker container and Google Colab notebook are also provided for zero-setup inference and small-scale fine-tuning [2507.06761].

The paper identifies several deployment consequences. The open-source paradigm is said to avoid high API costs, ensure data privacy, and offer full control over tokenization and architecture [2507.06761]. It also states that synthetic data plus parameter-efficient fine-tuning of VLMs yields state-of-the-art Manchu OCR with minimal real-world handwriting samples, and that LLaMA-3.2-11B with LoRA achieves more than 93% WA on historical documents with a single GPU [2507.06761].

Within the paper’s stated scope, ManchuAI-OCR is therefore significant in two linked senses. First, it provides a concrete OCR system for a historically important endangered language using synthetic data and parameter-efficient multimodal adaptation [2507.06761]. Second, it is presented as a transferable framework for endangered language OCR that lowers technical and financial barriers in digital humanities and may be extendable to other low-resource scripts [2507.06761]. This suggests a broader methodological relevance: the contribution lies not only in the specific Manchu recognizer, but also in the demonstration that synthetic corpora and PEFT-tuned VLMs can support historically grounded OCR under low-resource conditions.

Source: https://www.emergentmind.com/topics/manchuai-ocr