ManchuAI-OCR: Fine-Tuning VLMs for Manchu Transcription

• The paper introduces ManchuAI-OCR, an OCR system that employs parameter-efficient LoRA adapters on open-source VLMs to process synthetic Manchu word images.
• It demonstrates robust synthetic-to-real domain transfer by achieving up to 93.1% word accuracy on historical handwritten documents despite scarce labeled data.
• The approach minimizes computational overhead and deploys on accessible hardware, offering a scalable and cost-effective solution for digital humanities research.

Searching arXiv for the specified paper to ground the article in the cited source. ManchuAI-OCR is an OCR framework for the low-resource and critically endangered Manchu language that is presented as a case study in fine-tuning open-source vision-LLMs for historical document transcription. It is described in “Finetuning Vision-LLMs as OCR Systems for Low-Resource Languages: A Case Study of Manchu” (Chung et al., 9 Jul 2025). The system fine-tunes three open-source vision-LLMs—LLaMA-3.2-11B-Vision, Qwen2.5-VL-7B-Instruct, and Qwen2.5-VL-3B-Instruct—on 60,000 synthetic Manchu word images using parameter-efficient training, with the stated goal of producing effective OCR for real-world historical documents while remaining deployable on accessible infrastructure (Chung et al., 9 Jul 2025).

1. Scope and problem setting

ManchuAI-OCR is situated in the context of Manchu document transcription, where the central technical problem is the absence of effective OCR systems that can handle real-world historical documents (Chung et al., 9 Jul 2025). The paper characterizes Manchu as a critically endangered language essential for understanding early modern Eastern Eurasian history, and frames OCR as a practical requirement for historians and linguists working with archival sources (Chung et al., 9 Jul 2025).

The framework is explicitly designed around low-resource constraints. Rather than relying on large manually annotated corpora of historical scans, it uses synthetic Manchu word images for training and evaluates transfer to real handwritten material (Chung et al., 9 Jul 2025). This design choice is closely tied to the paper’s deployment argument: the approach is presented as a cost-effective solution that removes technical and financial barriers in digital humanities and can be used without specialized computing resources (Chung et al., 9 Jul 2025).

A plausible implication is that ManchuAI-OCR is not merely a benchmark exercise but a methodology for operating under the joint constraints of scarce labeled data, historically variable handwriting, and limited hardware budgets. The paper also presents the framework as transferable to other low-resource scripts, though that extension is stated as a framework-level possibility rather than an evaluated cross-script result (Chung et al., 9 Jul 2025).

2. Model architecture and parameter-efficient adaptation

ManchuAI-OCR fine-tunes three open-source vision-LLMs with Low-Rank Adaptation (LoRA) adapters: LLaMA-3.2-11B-Vision, Qwen2.5-VL-7B-Instruct, and Qwen2.5-VL-3B-Instruct (Chung et al., 9 Jul 2025). In all three cases, the model processes an input image through a vision encoder, projects visual features into a joint multimodal space, and then decodes into a text sequence under a fixed “OCR” instruction prompt (Chung et al., 9 Jul 2025).

The parameter-efficient training scheme avoids full fine-tuning by inserting LoRA modules into both the visual projection layers and the language self-attention weights (Chung et al., 9 Jul 2025). The low-rank update is given as

$$\Delta W = AB,\qquad A\in\mathbb{R}^{d\times r},\; B\in\mathbb{R}^{r\times k},$$

with $r=64$ and scaling $\alpha=64$ (Chung et al., 9 Jul 2025). This formulation is used for any weight matrix $W_0\in\mathbb{R}^{d\times k}$.

The training objective combines standard cross-entropy loss over the target token sequence with weight-decay regularization (Chung et al., 9 Jul 2025):

$$L(\theta) = \frac{1}{N}\sum_{i=1}^{N}\left[-\sum_{t=1}^{T_i}\log p_\theta(y_i^t\mid y_i^{<t},x_i)\right] + \lambda\|\theta\|_2^2.$$

The paper also provides the expanded form

$$\mathcal{L}_{CE}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^{T_i}\sum_{v\in V} y_{i,t,v}\log p_\theta(v\mid y_{i,<t},x_i), \qquad \mathcal{L}_{reg}=\lambda\|\theta\|_2^2.$$

Within the paper’s experimental logic, this architecture is notable because it repurposes general multimodal models as OCR systems through instruction-conditioned decoding rather than through a conventional OCR-specific recognizer design alone (Chung et al., 9 Jul 2025). The comparison against a CRNN baseline later in the paper makes this architectural distinction central to the interpretation of synthetic-to-real transfer.

3. Synthetic data construction and dataset composition

The training corpus is derived from Zhang Zhuohui’s open-source dataset of 750 K rendered word images sampled from a 130,917-entry Manchu lexicon, from which 60 K training images were randomly subsampled (Chung et al., 9 Jul 2025). The synthetic generation pipeline uses 12 distinct Manchu TrueType typefaces covering a variety of historical styles (Chung et al., 9 Jul 2025).

Rendering is performed in grayscale on a black background, after which the images are overlaid with pixel-level noise modeled roughly as additive Gaussian noise $N(0,\sigma^2)$ with $\sigma\in[5,15]$ intensity levels and random geometric distortions consisting of elastic warping and slight rotation of $\pm 3^\circ$ (Chung et al., 9 Jul 2025). Post-processing then applies color inversion, median denoising, contrast stretching, and fixed resizing to $112\times112$ pixels (Chung et al., 9 Jul 2025). The paper notes that one can view each synthetic sample as

$r=64$0

where $r=64$1 and geometric transforms are sampled from small-angle $r=64$2 (Chung et al., 9 Jul 2025).

The dataset statistics reported in the paper are as follows.

Dataset component	Size / description
Vocabulary size	130,917 unique Manchu words
Training split	60,000 synthetic images
Validation split	15,000 synthetic images
Synthetic test subset	1,000 randomly sampled synthetic images
Real handwritten test	218 word images extracted from Neige Cangben Manwen Laodang
Word length	1–14 characters, mean length $r=64$3 characters

The real handwritten test set consists of 218 word images extracted from Neige Cangben Manwen Laodang, described as 18th c. Qing court archives (Chung et al., 9 Jul 2025). This is the principal real-world evaluation set for the study. The use of synthetic training data paired with a real handwritten test set is the basis for the paper’s synthetic-to-real domain transfer analysis.

4. Training protocol and computational profile

All three models are fine-tuned with the Unsloth “FastVisionModel” wrapper and HuggingFace’s SFTTrainer (Chung et al., 9 Jul 2025). The optimizer is AdamW (paged_adamw_8bit) with weight decay $r=64$4, a learning rate of $r=64$5, cosine decay, and 50-step linear warm-up (Chung et al., 9 Jul 2025). The training uses batch size 4 images per GPU with 2 gradient-accumulation steps, giving an effective batch size of 8, and employs bfloat16 mixed precision (Chung et al., 9 Jul 2025). LoRA hyperparameters are rank $r=64$6, $r=64$7, and dropout $r=64$8 (Chung et al., 9 Jul 2025).

The epoch schedules differ by model, with early stopping by validation WA every 1 K steps: Qwen2.5-VL-3B for 15 epochs, Qwen2.5-VL-7B for 10 epochs, and LLaMA-3.2-11B for 5 epochs (Chung et al., 9 Jul 2025). The reported hardware is an NVIDIA RTX A6000 with 48 GB VRAM, an Intel i9-13900KS, 188 GB RAM, Ubuntu 24.04, and PyTorch 2.6 (Chung et al., 9 Jul 2025).

Evaluation uses four metrics: Word Accuracy (WA), Character Error Rate (CER), Character F1, and Inference Time (Chung et al., 9 Jul 2025). WA is defined as the number of exact matches divided by total samples, while CER uses standard Levenshtein-distance normalization (Chung et al., 9 Jul 2025):

$r=64$9

Character F1 is described as block-based matching of character sequences, and inference time is the average wall-clock time per image including romanization decoding (Chung et al., 9 Jul 2025).

The deployment profile is a recurring part of the paper’s argument. LoRA adapters increase model size by less than 1% beyond base, and full model inference for the 11 B model requires approximately 20 GB GPU memory (Chung et al., 9 Jul 2025). The codebase is packaged in Python and PyTorch and is described as runnable on a single A6000 or Google Colab Pro+ without special hardware (Chung et al., 9 Jul 2025).

5. Quantitative performance and baseline comparison

The paper reports quantitative results on a synthetic validation set of 1,000 samples and a real handwritten test set of 218 samples (Chung et al., 9 Jul 2025). The measured values are reproduced below.

Model	Synthetic WA (%)	CER	Real WA (%)
LLaMA-3.2-11B	98.3	0.0024	93.1
Qwen2.5-VL-7B	87.5	0.0264	43.1
Qwen2.5-VL-3B	84.4	0.0329	23.9
CRNN Baseline	99.8	0.0001	72.5

The full table also reports real CER values and inference times: LLaMA-3.2-11B has real CER 0.0219 and 8.9 s/image, Qwen2.5-VL-7B has real CER 0.254 and 0.9 s/image, Qwen2.5-VL-3B has real CER 0.368 and 0.7 s/image, and the CRNN baseline has real CER 0.093 and 0.05 s/image (Chung et al., 9 Jul 2025).

LLaMA-3.2-11B is the top-performing model on the real handwritten test, achieving 93.1% word accuracy with 0.0219 CER (Chung et al., 9 Jul 2025). On synthetic data, the CRNN baseline reaches the highest reported word accuracy, 99.8%, with CER 0.0001, but its real-document performance falls to 72.5% word accuracy and 0.093 CER (Chung et al., 9 Jul 2025). The paper therefore places emphasis not on synthetic-only recognition quality, but on robustness under domain shift.

The speed-accuracy trade-off is explicit. Qwen2.5-VL-3B and Qwen2.5-VL-7B are substantially faster than LLaMA-3.2-11B, but the paper states that this comes at the cost of dramatically lower real-world accuracy (Chung et al., 9 Jul 2025). In this sense, ManchuAI-OCR is not a single model but a comparative framework in which the 11 B LLaMA variant is identified as the preferred operating point for historical handwritten OCR.

6. Synthetic-to-real transfer, error structure, and limitations

The study’s central empirical claim is effective synthetic-to-real domain transfer (Chung et al., 9 Jul 2025). LLaMA-3.2-11B retains high word accuracy with only a –5.2 percentage-point drop from synthetic to real data, along with a low CER increase, which the paper interprets as strong generalization from synthetic fonts plus noise to real handwriting (Chung et al., 9 Jul 2025). By contrast, the Qwen models are described as collapsing on real data, with 44–60 percentage-point WA drops that are taken to reflect overfitting to synthetic artifacts (Chung et al., 9 Jul 2025).

The CRNN baseline also degrades, with a –27 percentage-point WA change, but still outperforms the Qwen models on real data (Chung et al., 9 Jul 2025). The paper states that this confirms that multimodal pretraining in VLMs offers superior cross-domain robustness (Chung et al., 9 Jul 2025). Because this comparison is internal to the reported experiments, it is one of the article’s clearest architectural conclusions.

Error analysis identifies systematic confusion among visually similar vowels, including Manchu “A” and “E” (Chung et al., 9 Jul 2025). The paper reports that approximately 55–60% of all errors in the LLaMA test set are concentrated in these confusions (Chung et al., 9 Jul 2025). Morphological similarity of Manchu glyphs, especially vowels, is presented as a persistent failure mode, and isolated characters are often mis-recognized (Chung et al., 9 Jul 2025).

Several limitations and future improvements are explicitly enumerated. No explicit real-to-real domain adaptation was applied, and future work could incorporate unsupervised domain adaptation, adversarial style transfer, or fine-tuning on scanned archival pages (Chung et al., 9 Jul 2025). The augmentation scheme could be extended with elastic distortions, blur, bleed-through modeling, and multiple scanning resolutions (Chung et al., 9 Jul 2025). The concentration of errors in targeted confusion classes suggests that character-level discriminators or contrastive pretraining on similar pairs may be useful (Chung et al., 9 Jul 2025). These proposals are framed as future directions rather than demonstrated components of the current system.

7. Reproducibility, deployment, and significance

The system is distributed through an open-source ManchuAI-OCR codebase with code, training scripts, and pretrained model weights publicly available at the project repository listed in the paper (Chung et al., 9 Jul 2025). The README is described as covering environment setup for Ubuntu 24.04, CUDA 12.4, and PyTorch 2.6, along with dataset download and reproduction of training and evaluation (Chung et al., 9 Jul 2025). A prebuilt Docker container and Google Colab notebook are also provided for zero-setup inference and small-scale fine-tuning (Chung et al., 9 Jul 2025).

The paper identifies several deployment consequences. The open-source paradigm is said to avoid high API costs, ensure data privacy, and offer full control over tokenization and architecture (Chung et al., 9 Jul 2025). It also states that synthetic data plus parameter-efficient fine-tuning of VLMs yields state-of-the-art Manchu OCR with minimal real-world handwriting samples, and that LLaMA-3.2-11B with LoRA achieves more than 93% WA on historical documents with a single GPU (Chung et al., 9 Jul 2025).

Within the paper’s stated scope, ManchuAI-OCR is therefore significant in two linked senses. First, it provides a concrete OCR system for a historically important endangered language using synthetic data and parameter-efficient multimodal adaptation (Chung et al., 9 Jul 2025). Second, it is presented as a transferable framework for endangered language OCR that lowers technical and financial barriers in digital humanities and may be extendable to other low-resource scripts (Chung et al., 9 Jul 2025). This suggests a broader methodological relevance: the contribution lies not only in the specific Manchu recognizer, but also in the demonstration that synthetic corpora and PEFT-tuned VLMs can support historically grounded OCR under low-resource conditions.