DLoRA-TrOCR: Efficient Hybrid OCR

• The paper presents a hybrid OCR system by integrating DoRA in the vision encoder and LoRA in the text decoder.
• It leverages low-rank updates in both modules to adapt to diverse text types while updating under 0.7% of parameters.
• The design balances efficiency and performance, reducing GPU memory footprint by 30% and enhancing generalization.

DLoRA-TrOCR is a parameter-efficient hybrid text recognition system designed for robust mixed-scene Optical Character Recognition (OCR). It extends TrOCR, the end-to-end pre-trained Transformer-based OCR model, by integrating two distinct Parameter-Efficient Fine-Tuning (PEFT) modules: the weight-decomposed DoRA in the vision encoder and LoRA in the text decoder. DLoRA-TrOCR enables rapid adaptation to handwritten, printed, and street-view text scenes while updating less than 0.7% of model parameters, with notable improvements in efficiency, generalization, and resource utilization (Chang et al., 2024).

1. System Architecture and Module Integration

DLoRA-TrOCR builds upon the TrOCR-Base architecture, which couples a ViT-style image Transformer encoder (patch embedding, multi-head self-attention) with a BERT-style text Transformer decoder (masked self-attention, encoder–decoder cross-attention). PEFT modules are systematically incorporated to optimize training efficiency:

• DoRA in Encoder: Each self-attention projection (Query, Key, Value) in the image encoder is wrapped with the DoRA module. DoRA decomposes the pre-trained weight matrix $W_0 \in \mathbb{R}^{m \times n}$ into a column-norm scalar $\|W_0\|_c$ and a direction matrix $V$. During fine-tuning, only the direction is updated via a low-rank increment $\Delta V = BA$ (with $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$, $r \ll \min(m,n)$) while the magnitude remains fixed. The updated weight is normalized using the column norm, which suppresses background noise and stabilizes training.
• LoRA in Decoder: All principal linear layers in the decoder—attentional projections (Q, K, V) for both self- and cross-attention, plus the output projection—are adapted using LoRA. Each weight is reparameterized by adding a low-rank offset $\Delta W = BA$. This provides lightweight adaptation for the text generation components while operating on a minimal parameter subset (rank $r$ typically 4–8).

PEFT Module Locations in DLoRA-TrOCR

Module	Location	Operation
DoRA	Encoder	Image Q/K/V
LoRA	Decoder	Q/K/V, output linear

2. Mathematical Formulation

DoRA Weight Decomposition and Fine-Tuning

For a pre-trained weight $W_0$, DoRA computes: $$W_0 = \|W_0\|_c\,\frac{V}{\|V\|_c}\,, \quad V \equiv W_0,\quad \|V\|_c = \sum_j |V_{ij}|\text{ (per column)}$$ Fine-tuning modifies $V$ with a rank-$r$ update: $$W = \|W_0\|_c\,\frac{V + \Delta V}{\|V + \Delta V\|_c},\quad \Delta V = BA$$ This is instanced for each encoder self-attention matrix.

LoRA Weight Adaptation

Each targeted decoder weight evolves as: $W = W_0 + \Delta W,\quad \Delta W = BA$ Q/K/V projections and output linear transformation within the decoder use this low-rank adaptation, both for self-attention and cross-attention.

3. Parameter Efficiency and Fine-Tuning Protocol

DLoRA-TrOCR achieves significant reduction in trainable weights:

• TrOCR-Base total: 333.9M parameters.
• DLoRA total stored: ~335.9M (includes frozen backbone), but only ~2.0M trainable ($0.594\%$ of full).

Resource Benefits

• GPU memory footprint (fp16): 15.3GB versus 23.3GB (–30%) compared to full fine-tuning.
• Only low-rank adapters require updates; the Transformer backbone remains frozen.

Training Recipe

• Datasets:
  • IAM (handwritten English): 6,482 train, 2,915 test lines.
  • SROIE-Task2 (printed receipts): 10,682 train, 6,897 test lines.
  • STR-Benchmark (scene text): IIIT5K, SVT, IC13, IC15, SVTP, CUTE80 (7,573 train, 11,435 test lines).
  • Mixed dataset: uniform merge/sampling of above, 90% train, 10% validation.
• Loss: Cross-entropy (NLL) on token sequences.
• Optimizer: AdamW, 20 epochs, batch size 16, $5\times10^{-5}$ learning rate for PEFT, $1\times10^{-5}$ for full fine-tuning.
• Inference: beam search width 5, greedy decoding as fallback.

4. Performance Evaluation and Comparative Results

Key Metrics

• *Character Error Rate (CER): * $\mathrm{CER} = \frac{S+D+I}{N_{chars}}$
• F1 Score (word-level, case-insensitive): $$\mathrm{F1} = \frac{2 \cdot \mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$
• Word Accuracy Rate (WAR): $$\mathrm{WAR} = \frac{\# \text{correctly recognized words}}{\text{total \# words}}$$

Benchmark Results

Model	IAM CER	SROIE F1	STR WAR
Full fine-tune	10.47%	91.48%	82.57%
LoRA only	7.47%	92.74%	83.31%
DoRA only	7.57%	92.60%	83.26%
DLoRA (full)	7.56%	92.93%	83.45%

On the mixed-dataset test split, DLoRA achieves 84.63% WAR. The DLoRA configuration yields superior average CER (5.42%) and F1 (85.07%) across ablation experiments, with cross-dataset generalization performance drop $<$1% versus dedicated full fine-tuning (Chang et al., 2024).

5. Generalization, Robustness, and Ablation Studies

The combined use of DoRA (encoder) and LoRA (decoder) modules produces the most favorable trade-off between accuracy and efficiency among tested PEFT strategies. Cross-dataset results indicate strong generalization, with minimal (<1%) drops in performance when adapting from a mixed corpus to specialized domains (such as handwritten, printed, or scene text).

DoRA’s direction-magnitude decoupling in the encoder aids in suppressing domain-specific noise, while LoRA’s lightweight adjustment in the decoder preserves efficient adaptation on diverse language domains. Ablations confirm that neither DoRA-only nor LoRA-only setups achieve the overall improvements realized by their combination.

6. Deployment Advantages and Practical Applications

DLoRA-TrOCR’s advantages in deployment settings include:

• Lower GPU memory use during training (by 30%), allowing for increased batch size and operation on constrained hardware.
• Minimal inference-time overhead: low-rank adapters can be fused into TrOCR at load, maintaining original computational demand (FLOPs).
• Accelerated domain adaptation: convergence in 20 epochs (~30,000 lines), practical for rapid on-site customization (e.g., new font or camera).

Core Applications:

• Automated mailroom processing (mixed handwritten/typed addresses).
• Mobile device capture of annotated printed forms.
• Signage recognition in autonomous vehicles (street scenarios, low resolution, motion blur).

7. Summary and Significance

DLoRA-TrOCR unites the generalization strengths of pre-trained, two-stage TrOCR architecture with the efficiency of DoRA (noise-robust low-rank tuning for the encoder) and LoRA (parametric-efficient decoder adaption). It achieves state-of-the-art text recognition in heterogeneous scenes with less than 0.7% parameter updates, maintains competitive performance across multiple text domains, and supports flexible, efficient deployment in resource-constrained contexts (Chang et al., 2024).