PorTEXTO: A European Portuguese Benchmark for Visual Text Extraction

Abstract: European Portuguese (pt-PT) is largely absent from OCR benchmarks, which skew toward high-resource languages. The few benchmarks that cover pt-PT focus on historical artifacts and literature. This work addresses modern OCR applications, introducing PorTEXTO, the first benchmark for contemporary and culturally relevant pt-PT visual text extraction. To ascertain quality, we employ an annotation pipeline combining transcriptions from a frontier LVLM with exhaustive review by native speakers. We observe a sharp performance drop from synthetic to real world samples in most models, and find that, currently, specialized multilingual data is a better driver for pt-PT performance than model size or resolution budget, motivating the release of open pt-PT OCR resources.

• The paper presents PorTEXTO, a comprehensive benchmark for visual text extraction from contemporary European Portuguese documents in diverse real-world scenarios.
• It details a rigorously annotated dataset of 779 samples covering handwritten, in-the-wild, and synthetic texts, validated by advanced LVLMs and native speakers.
• Evaluation highlights that specialized multilingual training and targeted data quality are crucial to overcoming OCR challenges in low-resource pt-PT environments.

PorTEXTO: A Benchmark for Contemporary European Portuguese Visual Text Extraction

Motivation and Benchmark Design

PorTEXTO addresses a significant gap in OCR benchmarks by focusing on contemporary European Portuguese (pt-PT) text extraction, diverging from datasets which predominately cover high-resource languages (e.g., English, Chinese), or are limited to historical artifacts and literature for pt-PT. The benchmark explicitly targets usage scenarios representative of modern pt-PT contexts: handwritten notes, in-the-wild photographs and screenshots, and synthetically rendered documents. This diversity is required to assess OCR systems under authentic, varied conditions typically faced by pt-PT users.

Figure 1: Modern OCR applications illustrating the breadth of professional to casual use cases relevant for contemporary pt-PT text extraction.

Dataset Composition and Annotation Pipeline

The benchmark encompasses three distinct data sources, each crafted to challenge OCR models with real-world idiosyncrasies:

• Handwritten Subsets: Sourced from student notes and personal writings, exposing models to authentic handwriting, prevalent abbreviations, and mixed prose. Pages are scanned and further subdivided into logical content blocks for granular annotation and evaluation.

Figure 2: Samples from Full Page and Region Handwritten subsets highlight challenges posed by authentic handwriting and contemporary abbreviations.

• In-The-Wild Subset: Includes images depicting actual everyday and culturally-relevant phenomena, ranging from exams and invoices to public signage, social memes, and informal photographs. This subset incorporates a range of resolutions, aspect ratios, and environmental artifacts (noise, lighting, perspective), mirroring practical OCR scenarios.

Figure 3: Samples from In-The-Wild subset demonstrating real-world variability and cultural specificity for pt-PT.

• Synthetic Subset: Generated by compositing pt-PT text from Wikipedia and parliamentary transcripts onto diverse backgrounds using domain-aware layouts and contrast-adjusted rendering. This subset biases toward distributions familiar to most text recognition models, facilitating controlled evaluation.

Figure 4: Synthetic subset samples illustrating standardized document layouts aligned with typical model training distributions.

Annotation combines an LVLM (Gemini 3.1 Pro Preview) for initial transcription, followed by exhaustive review and correction by native pt-PT speakers (17 contributors), ensuring high fidelity reference texts. For handwritten and in-the-wild sets, bounding boxes and page segmentation are extracted automatically to augment granularity, whereas synthetic subsets leverage ground-truth transcripts.

Figure 5: Annotation platform supporting visualization, redaction, transcription correction, validation, and sample iteration.

Statistical Properties and Data Diversity

PorTEXTO features 779 finalized samples with broad diversity: 121 handwritten pages, 351 region-level crops, 107 in-the-wild images, and 200 synthetic images. The dataset contains 53,479 words, averaging 68.7 per sample. Distributions of transcription length and resolution reflect the wide range of pt-PT visual contexts, testing both layout robustness and linguistic comprehension.

Figure 6: Distributions of transcription length and source image resolution illustrating the diversity inherent in PorTEXTO.

The curation process involved careful filtering and correction, with substantial rejection and redaction rates, attesting to stringent data quality standards critical for benchmark reliability.

Evaluation Protocol and Metric Selection

Evaluation leverages the lmms-eval framework for reproducible benchmarking. Models include state-of-the-art OCR models, first-tier LVLMs (≤12B parameters), and second-tier LVLMs (>12B parameters), covering a spectrum of architecture, resolution budgets, and training paradigms.

Performance metrics are ANLS (Average Normalised Levenshtein Similarity) and BLEU-1. ANLS quantifies character-level accuracy, robust to minor OCR errors, while BLEU-1 captures correct unigram alignment, tolerant to reading order variance—a necessity for diverse layout structures.

Model Performance Analysis

PorTEXTO reveals strong performance on synthetic samples across all evaluated models, but a pronounced performance drop for in-the-wild and handwritten subsets, especially for generic OCR models. LVLMs demonstrate significantly superior resilience to the challenges posed by culturally nuanced pt-PT sources.

Noteworthy findings:

• Model Architecture and Data Mixing: Qwen3.5-9B leads first-tier LVLMs, Qwen3.6-27B and Gemma4-31B-it dominate the second tier. Performance gains are not linearly correlated with resolution budget or model size, suggesting that specialized multilingual training data is the primary driver for robust pt-PT OCR.
• Open Weights vs Closed Training Data: Fully open models (Molmo2-8B, PerceptionLM-8B) consistently underperform, indicating the detrimental effect of lacking pt-PT-centric OCR data. By contrast, models trained with focused multilingual corpora attain best results.
• Subset Difficulty: Synthetic subset results mask real-world fragility; in-the-wild and handwritten tasks, with high cultural and orthographic complexity, are not adequately solved by mainstream OCR or even some LVLMs.
• Benchmark Transferability: Cross-benchmark regression between PorTEXTO and OCRBench (EN/ZH) exposes minimal transferability ($R^2=0.05$); high-resource performance does not predict success in low-resource, culturally specialized domains.

Figure 7: Mean ANLS comparison between OCRBench (EN/ZH) and PorTEXTO, highlighting negligible correlation and underscoring dataset specialization.

Reasoning Augmentation in LVLMs

Explicit reasoning—when enabled—leads to modest performance gain in ANLS for most unified models, especially in the Handwritten (Region) subset. However, mixed or even negative effects can arise in the most challenging subsets, suggesting interaction between reasoning skills and the visual-linguistic complexity of pt-PT inputs. This warrants further investigation into reasoning strategies for linguistically challenging OCR domains.

Implications and Future Directions

PorTEXTO’s findings underscore:

• Data-Centric Bottlenecks: Progress in pt-PT visual text extraction relies on the availability of open, high-quality pt-PT-specific OCR training data. Model scaling and architectural innovation are secondary to targeted multilingual data mixing for this task.
• Benchmark Necessity: PorTEXTO provides a realistic, diverse ground-truth for pt-PT OCR, contrasting synthetic and historical alternatives, and highlights real-world fragilities in existing approaches.
• Transfer Limitations: OCR model generalization from high-resource languages to pt-PT is empirically limited. Domain adaptation and targeted multilingual training remain necessary.
• Future Research Opportunities: Research should prioritize dataset expansion, annotation quality, and reasoning-aware OCR modeling for low-resource, culturally-specific tasks. Robustly enabling extraction in handwritten and environment-rich scenarios remains unsolved.

Conclusion

PorTEXTO introduces a comprehensive, realistic benchmark for visual text extraction in European Portuguese, spanning handwritten, in-the-wild, and synthetic scenarios. The analytical results demonstrate sharp limitations in mainstream OCR approaches for pt-PT, highlight the importance of specialized data, and establish a foundation for future exploration of open multilingual OCR datasets and advanced multimodal models. PorTEXTO is an essential step for enabling robust, culturally-appropriate OCR solutions in low-resource language contexts.