MUDIDI: Multilingual Dictionary Digitization
- MUDIDI is a benchmark dataset and two-stage framework that transforms scanned multilingual dictionaries with complex layouts and historical orthographies into machine-readable lexicographic data.
- It decomposes the digitization process into faithful OCR transcription and structured lexicographic parsing according to SIL’s MDF schema, ensuring accurate entry segmentation and field mapping.
- The evaluation shows that general-purpose LLMs outperform traditional OCR systems in both transcription fidelity and parsing accuracy, supporting cost-effective workflows for endangered language documentation.
Searching arXiv for MUDIDI and closely related dictionary digitization work. arXiv search query: "MUDIDI multilingual dictionary digitization" MUDIDI is a benchmark dataset and a two-stage framework for converting scanned multilingual dictionaries into machine-readable lexicographic data. It targets a document class that is especially difficult for OCR-centric pipelines: multilingual dictionaries with language-specific scripts, dense abbreviations, cross-references, multi-column layouts, historical orthographies, and lexicographic conventions that are often explained only in introductory matter. The framework decomposes digitization into faithful page transcription and lexicographic parsing, and the accompanying dataset comprises human-annotated material from 30 public-domain dictionaries chosen for diversity of writing systems, language families, and formats (Setiawan et al., 8 Jun 2026).
1. Conceptual scope and problem formulation
MUDIDI treats dictionary digitization as a structured document-understanding problem rather than a pure OCR task. In its formulation, a scanned page image must first be transcribed with character fidelity and lightweight typography preservation, and then reorganized into dictionary entries mapped into SIL’s Multi-Dictionary Formatter (MDF). This decomposition is explicit: Stage 1 receives a page image and outputs a Unicode transcription in fixed line order with <b> and <i> tags preserved when possible; Stage 2 uses the gold Stage 1 transcript together with the page image to produce ordered entries in an MDF-compatible schema (Setiawan et al., 8 Jun 2026).
The target lexicographic object is formalized as a structured representation consisting of ordered entries, with an entry containing fields from a lexicographic inventory such as
The paper also illustrates MDF markers including \lx headword, \va variant form, \hm homonym number, \sn sense number, \gn gloss, \un usage note, and \se subentry. This representation makes clear that the second stage is not merely segmentation. It requires field typing, recovery of entry-internal structure, and compatibility with an existing lexicographic interchange format (Setiawan et al., 8 Jun 2026).
This two-stage design is diagnostically important. If a system fails in Stage 1, the error concerns character recognition, read order, or markup preservation; if it fails in Stage 2, the error concerns interpretation of lexicographic structure. The separation suggests that dictionary digitization accuracy should be analyzed along at least two axes: transcription fidelity and lexicographic parsing fidelity.
2. Corpus design and annotation architecture
The dataset is built from about 30 public-domain multilingual dictionaries, mainly from 19th- and early-20th-century materials, largely via HathiTrust. The Stage 1 set contains three annotated content pages per dictionary and covers 30 dictionaries and 30 source languages: Assyrian, Bengali, Canala (Xârâcùù), Chepang, Chukchi, Circassian (Adyghe), Efik, Evenki, Georgian, Gojri, Greek, Gujarati, Iñupiatun Eskimo, Japanese, Kashmiri, Khmer, Malay, Na (Mosuo), Nahuatl, Punjabi, Reel, Ritharngu, Sanskrit, Shilluk, Syriac, Telugu, Thai, Tiri, Vernacular Syriac, and Yiddish (Setiawan et al., 8 Jun 2026).
The Stage 2 subset is narrower because it requires lexicographic expertise. It contains 10 dictionaries: Chukchi–Russian, Circassian–English–Turkish, Efik–English, Evenki–Russian, Greek–English, Iñupiatun Eskimo–English, Kashmiri–English, Na–English–Chinese–French, Nahuatl–French, and Tiri–English. The dataset includes writing systems ranging from Bengali, Georgian, Gurmukhi, Syriac, Telugu, Thai, and Hebrew to Arabic-based orthographies, Kana+Kanji, Han+IPA mixtures, and cuneiform (Setiawan et al., 8 Jun 2026).
Stage 1 annotation follows a silver-to-gold workflow. Silver transcripts are generated with Gemini 3.5 Flash using the page image and a source-language alphabet list, then corrected by language experts in Label Studio. Annotation units are header, body, and footer, with multi-column layouts further separated by column. Gold OCR text is then linearized in canonical reading order: usually column-major, but row-wise when a single entry spans columns. Stage 2 also begins with silver data, using Gemini 3.1 Pro prompted with the gold Stage 1 transcript, the page image, and dictionary introduction pages where available. Annotators then correct an intermediate field-discovery JSON and the resulting MDF according to SIL MDF guidelines (Setiawan et al., 8 Jun 2026).
The annotation architecture therefore couples model-assisted bootstrapping with expert correction. That arrangement is consequential for low-resource lexicography, because fully manual creation of structured gold data at this scale would be substantially more expensive.
3. Stage 1: faithful transcription, read order, and markup preservation
Stage 1 is defined as faithful OCR rather than entry parsing. Its output schema contains three ordered lists—header, lines, and footer—and the prompt explicitly instructs the model not to interpret entries, summarize, merge lines, or fix typos. For multi-column pages, the required reading order is “the full left column top-to-bottom, then the next column, and so on,” while bold and italic text should be preserved with <b>...</b> and <i>...</i> when the model is confident (Setiawan et al., 8 Jun 2026).
The evaluated systems include specialized OCR or document models—MinerU2.5 Pro, PaddleOCR-VL-1.5, GLM-OCR, and Mathpix—and general-purpose models—Gemini 3 Flash, Gemini 3.1 Pro, GPT-5.5, Claude Opus 4.7, and Qwen3-VL-235B-A22B-Instruct. Because OCR outputs often merge or split lines, evaluation begins with alignment using a “quick match” procedure derived from OmniDocBench: normalized grapheme edit distances are computed at line granularity, adjacent predicted lines are greedily merged when NED improves, pairs are assigned by Hungarian matching, and remaining units are linked by fuzzy subset search (Setiawan et al., 8 Jun 2026).
The metrics are GCER, WER, TextEdit, Markup F1, and ReadOrderEdit. Character distances are computed on aligned units using Unicode grapheme clusters; unmatched gold or predicted lines count as errors. Markup F1 pools bold and italic matches after word-level alignment. ReadOrderEdit measures agreement with gold line order, with lower values indicating better order accuracy (Setiawan et al., 8 Jun 2026).
The aggregate results strongly favor general-purpose LLMs. Gemini 3.1 Pro with alphabet conditioning is the best overall Stage 1 model, with Edit $0.05$, GCER $0.05$, WER $0.14$, Markup F1 $0.64$, and Order $0.06$. Gemini 3 Flash is nearly as strong and slightly better on markup, while GPT-5.5 is especially strong on order accuracy. By contrast, Mathpix is the strongest conventional OCR baseline but is far weaker overall, with Edit $0.30$, GCER 0, WER 1, Markup F1 2, and Order 3 (Setiawan et al., 8 Jun 2026).
The difficult cases are not uniformly distributed across scripts. Assyrian cuneiform is reported as the hardest; Arabic-based and Syriac-family scripts also remain challenging, as do dictionaries mixing Thai with an obsolete Cyrillic-based transcription. The alphabet-list ablation is not uniformly beneficial. Aggregate gains for some LLMs are small, and dictionary-level effects may be positive, neutral, or negative. OCR-hint conditioning using Mathpix output usually degrades performance on average, apparently because models anchor on OCR errors. This suggests that Stage 1 performance is highly sensitive to dictionary-specific script ecology rather than only to generic OCR capacity (Setiawan et al., 8 Jun 2026).
4. Stage 2: entry segmentation and MDF mapping
Stage 2 assumes gold Stage 1 transcripts and evaluates lexicographic interpretation in isolation from OCR noise. The model is instructed to copy characters exactly from the transcript and use the page image only for structure. The only permitted modifications are structural: removal of inline markup, rejoining of hyphenated line breaks, and splitting or merging of spans across MDF fields (Setiawan et al., 8 Jun 2026).
The pipeline has two passes. Pass 1 runs once per dictionary and infers parse-rules from dictionary introduction pages and one sample page. Its output is a JSON specification of MDF markers, abbreviations, and structural conventions. Pass 2 runs once per page and combines those parse-rules with the page snippet, the gold Stage 1 transcript, and optionally the MDF guidelines and introduction pages to generate MDF-formatted output. All Stage 2 experiments use extended reasoning, and the main models are Gemini 3.1 Pro, Claude Opus 4.7, GPT-5.5, and Qwen3-VL-235B-Thinking (Setiawan et al., 8 Jun 2026).
The evaluation uses three metrics: Entry Accuracy, MDF Field F1, and ReadOrderEdit at the entry level. Entry matching is based on normalized field-content similarity. Within matched entries, field lines are aligned with thresholded similarity matching; equivalent gloss or definition markers such as \ge vs \de, or \ge vs \gn for English, are treated as correct. Entry order is measured as normalized Levenshtein distance between predicted and gold entry sequences, with unmatched entries contributing insertions or deletions (Setiawan et al., 8 Jun 2026).
The main finding is that entry segmentation is almost solved for the strongest LLMs in this controlled setting, whereas field assignment remains the harder subproblem. Gemini 3.1 Pro is the strongest overall Stage 2 model. With both dictionary introduction and MDF guidelines, it reaches Entry Accuracy 4, MDF Field F1 5, and ROE 6. Claude and GPT-5.5 also achieve near-perfect entry accuracy but lower field F1, while Qwen3-VL trails substantially, particularly on field assignment (Setiawan et al., 8 Jun 2026).
Dictionary-level variation remains substantial. Chukchi is essentially solved for top models. Circassian, Kashmiri, Na, and Nahuatl are much more context-sensitive, and the introduction pages can be decisive because they explain abbreviations, part-of-speech keys, and entry conventions. A particularly informative diagnostic replaces automatically inferred parse-rules with gold parse-rules; on non-perfect dictionaries this raises F1 by about six points on average, from 7 to 8. This suggests that parse-rule induction, rather than page-level entry detection, is the primary residual bottleneck (Setiawan et al., 8 Jun 2026).
5. Benchmark configuration, practical guidelines, and workflow economics
MUDIDI is not presented only as a static benchmark; it also supplies an operational workflow. The Stage 1 prompt can be augmented with a source-language alphabet block and an OCR-reference block, while the Stage 2 workflow can be run in a direct MDF mode with dictionary-specific parse-rules. The repository linked in the paper provides the implementation path for these stages (Setiawan et al., 8 Jun 2026).
The practical recommendations are unusually specific. For Stage 1, Gemini 3 Flash is recommended as the default because it is nearly as accurate as Gemini 3.1 Pro, preserves markup slightly better, and is cheaper. The paper advises against always supplying an alphabet list: an initial run without the list is recommended, followed by dictionary-specific inspection for hallucinated or invalid characters. For Stage 2, the paper recommends including the SIL MDF guidelines and performing a brief human review of the automatically inferred parse-rules before rerunning the page-level parse. The empirical rationale is that correcting parse-rules is substantially cheaper than full MDF reannotation and fixes many recurring field-assignment errors (Setiawan et al., 8 Jun 2026).
The appendix also reports approximate per-page costs for end-to-end use: about \$Y$90.132 for Gemini 3.1 Pro, about \$e_i$01.635 for GPT-5.5. Stage 1 with Gemini 3 Flash alone is about \$0.005 per page. These figures matter because dictionary digitization projects often operate under archival or documentary budgets rather than commercial OCR budgets (Setiawan et al., 8 Jun 2026).
This operational emphasis implies that MUDIDI is intended not only for leaderboard comparison but also for deployment in actual lexicographic recovery workflows, especially where public-domain scans remain the primary lexical record for endangered or under-digitized languages.
6. Limitations, interpretive significance, and broader role
The paper is explicit about several limitations. Stage 1 metrics do not separate source-language errors from target-language errors, even though many pages are bilingual or multilingual. Stage 1 and Stage 2 are evaluated independently, so the benchmark does not yet quantify how OCR errors propagate into lexicographic parsing. Nor does it test whether Stage 1 markup preservation directly improves Stage 2 parsing quality (Setiawan et al., 8 Jun 2026).
Additional constraints are intrinsic to the benchmark design. Stage 2 uses gold transcripts, which isolates structure recovery but also removes a major source of end-to-end noise. The Stage 2 subset contains only 10 dictionaries, reflecting the cost of expert MDF annotation. Moreover, the hardest residual cases are precisely those where dictionary-specific conventions dominate: under-digitized scripts, obsolete orthographies, dense abbreviations, and idiosyncratic field inventories. This suggests that even strong general-purpose LLMs do not eliminate the need for dictionary-aware metadata and light expert correction.
Within those limits, MUDIDI establishes a technically clear result: current general-purpose LLMs outperform conventional OCR systems and specialized document VLMs on multilingual dictionary transcription, and they also perform strongly on lexicographic parsing when supplied with the relevant contextual aids. Its broader significance lies in reframing multilingual dictionary digitization as a coupled OCR-and-structure problem with separate failure modes, measurable stage-specific metrics, and practical interventions such as alphabet conditioning, introduction-page conditioning, and parse-rule review. For endangered-language documentation, that reframing is consequential because it turns legacy dictionary scans into candidates for machine-readable lexical infrastructure rather than static archival images (Setiawan et al., 8 Jun 2026).