---
title: 'DualGPT: A Hybrid Pixel-Text Model'
url: https://www.emergentmind.com/topics/dualgpt
type: topic
---

# DualGPT: A Hybrid Pixel-Text Model

DualGPT is a hybrid pixel-text autoregressive model used to test whether visual rendering of text as images can bypass tokenization constraints in pixel-based language modeling. In the experimental setting studied in "Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Models" [2602.06973], DualGPT is pretrained on text, images, and paired text-image data, and then fine-tuned for image-to-text transliteration. Its significance lies in the coexistence of two representational pathways: pixel-based rendering, which appears script-agnostic at the input level, and text-tokenizer-based autoregressive modeling, which reintroduces segmentation assumptions at the output side. Within that hybrid design, DualGPT serves as evidence that visual rendering does not decouple a model from tokenization constraints once a text tokenizer remains embedded in the architecture.

## 1. DualGPT as a hybrid pixel-text autoregressive architecture

DualGPT is not a pure pixel-only model. In the setup examined in [2602.06973], it combines pixel-based rendering of text into images with text-tokenizer-based autoregressive modeling. The model is therefore structurally hybrid: it includes a visual pathway that can represent arbitrary scripts through rendering, but its generation pathway still depends on discrete tokenization.

This architectural composition is central to the research question. Pixel-based language modeling is often motivated by the possibility of avoiding the sub-word tokenization bottleneck, especially for scripts that are poorly served by mainstream tokenizers. DualGPT provides a realistic test case because it retains the visual modality while also reintegrating a text tokenizer to improve autoregressive performance. The empirical conclusion is that once the tokenizer is reintroduced, the model again becomes sensitive to tokenizer design.

The paper frames this as a challenge to a common assumption in multimodal and pixel-based language modeling. Visual rendering is attractive because any script can be rendered, but autoregressive decoding still frequently relies on a tokenizer. DualGPT demonstrates that the visual side does not neutralize tokenizer effects in such a hybrid configuration. A plausible implication is that architectural multimodality does not, by itself, guarantee representational equity across scripts.

## 2. Task formulation, languages, and data regime

The evaluation task is image transliteration: the model receives rendered script images and must produce the correct text form [2602.06973]. This task is a direct probe of whether the model preserves surface form across script representations, making it especially suitable for testing script-tokenizer interactions.

The experiments focus on four Indonesian low-resource local languages that use non-Latin scripts: Javanese (`jav`), Balinese (`ban`), Sundanese (`sun`), and Lampungnese or Lampung (`ljp`). These languages were selected because tokenizer-script mismatch is especially visible in low-resource settings with non-Latin orthographies.

| Language | Train | Eval |
|---|---:|---:|
| Javanese | 400,726 | 816 |
| Sundanese | 293,933 | 823 |
| Balinese | 54,017 | 450 |
| Lampung | 945 | 84 |

Training data come from Wikidumps (July 2025) and digitized folklore for Javanese, Sundanese, and Balinese, while NusaAksara is used for evaluation. Lampung is split into train and eval from NusaAksara. The scale differences across languages are substantial, especially the contrast between Javanese and Lampung, and this suggests that tokenizer effects are being examined in a regime where data scarcity and orthographic specificity interact.

## 3. Rendering pipeline and tokenizer comparison

To isolate tokenization effects from rendering artifacts, the study uses a continuous rendering strategy rather than separating words with whitespace, particularly because Javanese and Balinese do not reliably mark word boundaries with spaces [2602.06973]. The visual renderer is a custom Pillow-based renderer rather than the default Pygame renderer, specifically to correctly render diacritics and additional characters. This design choice matters because rendering errors could otherwise confound any conclusion about tokenizer quality.

The comparison centers on script-tokenizer alignment. A tokenizer is treated as aligned when it segments a script in a way that matches the language’s orthography and character structure, and as misaligned when it fragments the script badly into many subword pieces or otherwise unnatural segmentations. In the paper’s framing, misalignment can lead to excessive subword fragmentation, degraded embeddings, poor generation quality, and weak transliteration performance.

Two tokenization regimes are compared. The first is the Llama 2 tokenizer, which is the default tokenizer used in PixelGPT/DualGPT-style modeling, is BPE-based, and has broad vocabulary coverage. The second is a custom tokenizer that is grapheme-based and designed to better fit the local scripts. The custom tokenizer is created by back-transliterating Latin training text into the local script, tokenizing at the grapheme level, transliterating back to Latin, and then applying word-level tokenization for token IDs. The contrast is therefore not merely between two vocabularies, but between two segmentation philosophies: broad-coverage subword tokenization versus script-sensitive grapheme-oriented tokenization.

## 4. Evaluation metrics and tokenizer-efficiency diagnostics

The main evaluation metric is chrF++, which the paper emphasizes because the dataset is low-resource and transliteration outputs may be short or morphologically unusual [2602.06973]. The appendix also reports BLEU and WER (Word Error Rate). chrF++ is preferred because it is character-sensitive and, according to the appendix, Balinese entries in NusaAksara are often very short, making BLEU and WER less stable or less informative.

The study also measures tokenizer behavior using OOV rate, fertility, and average text token length on training data. These are conventional efficiency metrics: OOV rate tracks vocabulary coverage, fertility measures how many tokenizer pieces are produced per word or unit, and average token length indicates how compressed or fragmented sequences become under a tokenizer.

A notable result is that the Llama 2 tokenizer appears stronger under these conventional diagnostics. It yields shorter sequences, lower fertility, and 0% OOV across all languages. By contrast, the custom tokenizers achieve near-0% OOV on their target languages but worse transfer to out-of-domain languages. The paper gives concrete examples: the Javanese tokenizer has near-0% OOV on Javanese but 22% OOV on Balinese, while the Sundanese tokenizer has near-0% OOV on Sundanese but 10% OOV on Lampung. It also reports tokenizer-space mismatch through vocabulary overlap, noting that the Balinese grapheme-based tokenizer has only 16.4% overlap with the Javanese tokenizer, computed as \( \frac{4,994}{30,346} = 16.4\% \).

These diagnostics establish the paper’s central paradox. Standard tokenizer-efficiency measures suggest that the Llama 2 tokenizer should be preferable, yet downstream transliteration performance shows the opposite.

## 5. Empirical results on transliteration performance

In monolingual DualGPT training and fine-tuning, the custom tokenizer substantially outperforms the Llama 2 tokenizer [2602.06973]. The reported chrF++ gains are \(+30.15\) for Javanese, \(+20.45\) for Balinese, \(+17.40\) for Sundanese, and \(+1.8\) for Lampung. These gains constitute the paper’s primary empirical finding.

The appendix-level examples reinforce the magnitude of the effect. On Javanese evaluation, the Llama 2 tokenizer obtains 64.83 chrF++, whereas the custom Javanese tokenizer reaches 94.98. On Sundanese evaluation, Llama 2 obtains 78.19 and the custom Sundanese tokenizer reaches 95.59. For Lampung, the Llama 2 tokenizer yields 0.77, while the custom Sundanese tokenizer yields 2.57. For Balinese, the paper reports both 4.10 versus 10.94 in one setup and 43.91 versus 64.36 in another.

The multilingual setting preserves the same qualitative conclusion. Multilingual exposure helps cross-lingual transfer somewhat, but tokenizer alignment still dominates. The paper highlights Java \(\rightarrow\) Bali as an especially notable case, improving to 69.2 chrF++ with the custom tokenizer, while performance with Llama 2 remains much lower.

The central comparative pattern is therefore stable across settings: the tokenizer that looks worse under OOV and fertility can perform much better in the actual transliteration task. This suggests that coverage-oriented metrics do not adequately characterize tokenization quality for non-Latin, low-resource scripts in hybrid pixel-text models.

## 6. Interpretation, misconceptions, and broader significance

The principal misconception addressed by DualGPT is the idea that pixel rendering automatically bypasses tokenization [2602.06973]. The paper rejects that claim for hybrid architectures. Once a text tokenizer is reintroduced, the model again depends on how that tokenizer splits the script. Misaligned tokenization produces poor text embeddings, and visual features do not compensate enough to eliminate the degradation.

This is why the low-OOV result for the Llama 2 tokenizer does not translate into better downstream performance. The paper’s interpretation is that vocabulary coverage and lower fertility do not measure whether a tokenizer is actually well matched to the script. A tokenizer may have 0% OOV and still be a poor representational fit because it fragments the script in ways that harm embedding quality and autoregressive modeling. The issue is therefore not only coverage but representation quality.

The broader implication is methodological as much as architectural. DualGPT shows that tokenizer choice must be evaluated as carefully as model architecture in multimodal and pixel-based systems. The paper presents this as a warning for future multimodal variants: text tokenizers remain a significant barrier to equitable models, and better script-aware tokenizers may be necessary for multilingual performance that does not systematically disadvantage low-resource scripts. In that sense, DualGPT functions less as proof that visual rendering solves tokenization and more as a demonstration that hybrid pixel-text models can inherit the same tokenizer inequities they were partly intended to avoid.

Source: https://www.emergentmind.com/topics/dualgpt