Dialect-Token Training

• Dialect-Token Training is a method that embeds explicit dialect identifiers—such as tags or feature labels—into the training signal.
• The approach leverages various conditioning formats, including prepended control codes and structured side strings, to guide model outputs in tasks like ASR and text generation.
• Empirical evidence shows that this training method improves control metrics and fidelity while reducing leakage into dominant standard varieties.

Dialect-Token Training denotes a family of dialect-aware learning schemes in which dialect identity, dialect features, or dialect-bearing discrete tokens are made explicit in the training signal rather than being left entirely to implicit inference. In contemporary practice, this explicit signal can take the form of prepended control tags such as "<HIJAZI>" or "<district>", output-side dialect symbols such as "<en-gb>", structured side strings such as "language: {language_tag}", token-level dialect labels in code-switching corpora, or routing signals derived from dialect-specific keywords in sparse Mixture-of-Experts systems. Across LLM generation, IPA transcription, ASR, metric learning, and token-based speech conversion, the common objective is to learn a dialect-aware conditional distribution—typically $p(y \mid x,d)$—that improves controllability, reduces leakage into dominant standard varieties, and preserves task fidelity under dialectal variation (Barmandah, 19 Aug 2025, Islam et al., 2024, Li et al., 2017, Sun et al., 2022, Pourreza et al., 2024).

1. Conceptual scope and historical positioning

The survey literature describes the core idea as conditioning models on an explicit dialect indicator $d$ at training and inference so that both understanding and generation become dialect-aware. Typical realizations include prepended sequence-level tags for monodialectal inputs, segment-level tags for code-switching, and token-level features that encode vernacular morphology. In this framing, dialect-token training is not restricted to one architecture family; it is a conditioning principle that can be integrated into encoder-only, encoder-decoder, decoder-only, adapter-based, or adversarial setups (Joshi et al., 2024).

Earlier Arabic dialect identification work already treated dialect at token granularity, especially in Linguistic Code Switching. There, token-level labels such as MSA, EGY, GLF, and LEV were predicted from LLMs, dictionary lookup, phonological rules, and morphological analysis, then aggregated upward to sentence- or document-level decisions. That literature is important because it shows that explicit dialect supervision at token level predates modern control-code prompting, even though appending special control tokens to raw inputs was not standard in the surveyed ADI systems (Althobaiti, 2020).

A second boundary case is dialect feature detection. In work on Indian English, the conditioning signal is not a dialect name but a feature description, using inputs of the form [CLS] feature description/name [[SEP](https://www.emergentmind.com/topics/semantic-entropy-production-sep-metric)] utterance [SEP] to detect constructions such as copula deletion, focus “only,” or invariant tags. This broadens the notion of dialect-token training from regional labels to interpretable dialect-feature conditioning, while remaining a sequence-level conditioning scheme over explicit linguistic metadata (Demszky et al., 2020).

2. Conditioning formats and token design

The defining implementation choice in dialect-token training is how dialect information is encoded and where it enters the model. The literature shows substantial heterogeneity: some systems use ordinary text passed through an unchanged tokenizer, some expand the vocabulary with new tokens, some place dialect symbols in the output sequence, and some avoid explicit indicator tokens while still using token-level dialect signals.

Source	Dialect signal	Implementation detail
Saudi-Dialect-ALLaM (Barmandah, 19 Aug 2025)	"<HIJAZI>", "<NAJDI>", also "<DIALECT=HIJAZI>", "<DIALECT=NAJDI>"	Prepended at the very beginning of the instruction; passed through the tokenizer as ordinary text without modifying the vocabulary
Bengali DGT (Islam et al., 2024)	"<district>"	Added at the very beginning of each input sentence; inserted into the tokenizer’s vocabulary as single units and the embedding layer is resized
Multi-dialect LAS (Li et al., 2017)	Dialect symbols such as "<en-gb>"	Added to the output label inventory and inserted into the target grapheme sequence, with end-of-sequence placement preferred
NANO (Sun et al., 2022)	"language: {language_tag}"	Structured side string retained in pretraining and metric fine-tuning, with dialect-specific or general language tags
Dolphin-CN-Dialect (Meng et al., 9 May 2026)	Prefix tokens such as "<ANHUI>", "<WU>", "<MINNAN>"	Used alongside task tokens such as "<asr>"; 80 extensible dialect token slots are reserved
SQL-GEN (Pourreza et al., 2024)	No explicit dialect indicator token	Dialect-specific keywords/functions initialize MoE gates and drive routing during SQL decoding
Whisper ADI label mapping (Radhakrishnan et al., 2023)	No tokenizer change in the PEL setup	Existing language tokens are reused as dialect proxies through a many-to-one hard-label mapping

These alternatives are not merely formatting differences. In Saudi-Dialect-ALLaM, treating the bracketed tag as ordinary text avoids tokenizer surgery and makes the dialect signal a plain-text control code. In Bengali District Guided Tokens, by contrast, the district marker is explicitly inserted as a single vocabulary item, with corresponding embedding-layer adjustment. LAS places the dialect symbol on the target side so that the decoder learns dialect jointly with grapheme prediction, whereas Dolphin-CN-Dialect uses prefix conditioning on the textual side of a joint CTC–AED ASR model. SQL-GEN departs further still: it does not append a dialect tag to the input, but initializes MoE gate weights from dialect-specific SQL keywords such as QUALIFY, DISTINCT ON, or PRAGMA, so that routing is activated when those tokens arise during decoding (Barmandah, 19 Aug 2025, Islam et al., 2024, Li et al., 2017, Meng et al., 9 May 2026, Pourreza et al., 2024).

A central misconception is therefore that dialect-token training always means “add a special token to the prompt.” The empirical record is narrower and more varied: some systems use special tokens, some use ordinary strings, some predict dialect symbols as outputs, and some infer dialectal routing from ordinary task tokens while never exposing an explicit dialect identifier at all (Sun et al., 2022, Radhakrishnan et al., 2023).

3. Objectives, architectures, and optimization patterns

The generic formalization is conditional modeling with an explicit dialect variable:

$$p(y \mid x,d)=\prod_{t=1}^{T} p(y_t \mid y_{<t},x,d).$$

For seq2seq and generation tasks, the usual training objective is teacher-forced cross-entropy,

$$\mathcal{L}=-\sum_{t=1}^{T}\log p_\theta(y_t \mid y_{<t},x,d),$$

with the dialect signal realized either as a prepended tag, a target token, or an auxiliary conditioning vector (Joshi et al., 2024).

Saudi-Dialect-ALLaM instantiates this directly in a decoder-only causal LM. Under Dialect-Token training, the instruction $x$ is prepended with a Hijazi or Najdi tag and the model optimizes a dialect-conditioned causal LM objective; under No-Token training, the same pairs are presented without the tag. The model is ALLaM-7B-Instruct-preview, LoRA is applied to projection layers while the backbone is frozen, and the paper reports that explicit conditioning on $d$ accelerates convergence and lowers validation loss relative to implicit dialect inference (Barmandah, 19 Aug 2025).

District Guided Tokens for Bengali IPA transcription implement the same logic in an encoder-decoder setting. The encoder consumes "<district> {contents}", the decoder predicts the IPA sequence, and training uses standard teacher-forced cross-entropy. Because the DGT tokens are added as single vocabulary entries, the embedding matrix is resized to accommodate them. The models evaluated include ByT5, mT5, BanglaT5, and umT5, with ByT5 distinguished by byte-level operation under high OOV conditions (Islam et al., 2024).

In the multi-dialect LAS ASR model, dialect-token training is combined with dialect-vector conditioning. The dialect symbol can be inserted at the beginning or end of the grapheme target sequence, and an 8D dialect vector—either 1-hot or learned embedding—can be injected into encoder and/or decoder layers as a dialect-dependent bias. The paper’s preferred formulation keeps the token at sequence end, so that early grapheme prediction is not forced to depend on an initial dialect decision (Li et al., 2017).

NANO uses a different objective class. Rather than conditioning a generator, it conditions a learned evaluation metric by appending a structured language/dialect string and training an mT5-based regressor to judge dialect/language acceptability with an RMSE objective over binary labels. The point is not output generation but making metric scores dialect-robust and, when tags are supplied at inference, dialect-aware (Sun et al., 2022).

Sparse routing introduces another architectural variant. SQL-GEN initializes MoE gate weights from hidden states of dialect-specific SQL keywords, then uses standard sparse routing:

$p(e_i \mid x)=\mathrm{softmax}_i(W_g h(x)+b_g).$

The full objective is sequence-to-sequence cross-entropy plus a load-balancing term. Here the “token” is not a separate dialect prefix but the dialect-specific SQL lexeme that steers expert selection during decoding (Pourreza et al., 2024).

Speech work extends the term further. TokAN performs accent normalization entirely over self-supervised discrete speech tokens extracted by a jointly trained VQ tokenizer on top of a frozen WavLM-Large feature extractor. An autoregressive encoder-decoder maps L2-accented token sequences to L1-native token sequences, then a flow-matching synthesizer reconstructs Mel-spectrograms conditioned on the source speaker embedding. This usage broadens dialect-token training from symbolic text tags to dialect-bearing discrete speech token sequences (Bai et al., 4 Jul 2026).

4. Empirical effects across applications

The clearest text-generation evidence comes from Saudi-Dialect-ALLaM. Relative to the base ALLaM-7B-Instruct-preview, the Dialect-Token model raises the Saudi rate from 47.97% to 84.21% and reduces MSA leakage from 32.63% to 6.21%. Fidelity also improves: chrF++ rises from 21.27 to 24.80 and BERTScore F1 from 0.6796 to 0.7386. Relative to the No-Token LoRA variant, explicit tags further improve control from 80.50% to 84.21% Saudi alignment and reduce MSA leakage from 9.26% to 6.21%, with slight fidelity gains. The same study also reports that both LoRA variants outperform Falcon-7B-Instruct, Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, AceGPT-v2-8B-Chat, and JAIS-13B-Chat in dialect control and fidelity while avoiding metadata-tag echoing frequently observed in those baselines (Barmandah, 19 Aug 2025).

In Bengali text-to-IPA transcription, ByT5 with District Guided Tokens obtains Public 1.995% and Private 2.072% WER, outperforming umT5, BanglaT5, and mT5. The paper attributes the strong result to DGT plus byte-level robustness under high OOV rates, noting that 4,926 words in the hidden test set are unseen in training, approximately 46.97% of test-set unique words. At the same time, the work explicitly states that it does not provide a no-DGT ablation, so the exact magnitude attributable to district conditioning alone cannot be isolated from architecture and tokenization effects (Islam et al., 2024).

For multi-dialect speech recognition, the single LAS model shows that token placement matters. Inserting the dialect symbol at sequence end outperforms placing it at sequence start, and combining encoder and decoder dialect-vector conditioning yields the best overall WER. Across seven English dialects, the resulting single model outperforms individually fine-tuned dialect-specific LAS systems with 3.1–16.5% relative WER reductions, the largest relative gain being on Australian English (Li et al., 2017).

Dialect-conditioned metric learning also shows measurable gains. In NANO, the regression coefficient difference $\phi_{\text{dialect}}-\phi_{\text{perturb}}$ improves after dialect/language acceptability pretraining: for mT5_XL, English goes from 0.01 to 0.06, Portuguese from 0.02 to 0.05, and Mandarin from 0.02 to 0.05. Dialect-vs-perturb win rates improve most strongly in Portuguese and Mandarin at XL scale, while standard benchmark performance is preserved or improved relative to finetuning-only at that scale (Sun et al., 2022).

In Chinese ASR, dialect-token conditioning is combined with temperature-based sampling and tokenizer redesign. Dolphin-CN-Dialect reports a 38% improvement in dialect recognition accuracy and a 16.3% relative overall CER reduction compared to the previous Dolphin, while standard Mandarin performance sees only about 0.2% degradation. This result is not attributable to tokens alone, but it demonstrates that explicit dialect markers can be integrated into a larger data-centric balancing strategy (Meng et al., 9 May 2026).

5. Evaluation methodology and representational interpretation

Evaluation protocols for dialect-token training vary sharply by task, but a recurring theme is the need to separate dialectal surface variation from semantic or task failure. Saudi-Dialect-ALLaM uses an external five-way MARBERTv2 Arabic Written Dialect Classifier, treating GLF as a practical proxy for Saudi usage, and reports Saudi rate, MSA leakage as mean classifier probability of MSA, low-confidence rate, chrF++, BERTScore, distinct-n, and Self-BLEU. The diversity results are instructive: No-Token has slightly higher distinct-2/3 and lower Self-BLEU than Dialect-Token, suggesting a mild lexical concentration trade-off in exchange for tighter dialect control (Barmandah, 19 Aug 2025).

NANO addresses a different problem: evaluation metrics themselves can penalize dialect variation more than real semantic perturbations. It therefore formalizes dialect robustness as invariance to dialectal surface change under fixed meaning, and dialect awareness as preference for the candidate matching a user-specified dialect tag. The proposed test suite uses mixed-effects regression and win/loss rates with one-tailed binomial tests to compare dialectal rewrites against genuine semantic perturbations. This moves dialect-token training beyond generation and into the meta-level question of how dialect-aware systems should be evaluated (Sun et al., 2022).

Work on literary orthographic variation clarifies why evaluation is difficult. Character-level models such as CANINE-c and CANINE-s best capture the “character-edit” channel, while token-level models such as BERT better preserve word-level and context-level semantics. The paper argues that the “dialect effect” is distributed across orthography, phonology as reflected through orthography, morphology, lexical choice, word-level semantics, and context-level semantics. No single modeling assumption surfaces all channels equally well (Messner et al., 2024).

Embedding-space analysis supplies an additional representational argument for explicit conditioning. In regional English embeddings trained with fastText, within-dialect nearest-neighbor overlap at $k=50$ ranges from 71.5% to 74.1%, whereas between-dialect overlap ranges from 19.7% to 26.8%, with paired $t$-tests significant at $d$0 across all pairings. This suggests that dialectal training data alters semantic neighborhoods far beyond baseline instability, and that explicit dialect conditioning can be understood as an attempt to control a real representational divergence rather than merely a stylistic nuisance (Dunn, 2023).

6. Limitations, misconceptions, and adjacent strategies

Dialect-token training is often presented as a simple control-code trick, but the literature makes three limitations clear. First, the token itself is rarely the whole intervention. Saudi-Dialect-ALLaM combines explicit tags with LoRA fine-tuning on a private synthetic dataset of 5,466 instruction–response pairs; its evaluation relies on a GLF proxy that does not perfectly separate Hijazi from Najdi, and the dataset, weights, and adapters are not released. The resulting gains are substantial, but they remain conditioned by dataset design, classifier proxy choice, and synthetic curation (Barmandah, 19 Aug 2025).

Second, explicit tags do not eliminate the need for correct labels. Dolphin-CN-Dialect recommends omitting dialect tokens when labels are missing or uncertain, because noisy dialect prefixes can induce cross-dialect confusion. The LAS experiments show the same principle from another angle: feeding mismatched dialect vectors degrades WER, more severely for encoder conditioning than decoder conditioning, which the paper interprets as evidence that acoustic variation across dialects is stronger than LM variation (Meng et al., 9 May 2026, Li et al., 2017).

Third, the effect of explicit conditioning is not always isolable. The Bengali DGT study does not include a no-DGT ablation, so its gains cannot be partitioned cleanly between district tokens, byte-level modeling, and the broader model choice. The literary-dialect corpus work likewise argues that tokenization scheme meaningfully changes which orthographic signals a model can surface, so “dialect-token training” should not be treated as separable from tokenizer and architecture assumptions (Islam et al., 2024, Messner et al., 2024).

There is also an adjacent family of methods that learns dialect without explicit tags. In Quebec French adaptation, compute-efficient continual pre-training with LoRA updates fewer than 1% of parameters and improves minority-dialect benchmarks under tight budgets, but it does so without inserting dialect-identifying control tokens during CPT. That work argues that dialect tokens could be layered on top later for controllability, rather than being a prerequisite for dialect acquisition. This positions dialect-token training as complementary to CPT+PEFT rather than universally superior (Khan et al., 26 Oct 2025).

Finally, token-level dialect supervision does not always take the form of a control tag. Arabic ADI literature often uses token-level labels, lexicons, LM scores, and morphological analyzer outputs as engineered dialect evidence rather than prompt-like tokens, while SQL-GEN routes tokens to dialect experts without ever appending a user-visible dialect indicator to the input. A plausible implication is that “Dialect-Token Training” is best understood as a broader design space for making dialect information explicit in the computational graph—through symbols, labels, side strings, routing cues, or discrete speech units—rather than as a single prompt-formatting recipe (Althobaiti, 2020, Pourreza et al., 2024).