HingMBERT: Code-Mixed mBERT Variant

• HingMBERT is a Hindi–English code-mixed transformer that refines mBERT by leveraging real-world Hinglish text from social media via masked language modeling.
• It preserves the standard mBERT architecture while undergoing continued pretraining on 52.93 million code-mixed sentences, achieving lower perplexity and enhanced performance.
• The model excels in downstream tasks like NER, LID, and sentiment analysis, demonstrating higher recall and outperforming non-code-mixed baselines in code-switching scenarios.

Searching arXiv for recent and foundational papers on HingMBERT and closely related Hinglish code-mixed modeling. HingMBERT is a Hindi–English code-mixed transformer LLM derived from multilingual BERT and adapted to Hinglish, especially Roman-script social-media text. In the literature, the term has two closely related uses. In the foundational L3Cube work, HingMBERT denotes a multilingual BERT model further pre-trained on L3Cube-HingCorpus, a large real-world Hinglish corpus, using masked language modelling (Nayak et al., 2022). In a later comparative study of code-mixed named entity recognition, HingMBERT is defined more generally as “a multilingual BERT model further trained on code-mixed data to enhance contextual understanding,” and is evaluated as a code-mixed-specific NER encoder fine-tuned for BIO tagging over person, organization, and location entities (Shirke et al., 2 Sep 2025). Across these usages, the common core is a domain-adapted mBERT checkpoint specialized for Hindi–English code-switching rather than a new architecture.

1. Definition and taxonomic position

HingMBERT occupies an intermediate position between general multilingual encoders and fully task-specific Hinglish systems. In the L3Cube model family, it is one of several code-mixed pretrained models released alongside HingBERT, HingRoBERTa, and HingGPT, with the distinguishing property that its initialization comes from multilingual BERT rather than English BERT or XLM-RoBERTa (Nayak et al., 2022). In the 2025 NER comparison, it is explicitly grouped under “Code-Mixed Specific Models,” in contrast to non-code-mixed baselines such as BERT Base Cased, RoBERTa, MuRIL, and IndicBERT (Shirke et al., 2 Sep 2025).

This distinction is consequential because vanilla mBERT, while multilingual, was not pretrained on real Hinglish code-switching. The L3Cube paper frames HingMBERT as mBERT continued on 52.93 million real Hindi–English code-mixed sentences in Roman script, specifically to address the mismatch between multilingual pretraining corpora and actual Hinglish usage on Twitter (Nayak et al., 2022). The later NER study adopts the same conceptual definition: HingMBERT is not the original released mBERT, but a domain-adapted variant intended to improve contextual understanding in informal, transliterated, and frequently code-switched text (Shirke et al., 2 Sep 2025).

A plausible implication is that HingMBERT should be understood less as a single immutable checkpoint name than as a model class: mBERT plus continued pretraining or downstream adaptation on Hinglish. This broader interpretation is reinforced by sentiment-classification work that does not use the exact name but implements a “HingMBERT-style” system by fine-tuning bert-base-multilingual-cased on Hinglish tweets for three-way sentiment classification (Garg et al., 8 Jan 2026).

2. Architecture and pretraining regime

Architecturally, HingMBERT preserves the standard mBERT encoder configuration: 12 transformer layers, hidden size 768, 12 attention heads, and multilingual WordPiece tokenization (Nayak et al., 2022). The L3Cube paper is explicit that HingMBERT is “mBERT → continued pretraining on 1B-token Hinglish corpus → HingMBERT,” with no architectural redesign beyond the inherited BERT-base/mBERT structure (Nayak et al., 2022). The 2025 NER study likewise states that the architecture is not modified for NER beyond adding a standard token-classification head (Shirke et al., 2 Sep 2025).

The pretraining objective in the foundational work is masked language modelling only, without next sentence prediction. A random 15% of tokens are selected for masking, and the model predicts the original token at masked positions under the standard MLM loss (Nayak et al., 2022). Pretraining starts from released pretrained mBERT weights and runs for 2 epochs on L3Cube-HingCorpus with batch size 64 and learning rate $1 \times 10^{-5}$, with the authors reporting that the models converged within 2 epochs (Nayak et al., 2022).

The principal resource enabling HingMBERT is L3Cube-HingCorpus. That corpus contains 52.93 million sentences and 1.04 billion tokens scraped from Twitter, filtered to retain code-mixed Hindi–English tweets in Roman script using token-level language identification and the rule that a sentence must contain at least 2 Hindi words and 2 English words (Nayak et al., 2022). Its measured code-mixing intensity is $\mathrm{CMI} \approx 31.21$, indicating substantial switching (Nayak et al., 2022). The same work also introduces a mixed-script extension, HingMBERT-mixed, trained on both Roman and Devanagari versions of the corpus (Nayak et al., 2022).

Intrinsic evaluation in the L3Cube paper uses MLM perplexity. HingMBERT attains perplexity 5.20 and HingMBERT-mixed 5.22, compared with 5.72 for HingBERT and 7.82 for HingRoBERTa (Nayak et al., 2022). This suggests that the mBERT initialization remains particularly well suited to continued language modeling on large Hinglish corpora, even though downstream task leadership is more heterogeneous.

3. Tokenization, domain adaptation, and code-mixed representation

A central technical feature of HingMBERT is its reliance on subword modeling to cope with transliteration, noisy orthography, and lexical heterogeneity. Because it inherits mBERT’s multilingual WordPiece vocabulary, it can represent both English forms and Romanized Hindi through subword decomposition rather than monolithic token lookup (Nayak et al., 2022). The NER study emphasizes that this is especially relevant for code-mixed text, where many forms are effectively OOV-like under monolingual assumptions and are split into multiple wordpieces; only the first subword is assigned the original NER tag, while subsequent subwords are masked during loss computation (Shirke et al., 2 Sep 2025).

The sentiment-classification work on Hinglish tweets, while not using the canonical HingMBERT checkpoint, clarifies why this design is effective in practice. It argues that WordPiece tokenization helps handle Romanized Hindi, spelling variation, slang, and out-of-vocabulary terms, with examples such as “likhna” being segmented into subwords and spelling variants like “shukriya” and “shukria” sharing overlapping pieces (Garg et al., 8 Jan 2026). This suggests that HingMBERT’s value is not merely multilingual coverage in the abstract, but multilingual subword composition after exposure to realistic Hinglish contexts.

The contrast with non-code-mixed baselines is recurrent across papers. English BERT and RoBERTa are pretrained on BookCorpus, Wikipedia, or large English corpora and therefore lack explicit exposure to Romanized Hindi or switch points. Vanilla mBERT includes Hindi and English but largely in formal, separated, and non-code-switched contexts (Shirke et al., 2 Sep 2025). The 2023 comparative study of code-mixed Hindi–English classification generalizes this point by arguing that vanilla BERT and mBERT suffer from vocabulary mismatch, lack of code-mixed pretraining, and domain mismatch, whereas Hinglish-specific models better encode Romanized Hindi and mixed-language constructions (Patil et al., 2023).

The same logic appears outside NER. In hate-speech identification, code-mixed embeddings derived from HingCorpus—both HingBERT-family models and Hing-FastText—are reported to outperform vanilla English BERT and FastText on Hinglish hate-speech datasets, with the authors attributing the gain to better handling of “atypical words in scrambled text” and real Hinglish usage patterns (Jagdale et al., 2024). Although that paper does not reproduce detailed architectural specifications, it reinforces the general conclusion that domain-matched code-mixed pretraining matters more than merely selecting a multilingual backbone.

4. Downstream performance across tasks

HingMBERT has been evaluated on multiple downstream tasks, especially those collected in the GLUECoS benchmark. In the L3Cube paper, all downstream systems add a task-specific dense layer on top of the encoder and are fine-tuned for 5 epochs with batch size 64, learning rate $3 \times 10^{-5}$, and early stopping on validation F1 (Nayak et al., 2022). On Roman-script GLUECoS and L3Cube-HingLID, HingMBERT achieves F1 scores of 84.90 for LID, 89.47 for POS-UD, 71.55 for POS-FG, 80.09 for NER, 63.51 for sentiment, and 96.27 for HingLID (Nayak et al., 2022). Relative to vanilla mBERT, these are consistent improvements: for example, NER improves from 76.64 to 80.09 and sentiment from 58.42 to 63.51 (Nayak et al., 2022).

On mixed-script evaluation, HingMBERT reaches 95.68 for LID, 86.71 for POS-UD, 70.15 for POS-FG, 78.78 for NER, and 60.72 for sentiment, while HingMBERT-mixed improves some of these results, especially when target inputs combine Roman and Devanagari scripts (Nayak et al., 2022). The mixed-script results support the paper’s claim that including Devanagari during continued pretraining benefits mixed-script targets (Nayak et al., 2022).

The most detailed task-specific analysis for HingMBERT in the provided corpus concerns Hinglish NER. The 2025 comparative study evaluates code-mixed-specific models, non-code-mixed baselines, and a zero-shot Google Gemini setup on a benchmark Hinglish NER dataset containing 3,637 code-mixed sentences labeled with BIO tags for [PER](https://www.emergentmind.com/topics/preference-informed-edge-reweighting-per), ORG, LOC, and Other (Shirke et al., 2 Sep 2025). HingMBERT is fine-tuned as a supervised token-level classifier using Hugging Face Transformers and PyTorch, with hyperparameter search over 40 Optuna trials per model, AdamW, linear warmup over 10% of steps, max sequence length 128, batch size 16–32, early stopping on validation F1, FP16 training on NVIDIA T4 GPUs, and averages over 3 random seeds (Shirke et al., 2 Sep 2025).

The results show a nuanced performance profile.

Setting	Precision	Recall	F1
Validation	73.27	81.43	77.14
Test	75.80	82.21	78.87

On validation, HingMBERT achieves the highest F1-score, highest recall, and highest accuracy among all evaluated models, including HingBERT, HingRoBERTa, MuRIL, IndicBERT, and Google Gemini (Shirke et al., 2 Sep 2025). On test, HingBERT slightly surpasses it in F1, but HingMBERT remains second overall with the highest recall of any model at 82.21 (Shirke et al., 2 Sep 2025). The same study notes that HingMBERT outperforms all general-purpose BERT variants on both validation and test F1, with gains of roughly 1–2 points over the best non-code-mixed baseline, MuRIL, and exceeds Google Gemini’s zero-shot test F1 by about 16 points (78.87 vs. 62.24) (Shirke et al., 2 Sep 2025).

This profile suggests a recurrent trade-off: HingMBERT often exhibits especially strong recall, possibly reflecting the multilingual coverage of mBERT combined with code-mixed adaptation, while HingBERT or HingRoBERTa may in some settings attain higher precision or slightly better overall F1 (Shirke et al., 2 Sep 2025). Because no significance tests are reported, this should be treated as a comparative descriptive pattern rather than a formally established superiority relation.

5. Methodological use in Hinglish NER

In the specific NER setting where HingMBERT has been most explicitly operationalized, the dataset consists of 3,637 Hindi–English code-mixed sentences split 80–20 at the sentence level and labeled in BIO format with B-Per, I-Per, B-Org, I-Org, B-Loc, I-Loc, and Other (Shirke et al., 2 Sep 2025). The label distribution is highly imbalanced: Other has 63,497 instances, compared with 2,138 for B-Per, 1,432 for B-Org, 762 for B-Loc, and much smaller counts for the inside tags (Shirke et al., 2 Sep 2025).

Preprocessing uses model-specific tokenization, which for HingMBERT is the mBERT tokenizer with subword tokenization and general normalization, including Unicode normalization and lowercasing as applied in that pipeline (Shirke et al., 2 Sep 2025). During token–label alignment, only the first subword of a split token retains the original label, while subsequent subwords are ignored in the loss. The paper characterizes the objective as standard token-level cross-entropy over labeled positions, consistent with token classification in Hugging Face Transformers (Shirke et al., 2 Sep 2025).

No special transliteration modules, language-identification features, or handcrafted cues are added. HingMBERT relies purely on its mBERT-plus-code-mixed pretraining, the multilingual WordPiece vocabulary, and supervised NER fine-tuning (Shirke et al., 2 Sep 2025). This is methodologically significant because it isolates the value of code-mixed domain adaptation from auxiliary engineering. The same paper reports that per-entity-type evaluation was performed, but does not provide the corresponding numerical breakdowns (Shirke et al., 2 Sep 2025).

A plausible implication is that HingMBERT functions as a strong baseline precisely because it avoids task-specific complication. In this respect it resembles the role played by the original L3Cube release on GLUECoS, where a single continued-pretrained encoder is repurposed across LID, POS, NER, and sentiment with only lightweight task heads (Nayak et al., 2022).

6. Comparative interpretation, limitations, and future directions

The main comparative conclusion across the cited literature is that real code-mixed pretraining improves downstream performance over both English-only and general multilingual baselines. In GLUECoS, HingMBERT improves on vanilla mBERT across all reported Roman-script tasks (Nayak et al., 2022). In the later NER study, code-mixed models collectively outperform non-code-mixed baselines and zero-shot Google Gemini, with HingMBERT either first or second among the code-mixed systems depending on split (Shirke et al., 2 Sep 2025). The 2023 comparative study of code-mixed classification reaches a parallel conclusion for HingBERT-family models more broadly, arguing that code-mixed pretraining is crucial and that multilingual coverage alone is insufficient (Patil et al., 2023).

At the same time, HingMBERT is not uniformly dominant. In the original L3Cube evaluations, HingRoBERTa usually provides the strongest overall numbers, especially for sentiment, while HingMBERT remains highly competitive and often stronger than vanilla mBERT (Nayak et al., 2022). In the NER benchmark, HingBERT yields the best test F1 whereas HingMBERT yields the best recall and second-best F1 (Shirke et al., 2 Sep 2025). This suggests that the multilingual initialization of HingMBERT may confer broader coverage, while other pretraining recipes can be more conservative or more precise on some tasks.

Several limitations recur. First, much of the pretraining data is Twitter-centric, so domain transfer to formal or longer-form text is uncertain (Nayak et al., 2022). Second, many downstream datasets are small, including the 3,637-sentence Hinglish NER corpus used in the 2025 comparison, which may amplify variance across fine-tuning runs (Shirke et al., 2 Sep 2025). Third, detailed qualitative error analyses are sparse. The NER study provides no concrete error examples and no model-specific breakdown by entity type, so inferences about false positives or boundary errors remain indirect (Shirke et al., 2 Sep 2025). Fourth, model comparison against LLMs is incomplete in the sense that zero-shot Gemini is evaluated, but instruction-tuned or domain-adapted LLM variants are not (Shirke et al., 2 Sep 2025).

Future directions stated in the literature include ensemble methods combining HingMBERT with HingBERT, HingRoBERTa, or even LLM outputs; extension of the same mBERT-plus-code-mixed-adaptation strategy to other Indian language pairs such as Tamil–English, Malayalam–English, and Bengali–English; more extensive cross-lingual pretraining and augmentation; and using LLMs to generate synthetic code-mixed labeled data or support instruction-tuned complementary systems (Shirke et al., 2 Sep 2025). The sentiment paper on Hinglish tweets implies another path: intermediate domain-adaptive pretraining on large unlabeled Hinglish social-media corpora even when starting from generic bert-base-multilingual-cased, which would move a task-specific “HingMBERT-style” model closer to a general-purpose Hinglish encoder (Garg et al., 8 Jan 2026).

In practical terms, HingMBERT is most appropriately used when the target data are Hindi–English code-mixed, especially in Roman script, and when retaining compatibility with the mBERT ecosystem is desirable. The L3Cube release makes this operational by providing public checkpoints l3cube-pune/hing-mbert and l3cube-pune/hing-mbert-mixed, together with related datasets and code (Nayak et al., 2022). Within the current Hinglish modeling landscape, HingMBERT therefore functions as a domain-adapted multilingual transformer that preserves the architectural and tooling familiarity of mBERT while materially improving performance on code-mixed NLP tasks.