HingBERT: Code-Mixed BERT for Hinglish

• HingBERT is a specialized BERT-base encoder pretrained on large-scale, naturally occurring Hindi-English code-mixed data in Roman script.
• It leverages continued masked language modeling on the L3Cube-HingCorpus, which includes 52.93M sentences and 1.04B tokens from Twitter.
• Benchmark results on GLUECoS tasks show HingBERT outperforms vanilla BERT, highlighting the benefits of domain-specific code-mixed pretraining.

HingBERT is a Hindi-English code-mixed BERT family centered on large-scale continued pretraining over naturally occurring Hinglish, especially Roman-script social-media text. In its original formulation, the term refers specifically to the BERT-base model further pretrained on L3Cube-HingCorpus, a real Twitter-derived Hindi-English code-mixed corpus containing 52.93M sentences and 1.04B tokens (Nayak et al., 2022). Subsequent literature uses the name in a broader way to denote code-mixed-specialized BERT systems for Hinglish, particularly in contrast to generic English or multilingual pretrained encoders, but the core identity of HingBERT remains a BERT-style encoder adapted to the distributional properties of real Hindi-English code-switching rather than to monolingual or broadly multilingual clean text (Patil et al., 2023).

1. Origin, scope, and naming

The original source of HingBERT is "L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT LLMs" (Nayak et al., 2022). That work presents L3Cube-HingCorpus, HingBERT, HingMBERT, HingRoBERTa, and HingGPT, with the BERT-family models pretrained on code-mixed HingCorpus using masked language modelling objectives and evaluated on downstream tasks from GLUECoS. Within that paper, HingBERT denotes the BERT-base model further pretrained on Hindi-English code-mixed text, whereas HingMBERT denotes multilingual BERT further pretrained on the same corpus, HingRoBERTa denotes XLM-RoBERTa further pretrained, and HingGPT denotes a GPT-2-style causal model trained for generation (Nayak et al., 2022).

The model was proposed because code-mixing is widespread on social media, real code-mixed pretraining data were described as scarce, and the dominant pretrained transformer ecosystem had weak exposure to real Hinglish, particularly in Roman script. The central methodological claim is not architectural novelty in the narrow transformer-block sense, but adaptation to the target distribution through large-scale continued pretraining on naturally occurring code-mixed data rather than monolingual English, monolingual Hindi, or generic multilingual corpora (Nayak et al., 2022).

A recurring source of confusion in later literature is the proximity between HingBERT and HindBERT. The Devanagari-focused paper "L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages" explicitly states that HingBERT is not the same model as HindBERT: HindBERT is a Hindi monolingual model, whereas HingBERT in prior work is a Hindi-English code-mixed BERT model trained on tweets (Joshi, 2022). This distinction is substantive rather than orthographic, because the former is a monolingual Hindi model and the latter is a Hinglish code-mixed model.

2. L3Cube-HingCorpus and data construction

The training resource behind HingBERT is L3Cube-HingCorpus, described as the first large-scale real Hindi-English code-mixed dataset in Roman script (Nayak et al., 2022). Its reported scale is:

• 52.93M sentences
• 1.04B tokens
• Training: 47.79M sentences, 944M tokens
• Validation: 5.13M sentences, 99M tokens (Nayak et al., 2022)

The corpus was collected from Twitter using Twint. The collection pipeline begins with an initial vocabulary of commonly spoken Hindi words, uses that vocabulary to scrape tweets, iteratively expands the vocabulary with newly found words, preprocesses the tweets to retain only code-switched Roman script, and then applies token-level language identification to determine whether each candidate tweet is genuinely code-mixed (Nayak et al., 2022). The filtering criteria reported for keeping a tweet as code-mixed are explicit: it must contain both Hindi and English words, with at least 2 Hindi words and at least 2 English words (Nayak et al., 2022).

Several preprocessing decisions are also stated directly. The corpus-construction pipeline removes non-English characters and user mentions for privacy, while retaining case, punctuation, and smileys (Nayak et al., 2022). The resulting corpus is therefore deliberately close to raw social-media usage rather than heavily normalized. The paper also reports a Mixed CMI index = 31.21, where 0 corresponds to monolingual data with no code-mixing and 100 corresponds to the highest degree of code-switching (Nayak et al., 2022). This is the principal quantitative indication supplied for the degree of code-mixing present in the corpus.

A mixed-script extension also exists. The paper states that it creates a Devanagari version via an in-house transliteration model, with the same number of sentences and approximately similar token count (Nayak et al., 2022). Later comparative work characterizes the Roman corpus as foundational and the mixed-script variant as an additional resource for settings in which users mix languages and scripts simultaneously (Patil et al., 2023).

The code-mixed status of the corpus depends heavily on language identification quality. The data-construction work first used a shallow subword-based LSTM language classifier trained in a semi-supervised fashion, starting with 5k sentences, iteratively expanding through pseudo-labeling and manual verification, and ultimately producing an LID dataset of around 44,455 sentences. A base BERT model was then fine-tuned on the expanded LID data and used to select code-mixed sentences for HingCorpus (Nayak et al., 2022). This suggests that corpus quality is tightly coupled to upstream LID performance.

3. Model formulation and training regime

In the original paper, HingBERT is the BERT-base architecture further pretrained on L3Cube-HingCorpus rather than a model trained from scratch (Nayak et al., 2022). The architecture details explicitly given for HingBERT are those of BERT-base:

• 12 transformer blocks
• 12 self-attention heads
• hidden size 768
• maximum input length 512 tokens
• special tokens [CLS] and [SEP] (Nayak et al., 2022)

The paper states that the authors load pretrained weights of the respective base models and continue pretraining them on Hinglish (Nayak et al., 2022). Accordingly, HingBERT should be understood as a continued-pretraining adaptation of BERT-base to Hindi-English code-mixed text. A plausible implication is that much of its empirical behavior derives from domain and language-distribution shift correction rather than from architectural divergence from standard BERT.

The pretraining objective reported for HingBERT is masked language modeling (MLM) with a masking probability of 15% (Nayak et al., 2022). The training setup reported for continued pretraining is concise but specific:

• epochs: 2
• learning rate: 1e-5
• batch size: 64 (Nayak et al., 2022)

The paper further states that 2 epochs were sufficient for convergence because the models were initialized from pretrained checkpoints and the authors saw no significant loss decrease after that point (Nayak et al., 2022). It does not provide optimizer type, warmup schedule, weight decay, total steps, hardware, wall-clock time, gradient accumulation, mixed precision, tokenizer vocabulary size, or parameter count. Later secondary descriptions preserve this distinction: the core methodological idea is continued pretraining on a large real code-mixed corpus, while many low-level engineering details remain underreported (Patil et al., 2023).

Validation perplexity is reported as an intrinsic pretraining metric. The paper gives HingBERT a validation perplexity of 5.72, compared with 5.20 for HingMBERT and 7.82 for HingRoBERTa (Nayak et al., 2022). The same source notes that downstream performance does not track perplexity exactly, which is borne out by later task comparisons in which RoBERTa-family variants often achieve the strongest end-task results despite weaker reported pretraining perplexity (Nayak et al., 2022).

For downstream fine-tuning, the original paper states that the encoder is followed by dense layers on top of either the [CLS] representation or token embeddings, depending on task type, with:

• epochs: up to 5
• early stopping: based on validation F1
• batch size: 64
• learning rate: 3e-5 (Nayak et al., 2022)

4. Evaluation on GLUECoS and immediate empirical profile

The original downstream evaluation is conducted mainly on the English-Hindi pair of GLUECoS and uses F1 score (Nayak et al., 2022). The tasks span token-level and sentence-level settings:

• Language Identification (LID) with tags EN / HI / OTHER
• POS tagging on POS-UD
• POS tagging on POS-FG
• NER
• Sentiment analysis with positive / negative / neutral (Nayak et al., 2022)

The reported dataset sizes are also explicit. For LID, the splits are 2631 train, 500 dev, 406 test. For POS-UD, they are 1384 train, 215 dev, 215 test. For POS-FG, they are 2104 train, 263 dev, 264 test. For NER, they are 2467 train, 308 dev, 309 test. For Sentiment, they are 10080 train, 1260 dev, 1261 test (Nayak et al., 2022).

The Roman-script GLUECoS test F1 results place HingBERT as a strong model relative to generic baselines:

Model	LID	POS-UD	POS-FG	NER	Sentiment
BERT	78.69	83.70	70.75	79.27	59.16
m-BERT	82.56	83.68	69.58	76.64	58.42
XLMRoBERTa	85.93	87.24	70.95	77.01	61.57
HingBERT	84.44	88.42	71.04	81.80	63.72

These numbers show that HingBERT improves over vanilla BERT on all five Roman-script tasks, including POS-UD: 88.42 vs 83.70, NER: 81.80 vs 79.27, and Sentiment: 63.72 vs 59.16 (Nayak et al., 2022). Compared with generic multilingual models, it is also competitive: HingBERT exceeds m-BERT on all listed Roman-script tasks and exceeds XLMRoBERTa on POS-UD, POS-FG, and NER, though not on LID or Sentiment (Nayak et al., 2022).

The mixed-script evaluation yields a different profile. HingBERT remains competitive but is not the strongest model in that setting, with HingRoBERTa and especially HingRoBERTa-mixed generally achieving the best mixed-script results (Nayak et al., 2022). This supports the paper’s stated pattern that Roman models do slightly better on Roman-script tasks, whereas mixed-script models do better on mixed-script tasks (Nayak et al., 2022). A plausible implication is that script match is a major factor in code-mixed model transfer.

The same paper also releases HingBERT-LID, a production-quality LID model trained on the L3Cube-HingLID corpus, and reports 98.77 F1 on HingLID in the results table (Nayak et al., 2022). Although that model is distinct from HingBERT, it is operationally important because it underpins the acquisition of additional code-mixed text.

5. Comparative studies and downstream reinterpretations

Subsequent studies have used HingBERT as a benchmark point for the broader proposition that code-mixed specialization improves performance on Hinglish tasks. The most systematic early comparative study is "Comparative Study of Pre-Trained BERT Models for Code-Mixed Hindi-English Data" (Patil et al., 2023). That paper describes HingBERT as a BERT model for Hindi-English code-mixed text, built using Roman-script Hindi-English mixed text, trained on L3Cube-HingCorpus, which it reports as containing 52.93 million phrases from Twitter and 1.04 billion tokens (Patil et al., 2023).

Its evaluations span five datasets over sentiment analysis, emotion recognition, and hate speech identification, and the reported pattern is that HingBERT is generally better than vanilla English BERT and usually better than ALBERT and mBERT, but generally worse than HingRoBERTa and HingRoBERTa-Mixed (Patil et al., 2023). The macro-F1 values reported for HingBERT in that study are:

• ICON: 0.62825
• Emotions: 0.81437
• Sentiment: 0.67471
• Hatespeech: 0.62860
• HASOC: 0.65204 (Patil et al., 2023)

The same study explicitly characterizes HingBERT as an improved version of mBERT on L3Cube-HingCorpus (Patil et al., 2023). This wording is somewhat looser than the original paper’s distinction between HingBERT and HingMBERT, but the comparative conclusion is consistent: code-mixed pretraining materially improves over vanilla pretrained baselines, while within the L3Cube family the RoBERTa variants are often strongest.

Later task-specific work continues this pattern. In code-mixed hate speech identification, a 2024 study reports that HingBERT models, benefiting from training on the extensive Hindi-English dataset L3Cube-HingCorpus, outperform BERT models when tested on hate speech text datasets (Jagdale et al., 2024). That paper emphasizes the same underlying argument as the original resource paper: distributional alignment to real-world Hinglish text from Twitter is more important than using a more sophisticated architecture trained on the wrong distribution (Jagdale et al., 2024).

In cybercrime complaint classification for Hinglish texts, HingBERT is treated as a pretrained Hinglish-adapted BERT variant fine-tuned on a real-world I4C dataset. The reported transformer comparison shows BERT: 71.03 accuracy / 70.73 F1, RoBERTa: 71.64 / 70.90, HingBERT: 72.82 / 71.02, and HingRoBERTa: 74.41 / 71.49 (Rani et al., 2024). Here the significance of HingBERT is not that it is the best model overall, but that it isolates the gain attributable to Hinglish adaptation within the BERT family.

6. HingBERT in code-mixed sequence labeling and alternative formulations

HingBERT has also been examined directly in code-mixed NER. The 2025 study "Comparative Study of Pre-Trained BERT and LLMs for Code-Mixed Named Entity Recognition" treats HingBERT as a code-mixed-specific transformer model for Hindi-English (Hinglish) and compares it with HingMBERT, HingRoBERTa, BERT Base Cased, IndicBERT, RoBERTa, MuRIL, and zero-shot Google Gemini (Shirke et al., 2 Sep 2025). The dataset used there contains 3,637 sentences with BIO-format NER tags over PER, ORG, and LOC, with an 80/20 train-test split and heavy imbalance toward the Other label (Shirke et al., 2 Sep 2025).

The fine-tuning framework in that study is more fully specified than in many earlier HingBERT evaluation papers. It uses Hugging Face Transformers, Optuna for hyperparameter search, 40 optimization trials per model, three different random seeds, AdamW, a linear learning rate scheduler, early stopping with patience 5 epochs, maximum sequence length: 128, warmup: first 10% of training steps, batch size: between 16 and 32, and FP16 mixed precision (Shirke et al., 2 Sep 2025). Although those settings are shared experimental settings rather than original HingBERT design parameters, they define one concrete modern fine-tuning regime for the model.

The reported test-set performance in that study places HingBERT first overall on F1:

• Accuracy: 97.38
• Precision: 78.81
• Recall: 80.69
• F1-score: 79.74 (Shirke et al., 2 Sep 2025)

This exceeds MuRIL: 77.50 F1, IndicBERT: 76.47, RoBERTa: 76.24, BERT Base Cased: 72.12, HingRoBERTa: 78.39, HingMBERT: 78.87, and Google Gemini: 62.24 on the test set (Shirke et al., 2 Sep 2025). The authors interpret this as evidence that code-mixed specialization improves handling of informal structure, transliteration, frequent code-switching, and noisy patterns in Hinglish (Shirke et al., 2 Sep 2025).

A distinct strand of work uses the label “HingBERT” in a looser pipeline sense rather than as the original L3Cube pretrained checkpoint. "Leveraging Language Identification to Enhance Code-Mixed Text Classification" presents a pipeline in which BERT-based models are augmented with explicit token-level language tags from HingLID, using two strategies: Interleaved Word-Language and Adjacent Sentence-Language (Takawane et al., 2023). In that paper, “HingBERT” is best understood as a BERT-based classifier with language-augmented input rather than as a newly pretrained transformer architecture. This alternative usage is important because it broadens the term from a specific checkpoint lineage to a family of language-aware Hinglish BERT pipelines.

7. Conceptual significance, limitations, and neighboring models

Across the literature, the core significance of HingBERT is consistent: it operationalizes the claim that real large-scale code-mixed pretraining matters for Hindi-English NLP (Nayak et al., 2022). The original evidence is comparative rather than ablative, but several patterns recur across papers:

• HingBERT > vanilla BERT on Roman-script GLUECoS tasks (Nayak et al., 2022)
• HingBERT generally improves over standard BERT-family baselines across classification tasks (Patil et al., 2023)
• code-mixed-specialized models outperform generic baselines in hate speech detection (Jagdale et al., 2024)
• HingBERT is highly competitive, and in one NER study best overall on test F1 (Shirke et al., 2 Sep 2025)

The main limitations are also stable. The original paper does not report optimizer, hardware, total training steps, tokenizer vocabulary details, or exact parameter count for HingBERT (Nayak et al., 2022). Later comparative papers likewise often supply strong empirical conclusions but limited implementation detail for the checkpoint itself (Patil et al., 2023, Jagdale et al., 2024). Domain bias is another clear issue: the pretraining corpus is scraped from Twitter, so the model is aligned to social-media Hinglish rather than to formal or domain-general Hindi-English mixing (Nayak et al., 2022). Script dependence also remains central. Roman-script pretraining is particularly effective for Roman Hinglish, while mixed-script variants can be more appropriate when Roman and Devanagari are both present (Nayak et al., 2022).

Finally, the term should not be conflated with several neighboring but distinct models. It is not HindBERT, the monolingual Hindi model introduced in a Devanagari-focused paper (Joshi, 2022). It is also unrelated by name and scope to TwHIN-BERT, a multilingual tweet encoder enriched by Twitter heterogeneous information network supervision (Zhang et al., 2022), and to HyperBERT, a mixed text-hypergraph model for node classification on text-attributed hypergraphs (Bazaga et al., 2024). Those models share the general idea of augmenting BERT-style representations with structure or domain-specific supervision, but they are not HingBERT in the L3Cube Hinglish sense.

Taken together, the published record defines HingBERT as a BERT-base-derived, code-mixed-specialized encoder anchored in L3Cube-HingCorpus and in the empirical thesis that pretraining must match the actual linguistic ecology of Roman-script Hindi-English code-switching. Subsequent work has largely reinforced that thesis, even when absolute best performance within the same family is often achieved by HingRoBERTa rather than by HingBERT itself (Nayak et al., 2022, Patil et al., 2023).