---
title: 'HingBERT: Code-Mixed BERT for Hinglish'
url: https://www.emergentmind.com/topics/hingbert
type: topic
---

# HingBERT: Code-Mixed BERT for Hinglish

HingBERT is a Hindi-English code-mixed BERT family centered on large-scale continued pretraining over naturally occurring Hinglish, especially Roman-script social-media text. In its original formulation, the term refers specifically to the BERT-base model further pretrained on **L3Cube-HingCorpus**, a real Twitter-derived Hindi-English code-mixed corpus containing **52.93M sentences** and **1.04B tokens** [2204.08398]. Subsequent literature uses the name in a broader way to denote code-mixed-specialized BERT systems for Hinglish, particularly in contrast to generic English or multilingual pretrained encoders, but the core identity of HingBERT remains a BERT-style encoder adapted to the distributional properties of real Hindi-English code-switching rather than to monolingual or broadly multilingual clean text [2305.15722].

## 1. Origin, scope, and naming

The original source of HingBERT is **"L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models"** [2204.08398]. That work presents **L3Cube-HingCorpus**, **HingBERT**, **HingMBERT**, **HingRoBERTa**, and **HingGPT**, with the BERT-family models pretrained on code-mixed HingCorpus using masked language modelling objectives and evaluated on downstream tasks from GLUECoS. Within that paper, **HingBERT** denotes the **BERT-base model further pretrained on Hindi-English code-mixed text**, whereas **HingMBERT** denotes multilingual BERT further pretrained on the same corpus, **HingRoBERTa** denotes XLM-RoBERTa further pretrained, and **HingGPT** denotes a GPT-2-style causal model trained for generation [2204.08398].

The model was proposed because code-mixing is widespread on social media, real code-mixed pretraining data were described as scarce, and the dominant pretrained transformer ecosystem had weak exposure to **real Hinglish**, particularly in **Roman script**. The central methodological claim is not architectural novelty in the narrow transformer-block sense, but adaptation to the target distribution through large-scale continued pretraining on naturally occurring code-mixed data rather than monolingual English, monolingual Hindi, or generic multilingual corpora [2204.08398].

A recurring source of confusion in later literature is the proximity between **HingBERT** and **HindBERT**. The Devanagari-focused paper **"L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages"** explicitly states that **HingBERT is not the same model as HindBERT**: **HindBERT** is a Hindi monolingual model, whereas **HingBERT** in prior work is a Hindi-English code-mixed BERT model trained on tweets [2211.11418]. This distinction is substantive rather than orthographic, because the former is a monolingual Hindi model and the latter is a Hinglish code-mixed model.

## 2. L3Cube-HingCorpus and data construction

The training resource behind HingBERT is **L3Cube-HingCorpus**, described as the **first large-scale real Hindi-English code-mixed dataset in Roman script** [2204.08398]. Its reported scale is:

- **52.93M sentences**
- **1.04B tokens**
- **Training:** **47.79M sentences**, **944M tokens**
- **Validation:** **5.13M sentences**, **99M tokens** [2204.08398]

The corpus was collected from **Twitter** using **Twint**. The collection pipeline begins with an initial vocabulary of commonly spoken Hindi words, uses that vocabulary to scrape tweets, iteratively expands the vocabulary with newly found words, preprocesses the tweets to retain only **code-switched Roman script**, and then applies token-level language identification to determine whether each candidate tweet is genuinely code-mixed [2204.08398]. The filtering criteria reported for keeping a tweet as code-mixed are explicit: it must contain both Hindi and English words, with at least **2 Hindi words** and at least **2 English words** [2204.08398].

Several preprocessing decisions are also stated directly. The corpus-construction pipeline removes **non-English characters** and **user mentions** for privacy, while retaining **case**, **punctuation**, and **smileys** [2204.08398]. The resulting corpus is therefore deliberately close to raw social-media usage rather than heavily normalized. The paper also reports a **Mixed CMI index = 31.21**, where **0** corresponds to monolingual data with no code-mixing and **100** corresponds to the highest degree of code-switching [2204.08398]. This is the principal quantitative indication supplied for the degree of code-mixing present in the corpus.

A mixed-script extension also exists. The paper states that it creates a **Devanagari version** via an in-house transliteration model, with the same number of sentences and approximately similar token count [2204.08398]. Later comparative work characterizes the Roman corpus as foundational and the mixed-script variant as an additional resource for settings in which users mix languages and scripts simultaneously [2305.15722].

The code-mixed status of the corpus depends heavily on language identification quality. The data-construction work first used a **shallow subword-based LSTM** language classifier trained in a **semi-supervised** fashion, starting with **5k sentences**, iteratively expanding through pseudo-labeling and manual verification, and ultimately producing an LID dataset of around **44,455 sentences**. A base BERT model was then fine-tuned on the expanded LID data and used to select code-mixed sentences for HingCorpus [2204.08398]. This suggests that corpus quality is tightly coupled to upstream LID performance.

## 3. Model formulation and training regime

In the original paper, HingBERT is the **BERT-base architecture** further pretrained on L3Cube-HingCorpus rather than a model trained from scratch [2204.08398]. The architecture details explicitly given for HingBERT are those of BERT-base:

- **12 transformer blocks**
- **12 self-attention heads**
- **hidden size 768**
- **maximum input length 512 tokens**
- special tokens **[CLS]** and **[SEP]** [2204.08398]

The paper states that the authors **load pretrained weights of the respective base models** and continue pretraining them on Hinglish [2204.08398]. Accordingly, HingBERT should be understood as a continued-pretraining adaptation of BERT-base to Hindi-English code-mixed text. A plausible implication is that much of its empirical behavior derives from domain and language-distribution shift correction rather than from architectural divergence from standard BERT.

The pretraining objective reported for HingBERT is **masked language modeling (MLM)** with a **masking probability of 15%** [2204.08398]. The training setup reported for continued pretraining is concise but specific:

- **epochs:** **2**
- **learning rate:** **1e-5**
- **batch size:** **64** [2204.08398]

The paper further states that **2 epochs were sufficient for convergence** because the models were initialized from pretrained checkpoints and the authors saw no significant loss decrease after that point [2204.08398]. It does not provide optimizer type, warmup schedule, weight decay, total steps, hardware, wall-clock time, gradient accumulation, mixed precision, tokenizer vocabulary size, or parameter count. Later secondary descriptions preserve this distinction: the core methodological idea is continued pretraining on a large real code-mixed corpus, while many low-level engineering details remain underreported [2305.15722].

Validation perplexity is reported as an intrinsic pretraining metric. The paper gives **HingBERT** a **validation perplexity of 5.72**, compared with **5.20** for HingMBERT and **7.82** for HingRoBERTa [2204.08398]. The same source notes that downstream performance does not track perplexity exactly, which is borne out by later task comparisons in which RoBERTa-family variants often achieve the strongest end-task results despite weaker reported pretraining perplexity [2204.08398].

For downstream fine-tuning, the original paper states that the encoder is followed by dense layers on top of either the **[CLS] representation** or **token embeddings**, depending on task type, with:

- **epochs:** up to **5**
- **early stopping:** based on validation **F1**
- **batch size:** **64**
- **learning rate:** **3e-5** [2204.08398]

## 4. Evaluation on GLUECoS and immediate empirical profile

The original downstream evaluation is conducted mainly on the **English-Hindi pair of GLUECoS** and uses **F1 score** [2204.08398]. The tasks span token-level and sentence-level settings:

- **Language Identification (LID)** with tags **EN / HI / OTHER**
- **POS tagging** on **POS-UD**
- **POS tagging** on **POS-FG**
- **NER**
- **Sentiment analysis** with **positive / negative / neutral** [2204.08398]

The reported dataset sizes are also explicit. For **LID**, the splits are **2631 train**, **500 dev**, **406 test**. For **POS-UD**, they are **1384 train**, **215 dev**, **215 test**. For **POS-FG**, they are **2104 train**, **263 dev**, **264 test**. For **NER**, they are **2467 train**, **308 dev**, **309 test**. For **Sentiment**, they are **10080 train**, **1260 dev**, **1261 test** [2204.08398].

The Roman-script GLUECoS test F1 results place HingBERT as a strong model relative to generic baselines:

| Model | LID | POS-UD | POS-FG | NER | Sentiment |
|---|---:|---:|---:|---:|---:|
| BERT | 78.69 | 83.70 | 70.75 | 79.27 | 59.16 |
| m-BERT | 82.56 | 83.68 | 69.58 | 76.64 | 58.42 |
| XLMRoBERTa | 85.93 | 87.24 | 70.95 | 77.01 | 61.57 |
| **HingBERT** | **84.44** | **88.42** | **71.04** | **81.80** | **63.72** |

These numbers show that HingBERT improves over vanilla BERT on all five Roman-script tasks, including **POS-UD: 88.42 vs 83.70**, **NER: 81.80 vs 79.27**, and **Sentiment: 63.72 vs 59.16** [2204.08398]. Compared with generic multilingual models, it is also competitive: HingBERT exceeds **m-BERT** on all listed Roman-script tasks and exceeds **XLMRoBERTa** on **POS-UD**, **POS-FG**, and **NER**, though not on **LID** or **Sentiment** [2204.08398].

The mixed-script evaluation yields a different profile. HingBERT remains competitive but is not the strongest model in that setting, with **HingRoBERTa** and especially **HingRoBERTa-mixed** generally achieving the best mixed-script results [2204.08398]. This supports the paper’s stated pattern that **Roman models** do slightly better on **Roman-script tasks**, whereas **mixed-script models** do better on **mixed-script tasks** [2204.08398]. A plausible implication is that script match is a major factor in code-mixed model transfer.

The same paper also releases **HingBERT-LID**, a production-quality LID model trained on the **L3Cube-HingLID** corpus, and reports **98.77 F1** on HingLID in the results table [2204.08398]. Although that model is distinct from HingBERT, it is operationally important because it underpins the acquisition of additional code-mixed text.

## 5. Comparative studies and downstream reinterpretations

Subsequent studies have used HingBERT as a benchmark point for the broader proposition that code-mixed specialization improves performance on Hinglish tasks. The most systematic early comparative study is **"Comparative Study of Pre-Trained BERT Models for Code-Mixed Hindi-English Data"** [2305.15722]. That paper describes HingBERT as a **BERT model for Hindi-English code-mixed text**, built using **Roman-script Hindi-English mixed text**, trained on **L3Cube-HingCorpus**, which it reports as containing **52.93 million phrases from Twitter** and **1.04 billion tokens** [2305.15722].

Its evaluations span five datasets over sentiment analysis, emotion recognition, and hate speech identification, and the reported pattern is that HingBERT is generally better than vanilla English BERT and usually better than ALBERT and mBERT, but generally worse than **HingRoBERTa** and **HingRoBERTa-Mixed** [2305.15722]. The macro-F1 values reported for HingBERT in that study are:

- **ICON**: **0.62825**
- **Emotions**: **0.81437**
- **Sentiment**: **0.67471**
- **Hatespeech**: **0.62860**
- **HASOC**: **0.65204** [2305.15722]

The same study explicitly characterizes HingBERT as an **improved version of mBERT** on L3Cube-HingCorpus [2305.15722]. This wording is somewhat looser than the original paper’s distinction between HingBERT and HingMBERT, but the comparative conclusion is consistent: code-mixed pretraining materially improves over vanilla pretrained baselines, while within the L3Cube family the RoBERTa variants are often strongest.

Later task-specific work continues this pattern. In code-mixed hate speech identification, a 2024 study reports that **HingBERT models, benefiting from training on the extensive Hindi-English dataset L3Cube-HingCorpus, outperform BERT models when tested on hate speech text datasets** [2411.18577]. That paper emphasizes the same underlying argument as the original resource paper: distributional alignment to **real-world Hinglish text from Twitter** is more important than using a more sophisticated architecture trained on the wrong distribution [2411.18577].

In cybercrime complaint classification for Hinglish texts, HingBERT is treated as a pretrained **Hinglish-adapted BERT variant** fine-tuned on a real-world I4C dataset. The reported transformer comparison shows **BERT: 71.03 accuracy / 70.73 F1**, **RoBERTa: 71.64 / 70.90**, **HingBERT: 72.82 / 71.02**, and **HingRoBERTa: 74.41 / 71.49** [2412.16614]. Here the significance of HingBERT is not that it is the best model overall, but that it isolates the gain attributable to Hinglish adaptation within the BERT family.

## 6. HingBERT in code-mixed sequence labeling and alternative formulations

HingBERT has also been examined directly in code-mixed NER. The 2025 study **"Comparative Study of Pre-Trained BERT and Large Language Models for Code-Mixed Named Entity Recognition"** treats HingBERT as a **code-mixed-specific transformer model for Hindi-English (Hinglish)** and compares it with HingMBERT, HingRoBERTa, BERT Base Cased, IndicBERT, RoBERTa, MuRIL, and zero-shot Google Gemini [2509.02514]. The dataset used there contains **3,637 sentences** with BIO-format NER tags over **PER**, **ORG**, and **LOC**, with an **80/20** train-test split and heavy imbalance toward the `Other` label [2509.02514].

The fine-tuning framework in that study is more fully specified than in many earlier HingBERT evaluation papers. It uses **Hugging Face Transformers**, **Optuna** for hyperparameter search, **40 optimization trials per model**, **three different random seeds**, **AdamW**, a **linear learning rate scheduler**, **early stopping** with patience **5 epochs**, **maximum sequence length: 128**, **warmup: first 10% of training steps**, **batch size: between 16 and 32**, and **FP16** mixed precision [2509.02514]. Although those settings are shared experimental settings rather than original HingBERT design parameters, they define one concrete modern fine-tuning regime for the model.

The reported test-set performance in that study places HingBERT first overall on F1:

- **Accuracy:** **97.38**
- **Precision:** **78.81**
- **Recall:** **80.69**
- **F1-score:** **79.74** [2509.02514]

This exceeds **MuRIL: 77.50 F1**, **IndicBERT: 76.47**, **RoBERTa: 76.24**, **BERT Base Cased: 72.12**, **HingRoBERTa: 78.39**, **HingMBERT: 78.87**, and **Google Gemini: 62.24** on the test set [2509.02514]. The authors interpret this as evidence that code-mixed specialization improves handling of informal structure, transliteration, frequent code-switching, and noisy patterns in Hinglish [2509.02514].

A distinct strand of work uses the label “HingBERT” in a looser pipeline sense rather than as the original L3Cube pretrained checkpoint. **"Leveraging Language Identification to Enhance Code-Mixed Text Classification"** presents a pipeline in which BERT-based models are augmented with explicit token-level language tags from **HingLID**, using two strategies: **Interleaved Word-Language** and **Adjacent Sentence-Language** [2306.04964]. In that paper, “HingBERT” is best understood as a BERT-based classifier with language-augmented input rather than as a newly pretrained transformer architecture. This alternative usage is important because it broadens the term from a specific checkpoint lineage to a family of language-aware Hinglish BERT pipelines.

## 7. Conceptual significance, limitations, and neighboring models

Across the literature, the core significance of HingBERT is consistent: it operationalizes the claim that **real large-scale code-mixed pretraining matters** for Hindi-English NLP [2204.08398]. The original evidence is comparative rather than ablative, but several patterns recur across papers:

- **HingBERT > vanilla BERT** on Roman-script GLUECoS tasks [2204.08398]
- HingBERT generally improves over standard BERT-family baselines across classification tasks [2305.15722]
- code-mixed-specialized models outperform generic baselines in hate speech detection [2411.18577]
- HingBERT is highly competitive, and in one NER study best overall on test F1 [2509.02514]

The main limitations are also stable. The original paper does not report optimizer, hardware, total training steps, tokenizer vocabulary details, or exact parameter count for HingBERT [2204.08398]. Later comparative papers likewise often supply strong empirical conclusions but limited implementation detail for the checkpoint itself [2305.15722; 2411.18577]. Domain bias is another clear issue: the pretraining corpus is scraped from **Twitter**, so the model is aligned to social-media Hinglish rather than to formal or domain-general Hindi-English mixing [2204.08398]. Script dependence also remains central. Roman-script pretraining is particularly effective for Roman Hinglish, while mixed-script variants can be more appropriate when Roman and Devanagari are both present [2204.08398].

Finally, the term should not be conflated with several neighboring but distinct models. It is not **HindBERT**, the monolingual Hindi model introduced in a Devanagari-focused paper [2211.11418]. It is also unrelated by name and scope to **TwHIN-BERT**, a multilingual tweet encoder enriched by Twitter heterogeneous information network supervision [2209.07562], and to **HyperBERT**, a mixed text-hypergraph model for node classification on text-attributed hypergraphs [2402.07309]. Those models share the general idea of augmenting BERT-style representations with structure or domain-specific supervision, but they are not HingBERT in the L3Cube Hinglish sense.

Taken together, the published record defines HingBERT as a BERT-base-derived, code-mixed-specialized encoder anchored in **L3Cube-HingCorpus** and in the empirical thesis that pretraining must match the actual linguistic ecology of **Roman-script Hindi-English code-switching**. Subsequent work has largely reinforced that thesis, even when absolute best performance within the same family is often achieved by **HingRoBERTa** rather than by HingBERT itself [2204.08398; 2305.15722].

Source: https://www.emergentmind.com/topics/hingbert