HingRoBERTa: A Code-Mixed Hinglish Model
- HingRoBERTa is a code-mixed Hindi-English language model built on XLM-RoBERTa, designed for authentic Hinglish text processing.
- It employs 12 transformer layers with dynamic masked language modeling, optimizing performance for code-switching scenarios.
- Empirical evaluations on GLUECoS benchmarks and cybercrime classification demonstrate its competitive edge over traditional multilingual models.
HingRoBERTa is a code-mixed Hindi-English LLM based on the XLM-RoBERTa architecture and further pre-trained on L3Cube-HingCorpus, a large real-world Hinglish corpus collected from Twitter. It was introduced to address the scarcity of large-scale real code-mixed data for pre-training and to improve downstream performance on tasks where Roman-script transliteration, lexical borrowing, and rapid code-switching degrade monolingual and generic multilingual encoders. Subsequent studies have evaluated HingRoBERTa on GLUECoS tasks, sentiment analysis, emotion recognition, hate speech identification, named entity recognition, and cybercrime complaint classification, making it a recurrent benchmark for Hinglish NLP (Nayak et al., 2022).
1. Corpus foundation and data regime
The immediate substrate for HingRoBERTa is L3Cube-HingCorpus, described as the first large-scale real Hindi-English code mixed data in a Roman script. The corpus consists of 52.93 million sentences and 1.04 billion tokens, scraped from Twitter; the reported split is 47.79 million sentences and 944 million tokens for training, and 5.13 million sentences and 99 million tokens for validation. Collection used iterative keyword-based search and lexical expansion, while filtering employed a language identification pipeline that required at least 2 Hindi and 2 English words per sentence for genuine code-mixing. For the Roman set, user mentions and non-English alphabetic characters were removed, and a mixed Roman+Devanagari variant was also created through in-house transliteration. The corpus reports a mixed Code-Mixing Index of 31.21, where 0 denotes monolingual text and 100 maximal code-mixing (Nayak et al., 2022).
These corpus properties matter because prior code-mixed resources were typically synthetic or substantially smaller. The L3Cube work explicitly positions HingCorpus as a real-world alternative for pre-training, rather than a synthetic approximation of code-switching. A plausible implication is that much of HingRoBERTa’s subsequent performance derives not only from model architecture but from exposure to authentic code-mixed distributional patterns during pre-training. The L3Cube Pune group also released the corpus and model artifacts publicly through GitHub and Hugging Face, which made the model reproducible and reusable in later studies (Nayak et al., 2022).
2. Model architecture and pre-training protocol
HingRoBERTa is described as being based on XLM-RoBERTa, a multilingual extension of RoBERTa. In the L3Cube formulation, the model uses 12 Transformer blocks, hidden size 768, and 12 self-attention heads, and is optimized with the masked language modeling objective. During pre-training, a random 15% of tokens in each input sentence are masked, and the model learns to predict them from surrounding context. The reported setup continues from pre-trained XLM-RoBERTa weights and further trains on HingCorpus for 2 epochs with learning rate , batch size 64, and later fine-tuning at . HingRoBERTa supports both Roman-only and mixed Roman+Devanagari settings, yielding sequence embeddings for token-level and sentence-level tasks (Nayak et al., 2022).
Later application work characterizes HingRoBERTa more generally as a Hinglish-specific variant of RoBERTa that has been pre-trained on Hinglish code-mixed corpora, inheriting RoBERTa’s optimized pretraining, dynamic masking, and removal of Next Sentence Prediction. That description emphasizes its intended fit for lexical borrowing, code-switching, and non-standard syntax in Indian digital communication, especially complaint text and social media content (Rani et al., 2024).
Validation perplexity during MLM pretraining was reported as 7.82 for HingRoBERTa and 9.39 for HingRoBERTa-mixed, while HingMBERT and HingMBERT-mixed attained 5.20 and 5.22 respectively. This establishes that HingRoBERTa was not the lowest-perplexity model within the L3Cube family, a point that becomes important when comparing language-model fit to downstream task behavior (Nayak et al., 2022).
3. Original benchmark behavior on GLUECoS
The initial multi-task evaluation of HingRoBERTa was conducted on GLUECoS and on L3Cube-HingLID. The benchmark covered language identification, part-of-speech tagging in Universal Dependencies and fine-grained settings, named entity recognition, and three-way sentiment analysis, with F1 used as the principal test metric. On Roman-script test data, HingRoBERTa reported F1 scores of 86.69 for LID, 90.17 for POS-UD, 71.69 for POS-FG, 81.13 for NER, 66.43 for sentiment, and 96.15 for HingLID. On mixed-script test data, HingRoBERTa reported 96.30 for LID, 89.97 for POS-UD, 69.90 for POS-FG, 80.28 for NER, and 64.43 for sentiment; HingRoBERTa-mixed reported 94.96, 90.81, 70.61, 81.72, and 66.07 on those same mixed-script tasks (Nayak et al., 2022).
In the same study, HingRoBERTa and HingRoBERTa-mixed were reported to achieve or better previous state of the art on NER and sentiment analysis, while remaining highly competitive on LID and POS-UD. The ablation discussion further states that Roman-only models work best on Roman-only test sets and mixed-script models on mixed-script test sets. This is significant because it locates script compatibility as an independent design variable: the pre-training corpus is not only code-mixed, but script-conditioned, and downstream gains follow that conditioning (Nayak et al., 2022).
The original GLUECoS results also make clear that code-mixed pre-training on real data consistently improves over vanilla BERT, m-BERT, and XLM-RoBERTa baselines. For example, on Roman-script sentiment analysis, HingRoBERTa reached 66.43 test F1 compared with 59.16 for BERT, 58.42 for m-BERT, and 61.57 for XLM-RoBERTa; on Roman POS-UD it reached 90.17 compared with 83.70, 83.68, and 87.24 respectively (Nayak et al., 2022).
4. Subsequent comparative evaluations across classification tasks
A later comparative study evaluated HingRoBERTa on five annotated code-mixed datasets: ICON sentiment analysis with 18,461 samples, an emotion recognition dataset with 151k samples, a sentiment dataset from Prabhu et al. with 3,879 samples, a hate speech dataset with 4,578 samples, and the HASOC dataset with 4,864 samples. The study used Simple Transformers as the model interface, Weights and Biases for hyperparameter optimization, a learning-rate search in , 1–5 epochs, batch sizes of 32 or 64, and reported accuracy, precision, recall, and macro F1-score (Patil et al., 2023).
| Dataset/task | HingRoBERTa F1 | Comparator note |
|---|---|---|
| ICON sentiment | 0.6522 | HingRoBERTa-Mixed: 0.6529 |
| Emotions | 0.9819 | HingRoBERTa-Mixed: 0.9828 |
| Prabhu sentiment | 0.7471 | Higher than HingBERT 0.6747 and RoBERTa 0.6238 |
| Hate speech | 0.6699 | Highest F1 in that comparison |
| HASOC | 0.7327 | Highest F1 and accuracy 0.7329 |
Across these datasets, HingRoBERTa consistently outperformed vanilla BERT, RoBERTa, and mBERT, while usually matching or narrowly trailing HingRoBERTa-Mixed. On the ICON dataset, for example, HingRoBERTa reported F1 0.6522 against RoBERTa 0.5873 and BERT 0.5800; on the Prabhu sentiment dataset it reported 0.7471 against RoBERTa 0.6238, BERT 0.5955, and mBERT 0.6246. The study interprets this as strong evidence that code-mixed pre-training on large real Roman-script data materially improves sentiment-, emotion-, and hate-speech-related transfer (Patil et al., 2023).
5. Model-selection questions and empirical caveats
The case for HingRoBERTa is strongest when it is compared with generic RoBERTa-family models that lack dedicated code-mixed pre-training. An earlier Hinglish sentiment study fine-tuned a generic RoBERTa-base language-model backbone on a 1.9 million tweet Hinglish corpus and then trained a classifier on Sentimix. That RoBERTa-base system reported F1 0.635 on the test set, with LM perplexity 7.54; NB-SVM reached F1 0.673, and the best majority-vote ensemble reached 0.707. The paper states explicitly that NB-SVM beats RoBERTa by 6.2% relative F1, underscoring how difficult code-mixed sentiment detection remained for generic RoBERTa-style adaptation in 2020 (Bhange et al., 2020).
Later comparisons show that HingRoBERTa alters that picture but does not eliminate task dependence. In a 2025 NER study, HingRoBERTa achieved validation metrics of accuracy 97.36, precision 74.50, recall 77.95, and F1 76.19, and test metrics of accuracy 97.21, precision 78.47, recall 78.30, and F1 78.39, with entity-level evaluation computed by the seqeval library. It outperformed non-code-mixed models such as BERT Base Cased at 72.12 F1, RoBERTa at 76.24, IndicBERT at 76.47, MuRIL at 77.50, and Google Gemini in a zero-shot setting at 62.24, but it was slightly below HingBERT at 79.74 and HingMBERT at 78.87 (Shirke et al., 2 Sep 2025).
Two caveats follow from these studies. First, code-mixed pre-training is more consequential than merely selecting a RoBERTa-family architecture: the contrast between generic RoBERTa in early sentiment work and HingRoBERTa in later evaluations is substantial. Second, lower MLM perplexity is not a sufficient proxy for downstream quality: in the L3Cube study, HingMBERT reported lower pre-training perplexity than HingRoBERTa, yet HingRoBERTa led on several downstream tasks. This suggests that corpus match, script match, and task structure govern downstream performance more strongly than perplexity alone (Nayak et al., 2022).
6. Operational deployment in cybercrime complaint triage
HingRoBERTa has also been used in an applied framework for automated classification of cybercrime complaints. In that setting, the source data came from the Indian Cybercrime Coordination Centre during the CyberGuard AI Hackathon 2024 and comprised 85,875 real cybercrime complaints in Hinglish spanning 14 crime categories. Because the class distribution was highly imbalanced, with “Financial Fraud” vastly overrepresented relative to rare categories such as “Ransomware” and “Cyber Terrorism,” the training data was augmented using LLama3.1-7b, producing a final dataset of 109,294 complaints. Privacy-aware preprocessing replaced personally identifiable information such as names, emails, phone numbers, and addresses with placeholders like <PERSON> and <EMAIL> using regex and spaCy; the pipeline also included stopword removal, lemmatization, and label standardization (Rani et al., 2024).
The augmentation pipeline used LLama3.1-7b to paraphrase minority-class complaints and retained only those synthetic samples whose semantic similarity exceeded a BERTF1-based cosine similarity threshold of 97%. Fine-tuning began from pre-trained HingRoBERTa weights and used grid search over learning rates from to , batch sizes from 8 to 32, maximum sequence lengths of 128 or 256, AdamW optimization, a 20% validation split, and early stopping after 5 epochs without validation improvement. The model head was a fully connected linear classification layer over the representation (Rani et al., 2024).
In this application, HingRoBERTa attained accuracy 74.41%, precision 70.86%, recall 74.41%, and F1-score 71.49%, exceeding BERT at 70.73 F1, RoBERTa at 70.90, HingBERT at 71.02, and the best classical TF-IDF/SVD + XGBoost result at 68% accuracy and 68% F1. The study coupled the classifier with a Django REST Framework backend and a modern JavaScript frontend using Tailwind CSS, and described the system as scalable, modular, and suitable for batch and single-instance prediction, with integration envisioned for platforms such as the National Cyber Crime Reporting Portal. The literature comparison in that work presents the framework as the first to combine Hinglish-specific modeling, privacy compliance, and deployed-tool support for real-world cybercrime complaint triage at that scale (Rani et al., 2024).