---
title: 'IndoBERT: Indonesian Pretrained Language Models'
url: https://www.emergentmind.com/topics/indobert
type: topic
---

# IndoBERT: Indonesian Pretrained Language Models

Searching arXiv for IndoBERT-related papers to ground the article in the latest and foundational literature.
arxiv_search({"query":"IndoBERT Indonesian BERT IndoNLU IndoLEM IndoBERTweet NusaBERT", "max_results": 10, "sort_by": "relevance"})
IndoBERT is the name used in the literature for Indonesian monolingual pretrained Transformer language models built in the style of BERT, first established as part of benchmark-building efforts for Indonesian natural language understanding and later reused as the backbone for a wide range of downstream systems and adapted variants. In its benchmark role, IndoBERT functions both as a released pretrained resource and as a baseline against multilingual encoders such as mBERT and XLM-R; in later work, checkpoints such as `indobenchmark/indobert-base-p1`, `indobert-base-p1`, and IndoBERT Large P2 are fine-tuned for sequence classification, token labeling, extractive question answering, context-conditioned pair classification, and deployment-oriented web services [2009.05387][2011.00677].

## 1. Origins and model family

IndoBERT emerged from two closely related infrastructure-building efforts for Indonesian NLP. In "IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding" [2009.05387], IndoBERT is introduced as a family of Indonesian monolingual pretrained language models released alongside a 12-task benchmark. In "IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP" [2011.00677], IndoBERT is presented as a monolingual Indonesian BERT model released with a seven-task benchmark suite. In both cases, the motivating diagnosis is similar: Indonesian is heavily used online but had lacked consolidated benchmarks, standardized splits, and strong monolingual pretrained models.

The IndoNLU release distinguishes between **IndoBERT** and **IndoBERT-lite**, each with **BASE** and **LARGE** variants. The published configurations are as follows [2009.05387].

| Variant | Parameters | Core configuration |
|---|---:|---|
| IndoBERT-lite BASE | 11.7M | 12 layers, 12 heads, hidden size 768 |
| IndoBERT BASE | 124.5M | 12 layers, 12 heads, hidden size 768 |
| IndoBERT-lite LARGE | 17.7M | 24 layers, 16 heads, hidden size 1024 |
| IndoBERT LARGE | 335.2M | 24 layers, 16 heads, hidden size 1024 |

The IndoLEM release describes IndoBERT as following the **BERT-Base (uncased)** configuration, with **12 hidden layers**, **hidden size 768**, **12 attention heads**, and **feed-forward hidden size 3,072**, and with an Indonesian **WordPiece vocabulary** of size **31,923** [2011.00677]. Later downstream papers often refer instead to concrete checkpoints such as **`indobenchmark/indobert-base-p1`** or **IndoBERT Large P2**, indicating that “IndoBERT” functions as a model family label rather than a single immutable artifact.

A recurring theme across these releases is that IndoBERT is intended to specialize model capacity for Indonesian rather than distribute it across many languages. This suggests that the central design objective is not architectural novelty, but language-matched pretraining and evaluation infrastructure.

## 2. Pretraining data, tokenization, and optimization

The published pretraining recipes under the IndoBERT name differ by release. In the IndoNLU line, IndoBERT and IndoBERT-lite are trained on **Indo4B**, a corpus of **3,581,301,476 words**, **275,301,176 sentences**, and **23.43 GB** of preprocessed text assembled from sources including **OSCAR**, **CoNLLu Common Crawl**, **OpenSubtitles**, **Twitter Crawl**, **Wikipedia Dump**, **Wikipedia CoNLLu**, **Twitter UI**, **OPUS JW300**, **Tempo**, **Kompas**, **TED**, **BPPT**, **Parallel Corpus**, **TALPCo**, and **Frog Storytelling** [2009.05387]. This corpus is explicitly mixed in style, combining formal and colloquial Indonesian.

The same release uses **SentencePiece** with **byte pair encoding (BPE)**, with vocabulary sizes **30,522** for IndoBERT and **30,000** for IndoBERT-lite. Pretraining is conducted in **two phases**: first with maximum sequence length **128**, then with maximum sequence length **512**, on **TPUv3-8** hardware. The paper states that **IndoBERT models were trained using masked language modeling loss**, while **IndoBERT-lite** follows ALBERT-style pretraining and uses **whole word masked loss**; both limit the **maximum prediction per sequence** to **20 tokens** [2009.05387].

The IndoLEM line describes a different pretraining setup. There, IndoBERT is trained on approximately **220 million words** drawn from **Indonesian Wikipedia (74M words)**, **news articles from Kompas, Tempo, and Liputan6 (55M words)**, and the **Indonesian Web Corpus (idWac, 90M words)** [2011.00677]. The corpus is **preprocessed into 512-token document blocks**, yielding **1,067,581 training instances** and **13,985 development instances**. The authors also state that they **modified the Huggingface framework to read a separate text stream for different document blocks**, because the existing implementation merged all documents into one stream. The reported optimization setup is **Adam**, **learning rate 1e-4**, **batch size 128**, **linear scheduler**, **2.4M steps**, **180 epochs**, on **4 NVIDIA V100 GPUs, 16GB each**, with final development-set **perplexity 3.97** [2011.00677].

These differences matter because later literature cites “IndoBERT” in both senses: as the IndoNLU/Indo4B family and as the IndoLEM WordPiece-based release. A plausible implication is that downstream papers using the IndoBERT name may inherit distinct tokenizers, vocabulary statistics, or checkpoint histories even when the architectural label is the same.

## 3. Extensions, lightweight variants, and post-training

Several later models are best understood as deliberate extensions of IndoBERT rather than replacements. The most explicit domain adaptation is **IndoBERTweet**, introduced as a Twitter-specialized extension of **`indobert-base-uncased`** [2109.04607]. Its premise is that general Indonesian BERT suffers from **vocabulary mismatch** on social media text. The paper constructs a Twitter vocabulary of **31,984 WordPiece types**, of which **17,400** are shared with the original IndoBERT vocabulary and **14,584** are new. After comparing several embedding-initialization strategies for new vocabulary items, the authors report that **initializing with the average BERT subword embedding makes pretraining five times faster** and yields the strongest downstream average after **200K** domain-adaptive steps, surpassing a from-scratch Twitter model trained for **1M** steps [2109.04607].

A second extension is **NusaBERT**, framed explicitly as “teaching IndoBERT to be multilingual and multicultural” [2403.01817]. NusaBERT keeps the original BERT architecture and instead expands IndoBERT’s vocabulary from **30,521** to **32,032** by adding **1,511 new non-overlapping tokens**, then continues pretraining on a **13-language** corpus containing **28,432,448** documents. The motivating claim is that IndoBERT is “too Indonesian-centric” for an ecology characterized by **700+ languages and dialects**, code-switching, and extremely low-resource regional languages. The result is a continued-pretrained model family, **NusaBERTBASE (111M)** and **NusaBERTLARGE (337M)**, that generally improves multilingual local-language sentiment and emotion tasks while introducing trade-offs on some Indonesian-only classification benchmarks [2403.01817].

IndoBERT has also been adapted through domain-specific continual pretraining without vocabulary change. In Indonesian financial NLP, generic **IndoBERT base** and **IndoBERT large** are post-trained with **masked language modeling** for **20 epochs**, **batch size 8**, **learning rate 2e-5**, **weight decay 0.01**, and **MLM masking probability 0.15** on a **647,586-word** financial corpus assembled from news and corporate reports [2310.09736]. The paper reports that post-training increases effectiveness on downstream financial sentiment and topic classification, with the clearest gains for **IndoBERT base**, especially when labeled downstream data are limited.

Lightweight deployment-oriented use also appears in work on clickbait detection. There, the authors switch from an earlier Multilingual BERT backbone to **IndoBERT Lite Base**, described as **12 layers** and about **11.7 million parameters**, in order to reduce memory usage [2102.10601]. The paper reports a **mean ROC-AUC of 89%** and a reduction of approximately **800 MB** in memory usage relative to the earlier system.

## 4. Fine-tuning paradigms and task formulations

The downstream use of IndoBERT is methodologically diverse, but most formulations fall into a small number of recurrent patterns. The most common is standard sequence classification. In Indonesian Twitter sentiment analysis on discourse about **Ibu Kota Nusantara (IKN)**, the model **`indobenchmark/indobert-base-p1`** is described as having **12 Transformer layers, 768 hidden units, 12 attention heads, and about 111M parameters**, with a **linear classifier head on top of the `[CLS]` representation**. Fine-tuning uses **MAX_LEN = 128**, **BATCH_SIZE = 16**, **EPOCHS = 5**, **AdamW**, **learning rate \(2 \times 10^{-5}\)**, **weight decay 0.01**, **linear warmup for 10% of total steps**, **linear decay**, and **gradient clipping `max_norm = 1.0`** [2604.25392].

A related but more structured pattern is context-conditioned pair classification. **IndoBERT-Relevancy** and **IndoBERT-Sentiment** both encode paired inputs in the format **`[CLS] context [SEP] text [SEP]`**, using **IndoBERT Large P2** as a cross-encoder over topic-text pairs rather than classifying text in isolation [2603.26095][2604.07057]. In these models, relevance or sentiment is explicitly defined as a property of a text relative to a topic description. This is a substantial reformulation of the task even though the encoder remains a standard BERT-family model.

IndoBERT also appears in extractive question answering. In **QASiNa**, the model **`indobert-base-p1` (124.5M parameters)** is fine-tuned with **Hugging Face `Trainer`** and **`AutoModelForQuestionAnswering`** on **Indonesian translation of SQuAD v2.0 (SQuAD-ID)**, then evaluated on a religion-specific extractive QA dataset built from **Sirah Nabawiyah** literature [2310.08102]. The labels are answer token positions in context, so the task is standard span extraction rather than generative answering.

A further usage pattern treats IndoBERT as a contextual feature extractor rather than the end classifier itself. In Indonesian COVID-19 misinformation detection, IndoBERT provides representations both for a fully fine-tuned BERT sequence classifier and for downstream models such as **Bi-LSTM**, **CNN**, **DNN**, **SVM**, **Logistic Regression**, **Random Forest**, and **XGBoost** [2206.15359]. The strongest overall system is a two-stage pipeline: a BERT sequence classifier with IndoBERT for relevance prediction, followed by a **Bi-LSTM** using IndoBERT features for misinformation detection.

Deployment papers show that these fine-tuning paradigms are not confined to offline experiments. The clickbait-detection system exposes an IndoBERT-based classifier through a **RESTful API** with a **Flask** backend, **Gunicorn**, **Nginx**, **MySQL**, and a **DigitalOcean** deployment target, while the final sentiment-classification paper on Tokopedia reviews pushes a fine-tuned IndoBERT model and tokenizer to the **Hugging Face Hub** and wraps inference in **Gradio** [2102.10601][2605.03439].

## 5. Benchmark results and downstream empirical profile

On the IndoNLU benchmark, **IndoBERT\(_\text{LARGE}\)** achieves the highest reported **average classification** score, **88.43 macro-F1**, across **EmoT**, **SmSA**, **CASA**, **HoASA**, and **WReTE** [2009.05387]. It is also highly competitive on sequence labeling, with **81.21** average for **IndoBERT\(_\text{LARGE}\)** and **81.26** for the phase-two version, though **XLM-R\(_\text{LARGE}\)** is slightly higher at **81.92** average. The same paper states that IndoBERT models outperform multilingual models on **8 out of 12 tasks**, indicating that monolingual Indonesian pretraining is especially strong for semantics-heavy classification tasks, though not uniformly dominant.

On the IndoLEM benchmark, IndoBERT is strongest on nearly every task reported. It reaches **84.13 F1** on sentiment analysis, **90.1** on **NER UI**, **74.9** on **NER UGM**, **93.7** on **Next Tweet Prediction**, and **0.59** Spearman correlation on **Tweet Ordering** [2011.00677]. The main exception is **UD-Indo-PUD**, where **mBERT** reaches **90.58 / 85.44** for **UAS / LAS**, compared with **89.23 / 83.95** for IndoBERT. The paper attributes this difference to the translated and foreign-name-heavy character of the PUD data.

Later downstream evaluations reinforce this picture but also add caveats. On informal Indonesian Twitter sentiment about **IKN**, fine-tuned IndoBERT reaches **89.59% accuracy** and **89.37% macro F1**, compared with **77.57% accuracy** and **77.17% F1** for the strongest classical baseline, **Logistic Regression**, a gain of about **12.02 percentage points in accuracy** [2604.25392]. In a PPKM-policy sentiment study, IndoBERT reports **F1-score 84%**, compared with **70%** for **SVM** and **83%** for **Multinomial Naive Bayes** [2301.00096]. In Indonesian clickbait detection, **IndoBERT Lite Base** supports a deployed system with **mean ROC-AUC of 89%** while reducing memory usage relative to a prior Multilingual BERT setup [2102.10601].

Context-conditioned models built on IndoBERT produce some of the largest reported downstream margins. **IndoBERT-Relevancy** reaches **F1 = 0.948** and **accuracy = 96.5%** on **31,360** context-text pairs spanning **188 topics** [2603.26095]. **IndoBERT-Sentiment**, trained on the same paired-input design but relabeled for sentiment, reports **88.1% accuracy** and **0.856 macro-F1**, outperforming the best context-free baseline by **35.6** macro-F1 points [2604.07057].

At the same time, IndoBERT is not universally best. In **QASiNa**, after fine-tuning on **SQuAD-ID**, IndoBERT reaches **EM 42.40**, **F1 57.77**, and **Substring Match 49.00** on the religious-domain test set, trailing both **mBERT** and **XLM-R** [2310.08102]. In Tokopedia review sentiment classification, fine-tuned IndoBERT reports **Accuracy 0.8870**, **Macro F1 0.5088**, and **Weighted F1 0.9268**, while **TF-IDF + Linear SVC** reaches **Accuracy 0.9760** and **Macro F1 0.5510** [2605.03439]. That paper explicitly argues that the gap is “primarily driven by discrepancies in the data sampling regimes,” because the Transformer was trained on a reduced subset of approximately **1,800 training records** and **3,000 test records**, while the baselines used the full **65,335-record** corpus.

The empirical record therefore supports a precise formulation: IndoBERT is frequently a strong or leading Indonesian backbone, but published outcomes remain sensitive to domain match, sampling parity, model size, and evaluation protocol.

## 6. Limitations, trade-offs, and open directions

Several limitations recur across the literature. The first is **compute and deployment cost**. Base IndoBERT variants are reported as **about 111M**, **124.5M**, or similar magnitudes depending on release, while large variants reach **335M** or **337M** parameters [2604.25392][2009.05387][2403.01817]. Downstream work notes that even CPU-only fine-tuning can outperform classical models, but at higher computational cost; this is why lighter variants such as **IndoBERT Lite Base** remain attractive in deployment-oriented systems [2604.25392][2102.10601].

The second is **sensitivity to data regime and overfitting**. In the IKN sentiment study, the best checkpoint is obtained already at **epoch 1** with **validation accuracy 85.07%**, after which validation loss increases while training accuracy approaches **100%**, which the authors interpret as overfitting [2604.25392]. In Indonesian emotion classification on PRDECT-ID, the best IndoBERT configuration reaches **0.8000** evaluation accuracy only after aggressive data augmentation and regularization tuning, while rising evaluation loss with more epochs again suggests overfitting [2509.14611]. The same paper also reports that removing stop words harms IndoBERT, which the authors attribute to the loss of sentence context needed by contextual transformer models.

The third is **domain and language coverage**. IndoBERTweet is motivated by nearly half of the Twitter vocabulary being absent from the original IndoBERT vocabulary, while NusaBERT is motivated by Indonesia’s **700+ languages and dialects**, regional multilingualism, and code-switching [2109.04607][2403.01817]. NusaBERT improves multilingual local-language performance but slightly underperforms IndoBERT on some Indonesian-only classification tasks and yields task-dependent code-mixing robustness. This suggests that IndoBERT’s monolingual specialization is both a strength and a boundary condition.

The fourth is **evaluation asymmetry and reporting incompleteness** in downstream practice. Several papers using IndoBERT omit optimizer identity, scheduler type, maximum sequence length, validation strategy, or complete confusion-matrix analysis [2605.03439][2509.14611]. A plausible implication is that reproducibility and fair comparison remain uneven even when the backbone itself is standardized.

Open directions stated in the literature include continued domain-adaptive pretraining, richer treatment of slang and abbreviations, more balanced multilingual coverage, better handling of implicit and context-dependent semantics, stronger diagnostics for minority classes, and targeted data construction rather than indiscriminate scaling [2310.09736][2403.01817][2603.26095]. Across these developments, IndoBERT remains less a single model than a durable Indonesian-language modeling paradigm: BERT-style pretraining anchored in Indonesian data, then specialized through vocabulary adaptation, continual pretraining, pairwise conditioning, or deployment-aware compression.

Source: https://www.emergentmind.com/topics/indobert