IndoBERT: Indonesian Pretrained Language Models

• IndoBERT is a family of Indonesian monolingual pretrained Transformer models designed for natural language understanding and adapted as a backbone for numerous NLP tasks.
• Models are pretrained on extensive Indonesian corpora using masked language modeling and fine-tuned for tasks like sentiment analysis, token labeling, extractive QA, and context-conditioned pair classification.
• Extensions such as IndoBERTweet and NusaBERT address vocabulary mismatches and multilingual challenges, while lightweight variants enable efficient deployment.

Searching arXiv for IndoBERT-related papers to ground the article in the latest and foundational literature. arxiv_search({"query":"IndoBERT Indonesian BERT IndoNLU IndoLEM IndoBERTweet NusaBERT", "max_results": 10, "sort_by": "relevance"}) IndoBERT is the name used in the literature for Indonesian monolingual pretrained Transformer LLMs built in the style of BERT, first established as part of benchmark-building efforts for Indonesian natural language understanding and later reused as the backbone for a wide range of downstream systems and adapted variants. In its benchmark role, IndoBERT functions both as a released pretrained resource and as a baseline against multilingual encoders such as mBERT and XLM-R; in later work, checkpoints such as indobenchmark/indobert-base-p1, indobert-base-p1, and IndoBERT Large P2 are fine-tuned for sequence classification, token labeling, extractive question answering, context-conditioned pair classification, and deployment-oriented web services (Wilie et al., 2020, Koto et al., 2020).

1. Origins and model family

IndoBERT emerged from two closely related infrastructure-building efforts for Indonesian NLP. In "IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding" (Wilie et al., 2020), IndoBERT is introduced as a family of Indonesian monolingual pretrained LLMs released alongside a 12-task benchmark. In "IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained LLM for Indonesian NLP" (Koto et al., 2020), IndoBERT is presented as a monolingual Indonesian BERT model released with a seven-task benchmark suite. In both cases, the motivating diagnosis is similar: Indonesian is heavily used online but had lacked consolidated benchmarks, standardized splits, and strong monolingual pretrained models.

The IndoNLU release distinguishes between IndoBERT and IndoBERT-lite, each with BASE and LARGE variants. The published configurations are as follows (Wilie et al., 2020).

Variant	Parameters	Core configuration
IndoBERT-lite BASE	11.7M	12 layers, 12 heads, hidden size 768
IndoBERT BASE	124.5M	12 layers, 12 heads, hidden size 768
IndoBERT-lite LARGE	17.7M	24 layers, 16 heads, hidden size 1024
IndoBERT LARGE	335.2M	24 layers, 16 heads, hidden size 1024

The IndoLEM release describes IndoBERT as following the BERT-Base (uncased) configuration, with 12 hidden layers, hidden size 768, 12 attention heads, and feed-forward hidden size 3,072, and with an Indonesian WordPiece vocabulary of size 31,923 (Koto et al., 2020). Later downstream papers often refer instead to concrete checkpoints such as indobenchmark/indobert-base-p1 or IndoBERT Large P2, indicating that “IndoBERT” functions as a model family label rather than a single immutable artifact.

A recurring theme across these releases is that IndoBERT is intended to specialize model capacity for Indonesian rather than distribute it across many languages. This suggests that the central design objective is not architectural novelty, but language-matched pretraining and evaluation infrastructure.

2. Pretraining data, tokenization, and optimization

The published pretraining recipes under the IndoBERT name differ by release. In the IndoNLU line, IndoBERT and IndoBERT-lite are trained on Indo4B, a corpus of 3,581,301,476 words, 275,301,176 sentences, and 23.43 GB of preprocessed text assembled from sources including OSCAR, CoNLLu Common Crawl, OpenSubtitles, Twitter Crawl, Wikipedia Dump, Wikipedia CoNLLu, Twitter UI, OPUS JW300, Tempo, Kompas, TED, BPPT, Parallel Corpus, TALPCo, and Frog Storytelling (Wilie et al., 2020). This corpus is explicitly mixed in style, combining formal and colloquial Indonesian.

The same release uses SentencePiece with byte pair encoding (BPE), with vocabulary sizes 30,522 for IndoBERT and 30,000 for IndoBERT-lite. Pretraining is conducted in two phases: first with maximum sequence length 128, then with maximum sequence length 512, on TPUv3-8 hardware. The paper states that IndoBERT models were trained using masked language modeling loss, while IndoBERT-lite follows ALBERT-style pretraining and uses whole word masked loss; both limit the maximum prediction per sequence to 20 tokens (Wilie et al., 2020).

The IndoLEM line describes a different pretraining setup. There, IndoBERT is trained on approximately 220 million words drawn from Indonesian Wikipedia (74M words), news articles from Kompas, Tempo, and Liputan6 (55M words), and the Indonesian Web Corpus (idWac, 90M words) (Koto et al., 2020). The corpus is preprocessed into 512-token document blocks, yielding 1,067,581 training instances and 13,985 development instances. The authors also state that they modified the Huggingface framework to read a separate text stream for different document blocks, because the existing implementation merged all documents into one stream. The reported optimization setup is Adam, learning rate 1e-4, batch size 128, linear scheduler, 2.4M steps, 180 epochs, on 4 NVIDIA V100 GPUs, 16GB each, with final development-set perplexity 3.97 (Koto et al., 2020).

These differences matter because later literature cites “IndoBERT” in both senses: as the IndoNLU/Indo4B family and as the IndoLEM WordPiece-based release. A plausible implication is that downstream papers using the IndoBERT name may inherit distinct tokenizers, vocabulary statistics, or checkpoint histories even when the architectural label is the same.

3. Extensions, lightweight variants, and post-training

Several later models are best understood as deliberate extensions of IndoBERT rather than replacements. The most explicit domain adaptation is IndoBERTweet, introduced as a Twitter-specialized extension of indobert-base-uncased (Koto et al., 2021). Its premise is that general Indonesian BERT suffers from vocabulary mismatch on social media text. The paper constructs a Twitter vocabulary of 31,984 WordPiece types, of which 17,400 are shared with the original IndoBERT vocabulary and 14,584 are new. After comparing several embedding-initialization strategies for new vocabulary items, the authors report that initializing with the average BERT subword embedding makes pretraining five times faster and yields the strongest downstream average after 200K domain-adaptive steps, surpassing a from-scratch Twitter model trained for 1M steps (Koto et al., 2021).

A second extension is NusaBERT, framed explicitly as “teaching IndoBERT to be multilingual and multicultural” (Wongso et al., 2024). NusaBERT keeps the original BERT architecture and instead expands IndoBERT’s vocabulary from 30,521 to 32,032 by adding 1,511 new non-overlapping tokens, then continues pretraining on a 13-language corpus containing 28,432,448 documents. The motivating claim is that IndoBERT is “too Indonesian-centric” for an ecology characterized by 700+ languages and dialects, code-switching, and extremely low-resource regional languages. The result is a continued-pretrained model family, NusaBERTBASE (111M) and NusaBERTLARGE (337M), that generally improves multilingual local-language sentiment and emotion tasks while introducing trade-offs on some Indonesian-only classification benchmarks (Wongso et al., 2024).

IndoBERT has also been adapted through domain-specific continual pretraining without vocabulary change. In Indonesian financial NLP, generic IndoBERT base and IndoBERT large are post-trained with masked language modeling for 20 epochs, batch size 8, learning rate 2e-5, weight decay 0.01, and MLM masking probability 0.15 on a 647,586-word financial corpus assembled from news and corporate reports (Maharani et al., 2023). The paper reports that post-training increases effectiveness on downstream financial sentiment and topic classification, with the clearest gains for IndoBERT base, especially when labeled downstream data are limited.

Lightweight deployment-oriented use also appears in work on clickbait detection. There, the authors switch from an earlier Multilingual BERT backbone to IndoBERT Lite Base, described as 12 layers and about 11.7 million parameters, in order to reduce memory usage (Fakhruzzaman et al., 2021). The paper reports a mean ROC-AUC of 89% and a reduction of approximately 800 MB in memory usage relative to the earlier system.

4. Fine-tuning paradigms and task formulations

The downstream use of IndoBERT is methodologically diverse, but most formulations fall into a small number of recurrent patterns. The most common is standard sequence classification. In Indonesian Twitter sentiment analysis on discourse about Ibu Kota Nusantara (IKN), the model indobenchmark/indobert-base-p1 is described as having 12 Transformer layers, 768 hidden units, 12 attention heads, and about 111M parameters, with a linear classifier head on top of the [CLS] representation. Fine-tuning uses MAX_LEN = 128, BATCH_SIZE = 16, EPOCHS = 5, AdamW, learning rate $2 \times 10^{-5}$, weight decay 0.01, linear warmup for 10% of total steps, linear decay, and gradient clipping max_norm = 1.0 (Mayzaroh et al., 28 Apr 2026).

A related but more structured pattern is context-conditioned pair classification. IndoBERT-Relevancy and IndoBERT-Sentiment both encode paired inputs in the format [CLS] context [[SEP](https://www.emergentmind.com/topics/semantic-entropy-production-sep-metric)] text [SEP], using IndoBERT Large P2 as a cross-encoder over topic-text pairs rather than classifying text in isolation (Saputra et al., 27 Mar 2026, Saputra et al., 8 Apr 2026). In these models, relevance or sentiment is explicitly defined as a property of a text relative to a topic description. This is a substantial reformulation of the task even though the encoder remains a standard BERT-family model.

IndoBERT also appears in extractive question answering. In QASiNa, the model indobert-base-p1 (124.5M parameters) is fine-tuned with Hugging Face Trainer and AutoModelForQuestionAnswering on Indonesian translation of SQuAD v2.0 (SQuAD-ID), then evaluated on a religion-specific extractive QA dataset built from Sirah Nabawiyah literature (Rizqullah et al., 2023). The labels are answer token positions in context, so the task is standard span extraction rather than generative answering.

A further usage pattern treats IndoBERT as a contextual feature extractor rather than the end classifier itself. In Indonesian COVID-19 misinformation detection, IndoBERT provides representations both for a fully fine-tuned BERT sequence classifier and for downstream models such as Bi-LSTM, CNN, DNN, SVM, Logistic Regression, Random Forest, and XGBoost (Faisal et al., 2022). The strongest overall system is a two-stage pipeline: a BERT sequence classifier with IndoBERT for relevance prediction, followed by a Bi-LSTM using IndoBERT features for misinformation detection.

Deployment papers show that these fine-tuning paradigms are not confined to offline experiments. The clickbait-detection system exposes an IndoBERT-based classifier through a RESTful API with a Flask backend, Gunicorn, Nginx, MySQL, and a DigitalOcean deployment target, while the final sentiment-classification paper on Tokopedia reviews pushes a fine-tuned IndoBERT model and tokenizer to the Hugging Face Hub and wraps inference in Gradio (Fakhruzzaman et al., 2021, Zahra et al., 5 May 2026).

5. Benchmark results and downstream empirical profile

On the IndoNLU benchmark, IndoBERT$_\text{LARGE}$ achieves the highest reported average classification score, 88.43 macro-F1, across EmoT, SmSA, CASA, HoASA, and WReTE (Wilie et al., 2020). It is also highly competitive on sequence labeling, with 81.21 average for IndoBERT$_\text{LARGE}$ and 81.26 for the phase-two version, though XLM-R$_\text{LARGE}$ is slightly higher at 81.92 average. The same paper states that IndoBERT models outperform multilingual models on 8 out of 12 tasks, indicating that monolingual Indonesian pretraining is especially strong for semantics-heavy classification tasks, though not uniformly dominant.

On the IndoLEM benchmark, IndoBERT is strongest on nearly every task reported. It reaches 84.13 F1 on sentiment analysis, 90.1 on NER UI, 74.9 on NER UGM, 93.7 on Next Tweet Prediction, and 0.59 Spearman correlation on Tweet Ordering (Koto et al., 2020). The main exception is UD-Indo-PUD, where mBERT reaches 90.58 / 85.44 for UAS / LAS, compared with 89.23 / 83.95 for IndoBERT. The paper attributes this difference to the translated and foreign-name-heavy character of the PUD data.

Later downstream evaluations reinforce this picture but also add caveats. On informal Indonesian Twitter sentiment about IKN, fine-tuned IndoBERT reaches 89.59% accuracy and 89.37% macro F1, compared with 77.57% accuracy and 77.17% F1 for the strongest classical baseline, Logistic Regression, a gain of about 12.02 percentage points in accuracy (Mayzaroh et al., 28 Apr 2026). In a PPKM-policy sentiment study, IndoBERT reports F1-score 84%, compared with 70% for SVM and 83% for Multinomial Naive Bayes (Fransiscus et al., 2022). In Indonesian clickbait detection, IndoBERT Lite Base supports a deployed system with mean ROC-AUC of 89% while reducing memory usage relative to a prior Multilingual BERT setup (Fakhruzzaman et al., 2021).

Context-conditioned models built on IndoBERT produce some of the largest reported downstream margins. IndoBERT-Relevancy reaches F1 = 0.948 and accuracy = 96.5% on 31,360 context-text pairs spanning 188 topics (Saputra et al., 27 Mar 2026). IndoBERT-Sentiment, trained on the same paired-input design but relabeled for sentiment, reports 88.1% accuracy and 0.856 macro-F1, outperforming the best context-free baseline by 35.6 macro-F1 points (Saputra et al., 8 Apr 2026).

At the same time, IndoBERT is not universally best. In QASiNa, after fine-tuning on SQuAD-ID, IndoBERT reaches EM 42.40, F1 57.77, and Substring Match 49.00 on the religious-domain test set, trailing both mBERT and XLM-R (Rizqullah et al., 2023). In Tokopedia review sentiment classification, fine-tuned IndoBERT reports Accuracy 0.8870, Macro F1 0.5088, and Weighted F1 0.9268, while TF-IDF + Linear SVC reaches Accuracy 0.9760 and Macro F1 0.5510 (Zahra et al., 5 May 2026). That paper explicitly argues that the gap is “primarily driven by discrepancies in the data sampling regimes,” because the Transformer was trained on a reduced subset of approximately 1,800 training records and 3,000 test records, while the baselines used the full 65,335-record corpus.

The empirical record therefore supports a precise formulation: IndoBERT is frequently a strong or leading Indonesian backbone, but published outcomes remain sensitive to domain match, sampling parity, model size, and evaluation protocol.

6. Limitations, trade-offs, and open directions

Several limitations recur across the literature. The first is compute and deployment cost. Base IndoBERT variants are reported as about 111M, 124.5M, or similar magnitudes depending on release, while large variants reach 335M or 337M parameters (Mayzaroh et al., 28 Apr 2026, Wilie et al., 2020, Wongso et al., 2024). Downstream work notes that even CPU-only fine-tuning can outperform classical models, but at higher computational cost; this is why lighter variants such as IndoBERT Lite Base remain attractive in deployment-oriented systems (Mayzaroh et al., 28 Apr 2026, Fakhruzzaman et al., 2021).

The second is sensitivity to data regime and overfitting. In the IKN sentiment study, the best checkpoint is obtained already at epoch 1 with validation accuracy 85.07%, after which validation loss increases while training accuracy approaches 100%, which the authors interpret as overfitting (Mayzaroh et al., 28 Apr 2026). In Indonesian emotion classification on PRDECT-ID, the best IndoBERT configuration reaches 0.8000 evaluation accuracy only after aggressive data augmentation and regularization tuning, while rising evaluation loss with more epochs again suggests overfitting (Christian et al., 18 Sep 2025). The same paper also reports that removing stop words harms IndoBERT, which the authors attribute to the loss of sentence context needed by contextual transformer models.

The third is domain and language coverage. IndoBERTweet is motivated by nearly half of the Twitter vocabulary being absent from the original IndoBERT vocabulary, while NusaBERT is motivated by Indonesia’s 700+ languages and dialects, regional multilingualism, and code-switching (Koto et al., 2021, Wongso et al., 2024). NusaBERT improves multilingual local-language performance but slightly underperforms IndoBERT on some Indonesian-only classification tasks and yields task-dependent code-mixing robustness. This suggests that IndoBERT’s monolingual specialization is both a strength and a boundary condition.

The fourth is evaluation asymmetry and reporting incompleteness in downstream practice. Several papers using IndoBERT omit optimizer identity, scheduler type, maximum sequence length, validation strategy, or complete confusion-matrix analysis (Zahra et al., 5 May 2026, Christian et al., 18 Sep 2025). A plausible implication is that reproducibility and fair comparison remain uneven even when the backbone itself is standardized.

Open directions stated in the literature include continued domain-adaptive pretraining, richer treatment of slang and abbreviations, more balanced multilingual coverage, better handling of implicit and context-dependent semantics, stronger diagnostics for minority classes, and targeted data construction rather than indiscriminate scaling (Maharani et al., 2023, Wongso et al., 2024, Saputra et al., 27 Mar 2026). Across these developments, IndoBERT remains less a single model than a durable Indonesian-language modeling paradigm: BERT-style pretraining anchored in Indonesian data, then specialized through vocabulary adaptation, continual pretraining, pairwise conditioning, or deployment-aware compression.