BERTomelo: Your Portuguese Encoder Best Friend

Abstract: Encoders have become the state of the art for multiple NLP tasks, especially those requiring deep contextual understanding. While multilingual models offer broad coverage, dedicated monolingual encoders are essential for capturing the unique lexical and syntactic nuances of specific languages. For Portuguese, however, existing monolingual options like BERTimbau and Albertina have not kept pace with recent architectural breakthroughs, often lagging behind English benchmarks in scalability and efficiency. This work introduces BERTomelo, a next-generation monolingual encoder pre-trained from scratch and specifically optimized for the Portuguese language. By leveraging the ModernBERT architecture, BERTomelo overcomes the limitations of previous models, offering Base and Large versions with a 1,024-token context window and hardware-level optimizations like FlashAttention and alternating attention mechanisms. The model was trained on ClassiCC-PT, a massive, high-quality Portuguese corpus of 106 million documents, ensuring superior alignment with the language's contemporary usage. The results demonstrate that BERTomelo not only outperforms previous Portuguese encoders but also provides a more robust and efficient alternative to massive multilingual models in downstream tasks such as STS and NER.

• The paper introduces BERTomelo, a novel monolingual encoder for Brazilian Portuguese that integrates cutting-edge techniques like FlashAttention and RoPE.
• The model achieves high performance on downstream tasks, with the Base variant reaching 79.26% MLM accuracy and outperforming prior benchmarks in NER and STS.
• Its scalable design, featuring a 1,024-token context window and an efficient training strategy, paves the way for improved long-document processing in Portuguese.

BERTomelo: A Modern Monolingual Encoder for Brazilian Portuguese

BERTomelo (2606.28999) presents a significant advancement in encoder-based language modeling for Brazilian Portuguese, addressing persistent limitations in prior monolingual architectures. The model suite—developed from scratch over a large, high-quality corpus—is architecturally engineered for scalability, efficiency, and high-quality downstream transfer. This analysis describes the motivation, model design, training strategy, evaluation benchmarks, and the broader implications for the field.

Motivation and Context

Encoder models remain central in a range of NLP tasks, particularly those requiring deep contextual embeddings for retrieval, classification, clustering, and semantic analysis. While transformer-based encoders like BERT have been extensively leveraged for English, the development of competitive monolingual equivalents for other languages has lagged. Prior Portuguese encoders such as BERTimbau and Albertina have not fully incorporated recent architectural enhancements, typically featuring limited context windows, lack of memory optimization, and suboptimal efficiency compared to their English counterparts.

Multilingual encoders (e.g., mBERT, XLM-RoBERTa) support a broad set of languages but introduce a performance trade-off for individual languages due to capacity limitations—commonly termed the "Curse of Multilinguality". Specialized monolingual training, as empirically validated in multiple languages (e.g., CamemBERT, FinBERT), yields superior results on language-specific tasks. Within the Portuguese NLP landscape, the need has emerged for a model that matches the architectural state of the art in high-resource languages.

Model Architecture and Training

BERTomelo is implemented in two variants—Base and Large—leveraging the ModernBERT[2024] architecture as a foundational blueprint. Both variants are structured with extended 1,024-token context windows, exceeding the standard 512-token limit of legacy BERT models.

Key architectural features include:

• FlashAttention-2: Efficient memory and compute-aware attention mechanism for accelerated pretraining and inference.
• Alternating Attention: Alternates between local and global attention layers to optimize sequence modeling and hardware utilization.
• Unpadding: Removes padding tokens for improved token throughput and reduced computational redundancy.
• RoPE Positional Embeddings: Enables position extrapolation and sequence length flexibility, a prerequisite for future context window extension.
• Bias-free linear layers, pre-normalization, GeGLU activation: Integration of advanced components to enhance stability, convergence, and expressive power.

The model is pre-trained from scratch over ClassiCC-PT, a curated 106-million-document corpus derived from contemporary Common Crawl data with rigorous language and quality filtering. The tokenizer is a Unigram-based model with a 32,768 token vocabulary, retaining compatibility with existing Portuguese resources.

Pretraining omits the Next Sentence Prediction objective, employing only Masked Language Modeling with a 30% masking ratio—a configuration reflecting evidence on the sample efficiency of pure MLM objectives. Extended context and local/global attention alternation are set with a window of 128 tokens. Both model variants are trained for 2M steps with large effective batch sizes, utilizing bf16 precision on AMD MI300X (Base) and MI325X (Large) GPUs.

Training Stability and Intrinsic Evaluation

Model convergence and training stability are extensively monitored. The loss curves indicate smooth convergence without instability or overfitting for both variants. The alignment between training and evaluation loss, as well as controlled gradient norms, demonstrates robust generalization and stable optimization throughout the process.

Figure 1: Training metrics for the BERTomelo variants.

Intrinsic assessment of tokenization confirms the Unigram tokenizer's appropriateness, with low subword fertility (1.506) and a negligible unknown token ratio (0.0017%). On MLM test evaluation (3.6M sequences), the Base model achieves 79.26% accuracy (loss 0.9495), while the Large model achieves 82.28% (loss 0.7799), reflecting strong internalization of Portuguese syntactic and semantic properties.

Downstream Results and Performance

BERTomelo is benchmarked against a comprehensive set of monolingual and multilingual models, including BERTimbau, BERTuguês, Albertina, mBERT, ModernBERT, XLM-RoBERTa, and ModBERTBr. Evaluations include:

• Semantic Textual Similarity (STS, ASSIN2 corpus): Measured via Mean Squared Error (MSE) and Pearson correlation.
• Recognizing Textual Entailment (RTE, ASSIN2): Accuracy and Macro F1.
• Named Entity Recognition (NER, LeNER-Br): Micro F1, recall, precision.

Key findings:

• BERTomelo delivers the lowest MSE and the highest Micro F1 and Precision scores in NER (Base variant), with the Large variant maintaining or exceeding the performance of all comparators except in RTE, where BERTimbau slightly surpasses it.
• The models' performance in NER and STS outperforms both prior monolingual and multilingual baselines—a direct consequence of the combination of monolingual training and advanced architecture.
• The extended context window (1,024 tokens) and optimized training pipeline are not fully stress-tested in these benchmarks, but represent substantial theoretical and practical improvements for long-sequence tasks.

Theoretical and Practical Implications

BERTomelo demonstrates that contemporary architectural innovations (FlashAttention, RoPE, alternating attention) deliver measurable performance and efficiency gains when transposed to low-resource language settings. The results highlight that dedicated monolingual models remain unrivaled for language-specific benchmarks, even against large multilingual competitors. The computational and architectural alignment with leading English encoders sets a new baseline for Portuguese NLP.

The open release of the models and codebase ensures reproducibility and community adoption, enabling further research into domain adaptation and specialized tuning for sub-domains such as law, healthcare, or social media.

Future Directions

Building on RoPE-based position encoding, BERTomelo is positioned for further extension to longer context windows (up to 8,192 tokens), matching the most advanced English encoders and opening new avenues for document-level understanding and retrieval in Portuguese. Planned adoption of sequence packing algorithms will maximize the efficiency of long-context inference. Additional evaluation on long document and information retrieval benchmarks is an anticipated direction, which will empirically validate the benefits of the architectural improvements for tasks inherently reliant on long-range dependencies.

Conclusion

BERTomelo (2606.28999) represents a marked evolution of Portuguese-language encoder models, systematically integrating cutting-edge architectural innovations into a rigorously trained monolingual suite. Achieving state-of-the-art results in downstream tasks—especially NER and semantic similarity—and demonstrating strong intrinsic modeling capacity, BERTomelo reaffirms the practical necessity of language-specific, modern encoder development. Its extensible architecture, strong numerical results, and open distribution are set to influence both research and applied NLP workflows in Portuguese, and offer a template for further progress in encoder model design for other under-resourced languages.