BERTino: Lightweight Italian DistilBERT Model

• BERTino is an Italian lightweight DistilBERT model that leverages knowledge distillation to reduce computational and memory requirements while maintaining strong NLP performance.
• The model is constructed by halving the depth of a BERTBASE teacher and employs a two-stage distillation process with combined KL, MLM, and cosine losses.
• BERTino achieves close-to state-of-the-art results on several Italian NLP benchmarks while significantly shortening fine-tuning and inference times.

BERTino is an Italian-specific, lightweight DistilBERT model designed to deliver much of the language understanding power of a BERTBASE architecture while achieving significant reductions in computational and memory requirements. Developed with knowledge distillation from a large Italian BERTBASE teacher model, BERTino enables efficient deployment and fine-tuning in settings with limited hardware resources, while maintaining performance that is close to state-of-the-art on several core Italian NLP tasks (Muffo et al., 2023).

1. Architectural Overview

BERTino closely follows the DistilBERT framework first proposed by Sanh et al. (2019). The architecture is derived by halving the depth of a standard BERTBASE model: BERTino contains 6 Transformer encoder layers (compared to 12 in BERTBASE), retains the hidden dimensionality of 768, and preserves 12 self-attention heads per layer. This results in a total parameter count that is approximately 40% lower than a BERTBASE—66 million in BERTino versus 110 million in the teacher model. The vocabulary and tokenization protocol are inherited unchanged from the teacher (dbmdz/bert-base-italian-xxl-uncased), which uses a WordPiece inventory trained on Italian corpora (Muffo et al., 2023).

2. Knowledge Distillation Procedure

BERTino employs a two-stage distillation approach in pre-training. The student model (BERTino) is initialized by selecting every other layer from the teacher model, as per standard DistilBERT methodology. During training, the loss is a weighted sum of three components:

• $\mathcal{L}_{KD}$: Kullback–Leibler divergence loss between the student’s and teacher’s softmax output distributions.
• $\mathcal{L}_{MLM}$: Standard masked language modeling (MLM) cross-entropy loss.
• $\mathcal{L}_{COS}$: Cosine embedding loss between student and teacher hidden-state representations.

The loss weighting is 0.45 for both distillation and MLM objectives, and 0.10 for hidden-state alignment:

$$\mathcal{L} = 0.45\,\mathcal{L}_{KD} + 0.45\,\mathcal{L}_{MLM} + 0.10\,\mathcal{L}_{COS}$$

A temperature of $T=2$ is used to soften the logits for the distillation loss, mirroring the original DistilBERT implementation. The approach ensures that the student model learns both from the distributional knowledge of the teacher and from the intrinsic structure of the MLM objective (Muffo et al., 2023).

3. Pre-Training Data and Protocol

BERTino is pre-trained on a concatenation of two Italian corpora: PAISA (7.5 million sentences, 223.5 million words) and a pre-processed ItWaC (6.5 million sentences, 1.6 billion words), yielding a final training set of approximately 14 million sentences and 1.9 billion words (about 12 GB). Sentences exceeding 400 words in ItWaC are split at semantically meaningful locations ensuring almost all sub-sentences remain under 512 tokens. Tokenization is performed with the teacher’s WordPiece vocabulary. Training proceeds for three full epochs over the corpus, using the Adam optimizer with an initial learning rate of $5 \times 10^{-4}$ and a per-GPU batch size of 6. Computation is distributed across 4 Tesla K80 GPUs for a total pre-training duration of 45 days (Muffo et al., 2023).

4. Downstream Benchmark Evaluation

BERTino was fine-tuned and evaluated on four standard Italian NLP tasks, benchmarking against its teacher and GilBERTo (another BERTBASE for Italian). Tasks include ISDT (POS tagging for dependency parsing), ParTUT (POS tagging), Italian WikiNER (named-entity recognition, 5-fold), and multi-class sentence classification (139-way). Fine-tuning utilized a learning rate of $5 \times 10^{-5}$, batch size 32, and task-tuned epoch counts.

The following table summarizes F1 scores and times for BERTino and baselines:

Task / Model	BERTino (F1)	Teacher (F1)	GilBERTo (F1)
ISDT POS tagging	0.9800	0.9829	0.9804
ParTUT POS tagging	0.9193	0.9708	0.9621
WikiNER NER	0.9039	0.9176	0.9136
Sentence Classification	0.7766	0.7954	0.7381

Fine-tuning and evaluation times are nearly halved for BERTino compared to the teacher across all tasks (e.g., 9′ vs. 16′ on ISDT, 38′ vs. 67′ on WikiNER). The performance drop versus the teacher is ≤1.9 percentage points in F1 for all but ParTUT, where it is approximately 5.15 points (Muffo et al., 2023).

5. Computational Efficiency and Resource Profile

BERTino is architected to minimize both training and inference resource requirements. With 40% fewer parameters, the model requires nearly half the memory of a full BERTBASE during fine-tuning and inference. This reduction enables deployment of the model on resource-constrained hardware and markedly shortens training and inference cycles—for instance, BERTino is approximately 1.8–1.9× faster than BERTBASE on the measured tasks. This makes it suitable for production and edge-device environments where compute and latency constraints are paramount (Muffo et al., 2023).

6. Trade-Offs, Limitations, and Perspectives

BERTino achieves a strong efficiency–performance trade-off, maintaining above 97% of teacher-level performance on most tasks while greatly reducing memory and time demands. The most notable performance degradation is seen on the Italian ParTUT task, which suggests certain subtasks may be more sensitive to the architectural reduction or distillation protocol. The initial pre-training remains non-negligible (45 days on 4 K80 GPUs), but the cost is amortized if the model is to be used widely. The authors suggest that further improvements might be gained by combining distillation with more aggressive architectural compression (e.g., reducing attention heads, smaller hidden size) or incorporating advanced distillation losses. Extending the model to low-resource or specialized Italian domains (legal, biomedical) is identified as a natural direction. Monolingual lightweight models such as BERTino are shown to be competitive with or superior to multilingual BERTBASE models at a fraction of the computational cost (Muffo et al., 2023).