MobileBERT: Compact Transformer for Mobile

• MobileBERT is a compact, task-agnostic transformer designed for resource-limited devices by compressing a 24-layer BERT into a lightweight model.
• It employs a deep-thin architecture with bottleneck structures and progressive layer-wise knowledge transfer, balancing performance with reduced computational demands.
• The model achieves competitive benchmarks on GLUE and SQuAD with low latency, demonstrating significant efficiency improvements over standard BERT versions.

MobileBERT is a compact, task-agnostic BERT for resource-limited devices. Introduced by Sun et al. as a thin version of BERT$_\text{LARGE}$, it preserves 24-layer depth while narrowing internal representations, inserting bottleneck structures, and relying on a specially designed teacher plus layer-wise knowledge transfer. In the original report, MobileBERT is 4.3x smaller and 5.5x faster than BERT$_\text{BASE}$, achieves a GLUEscore of 77.7, and reaches 62 ms latency on a Pixel 4 phone; on SQuAD v1.1/v2.0 it reports dev F1 90.0/79.2 for the optimized variant (Sun et al., 2020).

1. Origins and design objective

MobileBERT was proposed in response to the fact that BERT-class models have heavy model sizes and high latency and therefore cannot be deployed straightforwardly to resource-limited mobile devices. The design target was not a task-specific student, but a task-agnostic pre-trained model that can be fine-tuned on downstream tasks in the same way as BERT. In this sense, MobileBERT belongs to the same pre-train-then-fine-tune regime as the original BERT family, but with architectural and distillation choices explicitly oriented toward mobile inference (Sun et al., 2020).

The original paper frames MobileBERT as a compression and acceleration of BERT rather than as a shallow substitute. It keeps the 24-layer depth of BERT$_\text{LARGE}$, but compresses width aggressively and reallocates capacity through bottlenecks, stacked FFNs, embedding factorization, and teacher-guided training. A later survey of lightweight transformers places MobileBERT among the prominent compact NLP models for edge deployment and reports it in the 15–40M parameter range that is favorable for edge hardware utilization (Samson, 5 Jan 2026).

2. Architectural organization

The defining architectural idea is a deep but thin transformer. In the configuration summarized in the original paper, BERT$_\text{LARGE}$ has 24 layers, hidden size 1024, FFN size 4096, 16 attention heads, embedding size 1024, and 334M parameters; BERT$_\text{BASE}$ has 12 layers, hidden size 768, FFN size 3072, 12 heads, embedding size 768, and 109M parameters. By contrast, MobileBERT has 24 layers, an inter-block hidden size of 512, an intra-block hidden size of 128, FFN size 512, 4 stacked FFNs per layer, 4 heads, an embedding size of 128 with 3-conv expansion to 512, and 25.3M parameters. The companion teacher, IB-BERT$_\text{LARGE}$, keeps 24 layers and exposes the same 512-dimensional inter-block feature maps while using an intra-block hidden size of 1024 and 293M parameters (Sun et al., 2020).

This organization creates a bottlenecked student and an inverted-bottleneck teacher. MobileBERT uses a wide $\rightarrow$ narrow $\rightarrow$ wide pattern at the block level: the block input and output are 512-dimensional, while the main internal processing happens in a 128-dimensional bottleneck. IB-BERT$_\text{LARGE}$ does the converse internally, using a 512-dimensional interface but expanding to 1024 dimensions inside each block. The important consequence is that teacher and student expose aligned 512-dimensional feature maps at every block boundary, which makes layer-wise feature transfer direct rather than projection-heavy (Sun et al., 2020).

The paper also replaces standard LayerNorm with NoNorm, defined as

$$\mathtt{NoNorm}(\mathbf{h}) = \boldsymbol{\gamma} \circ \mathbf{h} + \boldsymbol{\beta},$$

and replaces GELU with ReLU for latency reasons. Embeddings are factorized by using a 128-dimensional embedding table and then expanding to the 512-dimensional inter-block space with a 1D convolution with kernel size 3. Within each layer, MobileBERT uses 4 sequential FFN blocks after MHA to restore a parameter balance between attention and feed-forward computation that would otherwise be distorted by the narrow bottleneck (Sun et al., 2020).

3. Teacher design and knowledge transfer

MobileBERT’s training procedure is inseparable from its teacher. The teacher, IB-BERT$_\text{BASE}$0, is a specially designed BERT$_\text{BASE}$1-style model whose inter-block hidden size is reduced to 512 while retaining a large intra-block hidden size of 1024. Architecture search in the original study found that reducing inter-block width from 1024 to 512 barely affected SQuAD v1.1 dev F1, whereas reducing intra-block width caused clear degradation. This is why the teacher keeps large internal capacity while exposing the same 512-dimensional interfaces as the student (Sun et al., 2020).

The student is trained with three transfer objectives. Feature Map Transfer (FMT) minimizes layer-wise MSE between teacher and student hidden states,

$_\text{BASE}$2

Attention Transfer (AT) matches teacher and student attention distributions with KL divergence,

$_\text{BASE}$3

Pre-training Distillation (PD) combines MLM, KD on MLM logits, and NSP:

$_\text{BASE}$4

with $_\text{BASE}$5 (Sun et al., 2020).

The paper evaluates three training schedules: Auxiliary Knowledge Transfer (AKT), Joint Knowledge Transfer (JKT), and Progressive Knowledge Transfer (PKT). PKT is the strongest. It trains MobileBERT in 24 stages, primarily aligning the $_\text{BASE}$6-th student layer to the $_\text{BASE}$7-th teacher layer at stage $_\text{BASE}$8, while lower layers are effectively retained with a much smaller learning rate. After this progressive layer-wise transfer, the model undergoes PD. The ablations further show that Feature Map Transfer contributes most: PD alone yields only moderate gains, while adding FMT moves MobileBERT close to BERT$_\text{BASE}$9; AT adds a smaller additional improvement (Sun et al., 2020).

4. Benchmark behavior and on-device optimization

On GLUE test, the original report gives BERT$_\text{LARGE}$0 a GLUE score of 78.3 and MobileBERT 77.7. On Pixel 4 at sequence length 128, BERT$_\text{LARGE}$1 requires 342 ms, whereas MobileBERT requires 62 ms. The same table reports 109M parameters and 22.5B FLOPs for BERT$_\text{LARGE}$2, against 25.3M parameters and 5.7B FLOPs for MobileBERT. On SQuAD dev, MobileBERT reaches 82.9/90.0 EM/F1 on v1.1 and 76.2/79.2 EM/F1 on v2.0, while the variant without the operational optimizations reaches 83.4/90.3 on v1.1 and 77.6/80.2 on v2.0 (Sun et al., 2020).

A notable result is that operator choice changes latency far more than FLOP counts alone would suggest. The original ablation reports 192 ms for LayerNorm + GELU, 167 ms for LayerNorm + ReLU, 92 ms for NoNorm + GELU, and 62 ms for NoNorm + ReLU, all at the same 5.7B FLOPs. This makes MobileBERT an example of co-design at the architecture and operator levels: the optimized variant sacrifices a small amount of benchmark accuracy relative to the non-optimized one, but obtains a much larger latency reduction (Sun et al., 2020).

MobileBERT is also robust to post-training 8-bit quantization. On dev metrics including MNLI-m, QNLI, MRPC, SST-2, and SQuAD v1.1 F1, the quantized model changes only marginally; for SQuAD v1.1 F1, the reported value remains 90.0 after quantization. A later survey summarizes the standard MobileBERT entry as 25.3M parameters, GLUE 77.7, SQuAD F1 90.3, and 62 ms latency on Pixel 4, and treats it as one of the strongest QA-oriented lightweight NLP transformers in the edge setting (Sun et al., 2020, Samson, 5 Jan 2026).

5. Later adaptations and deployments

Subsequent work uses MobileBERT both as a direct backbone and as a reference architecture. In an on-device emoji classifier for SwiftKey, MobileBERT is truncated to the bottom 2 transformer layers, quantized to INT8, and served in ONNX format. The resulting deployed model is about 3 MB, with final sizes 2.8 MB for 90C, 3.1 MB for 590C, and 3.3 MB for 1090C, and latencies of 19.3 ms, 20.4 ms, and 22.0 ms on a Samsung A23. In that system, MobileBERT supplies the context encoder, while GPT-based data augmentation and a favorites-based interpolation layer improve rare-emoji performance and personalization (Amer et al., 2024).

In a distinct line of work, MobileBERT is fine-tuned as a multi-label intent classifier for filtering multi-party conversation snippets before LLM processing. There it uses standard pre-trained MobileBERT with a two-label BCE objective, and on the Balanced Test set reports Precision 0.90, Recall 0.91, F1 0.90 for action-triggering and Precision 0.95, Recall 0.95, F1 0.95 for information-seeking. The same study reports token reduction figures close to the ideal case, indicating that a compact encoder can act as a cost-aware front-end to heavier LLM pipelines (Gody et al., 21 Mar 2025).

MobileBERT has also served as a deployment vehicle for nonstandard hardware targets. In analog in-memory computing, it is the central example for Hardware-Aware LoRA, with 25.3M parameters in total, 20.4M mapped to AIMC tiles and 4.9M left in digital cores; on SQuAD v1.1, the reported HWA LoRA numbers are F1 89.06, EM 81.93 at drift 0 s and F1 85.36, EM 76.92 at 10 years (Li et al., 2024). In a distributed MCU setting, MobileBERT is the encoder-only workload with embedding dimension $_\text{LARGE}$3, intermediate size $_\text{LARGE}$4, $_\text{LARGE}$5 attention heads, and sequence length $_\text{LARGE}$6; on 4 chips it reaches 38.8 ms runtime per inference with a 4.7x super-linear speedup over a single chip (Bochem et al., 2024).

6. Comparative standing, misconceptions, and limits

MobileBERT rapidly became a comparison point for later compact-transformer work. AutoDistill reports seven model architectures with better pre-trained accuracy (up to 3.2% higher) and lower inference latency (up to 1.44x faster) than MobileBERT on TPUv4i, and a 28.5M-parameter model with 81.69 average GLUE, higher than BERT$_\text{LARGE}$7, DistilBERT, TinyBERT, NAS-BERT, and MobileBERT (Zhang et al., 2022). EfficientBERT reports an average GLUE test score of 77.7, stated as 0.7 higher than MobileBERT$_\text{LARGE}$8, while operating at 15.7M parameters and 83 ms latency versus 15.1M and 96 ms for MobileBERT$_\text{LARGE}$9 (Dong et al., 2021). ZipLM states that it can match the performance of the heavily optimized MobileBERT model on SQuAD by pruning a baseline BERT model, without MobileBERT’s custom architecture search and bespoke modules (Kurtic et al., 2023). In a different design space, LIDSNet reports that it is 41x lighter and 30x faster during inference than MobileBERT on a Samsung Galaxy S20 for on-device intent detection (Agarwal et al., 2021).

At the same time, later evidence also sharpens MobileBERT’s limits. A benchmark on on-device fault detection with tabular-to-text encoding reports that MobileBERT scored 0% F1 across all datasets and training configurations, with 24.6M parameters, 94 MB model size, and 108 ms CPU latency, and attributes the failure to a mismatch between MobileBERT’s inverted bottleneck architecture and serialized numerical data (Patel, 23 Jun 2026). A broader survey notes two structural limitations common to MobileBERT-class models: quadratic attention remains a long-context bottleneck, and aggressive low-bit compression such as INT4 can incur substantial accuracy loss (Samson, 5 Jan 2026).

A recurrent misconception in later literature is to treat MobileBERT as synonymous with any compact on-device transformer. Some works cite it only as a conceptual baseline for compact-transformer design while the deployed system uses another model entirely; in an offline Vietnamese–English iOS translation prototype, MobileBERT is not used in the final iOS prototype and appears only as an initial research reference (Le, 12 May 2025). In a phishing-detection paper presented as “Using MobileBERT,” the model described in the body is reported to be DistilBERT, not MobileBERT (Roy et al., 2024). This suggests that “MobileBERT” has also functioned as a shorthand for a broader class of mobile-oriented transformer compression strategies, even when the concrete implementation differs.

Overall, MobileBERT’s historical importance lies in showing that a 24-layer, task-agnostic BERT descendant could be made small enough and fast enough for device-side inference without giving up competitive performance. Later work has either specialized it, surpassed it in particular hardware regimes, or used it as a reference point, but the original combination of deep-thin architecture, aligned teacher design, and progressive layer-wise transfer remains one of the canonical templates for compact transformer deployment (Sun et al., 2020).