BIBERT-Pipe: Modular Bilingual BERT Design

• The paper introduces BIBERT-Pipe as a modular pipeline design using dual BERT-based encoders coupled with task-specific modules for translation and biomedical entity linking.
• The approach decomposes tasks into distinct stages—such as encoding, bilingual mapping, and decoding—to efficiently leverage pretrained monolingual and bilingual priors.
• Empirical evaluations in machine translation and biomedical NER demonstrate competitive performance while reducing the bilingual training burden.

Bilingual BERT, often abbreviated in the literature as BIBERT-Pipe, denotes a family of pipeline-oriented architectures that use BERT-style contextual encoders across two languages and connect them to task-specific intermediate modules rather than relying on a single uniform end-to-end design. In machine translation, the term can refer either to a dual-encoder “thought-space” pipeline built from separate monolingual BERTs plus a lightweight bilingual mapping, or to a tailored bilingual masked LLM whose contextualized embeddings are injected into a neural MT encoder. In biomedical entity linking, the same label has been used for a bilingual BERT-based retrieval–ranking system with minimal modifications to an existing encoder. Taken together, these usages indicate that BIBERT-Pipe is best understood as a modular bilingual BERT pipeline pattern rather than a single standardized architecture (Cheng et al., 2019, Xu et al., 2021, Li et al., 10 Sep 2025).

1. Terminological scope and conceptual basis

In its earliest explicit pipeline formulation, BALM (“Bilingual is At Least Monolingual”) proposes that translation should be decomposed into three stages: encode the source sentence into a fixed-length sentence embedding, learn a simple mapping between source and target latent spaces, and decode from the target latent space back into text. The paper states that this is conceptually exactly what one might call a “Bilingual BERT pipeline” (BIBERT-Pipe): two BERT-based encoders, a bilingual mapping layer, and a decoder. Its motivating claim is that standard MT systems must learn source understanding, target generation, and cross-lingual alignment directly from parallel data, whereas BALM reuses monolingual priors already contained in pretrained BERT models (Cheng et al., 2019).

A second major use of the term appears in neural machine translation work where BiBERT denotes a bilingual English–German masked LLM, and BiBERT-Pipe denotes the integration of that model as a frozen external encoder whose contextualized embeddings replace standard encoder token embeddings in a Transformer NMT system. Here, the emphasis is not on a latent “thought-space” bridge but on inserting a bilingual pretrained encoder into an otherwise standard MT stack, augmented by stochastic layer selection and dual-directional training (Xu et al., 2021).

A third use is explicitly non-architectural in the pretraining sense. In the BioNNE-L 2025 multilingual biomedical nested entity linking system, the authors state that “BIBERT” here is not a new pretraining architecture, but a bilingual BERT-based EL pipeline that keeps the base encoder unchanged and adds three task-specific components: two-stage retrieval–ranking, boundary cues via learnable [Ms] and [Me] tags, and dataset augmentation (Li et al., 10 Sep 2025).

This terminological spread aligns with broader bilingual language-model research. A Finnish–English biBERT was presented as a single BERT-Base model jointly pretrained on two remotely related languages with a shared bilingual vocabulary, and language-specific BERT surveys position bilingual designs between monolingual specialists and massively multilingual generalists (Chang et al., 2020, Nozza et al., 2020). This suggests that “BIBERT-Pipe” names a design philosophy centered on bilingual contextual representation plus modular downstream composition, rather than one fixed checkpoint or training recipe.

2. Architectural archetypes

Across the literature, BIBERT-Pipe systems fall into a small number of recurring structural patterns.

Formulation	Encoder organization	Pipeline core
BALM / thought-space MT	Separate monolingual BERTs or multilingual BERT used monolingually	Source encoder → fixed-length mapping → target decoder
BiBERT-Pipe for NMT	One tailored bilingual RoBERTa-style model, frozen during NMT	BiBERT contextual embeddings → Transformer encoder
Biomedical BIBERT-Pipe	One shared biomedical multilingual encoder	Dense retrieval → cross-encoder ranking

The BALM formulation is explicitly compositional. For a language $L$, BERT is treated as a map $B_L : L_L \rightarrow S_{L,k}$, where $S_{L,k} \subseteq \mathbb{R}^k$ is a $k$-dimensional sentence embedding space. In the English–German setup, $k=768$, and the sentence vector is obtained by mean pooling the final token-level hidden states. Translation is then factored as

$$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$

with $F_{\text{German}\rightarrow\text{English}}$ implemented as a feedforward mapping between fixed-length latent spaces (Cheng et al., 2019).

The NMT BiBERT-Pipe architecture differs in a crucial respect: it does not collapse the source sentence into a single vector before translation. Instead, the bilingual pretrained model yields a full contextualized sequence $H_B^i(x)$ at layer $i$, and the NMT encoder consumes either the final-layer sequence or a stochastic choice among the top $K$ layers during training, with top-layer averaging at inference. The bilinguality is encoded in the shared English–German pretraining of the encoder itself rather than in a separate low-dimensional bridge (Xu et al., 2021).

The biomedical formulation is again distinct. It uses the same base encoder in two stages: first as a dense retriever over precomputed concept vectors, then as a fine-tuned ranker over mention–candidate pairs. The pipeline is organized around candidate generation and reranking, not around sequence generation or latent-space transformation (Li et al., 10 Sep 2025).

3. Machine translation instantiations

BALM formalizes a strong version of the pipeline view. The source and target languages each have a thought-space, and the bilingual learning burden is isolated to a simple mapping between them. In the reported English–German system, the English encoder is a pretrained 12-layer BERT with hidden size 768 and approximately 110M parameters, the German encoder is a multilingual BERT used as a German encoder, the sentence representation is the mean of final-layer token embeddings, the bilingual mapping is a $B_L : L_L \rightarrow S_{L,k}$0 MLP with ReLU, and the target-side inverse map is a single-layer GRU decoder with hidden size 768 and vocabulary size 28,996. Training proceeds in two phases: an English autoencoder $B_L : L_L \rightarrow S_{L,k}$1, then a German→English translator initialized from the autoencoder decoder. The autoencoder converges to approximately $B_L : L_L \rightarrow S_{L,k}$2 cross-entropy and BLEU $B_L : L_L \rightarrow S_{L,k}$3; the translator converges to approximately $B_L : L_L \rightarrow S_{L,k}$4 cross-entropy and BLEU $B_L : L_L \rightarrow S_{L,k}$5 on Multi30k, compared with a reported state-of-the-art near $B_L : L_L \rightarrow S_{L,k}$6 at that time (Cheng et al., 2019).

The 2021 BiBERT-Pipe work shifts from a fixed-length bottleneck to a token-sequence interface. BiBERT$B_L : L_L \rightarrow S_{L,k}$7 is a RoBERTa-based bilingual masked LLM with 12 layers, hidden size 768, and a shared 52k WordPiece vocabulary built from balanced English and German samples. Pretraining uses 146 GB of German OSCAR text and 146 GB of English OSCAR text, shuffles both languages together, and trains for 200k update steps with Adam, learning rate $B_L : L_L \rightarrow S_{L,k}$8, warmup for 20k steps, polynomial decay, and dropout $B_L : L_L \rightarrow S_{L,k}$9. During NMT training, BiBERT is kept frozen, and its contextualized outputs replace the NMT encoder’s learned embedding layer (Xu et al., 2021).

Two additional mechanisms define this BiBERT-Pipe. First, stochastic layer selection samples one of the top $S_{L,k} \subseteq \mathbb{R}^k$0 BiBERT layers per batch during training and uses the average of those top layers at inference:

$S_{L,k} \subseteq \mathbb{R}^k$1

Second, dual-directional translation trains one model on both En→De and De→En by concatenating original and swapped parallel pairs, then fine-tunes direction-specifically. On IWSLT’14, the best BiBERT-Pipe configuration with stochastic layer selection ($S_{L,k} \subseteq \mathbb{R}^k$2), dual-directional training, and fine-tuning achieves BLEU $S_{L,k} \subseteq \mathbb{R}^k$3 for En→De and $S_{L,k} \subseteq \mathbb{R}^k$4 for De→En. On WMT’14, BiBERT contextualized embeddings plus stochastic layer selection achieve $S_{L,k} \subseteq \mathbb{R}^k$5 for En→De and $S_{L,k} \subseteq \mathbb{R}^k$6 for De→En, while adding dual-directional training and fine-tuning yields the best En→De result of $S_{L,k} \subseteq \mathbb{R}^k$7 and a De→En result of $S_{L,k} \subseteq \mathbb{R}^k$8 (Xu et al., 2021).

These two MT lines illustrate complementary interpretations of bilingual pipelines. BALM minimizes the bilingual component by learning only a fixed-length bridge between monolingually structured latent spaces. BiBERT-Pipe for NMT retains a standard encoder–decoder backbone but replaces the source-side lexical interface with bilingual contextualized embeddings. The former emphasizes decomposition; the latter emphasizes bilingual pretraining and controlled integration.

4. Capacity, vocabulary, layers, and training dynamics

A central question for bilingual BERT pipelines is whether a single encoder can cover two languages without sacrificing monolingual competence. The Finnish–English biBERT study addresses precisely this point with a standard BERT-Base architecture: 12 layers, hidden size 768, 12 attention heads, and approximately 110M parameters excluding word embeddings. Its principal design choice is an 80,000-token shared bilingual vocabulary, exactly the sum of the monolingual English BERT and FinBERT vocabulary sizes. Pretraining uses roughly balanced corpora—3.8B English tokens and 3.3B Finnish tokens—and 1,000,000 training steps with whole-word masking and NSP. The resulting model matches English BERT-Base on GLUE in average score, $S_{L,k} \subseteq \mathbb{R}^k$9 vs $k$0, and nearly matches FinBERT on a range of Finnish tasks, $k$1 vs $k$2, while clearly outperforming mBERT at $k$3 (Chang et al., 2020).

This result is consistent with the broader survey finding that language-specific BERT models “almost always” outperform mBERT on the language they target. Averaged over surveyed tasks, the reported advantages of language-specific models over mBERT are $k$4 F1 for named entity recognition, $k$5 accuracy for natural language inference, $k$6 accuracy for paraphrase identification, $k$7 accuracy for part-of-speech tagging, $k$8 UPOS accuracy, $k$9 accuracy for sentiment analysis, and $k=768$0 accuracy for text classification (Nozza et al., 2020). Bilingual models are therefore motivated by a capacity tradeoff: they seek stronger pair-specific transfer than monolingual encoders while avoiding the dilution associated with 100+-LLMs.

Layer studies further complicate simple “best layer” narratives. Probing work on Dutch BERTje and mBERT shows that linguistic information is distributed across the network, that scalar mixing over layers is often preferable to selecting a single layer, and that mBERT’s task-relevant peaks occur earlier than BERTje’s. For POS, DEP, NER, and coreference, the model exhibits a pipeline-like ordering, but within-task information is spread across multiple layers rather than localized cleanly (Vries et al., 2020). This suggests that BIBERT-Pipe systems that consume contextualized representations may benefit from multi-layer aggregation or explicit layer selection rather than unconditional use of the final layer.

Training dynamics in multilingual token classification reinforce the same point. In multilingual NER, a jointly trained BERT-based model outperforms single-language fine-tuning, but zero-shot performance can degrade if all layers are updated aggressively. Freezing the lower layers of mBERT during fine-tuning is presented as a form of regularization that preserves pretrained cross-lingual alignment, and freezing 3–7 lower layers improves several zero-shot CoNLL NER results relative to unfrozen fine-tuning (Moon et al., 2019). For bilingual pipelines, this is a plausible implementation principle whenever labeled data are asymmetric across the two languages.

5. Biomedical BIBERT-Pipe

In biomedical NLP, BIBERT-Pipe denotes a two-stage bilingual entity-linking pipeline rather than a translation model or a new pretraining family. The BioNNE-L 2025 task concerns biomedical nested named entity linking in English and Russian, where the objective is to map a mention $k=768$1 in document $k=768$2 and language $k=768$3 to a gold UMLS concept $k=768$4. The bilingual system uses a shared multilingual encoder—SapBERT-UMLS-XLMR for RU and BI settings—and keeps the base encoder architecture intact while altering only three task-aligned components: two-stage retrieval–ranking, boundary cues, and dataset augmentation (Li et al., 10 Sep 2025).

The retrieval stage embeds mentions in context and concept strings, computes cosine similarity, and retrieves top-$k=768$5 candidates with $k=768$6. The ranking stage then scores mention–candidate pairs with a cross-encoder under either listwise ranking or contrastive/binary classification; the final system uses the contrastive formulation. A distinctive feature is the insertion of learnable [Ms] and [Me] tokens around the mention span, which serve as language-agnostic boundary cues in both retrieval and ranking (Li et al., 10 Sep 2025).

The reported development retrieval results show that the best English retriever, SapBERT–PubMedBERT (mean-token), reaches Acc@1 $k=768$7, Acc@5 $k=768$8, and Acc@10 $k=768$9; the best bilingual retriever, SapBERT-UMLS-XLMR-large, reaches Acc@1 $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$0, Acc@5 $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$1, and Acc@10 $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$2. Final hidden-test Acc@1 scores are $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$3 for English, $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$4 for Russian, and $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$5 for the bilingual track, where the system ranks third. The boundary-tag ablation shows gains from [Ms]/[Me] of $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$6 for English, $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$7 for Russian, and $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$8 for bilingual evaluation, with the largest effect in Russian (Li et al., 10 Sep 2025).

This usage extends BIBERT-Pipe beyond cross-lingual generation. The commonality is the modular use of a shared bilingual or multilingual contextual encoder inside a larger pipeline whose decisive operations occur in retrieval, ranking, or lightweight bridging modules rather than in a monolithic encoder–decoder.

6. Relation to monolingual, multilingual, and later bilingual encoders

BIBERT-Pipe occupies a middle ground between monolingual specialization and many-language multilinguality. The survey literature presents mBERT as a universal model with strong zero-shot transfer, but also records consistent in-language disadvantages relative to language-specific models (Nozza et al., 2020). The BiBERT NMT paper makes the same point in more pair-specific terms: under an identical integration framework, monolingual contextual embeddings such as RoBERTa or GottBERT help, mBERT and XLM-R show limited or negative gains in some directions, and the authors attribute this to the “curse of multilinguality”, namely the dilution of model capacity across 100+ languages (Xu et al., 2021).

The Finnish–English biBERT study provides direct evidence that the bilingual regime can avoid much of this degradation when vocabulary size and data balance are chosen carefully. The model’s 80k bilingual vocabulary is explicitly designed to preserve effective per-language subword capacity, and the reported outcome is that BERT-Base capacity is sufficient for two remotely related languages without visible English degradation and with only a small Finnish gap to a monolingual baseline (Chang et al., 2020).

A later encoder-focused line extends the idea beyond classical BERT-base recipes. NeoDictaBERT-bilingual is an English–Hebrew bilingual model using the NeoBERT architecture with 28 layers, width 768, RoPE, Pre-RMSNorm, SwiGLU, a 128k shared WordPiece vocabulary, and bilingual pretraining on $$T = B_{\text{English}}^{-1} \circ F_{\text{German}\rightarrow\text{English}} \circ B_{\text{German}},$$9B tokens in phase 1 and $F_{\text{German}\rightarrow\text{English}}$0B tokens in phase 2 with a $F_{\text{German}\rightarrow\text{English}}$1–$F_{\text{German}\rightarrow\text{English}}$2 English–Hebrew mix. On Hebrew QA it reports EM $F_{\text{German}\rightarrow\text{English}}$3, F1 $F_{\text{German}\rightarrow\text{English}}$4, and T1nls $F_{\text{German}\rightarrow\text{English}}$5; on Hebrew NER it reports F1 $F_{\text{German}\rightarrow\text{English}}$6; and on a Hebrew semantic retrieval benchmark it reaches NDCG@20 $F_{\text{German}\rightarrow\text{English}}$7 versus $F_{\text{German}\rightarrow\text{English}}$8 for multilingual-e5-base, despite English-only retrieval fine-tuning (Shmidman et al., 23 Oct 2025). Although not labeled BIBERT-Pipe in the same way as BALM or BioNNE-L, it represents a modern bilingual encoder that can function as the backbone for such pipelines.

A common misconception is therefore that BIBERT-Pipe always refers to a single architectural object. The literature instead supports at least three distinct possibilities: separate monolingual encoders connected by a bilingual bridge, a jointly pretrained bilingual encoder injected into a downstream model, or an application-specific bilingual pipeline that leaves the base encoder untouched.

7. Limitations, misconceptions, and open directions

The first limitation is definitional. Because BIBERT-Pipe has been used for substantially different systems, it should not be treated as a uniquely specified model family. BALM’s version uses two monolingual encoders plus a fixed-length translator; the NMT BiBERT-Pipe uses one frozen bilingual encoder on the source side; the biomedical system explicitly states that “BIBERT” is not a new pretraining architecture (Cheng et al., 2019, Xu et al., 2021, Li et al., 10 Sep 2025).

The second limitation is task and domain specificity. BALM is trained and evaluated on Multi30k image captions, where the paper emphasizes that the domain contains short descriptive captions and provides no evidence of generalization to other genres; its BLEU $F_{\text{German}\rightarrow\text{English}}$9 remains below the reported Multi30k state of the art near $H_B^i(x)$0 (Cheng et al., 2019). The biomedical system likewise operates in a narrow UMLS-centered setting with provided mention spans, and it treats nested mentions independently rather than modeling hierarchical relations among overlapping spans (Li et al., 10 Sep 2025).

The third limitation concerns scaling and computational tradeoffs. The NMT BiBERT-Pipe keeps BiBERT frozen and still reports approximately 20% training-time and 13% inference-time overhead from using BiBERT as a pipeline component (Xu et al., 2021). The biomedical contrastive ranker requires $H_B^i(x)$1 encoder passes per mention for $H_B^i(x)$2, which the authors note is expensive at scale (Li et al., 10 Sep 2025). More generally, bilingual specialization improves pair-specific performance, but extending this strategy to many language pairs would require either many separate bilingual checkpoints or new capacity-management strategies (Xu et al., 2021, Shmidman et al., 23 Oct 2025).

The literature therefore points to several open directions. One is better layer utilization, replacing fixed final-layer usage with learned or stochastic aggregation, as already explored in NMT and suggested by layer-probing work (Xu et al., 2021, Vries et al., 2020). Another is controlled fine-tuning, including partial freezing or auxiliary objectives, to preserve cross-lingual alignment under asymmetric supervision (Moon et al., 2019). A third is broader pair coverage without reverting to the curse of multilinguality, which remains unresolved across bilingual MT, bilingual language modeling, and modern bilingual encoder design (Chang et al., 2020, Xu et al., 2021).

In that sense, BIBERT-Pipe is less a single model than a recurring architectural answer to one research problem: how to exploit bilingual contextual priors while restricting the task-specific learning burden to a narrow, modular interface.