---
title: ELECTRA Base Discriminator Architecture
url: https://www.emergentmind.com/topics/electra-base-discriminator-architecture
type: topic
---

# ELECTRA Base Discriminator Architecture

ELECTRA Base Discriminator is a Transformer-based neural architecture introduced for discriminative pre-training through replaced token detection rather than masked language modeling. Analogous in depth and capacity to BERT-Base, the discriminator’s innovation is in its pre-training objective: it classifies each input token as “real” (unaltered) or “fake” (replaced by the generator), providing pre-training signals for all tokens instead of a masked subset. The architecture and workflows, matching BERT-Base in most respects, produce more sample-efficient self-supervised training and substantially improved downstream task performance under comparable compute budgets [2003.10555].

## 1. Model Hyperparameters and Configuration

ELECTRA-Base Discriminator adopts the standard BERT-Base architectural template, differing only in the pre-training task. The core configuration is as follows:

| Parameter                | Value  | Notes                |
|--------------------------|--------|----------------------|
| Number of layers ($L$)          | 12     | Transformer blocks      |
| Hidden size ($H$)               | 768    | per position           |
| FFN inner size ($H_{ff}$)       | 3072   |                       |
| Attention heads ($A$)           | 12     | $d_k = H/A = 64$ each  |
| Max sequence length             | 512    | tokens                 |
| Vocabulary size ($|V|$)         | 30,522 | WordPiece tokens       |
| Dropout (attention)             | 0.1    |                       |
| Dropout (hidden/FFN output)     | 0.1    |                       |
| Dropout (embeddings)            | 0.1    |                       |

This configuration is designed to maintain parity in representational power with BERT-Base, ensuring fair comparison of task and efficiency [2003.10555].

## 2. Embedding Layer Construction

The embedding layer integrates three sources:

- **Token Embeddings:** Lookup table $E \in \mathbb{R}^{|V| \times H}$ maps each WordPiece token $x_t$ to a 768-dimensional vector.
- **Positional Embeddings:** Learned vectors $P \in \mathbb{R}^{512 \times 768}$ are added element-wise to indicate position within the 512-token bound.
- **Segment (Token-Type) Embeddings:** Two learned $768$-dimensional vectors $T^{(0)}, T^{(1)}$ distinguish sentence A/B membership.

The summed embedding passes to:

$$
h_0 = \text{Dropout}(\text{LayerNorm}(E[x_t] + P[t] + T^{(s_t)}))
$$

where $s_t \in \{0,1\}$ specifies the segment. Dropout rate is fixed at $0.1$ for all three embedding categories [2003.10555].

## 3. Transformer Block Architecture

Each of the twelve stacked Transformer blocks is organized identically:

- **Multi-head Self-Attention:**
    - For each head $j$, queries, keys, and values are computed: $Q_j = H^{(\ell-1)} W_j^Q$, $K_j = H^{(\ell-1)} W_j^K$, $V_j = H^{(\ell-1)} W_j^V$.
    - Attention is $ \alpha_j = \text{Softmax}\left( \frac{Q_j K_j^T}{\sqrt{d_k}} \right) $, with output $\text{head}_j = \alpha_j V_j$.
    - All heads concatenated: $A^{(\ell)} = \text{Concat}(\text{head}_1, \dots, \text{head}_A) W^O$.
    - Output passes through post-norm residual block: $H'^{(\ell)} = \text{LayerNorm}(H^{(\ell-1)} + \text{Dropout}(A^{(\ell)}))$.

- **Feed-Forward Network (FFN):**
    - FFN for each position: $\text{FFN}(x) = W_2 \, \mathrm{GELU}(W_1 x + b_1) + b_2$, where $W_1 \in \mathbb{R}^{H_{ff} \times H}$, $W_2 \in \mathbb{R}^{H \times H_{ff}}$ and $\mathrm{GELU}$ is the Gaussian Error Linear Unit.
    - Output: $H^{(\ell)} = \text{LayerNorm}(H'^{(\ell)} + \text{Dropout}(\text{FFN}(H'^{(\ell)})))$.

This structure, including the residual/LayerNorm “post-norm” pattern and hidden/intermediate dimensionalities, is retained directly from BERT-Base [2003.10555].

## 4. Discriminator Output, Loss, and Training Signal

Upon completion of $L=12$ layers, $H^{(L)} = [h_1, \dots, h_n]$ encodes all positions. The discriminator head applies a sigmoid classifier:

$$
D_t = \sigma(w^T h_t + b)
$$

where $w \in \mathbb{R}^H$, $b \in \mathbb{R}$. Each $D_t$ predicts the “real” probability for input token $t$.

For ground-truth construction, let $x$ be the original, $\tilde{x}$ the corrupted sequence (masker and generator applied). Let $\delta_t = 1$ if $\tilde{x}_t \neq x_t$ (“fake”), otherwise $0$ (“real”). The replaced token detection (RTD) objective:

$$
L_{\mathrm{disc}} = -\sum_{t=1}^n \left[ \delta_t \log D_t + (1-\delta_t) \log (1-D_t) \right]
$$

Importantly, even if the generator accidentally resamples the original token, that position remains “real.” The overall pre-training loss combines RTD and a much-downweighted generator MLM loss ($L_{\mathrm{gen}}$, scale $\lambda=50$), but only the discriminator is used for downstream transfer [2003.10555].

## 5. Weight Initialization and Optimization

Weights for attention, FFN, and output heads are initialized identically to BERT: truncated normal (mean $0$, std $0.02$). The optimizer is Adam with $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-6}$, and weight decay $0.01$. The peak learning rate is $2 \times 10^{-4}$, with a linear warmup ($10^4$ steps) followed by linear decay to zero [2003.10555].

## 6. Pre-training Regimen and Data

Pre-training employs:

- **Batch size:** 256 sequences (max length 512), $\approx 200$ million tokens/step.
- **Masking rate:** 15% of tokens selected uniformly for corruption.
- **Discriminator steps:** 766,000 ($\approx250$ million updates).

This regimen, matching BERT-Base in batch and token count, but differing in objective and loss, supports direct empirical comparison [2003.10555].

## 7. Empirical Outcomes and Design Rationale

ELECTRA-Base, via replaced token detection, utilizes all tokens in the loss computation, in contrast to BERT’s restriction to $~15\%$ masked positions. This leads to greater sample efficiency and improved representation quality. Reported experiments show consistently faster convergence and improved GLUE/SQuAD results relative to BERT-Base, and strong GLUE performance even under modest computational budgets. When scaled, the approach achieves competitive or superior accuracy to RoBERTa and XLNet at substantially lower compute cost—less than one fourth when matching downstream performance. The architecture’s discriminative training principle is thus empirically validated as a more efficient pre-training paradigm within the BERT capacity envelope [2003.10555].

Source: https://www.emergentmind.com/topics/electra-base-discriminator-architecture