ELECTRA Base Discriminator Architecture

• The paper introduces replaced token detection, leveraging all tokens to boost sample efficiency compared to traditional masked language modeling.
• ELECTRA Base mirrors BERT-Base’s architecture with 12 Transformer layers and specific hyperparameters while employing a novel discriminative pre-training objective.
• Empirical results demonstrate faster convergence and competitive GLUE/SQuAD scores at lower computational costs relative to conventional models.

ELECTRA Base Discriminator is a Transformer-based neural architecture introduced for discriminative pre-training through replaced token detection rather than masked language modeling. Analogous in depth and capacity to BERT-Base, the discriminator’s innovation is in its pre-training objective: it classifies each input token as “real” (unaltered) or “fake” (replaced by the generator), providing pre-training signals for all tokens instead of a masked subset. The architecture and workflows, matching BERT-Base in most respects, produce more sample-efficient self-supervised training and substantially improved downstream task performance under comparable compute budgets (Clark et al., 2020).

1. Model Hyperparameters and Configuration

ELECTRA-Base Discriminator adopts the standard BERT-Base architectural template, differing only in the pre-training task. The core configuration is as follows:

Parameter	Value	Notes
Number of layers ($L$)	12	Transformer blocks
Hidden size ($H$)	768	per position
FFN inner size ($H_{ff}$)	3072
Attention heads ($A$)	12	$d_k = H/A = 64$ each
Max sequence length	512	tokens
Vocabulary size ($|V|$)	30,522	WordPiece tokens
Dropout (attention)	0.1
Dropout (hidden/FFN output)	0.1
Dropout (embeddings)	0.1

This configuration is designed to maintain parity in representational power with BERT-Base, ensuring fair comparison of task and efficiency (Clark et al., 2020).

2. Embedding Layer Construction

The embedding layer integrates three sources:

• Token Embeddings: Lookup table $E \in \mathbb{R}^{|V| \times H}$ maps each WordPiece token $x_t$ to a 768-dimensional vector.
• Positional Embeddings: Learned vectors $P \in \mathbb{R}^{512 \times 768}$ are added element-wise to indicate position within the 512-token bound.
• Segment (Token-Type) Embeddings: Two learned $768$-dimensional vectors $H$0 distinguish sentence A/B membership.

The summed embedding passes to:

$H$1

where $H$2 specifies the segment. Dropout rate is fixed at $H$3 for all three embedding categories (Clark et al., 2020).

3. Transformer Block Architecture

Each of the twelve stacked Transformer blocks is organized identically:

• Multi-head Self-Attention:
  • For each head $H$4, queries, keys, and values are computed: $H$5, $H$6, $H$7.
  • Attention is $H$8, with output $H$9.
  • All heads concatenated: $H_{ff}$0.
  • Output passes through post-norm residual block: $H_{ff}$1.
• Feed-Forward Network (FFN):
  • FFN for each position: $H_{ff}$2, where $H_{ff}$3, $H_{ff}$4 and $H_{ff}$5 is the Gaussian Error Linear Unit.
  • Output: $H_{ff}$6.

This structure, including the residual/LayerNorm “post-norm” pattern and hidden/intermediate dimensionalities, is retained directly from BERT-Base (Clark et al., 2020).

4. Discriminator Output, Loss, and Training Signal

Upon completion of $H_{ff}$7 layers, $H_{ff}$8 encodes all positions. The discriminator head applies a sigmoid classifier:

$H_{ff}$9

where $A$0, $A$1. Each $A$2 predicts the “real” probability for input token $A$3.

For ground-truth construction, let $A$4 be the original, $A$5 the corrupted sequence (masker and generator applied). Let $A$6 if $A$7 (“fake”), otherwise $A$8 (“real”). The replaced token detection (RTD) objective:

$A$9

Importantly, even if the generator accidentally resamples the original token, that position remains “real.” The overall pre-training loss combines RTD and a much-downweighted generator MLM loss ($d_k = H/A = 64$0, scale $d_k = H/A = 64$1), but only the discriminator is used for downstream transfer (Clark et al., 2020).

5. Weight Initialization and Optimization

Weights for attention, FFN, and output heads are initialized identically to BERT: truncated normal (mean $d_k = H/A = 64$2, std $d_k = H/A = 64$3). The optimizer is Adam with $d_k = H/A = 64$4, $d_k = H/A = 64$5, $d_k = H/A = 64$6, and weight decay $d_k = H/A = 64$7. The peak learning rate is $d_k = H/A = 64$8, with a linear warmup ($d_k = H/A = 64$9 steps) followed by linear decay to zero (Clark et al., 2020).

6. Pre-training Regimen and Data

Pre-training employs:

• Batch size: 256 sequences (max length 512), $|V|$0 million tokens/step.
• Masking rate: 15% of tokens selected uniformly for corruption.
• Discriminator steps: 766,000 ($|V|$1 million updates).

This regimen, matching BERT-Base in batch and token count, but differing in objective and loss, supports direct empirical comparison (Clark et al., 2020).

7. Empirical Outcomes and Design Rationale

ELECTRA-Base, via replaced token detection, utilizes all tokens in the loss computation, in contrast to BERT’s restriction to $|V|$2 masked positions. This leads to greater sample efficiency and improved representation quality. Reported experiments show consistently faster convergence and improved GLUE/SQuAD results relative to BERT-Base, and strong GLUE performance even under modest computational budgets. When scaled, the approach achieves competitive or superior accuracy to RoBERTa and XLNet at substantially lower compute cost—less than one fourth when matching downstream performance. The architecture’s discriminative training principle is thus empirically validated as a more efficient pre-training paradigm within the BERT capacity envelope (Clark et al., 2020).