---
title: 'FLANG-BERT/FLANG-ELECTRA: Financial NLP Models'
url: https://www.emergentmind.com/topics/flang-bert-flang-electra
type: topic
---

# FLANG-BERT/FLANG-ELECTRA: Financial NLP Models

FLANG-BERT and FLANG-ELECTRA are domain-adapted large pre-trained language models for the financial sector, extending the BERT-base and ELECTRA-base architectures respectively. Both incorporate explicit financial vocabulary expansion, finance-aware masking strategies, and additional objectives designed to better leverage the structural and semantic properties of financial texts. Their introduction coincides with the Financial Language Understanding Evaluation (FLUE) benchmark suite, which provides comprehensive multi-task evaluation for the financial domain. FLANG-BERT and FLANG-ELECTRA have demonstrated superior performance over prior approaches across a broad array of financial NLP tasks [2211.00083].

## 1. Architecture and Model Modifications

### FLANG-BERT

FLANG-BERT adopts the BERT-base configuration, comprising 12 Transformer encoder layers, 12 self-attention heads, a hidden size of 768, and approximately 110 million parameters. The major architectural adaptation is the vocabulary expansion: the standard WordPiece vocabulary is augmented by ~8.2k words and phrases targeting financial discourse. Pre-training introduces a two-stage masking procedure—first targeting single-word tokens and then contiguous financial phrases.

### FLANG-ELECTRA

FLANG-ELECTRA builds on ELECTRA-base, with distinct architectures for the generator and discriminator. The generator utilizes 12 layers (hidden size 256, 4 attention heads, ~14 million parameters), while the discriminator matches ELECTRA-base (12 layers, hidden size 768, 12 heads, ~110 million parameters). Modifications include:

- Preferential masking—masking of tokens/phrases selected for financial salience.
- Addition of a Span Boundary Objective to the generator (details below).
- Adoption of the two-stage masking protocol paralleling FLANG-BERT.

## 2. Pre-training Objectives and Loss Formulations

Both models integrate three core loss components, detailed below using the original mathematical formulations:

### 2.1 Financial Keyword and Phrase Masking

Let $N$ denote the sequence length, $D_f$ denote indices $i$ where token $w_i$ is in a financial-term dictionary, and $M$ denote positions selected for masking. The procedure is:

- Mask 15% of tokens per sequence.
- 30% of masked positions $M_f$ are reserved for financial terms ($|M_f| = 0.15 N \times 0.3$).
- 70% are drawn at random from non-financial terms ($|M_r| = 0.15 N \times 0.7$).

Probability a token $w_i$ is masked:

\[
P(i\in M) =
\begin{cases}
\tfrac{0.15\,N\times0.3}{|D_f|}, & i\in D_f \\
\tfrac{0.15\,N\times0.7}{N - |D_f|}, & i\notin D_f
\end{cases}
\]

During the phrase-masking stage, each multi-token financial phrase is masked as a contiguous span (single [MASK] token) with probability 0.3.

### 2.2 Masked Language Modeling (Infilling) Loss

The model predicts masked tokens from their context:

\[
L_{\mathrm{MLM}}(x ; \theta_G) =
\mathbb{E}\left[\sum_{i\in M} -\log P_G(x_i \mid x_{\setminus M})\right]
\]

### 2.3 Span Boundary Objective (SBO)

Specific to FLANG-ELECTRA, the generator reconstructs each masked token $x_i$ using only the boundary tokens of its masked span. For a span $[\mathrm{start}, \ldots, \mathrm{end}]$, compute

\[
y_i = f\left(x_{\mathrm{start}-1}, x_{\mathrm{end}+1}, \mathrm{pos}_{i-\mathrm{start}+1}\right)
\]
where
\[
\begin{aligned}
h_0 &= [x_{\mathrm{start}-1};\,x_{\mathrm{end}+1};\,\mathrm{pos}_i] \\
h_1 &= \mathrm{LayerNorm}\left(\mathrm{GELU}(W_1 h_0)\right) \\
y_i &= \mathrm{LayerNorm}\left(\mathrm{GELU}(W_2 h_1)\right)
\end{aligned}
\]

The span boundary loss:
\[
L_{\mathrm{SBO}}(x ; \theta_G) = \mathbb{E}\left[\sum_{i\in M} -\log P_G(x_i \mid y_i)\right]
\]

### 2.4 Discriminator Loss (ELECTRA)

The discriminator distinguishes real tokens from generator replacements:

\[
L_{\mathrm{Disc}}(x ; \theta_D) =
\mathbb{E}\left[\sum_{j=1}^N -\left[\mathbb{1}_{j\,\mathrm{real}}\log D_j + \mathbb{1}_{j\,\mathrm{fake}}\log(1 - D_j)\right]\right]
\]

### 2.5 Total Loss

The overall loss combines these components:

\[
L_{\mathrm{total}} =
L_{\mathrm{MLM}}(x ; \theta_G)
+ \lambda_1 L_{\mathrm{SBO}}(x ; \theta_G)
+ \lambda_2 L_{\mathrm{Disc}}(x ; \theta_D)
\]

This composite objective explicitly incorporates domain relevance, infilling, and span boundary prediction [2211.00083].

## 3. Datasets and Pre-training Protocol

FLANG models are pre-trained using a mixture of general English and financial-domain corpora.

### Pre-training Data Sources

| Dataset                      | # Documents | Years      | % Sampled/Epoch |
|------------------------------|------------:|-----------:|----------------:|
| BooksCorpus                  |      —      |     —      |      —          |
| Wikipedia                    |      —      |     —      |      —          |
| SEC 10-K filings             |     13,660  | 1993–2020  |       8 %       |
| SEC 10-Q filings             |     36,402  | 1993–2020  |       5 %       |
| Earnings call transcripts    |    151,359  | 2007–2019  |     1.5 %       |
| Reuters financial news       |    106,521  |   2007     |      10 %       |
| Bloomberg financial news     |    387,220  |   2009     |       5 %       |
| Analyst reports (LexisNexis) |        201  | 2017–2020  |     100 %       |
| Investopedia concept pages   |        638  |    N/A     |     100 %       |

- BooksCorpus (800M words) and Wikipedia (2.5B words) represent general English.
- Financial corpora are subsampled per epoch as indicated.

### Pre-training Regimen

- Models initialized from Huggingface BERT-base or ELECTRA-base.
- Four epochs of pre-training on the combined data.
    - Epochs 1–2: single-token (word) financial masking.
    - Epochs 3–4: token + multi-token phrase masking.
- Masking rate: 15%.
- Adam optimizer; learning rate warm-up as in standard Transformer pre-training.
- Batch size and specific learning rate schedule not fixed.

## 4. Financial Language Understanding Evaluation (FLUE) and Empirical Results

FLUE is an open-source benchmark suite covering five core financial NLP tasks:

1. Financial Sentiment Analysis (Financial PhraseBank, FiQA 2018 SA)
2. News Headline Classification (Gold Headlines dataset)
3. Named Entity Recognition (finance-domain NER set)
4. Structure Boundary Detection (FinSBD3)
5. Financial Question Answering (FiQA 2018 QA)

### Summary Results (averaged over 3 seeds)

| Model            | FPB (Acc) | FiQA SA (MSE) | Headline (F₁) | NER (F₁) | SBD (F₁) | FiQA QA (nDCG) |
|------------------|----------:|--------------:|--------------:|---------:|---------:|--------------:|
| BERT-base        |     85.6% |        0.073  |      0.967    |   0.79   |   0.95   |    0.46       |
| FinBERT          |     87.2% |        0.070  |      0.968    |   0.80   |   0.89   |    0.42       |
| FLANG-BERT       |     91.2% |        0.054  |      0.972    |   0.83   |   0.96   |    0.51       |
| ELECTRA-base     |     88.1% |        0.066  |      0.966    |   0.78   |   0.94   |    0.52       |
| FLANG-ELECTRA    |     91.9% |    **0.034**  |   **0.980**   |   0.82   | **0.97** | **0.55**      |

Across every task, FLANG-BERT and FLANG-ELECTRA deliver state-of-the-art metrics compared to non-domain-adapted and prior financial-domain baselines. Detailed ablation results are available in the paper's supplementary tables.

## 5. Deployment and Usage Recommendations

- **Model access:** FLANG-BERT and FLANG-ELECTRA are available for download on Huggingface. Complete code, data, and model configurations are hosted at https://salt-nlp.github.io/FLANG/.
- **Fine-tuning:** For classification tasks, it is recommended to use a combined loss—cross-entropy plus Supervised Contrastive Loss:
  \[
  L = \lambda\,L_{\mathrm{CE}} + (1-\lambda)\,L_{\mathrm{SCL}}
  \]
  Refer to Section –A.3 of [2211.00083] for the exact formula.
- Usual hyperparameters: 2–5 epochs, learning rate $1e^{-5}$–$5e^{-5}$, batch size 16–32, weight decay 0.01.
- For regression and QA tasks, standard task heads from the Transformers library may be used (MSE loss for regression, bi-encoder or cross-encoder for QA).

*This suggests that FLANG-BERT and FLANG-ELECTRA can be integrated into existing NLP pipelines for financial tasks with minimal adaptation, while yielding substantial gains where financial vocabulary and structure are critical.*

## 6. Distinctive Features and Performance Analysis

Distinctive aspects of FLANG-BERT and FLANG-ELECTRA include:

- Systematic augmentation of vocabulary for financial precision (~8.2k terms/phrases).
- Layered masking regime emphasizing domain relevancy, including span-based and phrase-level masking.
- Integration of the span boundary objective, especially in FLANG-ELECTRA, to leverage local context for masked span reconstruction.
- Empirical validation demonstrating consistent outperformance over BERT-base, ELECTRA-base, and domain-adapted baselines (FinBERT), especially on complex multi-sentence financial tasks and open-domain QA.
- The FLUE benchmark suite is designed to be more comprehensive and challenging than previously available financial NLP benchmarks.

A plausible implication is that targeted adaptation through vocabulary design and masking objectives is critical for extending pre-trained language models to specialized technical domains such as finance. These adaptations achieve superior empirical results while preserving compatibility with standard Transformer and ELECTRA frameworks [2211.00083].

Source: https://www.emergentmind.com/topics/flang-bert-flang-electra