---
title: 'GenBERT: Enhanced Numeracy in Transformers'
url: https://www.emergentmind.com/topics/genbert
type: topic
---

# GenBERT: Enhanced Numeracy in Transformers

GenBERT refers to a pre-trained language model specifically designed to inject numerical reasoning skills into transformer architectures via large-scale, automatically generated arithmetic and math-word-problem data, trained alongside the masked language modeling (MLM) objective. The core strategy is to harness controlled synthetic data and multitask learning to endow standard BERT-based models with robust arithmetic and comparative reasoning capabilities, addressing the limitations of standard language model pre-training for high-level numerical tasks [2004.04487].

## 1. Model Architecture

GenBERT is implemented as an encoder–decoder transformer, both components initialized from BERT-base-uncased. Each module comprises 12 layers, a hidden size of 768, 12 self-attention heads, and an intermediate feed-forward network (FFN) dimension of 3072. Encoder and decoder self-attention and FFN weights are tied, as are the decoder’s source-attention weights with the encoder’s weights. 

The encoder output is processed by a small FFN:
$$
H_{\mathrm{enc}} = \mathrm{LayerNorm}(\mathrm{GELU}(W_{\mathrm{enc}} L_{\mathrm{enc}}))
$$
with an analogous FFN for decoder states. Decoder-side generation is delimited by [SOS] and [EOS] tokens.

GenBERT utilizes a composite answer-head structure:
- **Two span-extraction heads** operating over $H_{\mathrm{enc}}$ for extracting answer spans from the question or passage.
- **A generative (“decoder”) head** for token-level answer emission outside input sequences, applied over $H_{\mathrm{dec}}$.
- **A type head** built over $H_{\mathrm{enc}}$ to select which answer head to use (with probabilities $p_q$, $p_c$, $p_{dec}$).

Two key modifications are included for numerical tasks:
- **Digit Tokenization (DT):** Each numerical token is split into its digit-wise components, aiding numeric-value representation learning.
- **Random Shift (RS):** Positional embeddings are randomly offset per batch for short synthetic examples, disrupting position-based memorization.

## 2. Synthetic Data Generation Pipeline

GenBERT’s numerical proficiency derives from curated synthetic datasets, split into numerical (ND) and textual (TD) regimes.

### Numerical Data (ND)
- **Templates (6 total):** Arithmetic expressions (signed float combinations, min/max/avg, argmin/argmax, date operations, percentage queries), mapping directly to numerical answers. 
- **Sampling:** Inputs sample floats, signs, dates, or labeled entities according to defined pools and vocabularies.
- **Scale:** 1 million training and 10,000 validation examples.

### Textual Data (TD)
- **Source templates:** Top ≈12 sentence templates derived from established math word problem (MWP) corpora, abstracted as slot-based templates (NUM, entity, container, verb-category, attribute).
- **Passage construction:** 3–6 templates per passage are sampled and instantiated, with slot fillers drawn to enforce intra-passage cohesion and maintain a world state mapping entities and quantities.
- **Question construction:** 13 interrogative templates encompassing seven numeric reasoning skills (comparisons, superlatives, percentages, etc.), instantiated from the evolving world state.
- **Scale:** 2.5 million training and 10,000 validation examples.

## 3. Multi-task Training Objectives

Joint pre-training is conducted over three tasks—ND, TD, and standard MLM—using a unified loss:
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{model}}(X_{ND}) + \mathcal{L}_{\text{model}}(X_{TD}) + \lambda \cdot \mathcal{L}_{\text{mlm}}(X_{MLM})
$$
where $\lambda$ ensures MLM and synthetic losses are on comparable scales ($\lambda \approx 1$). Here, $\mathcal{L}_{\text{model}}$ is the negative log likelihood over the span and generation answer heads, and $\mathcal{L}_{\text{mlm}}$ is the negative log likelihood on masked tokens.

## 4. Pre-training and Fine-tuning Regimen

Pre-training begins from BERT-base-uncased, with DT and RS applied to all synthetic batches. Training alternates across ND, TD, and MLM mini-batches, typically for one epoch over each synthetic corpus; hyperparameters such as learning rate and batch size follow standard BERT practice.

Fine-tuning is performed downstream on the DROP benchmark for 3–5 epochs ($\mathrm{LR}\approx 3 \times 10^{-5}$, $\text{max seq len}=512$). SQuAD v1 fine-tuning follows the same procedure for pure reading comprehension evaluation.

## 5. Empirical Results

### Performance on DROP

| Model / Setting                 | EM   | F₁   |
|---------------------------------|------|------|
| Baseline GenBERT (no synth)     | 46.1 | 49.3 |
| + ND only                       | 64.7 | 68.2 |
| + TD only                       | 64.4 | 67.8 |
| + ND + TD                       | 68.8 | 72.3 (test: 68.6/72.4) |

GenBERT with both ND and TD matches MTMSN_base (specialized architecture) F₁ on DROP while retaining a general-purpose encoder–decoder design.

Breakdown by answer type (DROP F₁):

| Answer type      | GenBERT+ND+TD | MTMSN_base |
|------------------|:-------------:|:----------:|
| number           | 75.2          | 75.0       |
| span             | 74.5          | 71.3       |
| date             | 56.4          | 44.2       |
| multi-span list  | 24.2          | 53.4       |

### Generalization and SQuAD

GenBERT maintains BERT-level performance on SQuAD v1 (EM 81.3, F₁ 88.6), indicating no loss in language understanding, while zero-shot generalization to MWPs (e.g., AddSub, SOp, SEq) is improved by ND/TD pre-training (EM improvements up to ~28.3), though multi-term reasoning remains challenging.

## 6. Ablation and Error Analysis

Ablation studies demonstrate that both DT and RS are critical; removing DT prevents arithmetic learning, and omitting RS yields overfitting to positional patterns. Excluding MLM slows convergence and impairs RC fine-tuning transfer.

Error analysis of random DROP failures with GenBERT+ND+TD reveals:
- 43% require unmodeled reasoning (sorting, complex lists, coreference),
- 23% due to minor answer format errors (off-by-span/partial digit),
- remainder from comparative/multi-span reasoning.

This suggests expanding template diversity (e.g., sorting, multiplication, coreference) would improve coverage.

## 7. Interpretation and Broader Impact

GenBERT demonstrates that automatic skill-focused data generation coupled with multitask pre-training enables injection of high-level numerical reasoning into standard transformer architectures without loss of language understanding. The approach avoids architectural specialization (remaining flexible between span extraction and answer generation) and is extensible whenever large volumes of skill-targeted synthetic examples can be generated. Full competence in mathematical reasoning beyond arithmetic remains limited by the coverage of synthetic templates, but further template development or inclusion of noisy real examples is straightforward [2004.04487].

Source: https://www.emergentmind.com/topics/genbert