---
title: 'Astro-HEP-BERT: BERT for HPSS Semantic Analysis'
url: https://www.emergentmind.com/topics/astbert
type: topic
---

# Astro-HEP-BERT: BERT for HPSS Semantic Analysis

Astro-HEP-BERT, often abbreviated as AstBERT in the project materials, is a transformer-based language model for generating contextualized word embeddings (CWEs) to study the meanings of concepts in astrophysics and high-energy physics. It is built by continuing the pretraining of a general BERT model on the Astro-HEP Corpus, a domain corpus curated from more than 600,000 arXiv articles and 21.84 million filtered paragraphs from astrophysics and high-energy physics. The model is positioned not as a general-purpose scientific foundation model, but as a bidirectional language model optimized for context-sensitive semantic analysis, especially for applications in the history, philosophy, and sociology of science (HPSS) such as word sense disambiguation, word sense induction, semantic change detection, and discourse analysis [2411.14877].

## 1. Conceptual scope and naming

Astro-HEP-BERT is explicitly designed for studying the meanings of concepts in astrophysics and high-energy physics through CWEs rather than for citation recommendation, document classification, or generative text production as primary endpoints. Its stated aim is to adapt a bidirectional transformer to HPSS-oriented semantic analysis, using domain-specific continued pretraining rather than training from scratch [2411.14877].

The abbreviation “AstBERT” is potentially ambiguous in the broader literature. In the present usage it denotes Astro-HEP-BERT, but the same shorthand also appears in work on AST-aware programming-language models, including a consolidated description based on TreeBERT and the separate financial-code model “AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees” [2105.12485] [2201.07984]. This makes the astrophysics/high-energy-physics expansion important in bibliographic and technical contexts.

A common assumption in domain adaptation is that competitive domain models require training from scratch on very large specialist corpora. Astro-HEP-BERT is presented as a counterexample to that assumption: preliminary evaluations indicate that continued pretraining of a general BERT model on a domain corpus can yield CWEs comparable to domain-adapted BERT models trained from scratch on larger datasets for domain-specific word sense disambiguation, induction, and related semantic change analyses [2411.14877].

## 2. Architecture and learning objective

Astro-HEP-BERT is initialized from the original uncased BERT-Base. The inherited backbone has 12 transformer layers, hidden size $d = 768$, 12 self-attention heads of dimension $d_k = d/12 = 64$ each, and approximately 110 million parameters [2411.14877].

The domain adaptation modifies the pretraining setup rather than the core transformer stack. The Next Sentence Prediction head is removed, and training is performed on Masked Language Modeling only. Whole-word masking is applied, so whenever one subword of a word is masked, all of its subwords are masked. The input format is also changed from “single sentence” or “document-sentence” inputs to a full-paragraph format in which each training example is one complete paragraph. Dynamic batch packing targets $8\,192 \pm 5\%$ tokens per batch, with at most 20% padding, in order to reduce wasted compute [2411.14877].

The MLM objective is the standard cross-entropy over masked positions. Let $M$ be the set of masked token positions and $\hat{x}$ the corrupted input. The minimized loss is

$$
L_{MLM} = - \sum_{i \in M} \log P(x_i \mid \hat{x}).
$$

For reference, each transformer block computes multi-head attention as

$$
\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V.
$$

The architecture therefore remains a standard bidirectional transformer encoder in the BERT family, but its training distribution and masking regime are tailored to scientific paragraphs in astrophysics and high-energy physics. This suggests that the principal innovation lies in corpus design, input granularity, and economical domain adaptation rather than in architectural novelty.

## 3. Astro-HEP Corpus and preprocessing pipeline

The Astro-HEP Corpus covers approximately 600,000 arXiv papers from 1986 to 2022 in astrophysics and high-energy physics, restricted to the 10 ASTRO-PH and HEP subcategories. After filtering, the corpus contains 21.84 million paragraphs and approximately 12.7 billion total tokens, of which approximately 1.9 billion were masked during training [2411.14877].

The preprocessing pipeline is specified as a sequence of operations:

1. Download LaTeX sources via Kaggle arXiv metadata and restrict to the 10 ASTRO-PH and HEP subcategories.  
2. Pre-normalize LaTeX, including spacing and macros, and convert to plain text with Pandoc.  
3. Replace in-text citations with `[CIT]`, replace block formulas with `FORMULA`, and retain inline math in the form `"$…$"`.  
4. Split on newlines to obtain paragraphs.  
5. Filter paragraphs shorter than 250 characters.  
6. Filter out paragraphs with whitespace-character rates below 0.1 or above 0.2 [2411.14877].

Tokenization uses the WordPiece tokenizer inherited from BERT-Base, with the full original BERT vocabulary of 30,522 tokens unchanged. No new domain-specific vocabulary is introduced [2411.14877]. This is notable in comparison with related domain models such as astroBERT, which retrains the WordPiece tokenizer on the ADS corpus to capture astrophysics-specific tokens such as “Planck,” “AGN,” and “H II” [2212.00744].

The paragraph-centric corpus construction is central to the model’s semantic aims. Because each example is a complete paragraph, the unit of contextualization is larger than a sentence but smaller than a full document. A plausible implication is that the model is optimized for local scientific discourse structure, where term meaning is often determined by paragraph-level argumentation, exposition, or methodological framing.

## 4. Pretraining regime and compute profile

Continued MLM pretraining is carried out for 3 epochs over the 21.84 million paragraphs, corresponding to 12.7 billion tokens seen. The masking rate is approximately 15%, following standard BERT masking [2411.14877].

Optimization uses AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.999$, and weight decay $= 0.01$. The learning rate follows linear decay from $1 \times 10^{-4}$ to $0$ over all steps, with no NSP warmup needed. Batching uses the same dynamic token budget of 8,192 tokens with a $\pm 5\%$ tolerance and no more than 20% padding per batch [2411.14877].

The full training run is conducted on a single MacBook Pro with Apple M2 and 96 GB RAM, with total wall-clock time of approximately 48 days for all 3 epochs [2411.14877]. This compute profile is integral to the project’s methodological claim. The model is presented as a cost-effective alternative to training domain-specialized BERTs from scratch, and the comparison is framed explicitly against astroBERT and PhysBERT, which are described as requiring several GPU-months [2411.14877].

The efficiency argument becomes sharper when set beside the reported training setup of astroBERT, which uses approximately 400,000 full-text astronomy and astrophysics papers, about 4 billion tokens, MLM plus NSP, and 40 epochs over about 50 days on two NVIDIA V100 GPUs [2212.00744]. Astro-HEP-BERT therefore exemplifies continued pretraining under comparatively modest hardware constraints rather than large-scale de novo pretraining.

## 5. Contextualized word embeddings and empirical behavior

For a given paragraph, CWE extraction proceeds by a forward pass through the model and collection of hidden states from layer $\ell$, with the default choice being the last layer, $\ell = 12$. When the target word is split into subwords, the hidden states of those subwords may be average-pooled to obtain a single vector. Code and examples are available on Hugging Face under `arnosimons/astro-hep-bert` [2411.14877].

The project includes an illustrative ambiguity example based on the word “light.” In the context “the particle is light,” $\mathrm{CWE}_{12}(\text{light})$ clusters near mass/energy contexts; in the context “light is a wave,” $\mathrm{CWE}_{12}(\text{light})$ clusters near electromagnetic-field contexts. The nearest-neighbor distances between the two senses increase by more than 0.7 cosine, indicating strong contextual separation [2411.14877].

Evaluation details and data are reported in Simons (2024), where “Planck” is used as a prototypical polysemous term. On the reported WSD/WSI test setting, Astro-HEP-BERT reaches precision 0.82, recall 0.80, and $F_1 = 0.81$. The comparison models score as follows: astroBERT 0.84/0.83/0.83, PhysBERT 0.80/0.78/0.79, SciBERT 0.75/0.73/0.74, and BERT-Base 0.70/0.68/0.69 [2411.14877].

The semantic change analysis tracks shifts in “Planck” before and after the 2009 Planck mission launch. Astro-HEP-BERT clusters split into senses corresponding to “constant $\hbar, c, k_B$” and “Planck satellite data,” with silhouette score greater than 0.65. The model achieves comparable change-point detection to astroBERT and PhysBERT [2411.14877]. The stated takeaway is that continued pretraining on a domain corpus yields virtually the same CLE performance as training from scratch on a much larger physics-only dataset.

## 6. Significance for HPSS and relation to adjacent models

Astro-HEP-BERT is explicitly framed as an instrument for HPSS research. Its CWEs are said to enable large-scale “distant reading” of conceptual evolution while preserving local context sensitivity. The intended applications include digital Begriffsgeschichte, semantic change studies, and discourse analysis in astrophysics and high-energy physics [2411.14877].

The significance of the project lies partly in accessibility. The model leverages existing BERT vocabulary and weights, uses freely available code, pretrained weights, and text inputs, and is associated with open-source code, model, and corpus for reproducible HPSS research. The paper identifies a reduced CO$_2$ footprint and a reduced barrier of entry for HPSS researchers as consequences of this reuse strategy [2411.14877].

In the surrounding ecosystem of scientific BERT variants, astroBERT provides a useful comparison point. That model, introduced in the NASA Astrophysics Data System context, is also initialized from BERT-base and tailored to astrophysics literature, but it retrains the tokenizer on ADS data, retains MLM and NSP, and targets tasks such as NER on the DEAL shared task, where the public release version reports test $F_1 = 0.609$ and MCC approximately 0.830 [2212.00744]. Astro-HEP-BERT differs in corpus scope, which includes both astrophysics and high-energy physics, in its unchanged BERT vocabulary, and in its emphasis on contextual lexical semantics for HPSS rather than on entity extraction benchmarks [2411.14877].

A recurring misconception is that an abbreviation such as “AstBERT” necessarily refers to an AST-aware code model because of the prominence of abstract syntax trees in programming-language research. The literature contains both AST-aware uses of the label and the astrophysics/high-energy-physics usage. Tree-based pretraining for programming languages has been summarized under the shorthand “AstBERT” in relation to TreeBERT, where ASTs are represented as composition paths and trained with Tree-Masked Language Modeling and Node Order Prediction [2105.12485]. Separately, the financial-code paper titled “AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees” integrates AST information into CodeBERT-derived representations and reports downstream results on code question answering, code clone detection, and code refinement [2201.07984]. In the astrophysics and high-energy-physics setting, however, AstBERT designates Astro-HEP-BERT and refers to “astro” rather than “AST.”

Source: https://www.emergentmind.com/topics/astbert