NoLBERT: Time-Scoped Language Model

• NoLBERT is a timestamped foundational language model that mitigates lookahead and lookback biases by restricting pretraining to a specific historical window.
• The model employs a custom ByteLevelBPE tokenizer and a controlled corpus, ensuring temporal integrity while achieving competitive NLP benchmark performance.
• NoLBERT demonstrates practical utility by enabling firm-level innovation network construction from patent texts, with empirical evidence linking centrality growth to future profit increases.

Searching arXiv for NoLBERT and closely related papers to ground the article and citations. arxiv_search(query="NoLBERT foundational LLM empirical research economics finance temporal consistency", max_results=10) NoLBERT is a timestamped foundational LLM for empirical research in the social sciences, particularly economics and finance, designed to preserve temporal consistency in text-based measurement and inference (Kakhbod et al., 1 Sep 2025). Its defining methodological premise is that conventional pretrained LLMs can compromise econometric work through two distinct forms of temporal contamination: lookahead bias, in which training data contain future language relative to the study period, and lookback bias, in which long-horizon pretraining collapses historically distinct meanings into a single representation. NoLBERT addresses both by restricting pretraining to text from 1976–1995, using 1996 as validation, while retaining competitive performance on standard NLP benchmarks and supporting downstream empirical analysis such as the construction of firm-level innovation networks from patent text (Kakhbod et al., 1 Sep 2025).

1. Concept and research motivation

NoLBERT was developed as a foundational LLM for settings in which the timing of textual information is itself part of the identification problem (Kakhbod et al., 1 Sep 2025). The model’s target use cases are empirical workflows in economics, finance, and related social-science domains where backtesting, causal inference, and historical measurement can be distorted if textual representations encode knowledge or semantics unavailable at the time.

The paper distinguishes two threats. Lookahead bias arises when a model has been trained on post-period text and therefore may indirectly exploit future information. Lookback bias arises when a model trained across long historical horizons learns blended embeddings that reinterpret earlier language through later usage. The motivating claim is not merely that future facts contaminate prediction; it is also that semantic drift contaminates interpretation. This framing places NoLBERT within a methodological agenda aimed at temporally disciplined representation learning rather than maximal cross-era coverage (Kakhbod et al., 1 Sep 2025).

A common misconception is to treat NoLBERT as a new transformer architecture. The paper explicitly states that it does not introduce a new transformer architecture; its novelty lies in being time-scoped, tokenized, and evaluated in a way intended to preserve temporal integrity (Kakhbod et al., 1 Sep 2025). This suggests that its contribution is primarily methodological: the design of a historically localized foundation model whose inductive bias is aligned with econometric constraints.

2. Temporal bias framework

The model’s central claim is that a narrow and timestamped pretraining window can mitigate both forward and backward temporal leakage (Kakhbod et al., 1 Sep 2025). NoLBERT is never trained on post-1995 text, so when it is applied to later-period data it cannot have learned from subsequent outcomes or later language usage. Conversely, because it is not trained on a century-spanning corpus, it does not force old and new meanings into a single ahistorical embedding space.

The paper formalizes this issue using paired masked-word evaluations for terms whose meanings shifted across eras. For old-era and new-era sentence contexts, the statistic is defined as

$$\frac{1}{n}\sum_{i=1}^n \big[\log P_{\text{old}(w_i)} - \log P_{\text{new}(w_i)}\big] = 0,$$

with one-sided paired $t$-tests used to evaluate directional alternatives (Kakhbod et al., 1 Sep 2025). Positive values and negative values are interpreted differently depending on whether the test is probing lookahead-type or lookback-type bias.

The reported results indicate temporal localization rather than universal historical generalization. For 2020–present vs. 1976–1995, the difference is $-2.78$ with $t=-5.56$ and $p<0.01$. For 1976–1995 vs. 19th century, the difference is 2.53 with $t=4.60$ and $p<0.01$ (Kakhbod et al., 1 Sep 2025). The paper interprets these findings as evidence that the model performs significantly better in its own training-era semantics than in either later or much earlier usage. A qualitative illustration uses the prompt “XXX is a United States <mask>,” for which presidents inside the 1976–1995 window are usually ranked highly, whereas those outside it often are not (Kakhbod et al., 1 Sep 2025). The intended implication is that NoLBERT behaves as a temporally bounded model rather than as an all-purpose ahistorical memory bank.

3. Architecture, corpus, and training protocol

NoLBERT is built on the DeBERTa v3 base architecture and has about 109 million parameters (Kakhbod et al., 1 Sep 2025). The paper characterizes it as lightweight relative to large contemporary foundation models and compact enough for academic use, while still competitive on language-understanding benchmarks.

Its pretraining corpus is restricted to 1976–1995 text, with 1996 used as validation (Kakhbod et al., 1 Sep 2025). The corpus is intentionally heterogeneous and timestamped, combining popular culture sources, formal prose, and economics-relevant material such as FOMC transcripts and patents. The rationale is to preserve broad linguistic coverage without sacrificing temporal control.

The tokenization pipeline is also historically localized. The authors train a custom ByteLevelBPE tokenizer from scratch with a 30,000-token vocabulary and minimum frequency threshold of 2 (Kakhbod et al., 1 Sep 2025). Pretraining uses a masked-language-model objective, mixed precision, and 15 epochs (Kakhbod et al., 1 Sep 2025). In the paper’s own framing, this makes NoLBERT better understood as a historically disciplined foundation model than as a novel neural architecture.

The training setup matters because temporal consistency is enforced at the level of the entire modeling stack rather than only at the corpus level. The tokenizer, vocabulary, and validation split are all aligned with the same temporal window. A plausible implication is that temporal leakage can enter not only through model weights but also through preprocessing and subword segmentation; the paper’s design aims to constrain those channels as well.

4. Benchmark behavior and empirical validation

The paper evaluates NoLBERT on GLUE tasks—CoLA, SST-2, QQP, MNLI, and QNLI—against FinBERT and StoriesLM (Kakhbod et al., 1 Sep 2025). NoLBERT uses the same 30k vocabulary size as those baselines and has slightly fewer parameters than FinBERT and StoriesLM, yet it performs best on average across the benchmark suite (Kakhbod et al., 1 Sep 2025).

Task	FinBERT / StoriesLM	NoLBERT
CoLA	0.29 / 0.49	0.43
SST-2	0.89 / 0.90	0.91
QQP	0.87 / 0.87	0.91
MNLI	0.79 / 0.80	0.82
QNLI	0.86 / 0.87	0.89

These results support a specific methodological claim: temporal discipline need not imply severe degradation on standard NLP tasks (Kakhbod et al., 1 Sep 2025). The benchmark pattern is uneven—StoriesLM exceeds NoLBERT on CoLA—but the aggregate comparison favors NoLBERT, especially on QQP, MNLI, and QNLI. In that sense, the model is positioned not as a specialized historical artifact but as a general-purpose encoder whose inductive constraints are tailored to empirical research.

The paper’s validation strategy therefore combines two criteria. One is temporal boundedness, demonstrated by the era-sensitive masked-word tests. The other is task competence, demonstrated by competitive GLUE performance (Kakhbod et al., 1 Sep 2025). Together, these evaluations are intended to show that temporal honesty and practical utility need not be in conflict.

5. Patent fine-tuning and innovation network construction

The paper’s principal downstream application is the use of patent text to construct a firm-level innovation similarity network (Kakhbod et al., 1 Sep 2025). NoLBERT is fine-tuned on patent abstracts using a pairwise classification design. For each patent abstract, the text is split into two chunks, $A$ and $B$. With probability 0.5, $A$ is paired with the true $t$0 from the same patent and labeled 1; otherwise, $t$1 is paired with a $t$2 from a different patent and labeled 0 (Kakhbod et al., 1 Sep 2025).

This fine-tuning is carried out year by year using patents granted from 1997 to 2021, with 70/30 train-test splits and one epoch of training per year (Kakhbod et al., 1 Sep 2025). The reported average accuracy is 98%, which the authors interpret as evidence that the $t$3 representation captures patent-level textual similarity effectively.

For the pair classifier, the feature representation is built from the two $t$4 embeddings as

$t$5

forming a $t$6-dimensional input to a two-layer MLP with ReLU and dropout (Kakhbod et al., 1 Sep 2025). Training uses AdamW with learning rate $t$7, weight decay 0.01, 10% warmup, gradient clipping at $t$8, and cross-entropy loss (Kakhbod et al., 1 Sep 2025).

After fine-tuning, patents are embedded within each year $t$9. If $-2.78$0 and $-2.78$1 denote the sets of patent embeddings for firms $-2.78$2 and $-2.78$3 in year $-2.78$4, firm-level innovation similarity is defined as

$-2.78$5

where $-2.78$6 and $-2.78$7 are the mean embeddings of the firms’ patents in that year (Kakhbod et al., 1 Sep 2025). This yields a sparse, weighted, undirected firm–firm innovation network $-2.78$8 with adjacency matrix $-2.78$9, where $t=-5.56$0 is the textual similarity between firms $t=-5.56$1 and $t=-5.56$2 (Kakhbod et al., 1 Sep 2025).

6. Centrality measures and econometric findings

The paper’s main network statistic is PageRank centrality computed on the innovation similarity graph (Kakhbod et al., 1 Sep 2025). The row-stochastic transition matrix is

$t=-5.56$3

where $t=-5.56$4 and $t=-5.56$5 (Kakhbod et al., 1 Sep 2025). PageRank is then computed by power iteration,

$t=-5.56$6

with damping factor $t=-5.56$7, initialized at

$t=-5.56$8

and iterated to convergence (Kakhbod et al., 1 Sep 2025). As a robustness measure, the paper also uses weighted-degree centrality,

$t=-5.56$9

and reports a Pearson correlation of 0.81 between PageRank and weighted-degree centrality (Kakhbod et al., 1 Sep 2025).

The main regressor is the one-year log change in PageRank centrality,

$p<0.01$0

which is related to future changes in log profit over horizons $p<0.01$1 with controls for firm-level innovation value, industry-level innovation value, log profit, log employment, and log capital stock, along with year fixed effects and Fama–French 30 industry fixed effects (Kakhbod et al., 1 Sep 2025). Standard errors are double-clustered by industry and year, and all regressors are standardized within industry-year cells (Kakhbod et al., 1 Sep 2025).

The reported coefficients on centrality growth are 0.001 at $p<0.01$2 and not significant, 0.005$p<0.01$3 at $p<0.01$4, 0.007$p<0.01$5 at $p<0.01$6, 0.007$p<0.01$7 at $p<0.01$8, and 0.003$p<0.01$9 at $t=4.60$0 (Kakhbod et al., 1 Sep 2025). Sample sizes range from 20,976 observations at $t=4.60$1 to 14,199 at $t=4.60$2, while $t=4.60$3 rises from 0.023 to 0.035 (Kakhbod et al., 1 Sep 2025). The paper interprets a one standard deviation increase in PageRank centrality growth as predicting about a 0.5% increase in profit growth by year 2 and about a 0.3% increase by year 5 (Kakhbod et al., 1 Sep 2025).

The temporal profile of the coefficients is important. The relationship is not immediate; it strengthens over medium and longer horizons. The paper treats this as consistent with the view that innovation centrality captures gradual repositioning within a firm’s technological ecosystem rather than instantaneous monetization of patent value (Kakhbod et al., 1 Sep 2025).

7. Interpretation, robustness, and significance

The paper argues that innovation centrality measures something distinct from the standalone value of a firm’s patents (Kakhbod et al., 1 Sep 2025). A firm whose patents become more central in the similarity network may become a reference point, complement, or standard for other firms’ innovation. The authors describe this as diffusion leverage and stronger complementarities, which can raise profits without requiring comparable growth in physical capital (Kakhbod et al., 1 Sep 2025).

To probe this interpretation, the study also examines profit margin growth, using profitability growth defined as $t=4.60$4 (Kakhbod et al., 1 Sep 2025). The reported pattern is that innovation value is positively related to contemporaneous profitability, whereas centrality growth predicts future profitability growth rather than merely current margin levels (Kakhbod et al., 1 Sep 2025). The paper presents this as evidence that centrality is forward-looking.

The main result is also subjected to robustness checks using 2-year changes in centrality and using weighted-degree centrality instead of PageRank; in each case, growth in innovation centrality remains positively associated with future profit growth, with stronger coefficients at horizons $t=4.60$5 to $t=4.60$6 (Kakhbod et al., 1 Sep 2025). Descriptively, level centrality is strongly correlated with firm size and performance—profit 0.497, capital stock 0.461, employment 0.483, and firm-level innovation value 0.404—whereas changes in centrality are less mechanically tied to size (Kakhbod et al., 1 Sep 2025). The most central firms increasingly concentrate in Personal and Business Services and Healthcare, Medical Equipment, and Pharmaceuticals (Kakhbod et al., 1 Sep 2025).

NoLBERT’s broader significance is therefore twofold. Methodologically, it proposes a foundation-model design in which timestamping is treated as a first-order requirement for empirical validity. Substantively, it shows that text-derived innovation networks can recover economically meaningful structure, with gains in innovation centrality predicting higher medium- and long-run profit growth even after controlling for patent value and standard firm characteristics (Kakhbod et al., 1 Sep 2025). Within that framing, NoLBERT is best understood not as a general-purpose knowledge model but as an instrument for temporally consistent representation learning in empirical research.