Fundamental Language Model (FLM) Overview

• Fundamental Language Model (FLM) is a modular paradigm that preserves core linguistic competence while delegating factual retrieval to external knowledge sources.
• Empirical evaluations reveal that while linguistic competence stabilizes at moderate model sizes, internal factual knowledge scales strongly with model size.
• FLM research underscores practical benefits such as improved efficiency, interpretability, and cost-effectiveness, while acknowledging challenges like source-selection bias and incomplete separation of competencies.

Fundamental LLM (FLM) most explicitly denotes a design paradigm in which a relatively compact model preserves the core linguistic competence of conventional LLMs while delegating factual retrieval to external knowledge sources, rather than internalizing both language ability and factual storage inside one monolithic system. In current literature, the term also names specific open model lines such as Tele-FLM/FLM-2 and FLM-101B, so it functions both as a conceptual program and as a family label within open foundation-model research (Collado-Montañez et al., 2 Sep 2025, Li et al., 2024, Li et al., 2023).

1. Definition and conceptual core

The explicit FLM paradigm is defined as a model that preserves the core linguistic competence of conventional LLMs while delegating factual retrieval to external knowledge sources. In this framing, linguistic competence is decomposed into lexical competence, grammatical competence, and semantic competence. The motivation is a critique of monolithic LLMs: coupling linguistic ability and factual storage in a single very large parameterized model is argued to contribute to hallucinations, bias, privacy concerns, high computational cost, and an entanglement of linguistic competence with factual memorization. FLMs therefore differ from standard retrieval-augmented generation by proposing “a more fundamental separation between linguistic and factual knowledge,” with the compact LLM serving as the linguistic core and external systems handling knowledge lookup (Collado-Montañez et al., 2 Sep 2025).

2. Empirical basis for the paradigm

The main empirical support for the FLM thesis comes from a cross-model evaluation of 23 released models ranging from 135M to 32.8B parameters, assessed in zero-shot mode with the LM Evaluation Harness across three capacities: linguistic competence, external factual knowledge, and internal factual knowledge. Linguistic competence is computed as the average of lexical competence, grammatical competence, and semantic competence; semantic competence is itself the average of RTE, MNLI, and QQP. External factual knowledge is the average across LAMBADA, BoolQ, COPA, MultiRC, and ReCoRD, while internal factual knowledge is the average across TriviaQA and three TruthfulQA variants (Collado-Montañez et al., 2 Sep 2025).

Capacity	Operationalization	Representative finding
Linguistic competence	WiC, BLiMP, and average of RTE/MNLI/QQP	Qwen2.5-32B reaches 0.7688
External factual knowledge	LAMBADA, BoolQ, COPA, MultiRC, ReCoRD	gemma-2-9b reaches 0.7961
Internal factual knowledge	TriviaQA and TruthfulQA variants	OLMo-2-0325-32B reaches 0.5784

The reported scaling pattern is asymmetric. Linguistic competence improves with scale, but the paper states that it “stabilizes at moderate model sizes” and shows diminishing returns beyond roughly 5–7B parameters. Internal factual knowledge grows much more strongly: the regression slope on $\log(\text{Size})$ is 0.059 for internal factual knowledge versus 0.029 for linguistic competence, and the regression for internal factual knowledge attains $R^2 = 0.81$. Quartile-based Mann–Whitney U tests at $\alpha = 0.05$ further show that linguistic competence has no significant gain from small to medium models ($p = 0.256$) or from medium to large models ($p = 0.062$), whereas internal factual knowledge is significant from small to medium ($p = 0.015$) and from small to large ($p = 0.009$). The paper interprets this as evidence that model size is more strongly associated with factual memorization than with core language ability (Collado-Montañez et al., 2 Sep 2025).

A related evaluation line operationalizes one core FLM-like property—fundamental knowledge competence—in Chinese LLMs. FoundaBench contains 3354 multiple-choice questions spanning common sense and K–12 educational subjects, and reports that reasoning questions are on average 28.69 points lower than memory-question scores, with $t=-10.11$ and $p<0.001$. Under CircularEval, score drops are negatively correlated with ordinary accuracy, $r=-0.47$ with $R^2 = 0.81$0, and Chinese-pretrained models outperform English-oriented ones on Chinese foundational knowledge (Li et al., 2024).

3. Modular system design and relation to retrieval

Within the FLM paradigm, the proposed division of labor is explicit. Inside the compact FLM reside lexical competence, grammatical competence, semantic competence, and the language interface for communication and interpretation; delegated externally are factual retrieval, knowledge lookup, and possibly reasoning over retrieved documents or tools. The claimed benefits are efficiency, interpretability, sustainability, privacy, and reliability, especially because external retrieval can provide fresher and more verifiable information than static internal memory. At the same time, the authors stress that linguistic and factual knowledge are not perfectly separable, since metaphors, cultural references, and domain-specific terminology may require world knowledge for genuine understanding (Collado-Montañez et al., 2 Sep 2025).

A closely related architectural thesis treats the LLM itself as a general-purpose interface or “universal task layer” to which pretrained encoders dock. MetaLM implements this with a causal LLM connected to bidirectional encoders through a semi-causal language modeling objective, preserving in-context learning and open-ended generation while improving finetuning through non-causal encoders. This suggests an implementation style for FLM systems in which the linguistic core remains the central interface while perception modules and other specialists remain modular (Hao et al., 2022).

4. Open model lines and scaling practice

Separate from the compact-and-modular FLM paradigm, the acronym FLM also labels a family of open large base models. Tele-FLM, explicitly identified as FLM-2, is a 52B open-sourced multilingual LLM with 52,850M parameters, 2.0 trillion training tokens, context length 4096, and vocabulary size 80,000. It is a decoder-only Transformer with pre-normalization, RMSNorm, SwiGLU, RoPE, untied embedding and language-model head, and no linear bias in attention or MLP modules. On foundation-model evaluation, the report states that it is comparable to strong open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B and DeepSeek-67B (Li et al., 2024).

The later Tele-FLM scaling report extends this line from 52.85B to 102.3B and then to 1083.74B parameters through staged, function-preserving growth. The full 52B→102B→1T process consumes 2318.7B training tokens, with 2003B at 52B, 300B at 102B, and 15.7B at 1T. The same report also argues for a “less is more” approach to supervised fine-tuning: starting from 1 million open-sourced Chinese instruction samples, the best-performing subset was 30k total samples, specifically 25k “maths” samples plus 5k coding problems and multi-turn dialogues, with the final selection filtered by Tele-FLM-52B perplexity and the best checkpoint appearing at the end of epoch 2 (Li et al., 2024).

FLM-101B is another open bilingual base model in this naming lineage. It is a 101B-parameter decoder-only Transformer trained in three sequential stages—16B, 51B, and 101B—over 311.54B total tokens. The paper reports 52.76 zettaFLOPs, about a \$100K budget, about 22 training days on 192 A800 GPUs, and an estimated 72% time saving and 3.56× speedup relative to training a fixed 101B model from scratch on the same token budget. Its central methodological claim is that function-preserving progressive growth can make 100B-scale training substantially cheaper while retaining a large fraction of more expensive baselines’ performance (Li et al., 2023).

5. Terminological ambiguity in the literature

The acronym FLM is not stable across recent arXiv literature. In encyclopedia usage, this ambiguity is important because “Fundamental LLM” is only one of several active expansions.

Usage context	Meaning of FLM	Example
FLM paradigm	Fundamental LLM	(Collado-Montañez et al., 2 Sep 2025)
Open model line	Tele-FLM / FLM-2 / FLM-101B	(Li et al., 2024, Li et al., 2024, Li et al., 2023)
Continual-learning ensembles	Foundational LLM	(Shiri et al., 2023)
ASR decoding	Forward LLM	(Ogawa et al., 2023)
Continuous generative modeling	Flow LLM	(Azangulov et al., 13 May 2026, Lee et al., 18 Feb 2026)
Vision-language pretraining objective	Free Language Modeling	(Wang et al., 2023)
Task-specific structured interface	Formal LLM	(Chen et al., 17 Jan 2025)
OW-CZSL feasibility module	Feasibility with LLM	(Kim et al., 16 May 2025)

This multiplicity often creates confusion. In some works, FLM means standard pretrained transformers such as BERT, DistilBERT, ELECTRA, and RoBERTa in a lifelong-learning ensemble; in ASR it denotes the forward LLM used in shallow fusion; in generative modeling it denotes flow-based LLMs; and in other task-specific systems it expands to formal LLM or feasibility with LLM (Shiri et al., 2023, Ogawa et al., 2023, Azangulov et al., 13 May 2026, Wang et al., 2023, Chen et al., 17 Jan 2025, Kim et al., 16 May 2025).

6. Limitations and open questions

The FLM paradigm is explicitly proposed as a promising modular alternative, not as a complete solution. Its own authors acknowledge that linguistic and factual knowledge are not perfectly separable, that the empirical study is mostly English-centric, that family effects are substantial because released models differ in data and architecture, and that external retrieval introduces source-selection bias and a need for transparency about what came from the model versus retrieved knowledge. For that reason, the practical recommendation is not “replace all large models with tiny ones,” but rather to use smaller linguistically strong models as the foundation for tool-augmented systems (Collado-Montañez et al., 2 Sep 2025).

The large-model FLM line has parallel limitations. The Tele-FLM 1T report does not compare against a from-scratch 1T baseline and explicitly says that comprehensive downstream benchmarking of the 1T checkpoint was not possible because of limited computational budget. FLM-101B, although important as a progressive-training demonstration, is also presented as a feasibility result rather than a Chinchilla-optimal recipe, and its paper leaves major details under-specified, including the exact intermediate architectures and the full data recipe (Li et al., 2024, Li et al., 2023).

Taken together, these works suggest that “Fundamental LLM” is best understood as a research program about reallocating functions across the language stack: preserving a strong linguistic core inside the model, moving factual and specialized operations outward to tools or modules, and treating language as the system interface. Whether that separation can remain robust across languages, domains, and high-stakes deployment remains an open empirical question (Collado-Montañez et al., 2 Sep 2025).