Internal Linguistic Spectrum in Language Models

Updated 17 July 2026

Internal Linguistic Spectrum is a conceptual framework that defines the layered organization of linguistic information across model structures and training checkpoints.
It illustrates how neural probing reveals the gradual emergence of syntax, morphology, and semantic interfaces in models such as GPT-2 XL and HuBERT.
The framework bridges diverse methodologies from neural, acoustic, and formal analyses to explain scaling effects, competency shifts, and memorization trends.

“Internal Linguistic Spectrum” denotes a family of research constructs for describing how linguistic information is internally organized, stratified, and transformed. In neural language modeling, it most often refers to the layer-wise and component-wise distribution of linguistic features inside a model, such as the emergence of syntactic grammaticality in early and intermediate layers and the deeper consolidation of morphology or semantics–syntax interface phenomena (He et al., 2024). In self-supervised speech modeling, it refers to the joint distribution of linguistic information across layers and training time, spanning a continuum from acoustic and phonetic structure to semantic and syntactic structure (Kloots et al., 2 Apr 2026). Other work uses the term for the balance between linguistic competence and internal factual knowledge as model size increases (Collado-Montañez et al., 2 Sep 2025), for the stratified organization of linguistic information from phonology through pragmatics in Functional Language Theory (Broekman et al., 2021), and for broader theoretical axes linking symbolic and distributed representations, cognition and computation, individual and social coordination, and synchronic and evolutionary analysis (Mansfield et al., 25 Feb 2025).

1. Conceptual scope and major usages

The term does not have a single standardized definition across the literature. Instead, it names a recurring idea: linguistic information is not monolithic, but internally stratified. Different traditions specify the strata differently and study different carriers of that structure.

Usage	Core definition	Principal organization
Neural probing	Layer-wise and component-wise distribution of linguistic information in model internals	Layers, hidden states, attention heads, frequency bands
Self-supervised speech	Distribution and evolution of linguistic structure across layers and training time	Abstraction level and temporal integration window
Scaling studies	Relative share of linguistic competence versus memorized facts in parameters	Model size and competence–memorization ratios
FLT and graph-theoretic work	Stratified linguistic information from phonology to pragmatics	Subconstructs and mental/realised metamodels
Broad theoretical syntheses	Intersecting axes of representation, cognition, sociality, and evolution	Symbolic/distributed, individual/social, synchronic/diachronic

A closely related theoretical synthesis defines the “internal spectrum” through four intersecting axes: representational format, cognitive mechanisms versus computational implementations, individual cognition versus intersubjective or social coordination, and synchronic structure versus evolutionary or cultural dynamics. In that formulation, symbolic grammars, distributed neural representations, Bayesian pragmatics, and phylogenetic or agent-based models occupy different regions of a shared conceptual map rather than belonging to separate domains (Mansfield et al., 25 Feb 2025).

A longer-range analogue appears in statistical-physics work on language dynamics, where language is described through a hierarchy of coupled scales running from individual learning and interaction to community, population, and historical timescales. On that view, phonology, lexicon, morphology, syntax, semantics, and discourse appear at multiple scales, but with different state variables and different effective descriptions at each scale (Blythe, 2015).

2. Layer-wise spectra in neural LLMs

The most explicit neural formulation is introduced by “decoding probing,” which treats the LLM as a “brain” and intermediate representations as “neural activations,” then decodes BLiMP grammaticality labels from those activations layer by layer. In GPT-2 XL, grammatical information becomes decodable gradually through the first third of the stack, approximately layers 1–16 of 48, and then remains distributed across later layers with a slight downward trend. On the 41-task subset obtained after excluding 26 tasks where GloVe’s F1 exceeds 0.9, sentence complexity correlates with feature capture depth at $r = 0.58, p < 1\times 10^{-4}$ . Mean feature capture depths are 19.50 layers for the semantics–syntax interface, 16.00 for morphology, and 14.25 for syntax, indicating that semantics–syntax interface phenomena are the most depth-demanding and syntax the least (He et al., 2024).

Within the same study, hidden states and attention behave differently. Hidden-state embeddings show a clearer hierarchical progression, whereas attention-based decoding oscillates across layers rather than accumulating smoothly. Concatenating all 20 GPT-2 XL attention heads often yields $F1 > 0.9$ , especially for syntax and semantics–syntax interface phenomena, while most single heads remain below $F1 < 0.8$ . A small subset, notably heads No. 8 and No. 16, consistently ranks highest. The same comparative analysis shows GPT-2 XL outperforming ELMo and GloVe overall, with ELMo close to GPT-2 XL on syntax but weaker on morphology and interface phenomena, and GloVe performing surprisingly well on some tasks such as quantifiers and ellipses (He et al., 2024).

A broader systematic review of 160 interpretability studies generalizes this pattern across pretrained Transformers. For many encoder PLMs, lower layers primarily encode lexical or subword and shallow morphosyntactic cues, middle layers maximize syntactic structure, and higher layers consolidate semantics and some discourse-like properties. The review also emphasizes that this organization is distributed rather than strictly localized: relations may be split across heads, and some phenomena occupy low-dimensional subspaces rather than single neurons. It also records important architecture-dependent exceptions, including earlier localization in XLNet, later localization in ELECTRA, and syntax concentrated in top layers 11–12 in mBART (López-Otal et al., 9 Apr 2025).

Minimal-pair difference analysis extends the same idea from task decoding to representational geometry. Over 104 LLMs and 150,000 minimal pairs in English, Chinese, and Russian, linguistic similarity is defined as cosine similarity between activation-difference vectors for minimal pairs. Cross-model agreement is highest for English, with a mean alignment score of 0.471 on BLiMP, lower for Chinese at 0.414 on SLING, and much lower for Russian at 0.139 on RuBLiMP. The resulting geometry aligns strongly with fine-grained theoretical categories but only weakly with broader linguistic fields, and it correlates only weakly with semantic similarity, indicating that the internal organization of linguistic contrasts is more phenomenon-specific than topic-like (Zhou et al., 2024).

Other diagnostics point to a similar stratification. Spectral probing decomposes contextual embeddings by frequency and finds that POS relies strongly on high frequencies, dependency relation classification on mid-high frequencies, and sequence-level tasks such as sentiment, topic classification, and natural language inference on low frequencies. Intrinsic-dimension analysis across six decoder-only LLMs finds a robust mid-layer ID peak on generic sequences; formal complexity in coordination versus subordination first separates just under or just before that peak and remains higher thereafter for subordination (Müller-Eberstein et al., 2022, Baroni et al., 7 Jan 2026).

3. Training-time emergence in self-supervised speech models

In self-supervised speech modeling, the internal linguistic spectrum is explicitly defined as the distribution and evolution of linguistic information across internal layers and training time, spanning a continuum from short-timescale, low-abstraction acoustic and phonetic features to long-timescale, high-abstraction syntactic and semantic structures. The study probes nine structures—acoustic alignment, phonetic categories, syllable form, syllable type, word form, part-of-speech, homophone disambiguation, semantic alignment, and syntactic dependency—across six Dutch Wav2Vec2 and HuBERT models (Kloots et al., 2 Apr 2026).

The layer-wise organization is highly structured. In Wav2Vec2 and HuBERT-I1, acoustic, phonetic, and syllabic information rises in early-to-mid layers, while lexical and syntactic information concentrates around T7; final layers show drops for linguistic probes, consistent with specialization for the acoustic pre-training objective. HuBERT-I2, trained on iteratively refined pseudo-labels taken from HuBERT-I1 layer T6, behaves differently: higher-level structures persist more strongly into later layers, more abstract probes peak before phonetic, syllable, and word-form peaks, and peak scores for all non-acoustic probes exceed those of HuBERT-I1 (Kloots et al., 2 Apr 2026).

The training trajectory is equally stratified. Acoustic alignment rises sharply and reaches maximum performance within the first 10k steps, while most linguistic probes surpass the nonspeech baseline around 10k steps. POS advantage over the nonspeech baseline appears around 25k steps, and syntactic dependency around 50k steps. Peak-layer locations stabilize between 10k and 50k steps, while additional training to 200k yields minimal improvements for most probes except consistent gains in syntax UUAS, suggesting that later training chiefly improves sentence-level contextualization (Kloots et al., 2 Apr 2026).

This temporal ordering is tied to temporal integration windows. Phone-level structure depends on tens of milliseconds, word-form structure on roughly word-scale windows, and semantics, POS, and syntax on beyond-word context. The study therefore argues that the spectrum is jointly organized by abstraction from the acoustic signal and by the timescale required to integrate input information (Kloots et al., 2 Apr 2026).

4. Scaling, competence, and memorization

A distinct usage treats the internal linguistic spectrum as a continuous view of what an LLM’s parameters predominantly encode as model size increases: core linguistic competence versus memorized facts. Linguistic competence is operationalized through lexical competence, grammatical competence, and semantic competence; internal factual knowledge through closed-book recall; and external factual knowledge through reasoning over provided context. The study evaluates models from 0.135B to 32.8B parameters across SmolLM2, Qwen2.5, Llama-3/3.1/3.2, OLMo-2, Falcon3, Gemma-2, and Yi-1.5 families (Collado-Montañez et al., 2 Sep 2025).

The central empirical result is a difference in scaling slopes. Linguistic competence follows $L(N) \approx a_L + b_L \log N$ with $b_L = 0.029$ and about $50\%$ of variance explained, whereas internal factual knowledge follows $F_{\mathrm{int}}(N) \approx a_F + b_F \log N$ with $b_F = 0.059$ and $R^2 = 0.81$ . Since $0.059 > 0.029$, internal factual knowledge grows faster than linguistic competence. The paper formalizes the resulting shift with $F1 > 0.9$ 0, which increases with size, and $F1 > 0.9$ 1, which decreases with size. Illustrative examples move from $F1 > 0.9$ 2 for SmolLM2-135M to $F1 > 0.9$ 3 for OLMo-2-0325-32B (Collado-Montañez et al., 2 Sep 2025).

Performance trends are asymmetric across the three evaluation dimensions. Linguistic competence reaches 0.7688 for Qwen2.5-32B, but multiple 3–13B models already lie around 0.71–0.72. External factual knowledge peaks in mid-range systems, including Gemma-2-9b at 0.7961 and Falcon3-10B-Base at 0.7746. Internal factual knowledge rises much more steeply and reaches 0.5784 for OLMo-2-0325-32B. Mann–Whitney U tests show significant differences from small to large models for EFK, IFK, and LC, but medium-to-large differences are not statistically significant for these benchmarks, which the paper interprets as diminishing returns in LC and EFK while IFK remains strongly size-tied (Collado-Montañez et al., 2 Sep 2025).

This formulation motivates the Fundamental LLM paradigm. The paper argues for compact models that already exhibit strong linguistic competence, paired with external retrieval or tools for factual knowledge, rather than further increasing parameter count primarily to expand memorization. The broader claim is not that scaling stops improving language ability, but that the internal balance of what parameters encode shifts toward memorized facts faster than it shifts toward linguistic competence (Collado-Montañez et al., 2 Sep 2025).

5. Formal and theoretical formulations beyond neural probing

In Functional Language Theory, the internal spectrum is the layered organization of linguistic information across phonology, morphology, syntax, semantics, and pragmatics, with each subconstruct modeled both as a mental metamodel and as a realised metamodel. In the associated graph-theoretic report, phonology maps to abstract and realised markings, morphology to entities and relationships instantiated as vertices or edges, and syntax to incidence rules, orderings, and trees. Semantics and pragmatics, however, do not map into canonical graph definitions, because the incidence function and standard graph primitives preserve structural information but do not encode meaning or usage context. The report therefore concludes that canonical graphs, “in their general definition,” do not allow for the expression of higher information levels of a source (Broekman et al., 2021).

A different formalization derives the spectrum from a predictive-information bottleneck on sequential processing. There, language is a function $F1 > 0.9$ 4, systematicity is expressed by the homomorphism $F1 > 0.9$ 5, and predictive information is excess entropy $F1 > 0.9$ 6. Minimizing $F1 > 0.9$ 7 over bijective mappings drives factorization of source distributions, systematic expression of approximately independent components, and locality of mutually predictive elements. The paper argues that the resulting multilevel factorization across phonology, morphology, syntax, and semantics is a sequential, discrete form of Independent Components Analysis, and reports cross-linguistic corpus evidence that human languages have low predictive information at those levels (Futrell et al., 2024).

Theoretical syntheses extend the same idea into broader maps of linguistic inquiry. One chapter defines the internal spectrum through the four axes of symbolic versus distributed representation, cognition versus computational implementation, individual cognition versus intersubjective or social coordination, and synchronic versus evolutionary dynamics. Another describes a hierarchy of scales running from individual learning and interaction, through community and population processes, to historical change on language trees. In both cases, the spectrum is not a neural-layer profile but a structured space linking multiple explanatory levels (Mansfield et al., 25 Feb 2025, Blythe, 2015).

More specialized formalizations push the concept further. The morphospace of language networks defines a design space of binary communication matrices $F1 > 0.9$ 8 relating signals to meanings, then studies its internal organization through hearer effort $F1 > 0.9$ 9, speaker effort $F1 < 0.8$ 0, effective vocabulary size $F1 < 0.8$ 1, polysemy $F1 < 0.8$ 2, synonymy $F1 < 0.8$ 3, component-size entropy $F1 < 0.8$ 4, random-walk entropies, and Zipf or power-law fits. For $F1 < 0.8$ 5, the Pareto front is the line $F1 < 0.8$ 6, and English WordNet-derived matrices lie near the one-to-one archetype unless particles are introduced, which shifts them toward more heterogeneous internal organization (Seoane et al., 2018). In a still more abstract formulation, sectoral coupling in linguistic state space defines the spectrum as the eigen-spectrum of intra-level coupling matrices $F1 < 0.8$ 7, with spectral radius, eigenmodes, and spectral entropy characterizing perceptual integration, memory, planning, meta-cognition, execution control, and affective modulation within an agent’s belief state (Dumbrava, 15 Jun 2025).

6. Operationalization, causal diagnostics, and recurring limitations

The neural literature operationalizes the spectrum with a wide range of probes. Decoding probing uses sentence-level representations $F1 < 0.8$ 8, logistic regression, cross-entropy loss, 10-fold cross-validation, averaged F1, and the feature capture depth $F1 < 0.8$ 9 to locate where a phenomenon becomes decodable (He et al., 2024). Minimal-pair similarity work instead computes hidden-state differences $L(N) \approx a_L + b_L \log N$ 0 at the last-but-two token and measures cosine similarity between those differences to build a phenomenon-level similarity matrix (Zhou et al., 2024). Spectral probing applies DCT-II along the sequence dimension, learns a sigmoid-bounded frequency filter $L(N) \approx a_L + b_L \log N$ 1, reconstructs filtered embeddings with IDCT, and treats the learned frequency weights as the task’s spectral profile (Müller-Eberstein et al., 2022). Intrinsic-dimension analysis estimates the effective dimensionality of layerwise representation manifolds with the TwoNN estimator, while sparse-autoencoder work identifies monosemantic base vectors and evaluates them with Feature Representation Confidence and Feature Intervention Confidence (Baroni et al., 7 Jan 2026, Jing et al., 27 Feb 2025).

Sparse-autoencoder results provide a fine-grained causal map. In Llama-3.1-8B, phonetics and phonology features such as sibilant, vowel, and stress appear in early layers 7–9; morphology in layers 8–12; syntax in layers 17–21; semantics in layers 22–26; and pragmatics in layers 27–31. FRC is 100.0 for sibilant, vowel, stress, past tense, inversion, genitive, causality, and progression, while metaphor is lower at 81.6, which the paper interprets as evidence that metaphors require more distributed mechanisms. Directly setting feature latents to 10 or 0 during forward propagation changes generated outputs, with politeness showing the highest reported FIC at 46.9 (Jing et al., 27 Feb 2025).

Several recurring limitations are also well established. Decoding probing on BLiMP isolates grammaticality in binary minimal pairs, but not conceptual or pragmatic content, and it reports only linear decodability because the decoder is logistic regression (He et al., 2024). Minimal-pair similarity analysis presently covers only three languages and does not systematically separate attention, MLP, or residual-stream subcomponents (Zhou et al., 2024). Speech-model probing uses a single Dutch speaker for aligned analyses, and its semantic RSA depends on FastText embeddings, which may retain residual orthographic or subword information (Kloots et al., 2 Apr 2026). Scaling studies acknowledge that the boundary between linguistic competence and factual knowledge is imperfect, especially for semantics, and that benchmarks are primarily English (Collado-Montañez et al., 2 Sep 2025). Survey work also emphasizes persistent methodological controversies around attention as explanation, probe faithfulness, control tasks, and the distinction between distributed encoding and genuine disentanglement (López-Otal et al., 9 Apr 2025).

A further caution comes from privacy-preserving pretraining. When BERT is trained on text privatized with metric differential privacy at $L(N) \approx a_L + b_L \log N$ 2, masked-LM perplexity rises from 6.82 to 61.45, layerwise RSA between baseline and privatized models falls from 0.9007 at the lexical embedding layer to 0.6784 in the deepest layers, and the largest probe deficits occur on contextual and relational tasks such as dependencies, semantic roles, entity relations, root prediction, and parse-tree UUAS. This pattern supports the view that localized lexical properties and some surface content survive perturbation better than high-layer contextual structure (Arnold et al., 2023).

Taken together, these literatures treat the internal linguistic spectrum as a structured distribution of linguistic organization rather than a single scalar property. In neural models, it is observed across layers, heads, frequencies, latents, and training checkpoints; in scaling studies, across competence and memorization; and in formal theory, across linguistic strata, communication trade-offs, abstraction levels, and dynamical couplings. The unifying theme is that linguistic structure appears internally differentiated, with different phenomena occupying different depths, timescales, representational subspaces, or functional sectors, and with those differentiations themselves becoming objects of measurement and theory.