Tiered Language Models (TLMs)

• Tiered Language Models are models in which a secret key rearranges select parameters to create distinct public and private computational tiers.
• They employ a two-stage training protocol that combines joint pretraining with private fine-tuning to maintain public performance while specializing private capabilities.
• Empirical results show effective separation in tasks such as language translation, instruction following, and factual recall, while resisting extraction attacks.

Searching arXiv for Tiered LLMs and closely related formulations. Search results unavailable via tool in this interface. Proceeding with the provided arXiv records as the evidentiary basis, which include the directly relevant Tiered LLMs paper (Feghali et al., 19 Jun 2026) and neighboring uses of the acronym TLM. Tiered LLMs (TLMs) are LLMs in which a single set of released weights supports multiple capability levels. In the formulation introduced for open-weight LLMs, the default public configuration behaves as a conventional model, while one or more private tiers are activated by a compact secret key that permutes a small parameter subset and thereby induces an alternative computation graph over the same weights. This places authorization in weight structure rather than in prompts, service-side routing, or separately distributed adapters, and was proposed as a mechanism for separating public and private capabilities under open-weight release (Feghali et al., 19 Jun 2026).

1. Definition and problem setting

The central problem addressed by Tiered LLMs is the tension between open-weight release and selective capability control. Conventional open-weight LLMs expose whatever capabilities are present in the checkpoint to any party that downloads it. Closed APIs, by contrast, can mediate access through service-layer controls, specialized model variants, monitors, and permissions, but that solution is incompatible with fully open-weight distribution. TLMs were proposed as an intermediate design: a single released checkpoint, but multiple capability tiers, with access determined by a secret structural key rather than by prompt tokens or API policy (Feghali et al., 19 Jun 2026).

In the two-tier case, the public model is the identity configuration and the private model is a keyed configuration. Both share the same parameter values and the same architecture. What differs is the arrangement of a small designated subset of parameters. The intended effect is that the public tier remains a usable conventional LLM, while the keyed tier exposes additional capabilities such as a new language, instruction-following, or memorized private facts.

The threat model is unusually strong. The design targets adversaries with full white-box access to the public checkpoint, possible access to partial private data, possible partial information about the key, and the ability to perform arbitrary fine-tuning on the public configuration, but without the full key. The security goal is therefore not mere prompt robustness; it is resistance to extraction when the attacker controls the released weights but not the structural authorization mechanism (Feghali et al., 19 Jun 2026).

2. Keyed configurations and the mechanics of tiers

Formally, the framework fixes a transformer architecture $\mathcal{M}$ and a structured parameter collection $\theta \in \Theta$. A configuration is a map $\mathcal{C} : \Theta \to \Theta$ that rearranges selected parameter positions without changing their numerical values. The public configuration is the identity,

$\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$

so the public model is simply $\mathcal{M}_\theta$. A private tier is induced by a secret key $K$ that specifies a permutation on a sparse parameter subset $S \subset \theta$, yielding $\mathcal{M}_{\mathcal{C}_K(\theta)}$ (Feghali et al., 19 Jun 2026).

The subset $S$ is defined over coherent functional units rather than arbitrary scalar weights. In the reported implementation, it comprises attention-head groups and FFN units. For an attention-head swap, the corresponding rows of the $Q$, $\theta \in \Theta$0, and $\theta \in \Theta$1 projections are exchanged, along with the matching columns of the output projection. For an FFN swap, the corresponding row and bias of the up projection and the corresponding column of the down projection are exchanged. The key therefore consists of index pairs that specify which units to swap across layers, not an auxiliary tensor of learned weights.

A notable design choice is that only about $\theta \in \Theta$2 of total parameters belong to the keyed subset in the reported experiments. This is small enough to keep the key compact, but large enough to support distinct keyed behavior after joint training. The authors also compare key size to LoRA-style alternatives and argue that the key is dramatically smaller than a separately distributed adapter (Feghali et al., 19 Jun 2026).

The construction extends naturally to more than two tiers. If disjoint subsets $\theta \in \Theta$3 are assigned to keys $\theta \in \Theta$4, then cumulative configurations

$\theta \in \Theta$5

define a hierarchy in which higher tiers inherit the capabilities of lower ones. Because the subsets are disjoint, the order of application is immaterial. This yields a genuinely hierarchical model family within one checkpoint rather than a mere public/private split (Feghali et al., 19 Jun 2026).

3. Joint pretraining and private specialization

A post-hoc permutation of an already trained model is not sufficient. The paper reports that randomly swapping even $\theta \in \Theta$6 of Qwen-3-8B head and MLP units in a TLM-like fashion reduces MMLU accuracy from $\theta \in \Theta$7 to $\theta \in \Theta$8, which motivates joint training of the public and keyed configurations from scratch (Feghali et al., 19 Jun 2026).

The training protocol has two stages. The first is asymmetric joint pretraining on public data. Let $\theta \in \Theta$9 denote the tier parameters and $\mathcal{C} : \Theta \to \Theta$0 the complement. During pretraining, gradients on $\mathcal{C} : \Theta \to \Theta$1 come only from the keyed configuration,

$\mathcal{C} : \Theta \to \Theta$2

whereas gradients on $\mathcal{C} : \Theta \to \Theta$3 mix public and keyed losses,

$\mathcal{C} : \Theta \to \Theta$4

In the reported experiments, $\mathcal{C} : \Theta \to \Theta$5 (Feghali et al., 19 Jun 2026).

This asymmetry forces the public model to depend mainly on $\mathcal{C} : \Theta \to \Theta$6, while the keyed configuration can exploit both $\mathcal{C} : \Theta \to \Theta$7 and $\mathcal{C} : \Theta \to \Theta$8. The result is a shared backbone with a parameter subset effectively reserved for private specialization.

The second stage is private fine-tuning. Only $\mathcal{C} : \Theta \to \Theta$9 is updated, and updates are applied through the keyed configuration on a private dataset $\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$0:

$\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$1

To preserve the public model’s behavior, a KL regularizer anchors the public configuration to its pretrained output distribution on public-domain inputs,

$\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$2

The fine-tuning objective is

$\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$3

with an additional $\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$4 penalty on $\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$5 in practice (Feghali et al., 19 Jun 2026).

For multi-tier models, fine-tuning proceeds sequentially from lower to higher private tiers, with both public anchors and tier-specific anchors so that later private specialization does not destroy earlier tier capabilities.

4. Empirical demonstrations of capability separation

The reported experiments use GPT-Neo–style decoder-only transformers at $\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$6 and $\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$7 parameters, pretrained on English FineWeb with approximately $\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$8 tokens per parameter. Relative to a non-tiered baseline, the public configuration converges somewhat more slowly, requiring about $\mathcal{C}_{\mathrm{pub}} = \mathrm{id},$9 more steps to reach the same validation loss, but final public-domain loss remains comparable. The keyed configuration also remains competent on public data after pretraining, though its loss is slightly higher than the public tier’s (Feghali et al., 19 Jun 2026).

Three capability-separation demonstrations are central.

Private capability	Setup	Separation result
New language	TLM-650M, 4B Spanish tokens	Keyed Spanish loss decreases; public shows no Spanish improvement
Instruction following	TLM-650M, Stanford Alpaca	Keyed AlpacaEval win rate rises beyond 85%; public remains around 15%
Synthetic factual memory	TLM-180M, 400-person biographies	Keyed reaches perfect exact match; public remains at zero exact match

In the language experiment, the keyed configuration learns Spanish while maintaining English performance, whereas the public configuration shows no improvement on Spanish validation loss. The same pattern is reported for Portuguese, German, and Turkish. In the instruction-tuning experiment, the keyed tier surpasses $\mathcal{M}_\theta$0 win rate on AlpacaEval within about $\mathcal{M}_\theta$1 fine-tuning steps, while the public tier behaves like an untuned base LM. In the synthetic factual memorization experiment, the keyed configuration reaches perfect exact-match accuracy over all $\mathcal{M}_\theta$2 biographies, while the public configuration stays at zero exact-match accuracy throughout fine-tuning (Feghali et al., 19 Jun 2026).

These results establish that the same checkpoint can support mutually different behaviors under different configurations. A plausible implication is that tiering can encode not just restricted knowledge but qualitatively distinct post-pretraining specializations.

5. Security analysis, attack surfaces, and limitations

The security argument for TLMs is empirical rather than cryptographic. Three attack classes are examined. The first is fine-tuning-based extraction. An attacker with the public weights and $\mathcal{M}_\theta$3 of the synthetic biography facts fine-tunes the public configuration. All tested starting points—including a TLM whose keyed tier had already memorized all $\mathcal{M}_\theta$4 biographies—quickly memorize the attacker’s train split but remain at about zero exact match on the held-out facts, indicating that public fine-tuning does not extract the private tier’s memorized knowledge (Feghali et al., 19 Jun 2026).

The second attack is partial key compromise. When only a random fraction of key swaps is known and applied, both token-level and exact-match accuracy remain near zero for all fractions up to $\mathcal{M}_\theta$5. Accuracy rises sharply only once more than about $\mathcal{M}_\theta$6 of the swaps are correct. The reported transition is abrupt rather than gradual, suggesting that partial knowledge of the permutation is largely useless until it is nearly complete (Feghali et al., 19 Jun 2026).

The third attack is weight-statistics analysis. After private fine-tuning, tier parameters can exhibit detectable magnitude signatures, especially in MLP units. Under a highly favorable attacker assumption—known key size and known unit grouping structure—simple norm-based ranking partially identifies tier parameters, with best F1 around $\mathcal{M}_\theta$7–$\mathcal{M}_\theta$8 in two-tier runs and around $\mathcal{M}_\theta$9 in a multi-tier run. This weakens obscurity but does not recover the permutation itself, and the partial-key results indicate that recovering most of the subset without the actual pairing still does not unlock the private behavior (Feghali et al., 19 Jun 2026).

The framework also has explicit limitations. Experiments were conducted at $K$0 and $K$1 rather than frontier scale. The attack suite does not include sophisticated permutation search, activation-level reconstruction, or stronger side-channel analysis. Tier parameters can leave detectable fingerprints in weight statistics. Key management, revocation, and rotation are external deployment problems rather than intrinsic parts of the method. The authors therefore present TLMs as raising the bar for extraction rather than providing formal guarantees (Feghali et al., 19 Jun 2026).

6. Terminological ambiguity and adjacent architectures

The acronym TLM is highly overloaded in recent literature. In multiple papers it refers to Transformer-based LLMs, not Tiered LLMs (Feldman et al., 2021, Gontier et al., 2020, Madusanka et al., 23 Aug 2025, Nighojkar et al., 2022, Zaczynska et al., 2020). In GeoTLM it refers to tactile-LLMs, specifically models connecting tactile perception to language and higher-level reasoning (Li et al., 14 Jun 2026). In a distinct architectural proposal, Tapered LLMs also abbreviate to TLMs, but there the meaning is depth-aware capacity allocation: MLP width is made larger in early layers and smaller in later layers under a fixed total parameter and FLOP budget (Bayat et al., 22 Jun 2026).

This terminological ambiguity matters because the access-control sense of Tiered LLMs is much narrower than a generic multi-level or hierarchical interpretation. Several papers do motivate broader tiered readings, but as interpretations rather than formal equivalences. Latent Thought Models, for example, introduce a latent thought tier optimized by variational Bayes and an autoregressive token tier, yielding an explicit two-level generative structure (Kong et al., 3 Feb 2025). GeoTLM describes a two-tier arrangement in which low-level geometric reasoning precedes downstream reasoning over a fused representation (Li et al., 14 Jun 2026). The semantic fluency work frames pretrained transformer models, prompts, and adaptive hyperparameter selection as a multi-level cognitive-modeling methodology rather than as a new tiered LM architecture (Nighojkar et al., 2022).

A common misconception is therefore to treat every occurrence of “TLM” as if it denoted the same model family. In current usage, it does not. “Tiered LLMs” in the strict sense denotes the keyed, multi-capability, single-checkpoint framework for public and private access tiers (Feghali et al., 19 Jun 2026).

7. Significance and open directions

Tiered LLMs expand the design space between fully open and fully closed model release. In the proposed deployment picture, a single checkpoint can be openly distributed for general-purpose use, while higher-capability tiers remain available only to parties holding the relevant key. The paper explicitly connects this to licensed or private data, safety-sensitive capabilities, and privacy-sensitive self-hosting, and suggests that keys could be integrated with existing access-control mechanisms such as legal agreements, authentication systems, audit trails, or hardware security modules (Feghali et al., 19 Jun 2026).

From a research standpoint, the framework makes capability partitioning an architectural property rather than a prompt-space behavior. That distinguishes it from password-locked prompting, refusal layers, or separately hosted model variants. A plausible implication is that future work on open-weight governance may treat capability control as a question of parameter-space organization, not only of safety tuning or inference-time filtering.

Open questions remain substantial. Larger-scale evaluations are needed. More adaptive attacks must be considered. Regularization that suppresses weight-space fingerprints is still undeveloped. Key management remains external to the model. And the relationship between access-tiered models and other kinds of tiering—such as latent-thought inference loops or depth-wise capacity tapering—remains conceptual rather than unified. Even so, Tiered LLMs in the strict sense establish a concrete technical proposal: multiple capability levels, one checkpoint, one architecture, and authorization encoded in the structure of the weights themselves (Feghali et al., 19 Jun 2026).