Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs

Abstract: Open-weight LLMs enable scientific progress and broad deployment. However, they make it difficult to control access to sensitive capabilities. Current practice either suppresses dangerous capabilities before release or mediates access through closed services that use specialized model variants, input/output monitors, and API permissions. The former is susceptible to jailbreaks while sacrificing capability for all users to mitigate the risks posed by a few, and the latter is fundamentally incompatible with open-weight release. In this paper, we propose Tiered LLMs (TLMs), where a single set of released weights supports multiple capability levels. In its default public configuration, a TLM behaves as a conventional LLM. A compact secret key specifies a permutation over a small parameter subset, inducing an alternative computation graph over the same weights that exposes additional capabilities. We develop a training protocol that jointly pretrains both configurations from scratch, then fine-tunes the keyed configuration on private data with regularization to preserve the public model's behavior. We pretrain 180M- and 650M-parameter TLMs and demonstrate that the keyed configuration can acquire a new language, gain instruction-following ability, and memorize private factual knowledge, whereas the public configuration exhibits none of these capabilities. Moreover, we show that our approach extends naturally to multiple hierarchical tiers. Because authorization operates on the model's weight structure rather than in the input space, the mechanism resists fine-tuning-based extraction and partial key compromise. In general, TLMs take a step toward reconciling open-weight release with selective capability control.

• The paper introduces Tiered Language Models (TLMs) that use a compact permutation key to enable differentiated public and private model behaviors.
• The method employs asymmetric joint pretraining and targeted private fine-tuning to achieve distinct behavioral separation in language transfer, instruction following, and factual memorization.
• Empirical results demonstrate minimal computational overhead, significant storage efficiency, and robust protection against adversarial key leakage.

Tiered LLMs: Parameter-Level Selective Access for Open-Weight LLMs

Introduction and Motivation

Open-weight LLMs facilitate reproducibility, democratize access, and accelerate scientific progress. However, their indiscriminate capability deployment complicates risk-sensitive use cases—either model maintainers must suppress all unsafe behaviors for everyone or limit access to privileged variants through proprietary APIs. Prompt-based password-locking and modular adapters have inherent weaknesses: keys in the input space are susceptible to imitation or fine-tuning, and adapter-based methods demand storage and deployment complexity that scales poorly with the number of secrets. This work introduces Tiered LLMs (TLMs), formalizing access control as a parameter permutation problem. Here, a single released set of weights supports both public and private behaviors, the latter unleashed only when a compact permutation key is provided. The key specifies a blockwise parameter permutation on a small, swappable subset of parameters, resulting in an alternate computation graph specific to authorized users.

Figure 1: Overview of the TLM approach. Released weights serve as a public LLM (default graph); authorized users apply a secret permutation key to expose private model capabilities. Training comprises public and keyed-path joint pretraining, then private fine-tuning through the keyed path.

TLM Formalism and Training Protocol

Let $\mathcal{M}$ be a decoder-only transformer with parameters $\theta$. A configuration function $\mathcal{C}: \Theta\rightarrow\Theta$ reorders selected parameter blocks in $\theta$, yielding either the public ($\mathcal{C}_{\rm pub}$: identity) or keyed ($\mathcal{C}_K$) model. The key $K$ specifies permutations of a small parameter subset $S\subset\theta$, typically comprising select attention heads and MLP neurons.

The TLM training protocol has two phases:

• Asymmetric Joint Pretraining: Both public and keyed configurations are trained simultaneously on public data. Gradients for $S$ are computed via the keyed configuration only; $\overline{S}$ parameters receive gradients from both paths. This drives the public path to avoid reliance on $\theta$0, ensuring robustness to later keyed fine-tuning.
• Private Fine-Tuning: Only $\theta$1 is updated through the keyed path on private data, with KL-regularization anchoring the public configuration's outputs on public data, constraining drift.

The key definition and update routing are compatible with multiple (hierarchical) keys, supporting granular, multi-tiered access control.

Empirical Evaluation

Behavioral Separation

TLMs are validated on three axes: language acquisition, instruction following, and factual memorization.

• Language Transfer: TLM-650M is fine-tuned on Spanish (private) data using the key; only the keyed path adapts, with the public path remaining English-centric and unchanged, demonstrating clean separation.
• Instruction Following: The keyed configuration, fine-tuned on Alpaca instructions, achieves >85% win rate (AlpacaEval), while the public configuration remains non-responsive.
• Fact Memorization: On a synthetic biography memorization task, the keyed path achieves perfect recall; the public path never exceeds random-guess accuracy.

Figure 2: Behavioral separation curves—keyed configuration learns Spanish/private tasks or instruction-following; public path remains unchanged or random.

Figure 3: Memorization task: only the keyed configuration achieves exact-match factual recall; public configuration is unaffected.

Computational and Storage Efficiency

Keyed pretraining overhead can be throttled by sparse keyed-path updates, causing a minimal public-path convergence slowdown (e.g., 6% more steps than non-tiered pretraining to match validation loss). Storage cost analysis shows that a full information-theoretic encoding of a permutation key is $\theta$2–$\theta$3 smaller than bf16 LoRA adapters targeting equivalent task performance, even at billion-parameter scale.

Figure 4: Left—Storage cost comparison: permutation key size vs. LoRA adapters across scales. Right—Public-domain pretraining efficiency: marginal loss increase over non-tiered baseline.

Adversarial Robustness

Mechanistic access control is only meaningful if security is robust to model release:

• Fine-Tuning Attacks: Adversaries retraining the public configuration on partial private data cannot induce leakage of hidden knowledge from the keyed path. No factual memorization transfer occurs—contrasting with prompt-locked methods, which are vulnerable to this attack surface.
• Partial Key Disclosure: Progressive partial application (up to 90%) of the key is insufficient to elicit the private capability, evidencing cryptographic-like fail-closed security.
• White-Box Inspection: The parameter subset $\theta$4 can be statistically identified at weak F1 via block-norm inspection, but permuting $\theta$5 correctly is combinatorially hard. Even near-complete recovery of the key (90% correct) fails to unlock private behaviors, demonstrating practical robustness.

Figure 5: Left—Fine-tuning extraction attack: zero leakage to unseen facts even after extensive adversarial fine-tuning. Right—Partial key attack: sharp threshold; only near-complete key knowledge yields factual recall.

Multi-Tier Extension

TLMs generalize to a hierarchy of $\theta$6 access-control tiers. Each tier operates on a disjoint parameter chunk; authorized users compose permutations cumulatively. Pretraining and fine-tuning are extended to maintain lower-tier properties while introducing new capabilities at each higher tier.

Figure 6: Multi-tier fine-tuning: each configuration cleanly acquires its target capability, and capabilities remain robust to later stages.

Practical and Theoretical Implications

• Deployment: A single checkpoint supports open research, commercial baselining, and granular, auditable privilege management. The permutation key distribution is trivial (kilobyte scale). Keyed adaptation is memory-light and introduces negligible inference latency; further implementation optimization is feasible.
• Model Security and Scientific Access: As secret knowledge is parameterized structurally, not through surface prompts, attacks that bypass prompt-based gating (e.g., toxic demonstration/fine-tuning, RL-driven jailbreaking, gradient leakage) are hampered. Leakage is abrupt, not gradual with increasing key exposure—approximating cryptographic fail-closed properties.
• Scalability: Theoretical efficiency and empirical non-interference scale to model sizes where adapter or API-based approaches become infeasible, supporting fine-grained control at web-scale release.

Limitations and Future Directions

The current evaluations are at 180M–650M parameter scale; cross-tier interference, fingerprinting, and adaptive search attacks must be re-validated at 10B+ scale. Weight-norm fingerprints in $\theta$7 demand mitigation, e.g., by regularization for indistinguishable parameter statistics. While permutation search is currently infeasible, cryptographic proofs or obfuscation mechanisms could further strengthen the access control and model security. Extension to more diverse tasks and alignment domains is warranted.

Conclusion

TLMs provide a practical mechanism for integrating robust, fine-grained access control into LLMs released with open weights, reconciling open scientific use and selective capability gating. Capabilities and secrets are encapsulated as alternate wiring of existing weights rather than in user prompts or withheld parameters, resulting in favorable scaling, storage, and robustness properties. This approach thus redefines the architecture of open-weight LLMs for settings with controlled-release requirements and sensitive domain knowledge (2606.21638).