Natively Unlearnable LLMs (NULLs)

• Natively Unlearnable LLMs (NULLs) are architectures that support precise, source-specific forgetting through dedicated sink neurons and security vectors.
• They employ deterministic masks during training to isolate unique data contributions, ensuring efficient and non-destructive unlearning at deployment.
• Empirical evaluations show that NULLs maintain overall performance while effectively erasing targeted information without the need for gradient-based retraining.

A Natively Unlearnable LLM (NULL) is an LLM architecture and training paradigm that supports precise, non-destructive, and computationally trivial unlearning of source-specific information at deployment, without the need for gradient updates or additional access to retained training data. NULLs are designed so that the contributions of distinct data sources or behaviors remain separable in the resulting model, enabling “surgical” forgetting by simple masking operations or parameter deactivation, thereby closely matching the output of hypothetical retraining that omitted the target source. This capability fundamentally differs from post-hoc unlearning, which attempts to erase already entangled knowledge via gradient-based updates and cannot achieve sharp, source-isolated forgetting without side effects.

1. Conceptual Foundations of Natively Unlearnable LLMs

NULLs address the entanglement problem inherent in conventional LLM training: the distributed, superposed nature of gradient updates means that unique and shared information from various data sources diffuses across model parameters. This property precludes clean, source-targeted removal, leading to proposals for architectures that isolate or localize knowledge. NULLs instantiate this by augmenting each layer’s feed-forward sublayer with a pool of “sink” neurons, alongside the standard backbone (“generator”). During training, sparse, per-source gating activates a fixed, deterministic subset of sinks for each source, while the backbone remains fully shared. Source-specific knowledge thus accumulates in assigned sinks, while shared knowledge aggregates in the backbone. Removal of a data source is realized by deactivating its associated sink neurons at inference. This methodology achieves both the joint learning capacity of backbone-sharing and the selective forgetting of “parameter disjointness” approaches (Ghosal et al., 11 Jun 2026).

Security vectors provide an orthogonal mechanism: a small, retrievable parameter vector is trained to “capture” an undesirable behavior, and during deployment, its nullification causes the model to behave as if the behavior had never been learned (Zhou et al., 2023). In contrast, framework-level machine unlearning can be implemented via loss-driven fine-tuning that pushes harmful outputs away and aligns responses towards benign clusters, but such approaches typically remain post-hoc unless integrated during pretraining (Chen et al., 2024).

2. Architectural Mechanisms: Sink Neurons, Security Vectors, and Loss Isolation

NULLs deform the Transformer MLP as follows. Each hidden layer is extended to $d_h = N_{\text{gen}} + N_{\text{pool}}$ neurons, partitioned into a backbone and a pool of sinks. Let $S_i$ denote the $i$-th source (e.g., article). The $N_{\text{pool}}$-dimensional binary mask $m_i$ is deterministically generated from the source ID, activating $N_{\text{source}} \ll N_{\text{pool}}$ sink neurons. Only these activated sinks receive gradient signal during training on $S_i$; backbone neurons accumulate gradients from all sources. The information flow ensures that unique information about $S_i$ is highly concentrated in its sinks, supporting removal by zeroing $m_i$. Critically, masks are generated on the fly with negligible storage and compute overhead (Ghosal et al., 11 Jun 2026).

Security vectors instead decompose the model parameters into a backbone $\theta_0$ and a trainable, source-specific vector $S_i$0 such that $S_i$1. During harmful behavior fine-tuning, $S_i$2 is frozen and only $S_i$3 is optimized on negative data; in subsequent training, $S_i$4 is held fixed and only $S_i$5 is updated. Deployment occurs with $S_i$6 set to zero. This ensures zero harmful gradient while preserving general utility (Zhou et al., 2023).

Loss-level approaches—such as those employing negative distance, cluster-alignment, and KL-preserving losses—can facilitate sharp unlearning when incorporated directly into the pretraining objective, yielding an LLM that is natively unlearnable for those behaviors or sources (Chen et al., 2024).

3. Training Objectives and Isolation of Source Contributions

NULLs are pre-trained with the standard autoregressive cross-entropy: $S_i$7 but with masked sinks so that only a designated subset corresponding to the current training source is active. There are no explicit sparsity or orthogonality regularizations beyond mask gating; the isolation arises solely from restricted sink activation (Ghosal et al., 11 Jun 2026). This sparse sink activation achieves a low expected overlap ratio $S_i$8, ensuring that source interference is minimal.

Security vectors are optimized solely on negative data with

$S_i$9

The downstream objective is

$i$0

ensuring vanishing gradients for harmful data (Zhou et al., 2023).

In loss-driven unlearning, the combined fine-tuning objective is

$i$1

where $i$2 is maximized to repel from harmful outputs, $i$3 minimizes distance to the cluster mean of benign responses, and $i$4 preserves general QA capability (Chen et al., 2024).

4. Deployment-Time Unlearning and Empirical Effectiveness

NULLs’ principal deployment procedure is masking: for a source $i$5, one sets the sink mask $i$6 in all layers, disabling all neurons that encode its unique information; the backbone and the sink neurons associated with other sources remain unchanged. No gradient computation or retraining is required (Ghosal et al., 11 Jun 2026). The effectiveness of this procedure is validated by truth ratio (TR) on fact probes (TR$i$70 for unique facts upon sink-off, TR$i$81 for backbone-shared facts), adversarial compression ratio (ACR) for extraction resistance, and functional equivalence to retrained models for removed sources.

Security vectors are nullified at deployment, so the backbone never “sees” the harmful subspace during inference (Zhou et al., 2023). Empirical evaluations reveal that training a security vector on as few as 100 negative examples suppresses harmful learning even when 1,000 negative samples are subsequently given, with harm rate (HR) and harm score (HS) restored to base (0%/∼1.0) and no loss in utility metrics such as MMLU or GSM8K.

Loss-level NULLs can achieve similar outcomes if integrated with continual or online learning, dynamically updating clustering or negative sets as new deletion requests or policies arrive (Chen et al., 2024).

5. Robustness, Limitations, and Scaling

NULLs achieve robustness to adversarial extraction, re-learning, and membership inference. For example, disabling Harry Potter book sinks renders QA accuracy and extractability indistinguishable from a retrained model, with the ACR matching the gold standard (Ghosal et al., 11 Jun 2026). Attempts to re-learn forgotten content require optimization comparable to initial pretraining, as the relevant subspace is genuinely inaccessible.

Potential weaknesses include residual memorization if backbone neurons encode unique content due to insufficient sink capacity or excessive overlap. Generalization across undesirable behaviors is immediate for security vectors (simply define a new $i$9 for each), and masking mechanisms scale to millions of sources (6M articles in Wikipedia) with negligible compute overhead (N_gen=500, N_pool=8000, N_source=100) (Ghosal et al., 11 Jun 2026). However, downstream fine-tuning that unfreezes sink or security dimensions may reintroduce harmful learning, requiring parameter freezing discipline (Zhou et al., 2023).

NULLs’ design leads to downstream benchmark parity with standard Transformers: ARC-Easy, Winogrande, PIQA, and SciQ metrics match within error, indicating that general language capabilities are preserved (Ghosal et al., 11 Jun 2026).

6. Extensions and Future Directions

NULLs support extensibility to any granular source definition, including semantic clusters, privacy partitions, or safety-relevant behaviors. Loss-driven unlearning can be synergistically integrated into pretraining for online or curriculum-based native unlearning, as new privacy or harm definitions arise (Chen et al., 2024). Theoretical questions remain about minimal sink or security vector dimensionality per class of behaviors, optimal mask overlap for scalability, and integration with certified unlearning or differential privacy. Combined approaches—mask-based NULLs, security vector injection, and loss-level unlearning—can be composed to build defense-in-depth safety stacks.

NULLs formalize a property for LLMs that traditionally has been relegated to post-hoc adjustment: reversibility at the informational level, enabling operational, legal, and societal compliance via source-level selective forgetting with negligible impact on retained capabilities. This opens new directions for both foundation model design and regulation-compliant deployment of generative AI systems (Ghosal et al., 11 Jun 2026, Zhou et al., 2023, Chen et al., 2024, Yao et al., 2023).