Locas-GLU: Architecture-Matched Memory Module

• Locas-GLU is a sideway parametric memory module that adds a parallel GLU branch to standard feed-forward networks, enabling efficient long-context storage.
• It employs activation-guided cloning to initialize the new branch from pretrained model parameters, ensuring fast convergence and reduced catastrophic forgetting.
• Empirical results show that Locas-GLU achieves competitive perplexity and enhanced parameter efficiency compared to alternative methods like TempLoRA in language modeling and QA tasks.

Locas-GLU is a sideway parametric memory module introduced as the practical, architecture-matched variant of Locas, a “Locally-Supported” memory mechanism designed to bridge test-time training with a parametric memory that can be offloaded from or merged into model parameters (Lu et al., 4 Feb 2026). It shares the gated linear unit feed-forward structure used in modern LLMs, can be attached to an existing transformer without modifying backbone weights, and is intended to store long-context information in parameters rather than exclusively in the prompt. In the formulation of the paper, Locas-GLU treats the transformer feed-forward network as a persistent key-value memory and extends it with a parallel GLU-shaped branch whose initialization is derived in a principled way from the pretrained model’s own parameters and activations (Lu et al., 4 Feb 2026).

1. Conceptual basis

The paper interprets a standard feed-forward network as a memory system. For an FFN written as

$$\mathrm{FFN}(\mathcal{A}_t^i)=V^\top \phi(K^\top \mathcal{A}_t^i),$$

the columns of $K$ and the rows of $V$ are treated as key-value memory slots (Lu et al., 4 Feb 2026). In this view, each intermediate neuron corresponds to a slot whose key determines when it fires and whose value determines what information is retrieved.

Locas extends this interpretation by adding a new FFN-like pathway that can be grown during test time and then updated by gradient descent. The stated aim is to store long-context information “into parameters” while preserving the pretrained backbone as a stable baseline (Lu et al., 4 Feb 2026). This design is presented as a way to support efficient continual learning and to reduce reliance on large context windows.

A central point of the framework is that initialization is not incidental. The paper argues that proper initialization of low-rank sideway-FFN-style memories—performed by reusing model parameters, activations, and/or gradients—is essential for fast convergence, improved generalization, and catastrophic forgetting prevention (Lu et al., 4 Feb 2026). This emphasis on principled initialization distinguishes Locas from memory additions that are randomly initialized and then optimized in isolation.

2. Architectural form of Locas-GLU

Locas-GLU is the GLU-shaped version of Locas and is intended to match the native FFN design of contemporary LLMs such as LLaMA, Qwen, and Mistral (Lu et al., 4 Feb 2026). Its operator is

$$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$

where $G \in \mathbb{R}^{d \times r}$ is the gate matrix, $K \in \mathbb{R}^{d \times r}$ is the up-projection or key matrix, $V \in \mathbb{R}^{r \times d}$ is the down-projection or value matrix, $r$ is the latent width, $\sigma$ is SiLU, and $\odot$ denotes elementwise multiplication (Lu et al., 4 Feb 2026).

The module is added in parallel to the backbone FFN rather than replacing it. The hidden output is written as

$K$0

so the pretrained path remains intact and the new memory branch contributes additively with scale $K$1 (Lu et al., 4 Feb 2026). This makes the mechanism a genuine capacity expansion rather than a low-rank perturbation of existing weights.

Two negative clarifications are important. First, the paper explicitly characterizes Locas-GLU as neither an external memory nor a low-rank perturbation of existing parameters (Lu et al., 4 Feb 2026). Second, because the new branch is parallel to the backbone, the backbone remains untouched at attachment time. This suggests a modular interpretation: Locas-GLU augments the model’s memory capacity without requiring architectural surgery on the original transformer.

3. Initialization by activation-guided cloning

The most technically distinctive component of Locas-GLU is its initialization scheme. For a memorization chunk $K$2, the pretrained backbone is run forward and the intermediate GLU activation at layer $K$3 is computed as

$K$4

The importance of each intermediate dimension $K$5 is then averaged over the chunk: $K$6 The top-$K$7 most activated dimensions are selected,

$K$8

and the Locas-GLU parameters are initialized by cloning the corresponding backbone rows: $K$9 with

$V$0

All of these steps are given explicitly in the paper (Lu et al., 4 Feb 2026).

The zero initialization of $V$1 has a specific operational consequence: immediately after creation, the Locas-GLU branch contributes nothing to the model output, so the model’s behavior is unchanged at initialization (Lu et al., 4 Feb 2026). The paper compares this safe start to the logic of LoRA initialization. Only subsequent updates cause the memory branch to affect predictions.

The paper interprets this top-$V$2 selection as a kind of nonlinear PCA in activation space (Lu et al., 4 Feb 2026). That interpretation is explicitly framed in the source as an interpretation rather than a formal theorem. A plausible implication is that the initialization preferentially aligns the new memory with context-relevant backbone directions instead of forcing optimization to discover them from scratch.

4. Optimization, control, and permanentization

After attachment, the Locas branch is updated during memorization with standard backpropagation on the language-modeling objective as new tokens arrive (Lu et al., 4 Feb 2026). The backbone parameters stay fixed, while the side branch absorbs context-specific information. The paper describes this as storing past context into parametric knowledge so that later inference can proceed with a much smaller context window.

The framework includes explicit control mechanisms. One is weight norm clipping,

$V$3

applied rowwise or columnwise to the key, gate, and value matrices (Lu et al., 4 Feb 2026). The other is the output scale $V$4, which is set from the backbone FFN’s down-projection norm statistics and normalized by $V$5 (Lu et al., 4 Feb 2026). These safeguards are described as ways to bound per-step behavioral change and to reduce the risk of destabilizing the pretrained model.

The paper also frames Locas as a memory that can be expanded during streaming context and later compressed or merged into model parameters (Lu et al., 4 Feb 2026). In this sense, “permanentization” denotes a shift from prompt-resident information to parameter-resident information. In the reported evaluations, this is reflected by using only a $V$6 context truncation for test-time-training baselines while relying on the memory branch to preserve information from much longer spans (Lu et al., 4 Feb 2026).

A recurring theme is that Locas-GLU is practical because it mirrors the GLU-FFN decomposition already present in many modern models. The paper states that this makes the module easy to attach “beside” existing FFNs and suitable for both parameter-efficient and computation-efficient continual learning (Lu et al., 4 Feb 2026).

5. Empirical evaluation

The paper validates Locas-GLU on PG-19 whole-book language modeling and LoCoMo long-context dialogue question answering, and further measures general capability retention with MMLU after memorizing an entire book (Lu et al., 4 Feb 2026). The abstract reports that, with only $V$7 additional parameters in the lowest case, Locas-GLU is capable of storing information from past context while maintaining a much smaller context window (Lu et al., 4 Feb 2026).

On PG-19, Locas-GLU is compared against context truncation, long-context attention, and TempLoRA. For Qwen3-0.6B-Base, TempLoRA uses $V$8 extra parameters and $V$9 relative time, whereas Locas-GLU uses $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$0 extra parameters and $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$1 relative time (Lu et al., 4 Feb 2026). At $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$2 context, the reported perplexities are $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$3 for truncation, $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$4 for TempLoRA, and $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$5 for Locas-GLU (Lu et al., 4 Feb 2026). For Qwen3-1.7B-Base, TempLoRA uses $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$6 parameters while Locas-GLU uses $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$7, and at $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$8 context the perplexities are $$\text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big),$$9, $G \in \mathbb{R}^{d \times r}$0, and $G \in \mathbb{R}^{d \times r}$1, respectively (Lu et al., 4 Feb 2026). The paper summarizes these results by stating that Locas-GLU reaches comparable or better performance than TempLoRA while using roughly $G \in \mathbb{R}^{d \times r}$2 of the extra parameters and about $G \in \mathbb{R}^{d \times r}$3 of the compute (Lu et al., 4 Feb 2026).

The width study reports the scaling $G \in \mathbb{R}^{d \times r}$4 for the $G \in \mathbb{R}^{d \times r}$5, $G \in \mathbb{R}^{d \times r}$6, and $G \in \mathbb{R}^{d \times r}$7 matrices (Lu et al., 4 Feb 2026). Empirically, $G \in \mathbb{R}^{d \times r}$8 corresponds to $G \in \mathbb{R}^{d \times r}$9 parameters, $K \in \mathbb{R}^{d \times r}$0 to $K \in \mathbb{R}^{d \times r}$1, $K \in \mathbb{R}^{d \times r}$2 to $K \in \mathbb{R}^{d \times r}$3, and $K \in \mathbb{R}^{d \times r}$4 to $K \in \mathbb{R}^{d \times r}$5 (Lu et al., 4 Feb 2026). The paper states that performance saturates early and that even $K \in \mathbb{R}^{d \times r}$6 is competitive with much larger TempLoRA ranks (Lu et al., 4 Feb 2026).

On LoCoMo, the reported gains are framed as evidence that dialogue facts have been internalized into parametric memory rather than merely reused from prompt access. For Qwen3-1.7B-Base with full context, Locas-GLU obtains $K \in \mathbb{R}^{d \times r}$7 on Single-hop, $K \in \mathbb{R}^{d \times r}$8 on Multi-hop, $K \in \mathbb{R}^{d \times r}$9 on Temporal, and $V \in \mathbb{R}^{r \times d}$0 on Adversarial, compared with $V \in \mathbb{R}^{r \times d}$1, $V \in \mathbb{R}^{r \times d}$2, $V \in \mathbb{R}^{r \times d}$3, and $V \in \mathbb{R}^{r \times d}$4 for full attention, and $V \in \mathbb{R}^{r \times d}$5, $V \in \mathbb{R}^{r \times d}$6, $V \in \mathbb{R}^{r \times d}$7, and $V \in \mathbb{R}^{r \times d}$8 for TempLoRA (Lu et al., 4 Feb 2026). In the no-context evaluation, Locas-GLU retains more information in parameters than TempLoRA, with $V \in \mathbb{R}^{r \times d}$9 versus $r$0 on Single-hop, $r$1 versus $r$2 on Multi-hop, and $r$3 versus $r$4 on Temporal (Lu et al., 4 Feb 2026).

The MMLU comparison is used to measure catastrophic forgetting. For Qwen3-1.7B-Base, the baseline MMLU score is $r$5, TempLoRA yields $r$6 $r$7, and Locas-GLU yields $r$8 $r$9 (Lu et al., 4 Feb 2026). At larger memory width $\sigma$0, TempLoRA gives $\sigma$1 $\sigma$2, whereas Locas-GLU gives $\sigma$3 $\sigma$4 (Lu et al., 4 Feb 2026). The paper interprets this as evidence that Locas-GLU exhibits much less catastrophic forgetting because it preserves the backbone and adds a parallel additive pathway rather than directly altering existing parameters.

6. Relation to Locas-MLP and scope of the method

The paper introduces two variants of Locas. The first, Locas-MLP,

$\sigma$5

is presented as the theoretically cleaner version, with a simpler two-layer MLP form and a step-wise optimal key/value initialization under assumptions (Lu et al., 4 Feb 2026). It also supports an explicit compression procedure, NL-SVD. The second, Locas-GLU, is presented as the practical version that matches the native FFN structure of modern LLMs (Lu et al., 4 Feb 2026).

This distinction matters because Locas-GLU is not merely a cosmetic rewrite of Locas-MLP. The paper explicitly states that MLP-style piecewise-linear separability at the FFN input is often poor for GLU-based backbones, whereas Locas-GLU reuses the gate/up/down decomposition already present in the pretrained model (Lu et al., 4 Feb 2026). This suggests that Locas-GLU is intended less as a minimal theoretical construction than as an architecture-compatible memory extension for current transformer families.

A common misconception would be to treat Locas-GLU as a variant of LoRA or as a standard retrieval memory. The paper rejects both descriptions. Locas-GLU is not a low-rank perturbation of existing weights and not an external memory; it is a parallel FFN-style module whose parameters can be updated during streaming context and whose content can later be permanentized into model parameters (Lu et al., 4 Feb 2026). Another misconception would be to treat it as a replacement for long-context attention. The reported experiments instead position it as a complementary mechanism: it stores information from past context while allowing the operational context window to remain much smaller (Lu et al., 4 Feb 2026).

Within the paper’s framing, the significance of Locas-GLU lies in three linked claims: modern FFNs can be interpreted as persistent key-value memories, architecture-matched sideway memories can be initialized from pretrained model structure rather than from scratch, and such memories can absorb long-context information with limited catastrophic forgetting and modest parameter overhead (Lu et al., 4 Feb 2026).