--- title: 'Locas-GLU: Architecture-Matched Memory Module' url: https://www.emergentmind.com/topics/locas-glu type: topic --- # Locas-GLU: Architecture-Matched Memory Module Locas-GLU is a sideway parametric memory module introduced as the practical, architecture-matched variant of Locas, a “Locally-Supported” memory mechanism designed to bridge test-time training with a parametric memory that can be offloaded from or merged into model parameters [2602.05085]. It shares the gated linear unit feed-forward structure used in modern large language models, can be attached to an existing transformer without modifying backbone weights, and is intended to store long-context information in parameters rather than exclusively in the prompt. In the formulation of the paper, Locas-GLU treats the transformer feed-forward network as a persistent key-value memory and extends it with a parallel GLU-shaped branch whose initialization is derived in a principled way from the pretrained model’s own parameters and activations [2602.05085]. ## 1. Conceptual basis The paper interprets a standard feed-forward network as a memory system. For an FFN written as \[ \mathrm{FFN}(\mathcal{A}_t^i)=V^\top \phi(K^\top \mathcal{A}_t^i), \] the columns of \(K\) and the rows of \(V\) are treated as key-value memory slots [2602.05085]. In this view, each intermediate neuron corresponds to a slot whose key determines when it fires and whose value determines what information is retrieved. Locas extends this interpretation by adding a new FFN-like pathway that can be grown during test time and then updated by gradient descent. The stated aim is to store long-context information “into parameters” while preserving the pretrained backbone as a stable baseline [2602.05085]. This design is presented as a way to support efficient continual learning and to reduce reliance on large context windows. A central point of the framework is that initialization is not incidental. The paper argues that proper initialization of low-rank sideway-FFN-style memories—performed by reusing model parameters, activations, and/or gradients—is essential for fast convergence, improved generalization, and catastrophic forgetting prevention [2602.05085]. This emphasis on principled initialization distinguishes Locas from memory additions that are randomly initialized and then optimized in isolation. ## 2. Architectural form of Locas-GLU Locas-GLU is the GLU-shaped version of Locas and is intended to match the native FFN design of contemporary LLMs such as LLaMA, Qwen, and Mistral [2602.05085]. Its operator is \[ \text{Locas-GLU}(\mathcal{A}) = V^\top \big(\sigma(G^\top \mathcal{A}) \odot K^\top \mathcal{A}\big), \] where \(G \in \mathbb{R}^{d \times r}\) is the gate matrix, \(K \in \mathbb{R}^{d \times r}\) is the up-projection or key matrix, \(V \in \mathbb{R}^{r \times d}\) is the down-projection or value matrix, \(r\) is the latent width, \(\sigma\) is SiLU, and \(\odot\) denotes elementwise multiplication [2602.05085]. The module is added in parallel to the backbone FFN rather than replacing it. The hidden output is written as \[ \mathcal{H}_{\text{out}}=\mathrm{FFN}(\mathcal{A})+\tau\cdot \text{Locas-GLU}(\mathcal{A}), \] so the pretrained path remains intact and the new memory branch contributes additively with scale \(\tau\) [2602.05085]. This makes the mechanism a genuine capacity expansion rather than a low-rank perturbation of existing weights. Two negative clarifications are important. First, the paper explicitly characterizes Locas-GLU as neither an external memory nor a low-rank perturbation of existing parameters [2602.05085]. Second, because the new branch is parallel to the backbone, the backbone remains untouched at attachment time. This suggests a modular interpretation: Locas-GLU augments the model’s memory capacity without requiring architectural surgery on the original transformer. ## 3. Initialization by activation-guided cloning The most technically distinctive component of Locas-GLU is its initialization scheme. For a memorization chunk \(x_{