Compact Gated Linear Unit (CGLU)

• Compact Gated Linear Unit (CGLU) is defined by decoupling multiplicative gating from linear prediction, achieving parameter efficiency through fixed gates or mask-based sharing.
• It reduces parameter counts and memory traffic by sharing weights or compressing gate representations, enabling cost-effective training and faster inference.
• The CGLU framework, encompassing GaLU and MGLU variants, offers convex optimization benefits and empirical performance improvements in large language model architectures.

Searching arXiv for the cited papers to ground the article in current records. arXiv search: "Decoupling Gating from Linearity" and "Masked Gated Linear Unit" Compact Gated Linear Unit (CGLU) is an Editor’s term for compact parameterizations of gated linear units that preserve multiplicative gating while reducing parameter count, storage, or memory traffic. Two distinct but related research lines motivate the term. In the first, the Gated Linear Unit (GaLU) decouples the gating vector from the linear predictor, fixes the gate after random initialization, and thereby converts one-hidden-layer training into convex linear regression on a random feature map (Fiat et al., 2019). In the second, the Masked Gated Linear Unit (MGLU) replaces the two projection matrices of a standard GLU with one shared matrix plus complementary binary masks, yielding a compact GLU implementation for LLM feed-forward networks (Tajima et al., 29 Jun 2025). Taken together, these lines define CGLU as a family of architectures in which gating and value computation remain multiplicative, but compactness is achieved either by decoupling-and-sharing in function space or by mask-based sharing in parameter space.

1. Conceptual definition and formal forms

A standard ReLU neuron can be written as

$$\sigma(w^\top x) = (w^\top x)\,\mathbf{1}\{w^\top x \ge 0\},$$

which exposes it as the product of a linear term and a hard gate controlled by the same vector $w$. GaLU separates these two roles by assigning different vectors $u,v \in \mathbb{R}^d$: $$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$ Under the hard gate, $\nabla_u g_{u,v}(x)=0$ almost everywhere, so the gate parameter $u$ is fixed after random initialization, typically $u \sim N(0,I_d)$, and training updates only the linear part $v$ (Fiat et al., 2019).

For transformer feed-forward networks, a standard GLU is written in the MGLU formulation as

$\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$

with separate gate and value matrices. MGLU makes this compact by introducing one shared weight matrix $W$ and a binary mask $w$0: $w$1 Here $w$2, so the shared matrix is partitioned into complementary gate and value subspaces. The source paper does not use the term CGLU, but this single-mask construction is explicitly identified as a compact GLU instantiation (Tajima et al., 29 Jun 2025).

A useful synthesis is that CGLU preserves the defining GLU operation—elementwise multiplication of a gate stream and a value stream—while compactness is induced by one of two mechanisms: fixed random gates plus reduced trainable structure, or shared weights plus learned binary partitions.

Variant	Core form	Compactness mechanism
ReLU	$w$3	None
GaLU	$w$4	Decoupled gate/value
Single-mask MGLU	$w$5	Shared matrix + mask

2. GaLU as the theoretical foundation for compact hard-gated units

A one-hidden-layer GaLU network with $w$6 hidden units and scalar output is

$w$7

Because the model is linear in $w$8, one may absorb the output weights into $w$9 and rewrite the network as

$u,v \in \mathbb{R}^d$0

where $u,v \in \mathbb{R}^d$1 and

$u,v \in \mathbb{R}^d$2

Training therefore reduces to linear regression on a random feature map with fixed gates (Fiat et al., 2019).

For a sample $u,v \in \mathbb{R}^d$3, the associated block design matrix is

$u,v \in \mathbb{R}^d$4

Optimization becomes the convex problem $u,v \in \mathbb{R}^d$5. The key data-dependent quantity is

$u,v \in \mathbb{R}^d$6

assumed positive. Under this assumption, if

$u,v \in \mathbb{R}^d$7

then with probability at least $u,v \in \mathbb{R}^d$8,

$u,v \in \mathbb{R}^d$9

which implies full row rank and zero training loss by linear regression.

For random Gaussian or spherical data, the bounds sharpen. When $$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$0 are i.i.d. on the unit sphere $$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$1 and $$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$2, there exist absolute constants $$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$3 such that

$$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$4

implies

$$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$5

with the stated high probability. This yields memorization at width $$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$6, improving on the cited ReLU theory requiring $$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$7 under bounded derivatives (Fiat et al., 2019).

The same rank perspective also characterizes the under-parameterized regime. When labels are i.i.d. $$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$8,

$$g_{u,v}(x) \;=\; (v^\top x)\,\mathbf{1}\{u^\top x \ge 0\}.$$9

When $\nabla_u g_{u,v}(x)=0$0, $\nabla_u g_{u,v}(x)=0$1, so the loss behaves like $\nabla_u g_{u,v}(x)=0$2. This makes explicit that the effective capacity is governed by the rank of the gated design matrix rather than by nominal width alone.

3. Optimization, kernel structure, and relation to ReLU

Because the GaLU reparameterization is linear in $\nabla_u g_{u,v}(x)=0$3, training with squared loss has a convex landscape. If $\nabla_u g_{u,v}(x)=0$4, $\nabla_u g_{u,v}(x)=0$5, and

$\nabla_u g_{u,v}(x)=0$6

then after

$\nabla_u g_{u,v}(x)=0$7

iterations of gradient descent, the squared loss is at most $\nabla_u g_{u,v}(x)=0$8 with probability at least $\nabla_u g_{u,v}(x)=0$9. The argument uses convexity together with spectral control of $u$0 (Fiat et al., 2019).

The corresponding normalized GaLU network induces the kernel

$u$1

The random features $u$2 approximate this kernel in expectation. If the data distribution is realizable by some $u$3, where $u$4, and if

$u$5

then with probability at least $u$6, the empirical risk minimizer satisfies

$u$7

Under the realizability assumption and fixed gates, this gives an $u$8 decay rate rather than the $u$9 rate emphasized in the comparison to cited ReLU analyses (Fiat et al., 2019).

The relation to ReLU is subtle. At initialization, under hinge loss and bounded outputs, small gradient at initialization occurs with the same probability for GaLU and ReLU: $u \sim N(0,I_d)$0 This aligns failure modes such as parity failure across the two models. A further result states that when ReLU training remains within $u \sim N(0,I_d)$1 of initialization—the “lazy training” regime—the fixed-gate GaLU is competitive with the trained ReLU. A plausible implication is that decoupled hard-gated models can capture a meaningful subset of ReLU behavior precisely when ReLU learning is dominated by a nearly fixed gating pattern.

4. Compact GaLU: proposed constructions and design rules

Within the GaLU line, a Compact GaLU (CGLU) is proposed by imposing parameter sharing or low-rank structure while keeping decoupled gating and linear prediction. One formal proposal is

$u \sim N(0,I_d)$2

where $u \sim N(0,I_d)$3 and $u \sim N(0,I_d)$4 are shared bases, $u \sim N(0,I_d)$5 is a per-neuron linear code, and $u \sim N(0,I_d)$6 is a per-neuron gate code. The associated one-hidden-layer network is

$u \sim N(0,I_d)$7

If $u \sim N(0,I_d)$8 is fixed, training in $u \sim N(0,I_d)$9 remains convex; if $v$0 is trained, the problem becomes bi-linear. This compactifies the trainable linear part from $v$1 to $v$2, with gate storage $v$3 if gates are stored (Fiat et al., 2019).

Three concrete compactification mechanisms are described.

First, gate sharing introduces a dictionary $v$4 with $v$5 and reuses gates across neurons: $v$6 This reduces gate storage to $v$7. However, duplicating gates does not increase $v$8, so memorization still requires sufficiently many distinct gates.

Second, low-rank linear predictors write $v$9 with $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$0, $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$1. The effective design matrix becomes $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$2, whose rank is at most $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$3. Exact memorization therefore requires $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$4, equivalently width $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$5.

Third, gate compression writes $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$6 with $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$7, $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$8, and fixed random $\mathrm{GLU}(x) = g(x W_g) \odot (x W_v),$9. This reduces gate storage but also reduces gate diversity. The synthesis notes that one should choose $W$0 so that the induced gate kernel maintains $W$1 bounded, for example by using a near-isometry on the data manifold.

These constructions lead to a compact design rule: because the achievable training loss depends on $W$2, a compact model with $W$3 distinct gates and per-unit linear dimension $W$4 should satisfy $W$5 to retain zero-loss memorization capacity. This is an inference from the rank analysis rather than a theorem stated in the original 2019 paper, but it follows directly from the stated bound $W$6 and the exact-loss characterization.

5. Masked compact GLUs in transformer feed-forward networks

In the transformer setting, compactness is framed differently. A standard GLU up-projection uses two FP16 matrices $W$7: $W$8 The MGLU construction replaces these with a single shared matrix $W$9 and one or more learned binary masks. The single-mask case,

$w$00

is the minimal compact GLU. To increase expressiveness, MGLU introduces a Mixture of Element-wise Gating (MoEG) with multiple masks: $w$01 Each route $w$02 induces a different gate/value partition while still using one shared FP16 matrix (Tajima et al., 29 Jun 2025).

The compactness advantage is explicit in the memory accounting. For the up-projection at inference, a standard GLU reads $w$03 bits per token, whereas MGLU with $w$04 masks reads $w$05 bits. At $w$06, the reduction is

$w$07

that is, 47% fewer bits transferred. The paper further notes that MGLU still breaks even with GLU up to $w$08 at FP16 inference. For Llama-1B with $w$09 and $w$10, the FFN up-projection weights per layer shrink from 96 MB to 64 MB, and one 1-bit mask adds approximately 2 MB for $w$11 (Tajima et al., 29 Jun 2025).

Parameter counts also change materially. A GLU up-projection has $w$12 FP16 parameters. MGLU uses $w$13 FP16 weights plus $w$14 mask bits at inference; during training, the masks are maintained as FP16 logits, adding $w$15 trainable FP16 parameters. Mask learning is performed through the straight-through estimator: the mask is thresholded to binary in the forward pass, while the backward pass treats the binarization as identity.

The compute trade-off is asymmetric between inference and training. For next-token inference, SwiMGLU incurs $w$16 multiply-adds per token, versus $w$17 for SwiGLU. Because low-batch decoding is memory-bound, the reduction in FP16 reads dominates latency. During training, however, the reported cost is approximately $w$18 per token for SwiMGLU versus $w$19 for SwiGLU, so large $w$20 can increase compute cost even as weight storage is reduced.

A central misconception is that compact GLU is equivalent to naive weight tying. The MGLU study explicitly reports that setting $w$21 in SwiGLU severely hurts perplexity, to 27.0 relative to 23.6 for SwiGLU, whereas masked sharing preserves multiplicative capacity with one shared $w$22. Compactness therefore comes not from simple equality constraints, but from structured partitioning of a shared weight tensor.

6. Kernel implementation, empirical behavior, and limitations

The efficiency of masked compact GLUs depends on kernel design. A naive MGLU implementation would launch $w$23 masked matrix-vector products and repeatedly reload $w$24, remaining memory-bound. FlashMGLU instead fuses these operations so that each weight is touched once. The reported implementation uses bit-packing of the $w$25 mask flags into a single 8-bit integer, single shared loading of $w$26 and mask words per tile, complementary masks to recover value sums from an unmasked total $w$27 and masked partial sums $w$28, Split-K tiling with on-chip accumulation, and fused arithmetic in registers. After reduction, the kernel writes gate sums $w$29 and value sums $w$30, after which the activation and elementwise multiplication are applied route-wise (Tajima et al., 29 Jun 2025).

On an RTX 5090 GPU in FP16, FlashMGLU achieves up to a 19.66$w$31 speed-up over a naive PyTorch MGLU at $w$32, reducing latency from 0.5210 ms to 0.0265 ms. Relative to standard PyTorch GLU, it is up to 1.51$w$33 faster in the $w$34 case and remains at least 1.24$w$35 faster for larger shapes. The abstract summarizes the result as 47% more memory-efficient and 34% faster than standard GLUs on an RTX5090 GPU. With four masks and Swish, the paper also reports a 29.1% reduction in projection-layer compute and 37.5% lower memory usage during inference, while matching or surpassing SwiGLU accuracy.

Model-level evaluations use Llama-style decoder-only architectures on FineWeb-Edu, with zero-shot and two-shot accuracy on ARC-E, ARC-C, HellaSwag, PiQA, SciQ, and Winogrande, together with validation perplexity. At the small scale, SwiGLU has 141M weights, whereas SwiMGLU has 113M weights plus mask bits; zero-shot averages are 46.20% for SwiGLU, 46.48% for SwiMGLU with $w$36, and 46.49% with $w$37. Validation perplexity improves from 25.0 at $w$38 to 23.5 at $w$39, close to SwiGLU’s 23.7. At the large scale, the baseline SwiGLU has 1.08B weights and SwiMGLU has 808M weights plus mask bits; zero-shot average improves from 56.00% to 56.85% at $w$40, and two-shot average improves from 57.36% to 57.87% (Tajima et al., 29 Jun 2025).

The empirical picture across the two source lines is internally consistent but not identical. In the GaLU experiments, minimal width $w$41 suffices for both GaLU and ReLU to reach MSE $w$42, GaLU loss follows $w$43 in the under-parameterized regime, and parity is failed by both models. On linearly separable data in $w$44, on MNIST, and on Fashion-MNIST, both GaLU and ReLU perform well, with ReLU slightly better. In the MGLU experiments, compact masked sharing is competitive or superior to dense SwiGLU, especially for $w$45 to $w$46. This suggests that “compactness” is not a single phenomenon: in hard-gated shallow models it refers to reduced trainable dimensionality under fixed random gates, whereas in LLM feed-forward blocks it refers to reduced memory traffic through shared weights and learned mask partitions.

Several limitations recur. The GaLU theory is restricted to one-hidden-layer networks with scalar outputs, hard binary gates $w$47, fixed Gaussian-initialized gates, and realizability assumptions for the RKHS result. Deeper GaLU networks, vector-valued outputs, and soft gates are not analyzed, and soft gates would break the hard-gate convex reduction. In compact GaLU variants, sharing or compressing gates can reduce diversity, so one must preserve enough distinct halfspaces to keep $w$48 bounded and $w$49. In MGLU, mask learning is discrete and depends on the straight-through estimator; training cost rises with $w$50, and the strongest kernel optimizations are implemented for CUDA GPUs. The masked models also exhibit diminishing returns beyond $w$51 to $w$52, so maximum compactness at $w$53 and maximum accuracy need not coincide.

In sum, CGLU denotes a compact gating principle rather than a single canonical architecture. In the GaLU formulation, compactness arises from decoupling gating from linearity and imposing shared or low-rank structure on the trainable linear component, producing convex optimization, rank-based memorization guarantees, and kernel-style generalization (Fiat et al., 2019). In the MGLU formulation, compactness arises from collapsing the two GLU matrices into one shared matrix plus complementary masks, producing substantial memory-bandwidth savings and hardware-efficient inference without sacrificing downstream accuracy (Tajima et al., 29 Jun 2025).