Periodic Linear Unit (PLU)

• Periodic Linear Unit (PLU) is a learnable activation combining a residual linear component with a periodic sine term to enhance neural network expressivity.
• It employs a repulsive reparameterization to maintain effective parameter bounds, preventing collapse into a mere linear identity.
• Empirical results, such as on spiral classification tasks, demonstrate PLU's ability to capture complex, curved patterns with smaller networks.

Searching arXiv for the cited PLU papers and closely related activation-function context. Periodic Linear Unit (PLU) denotes a learnable activation function introduced as a residual linear term plus a periodic sine component, with an associated repulsive reparameterization intended to prevent collapse into a non-expressive linear mapping (Kudo, 2 Aug 2025). In that formulation, PLU is explicitly periodic and non-monotone when needed, and is presented as a shift “from Taylor series to Fourier synthesis,” emphasizing expressive power inside individual neurons rather than relying primarily on large collections of monotone units (Kudo, 2 Aug 2025). The acronym is not historically unique: in 2018, “PLU” referred to the distinct Piecewise Linear Unit, a three-segment piecewise-linear activation (Nicolae, 2018).

1. Terminology and conceptual setting

The 2025 Periodic Linear Unit was proposed against the background of a dominant neural-network paradigm built around simple, generally monotone nonlinearities such as ReLU, GELU, and tanh. In that account, monotone activations act as “transparent” layers for gradient flow because their derivative with respect to the pre-activation is nonnegative, so the chain rule does not flip gradient signs, and single neurons are therefore limited to comparatively simple on/off, saturating, or hinge-like responses. PLU was introduced to exploit periodic non-monotonicity directly inside the activation, with the stated aim of increasing expressive power and parameter efficiency for strongly curved, periodic, or band-selective phenomena such as audio and spirals (Kudo, 2 Aug 2025).

The paper’s organizing perspective is that standard monotone activations induce a Taylor-like additive paradigm, whereas PLU supports a Fourier-like synthesis paradigm. Rather than approximating complex structure by summing many small linear or monotone pieces, PLU superposes a residual linear term with learnable sinusoidal structure. A central empirical claim is that a minimal multilayer perceptron with only two PLU neurons can solve a spiral classification task, whereas equivalent networks using standard activations fail (Kudo, 2 Aug 2025).

2. Formal definition and elementary properties

The initial formulation is given by

$$\mathrm{PLU}_{\mathrm{theoretical}}(x;\alpha,\beta)=x+\frac{\beta}{1+\lvert\beta\rvert}\,\sin(\lvert\alpha\rvert x).$$

Here, $\alpha$ controls frequency and $\beta$ controls amplitude through the soft-bounded map

$$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$

With $s=\frac{\beta}{1+\lvert\beta\rvert}$ and $\omega=\lvert\alpha\rvert$, the derivatives are

$$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$

The periodic component has period $T=2\pi/\omega$, but the full activation is not globally periodic because of the residual $x$. The periodic component is bounded by $|s|$, while the overall function is unbounded. The paper also states a global Lipschitz bound $\alpha$0, notes that the earlier formulation is odd, and records asymptotic regimes including $\alpha$1 identity, $\alpha$2, $\alpha$3, and $\alpha$4 identity (Kudo, 2 Aug 2025).

The residual path is not incidental. It preserves identity-like behavior and is used as a stability device for gradient flow, while the sine term provides the periodic, potentially non-monotone component. This combination yields a function that can remain close to a conventional residual activation when appropriate, yet admit oscillatory structure when optimization moves into that regime.

3. Repulsive reparameterization

The final formulation augments the initial activation with effective parameters

$\alpha$5

so that

$\alpha$6

Its derivatives with respect to $\alpha$7 are

$\alpha$8

where

$\alpha$9

The reparameterization map is

$\beta$0

with derivative

$\beta$1

Setting $\beta$2 yields $\beta$3, which the paper interprets as an effective lower bound $\beta$4. As $\beta$5, $\beta$6 and $\beta$7, creating what the paper describes as a repulsive barrier near zero. The stated purpose is to prevent the optimizer from taking the “path of least resistance” in which $\beta$8 and the activation collapses toward a linear identity. Away from zero, $\beta$9, so $$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$0, preserving the expressive regime far from the barrier (Kudo, 2 Aug 2025).

This design also addresses a specific numerical issue. Because the amplitude uses $$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$1 in the denominator, the formulation avoids division by zero in amplitude and is contrasted with Snake-like singularities at $$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$2. Conversely, near $$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$3, the effective frequency can become extremely large, so the repulsion is both the source of the anti-collapse mechanism and a potential source of high-frequency instability if not controlled.

4. Non-monotonicity, gradient structure, and Fourier-like synthesis

A defining property of Periodic Linear Unit is that it can generate negative local slopes through the $$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$4 factor. The paper frames this as non-monotone gradient shaping: unlike monotone activations whose derivative is nonnegative, PLU can flip the sign of local gradients and thereby implement bumps, bands, and interference-like responses within a single neuron (Kudo, 2 Aug 2025).

The monotonicity condition is explicit. PLU is strictly increasing if and only if

$$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$5

Otherwise there exist $$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$6 such that

$$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$7

and the activation has non-monotone segments. The proof sketch given in the paper is immediate from the fact that $$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$8: the minimum derivative is $$s(\beta)=\frac{\beta}{1+\lvert\beta\rvert}\in(-1,1).$$9, which remains positive exactly under the stated condition.

The same paper develops a “Fourier synthesis” interpretation by substituting PLU into a one-hidden-layer model: $s=\frac{\beta}{1+\lvert\beta\rvert}$0 In this expression, $s=\frac{\beta}{1+\lvert\beta\rvert}$1 sets the affine carrier, $s=\frac{\beta}{1+\lvert\beta\rvert}$2 determines frequency, and $s=\frac{\beta}{1+\lvert\beta\rvert}$3 scales amplitudes. The paper’s interpretation is that the network behaves less like a piecewise-polynomial or piecewise-linear approximator and more like a learnable spectral synthesizer. This suggests an alternative route to representational efficiency on highly curved or periodic structure, with constructive and destructive interference emerging inside and across neurons.

5. Empirical behavior on spiral classification

The reported experiments use binary spiral classification, MLP architectures $s=\frac{\beta}{1+\lvert\beta\rvert}$4 and $s=\frac{\beta}{1+\lvert\beta\rvert}$5, Adam with learning rate $s=\frac{\beta}{1+\lvert\beta\rvert}$6, Binary Cross-Entropy loss, and shared activation parameters across all neurons for both PLU and Snake. In the $s=\frac{\beta}{1+\lvert\beta\rvert}$7 model, with initialization $s=\frac{\beta}{1+\lvert\beta\rvert}$8 and $s=\frac{\beta}{1+\lvert\beta\rvert}$9, PLU reaches loss $\omega=\lvert\alpha\rvert$0 by epoch $\omega=\lvert\alpha\rvert$1, compared with Snake at $\omega=\lvert\alpha\rvert$2, while ReLU and GELU lag significantly; by epoch $\omega=\lvert\alpha\rvert$3, PLU attains $\omega=\lvert\alpha\rvert$4, reported as the lowest loss in that comparison (Kudo, 2 Aug 2025).

A separate “chaotic initialization” ablation sets $\omega=\lvert\alpha\rvert$5 and $\omega=\lvert\alpha\rvert$6, so $\omega=\lvert\alpha\rvert$7. Under that initialization, PLU begins in a high-frequency, chaotic state at epoch $\omega=\lvert\alpha\rvert$8. Convergence becomes harder; Snake and GELU solve the task, while PLU finishes with higher loss, though still with visibly spiral-shaped boundaries. The paper interprets this as evidence that PLU can impose strong high-frequency priors at initialization, giving rise to a “start complex, simplify” learning regime controlled by $\omega=\lvert\alpha\rvert$9.

The most distinctive result is the $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$0 model. At epoch $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$1, the reported losses are ReLU $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$2, GELU $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$3, Snake $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$4, and PLU $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$5, with a separate run achieving $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$6. The visual characterization is that PLU already produces multi-banded, “marbled” interference patterns by epoch $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$7, whereas monotone activations yield lines or single curves and fail on the task. The paper presents this result as evidence that non-monotone periodic activations can realize spiral-aligned decision structure with extremely small networks (Kudo, 2 Aug 2025).

6. Optimization, implementation, and failure modes

The recommended initialization is $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$8, $$\frac{d}{dx}\mathrm{PLU}_{\mathrm{theoretical}}(x)=1+s\,\omega\cos(\omega x),\qquad \frac{d^2}{dx^2}\mathrm{PLU}_{\mathrm{theoretical}}(x)=-s\,\omega^2\sin(\omega x).$$9, with moderate repulsion parameters such as $T=2\pi/\omega$0 and $T=2\pi/\omega$1. A more aggressive alternative, $T=2\pi/\omega$2 and $T=2\pi/\omega$3, is described as “chaotic” because it enforces stronger high-frequency priors. The rationale is that the reparameterization keeps effective frequency and amplitude away from the collapse regime by inducing approximate lower bounds $T=2\pi/\omega$4. Weight decay is not discussed; the paper notes that the soft amplitude bound already discourages runaway amplitudes (Kudo, 2 Aug 2025).

A minimal PyTorch implementation is given with scalar parameters $T=2\pi/\omega$5, $T=2\pi/\omega$6, $T=2\pi/\omega$7, and $T=2\pi/\omega$8, and an $T=2\pi/\omega$9 added in the denominators to avoid division-by-zero numerical exceptions while preserving the intended barrier behavior near zero. The same source notes that per-neuron PLU parameters can be obtained by instantiating separate modules or by registering vector parameters and broadcasting them appropriately. The activation can replace standard nonlinearities in MLPs after linear layers, be applied pointwise after convolutions in CNNs, and replace GELU or ReLU in Transformer feed-forward blocks. LayerNorm and BatchNorm are stated to remain compatible, though distributional shifts from periodic outputs should be monitored. Relative to matrix multiplications, the computational overhead is described as modest, consisting mainly of one sine and a few scalar operations.

The principal failure mode is collapse without repulsion: absent the repulsive reparameterization, the optimizer tends to drive $x$0, reducing PLU to a linear identity and halting learning. A second failure mode is excessive high-frequency complexity when the repulsion parameters are too large. The paper therefore recommends moderate $x$1, possible gradient clipping, and diagnostic tracking of $x$2, $x$3, $x$4, and $x$5. It also states that tasks requiring strictly monotone mappings may not benefit from periodic non-monotonicity (Kudo, 2 Aug 2025).

7. Distinction from the 2018 Piecewise Linear Unit

The acronym “PLU” was previously used for the Piecewise Linear Unit, introduced in 2018 as a hybrid between tanh and ReLU rather than as a periodic activation (Nicolae, 2018). Its definition is

$x$6

which yields the derived piecewise form

$x$7

In that paper, the breakpoints are at $x$8, the central slope is $x$9, the tail slopes are $|s|$0, and the illustrative parameters are $|s|$1 and $|s|$2. The activation is described there as “a crude piecewise approximation of the tanh” and as unbounded in order to avoid vanishing gradients.

The 2018 PLU is continuous, piecewise linear, and non-differentiable at $|s|$3, with derivative $|s|$4 in the tails and $|s|$5 in the central interval. It was proposed to avoid both tanh saturation and the zero-gradient negative half-axis of ReLU. The paper emphasizes that it is “inexpensive to compute,” has nonzero derivative everywhere in the practical sense away from kink points, and provides more linear pieces than ReLU, which in turn can improve approximation of highly nonlinear functions in shallow models (Nicolae, 2018).

Empirically, the piecewise-linear PLU and tanh were reported to achieve mean-squared error “two orders of magnitude lower than the ReLU” on a depth-3, width-3 sine-fitting task, with PLU yielding more linear segments in the learned function. On a depth-5, width-5 parametric fitting problem, ReLU was said to “struggle” with its “5 line segments,” while PLU fit much closer and kept up with tanh in loss. On a hyperbolic paraboloid fitting task, PLU again outperformed ReLU and tracked tanh more closely. In a shallow CIFAR-10 ConvNet, the results were described as “more favorable to the ReLU than the previous examples,” although tanh and PLU still trained faster than ReLU (Nicolae, 2018).

The nomenclature overlap is therefore substantive rather than merely stylistic. “PLU” may refer either to a three-segment piecewise-linear odd function from 2018 or to a residual periodic sine activation with Repulsive Reparameterization from 2025. In current technical usage, disambiguation by expansion—Piecewise Linear Unit versus Periodic Linear Unit—is necessary.