Exponential-Power Temporal Encoder

• Exponential-Power Temporal Encoder is a temporal coding scheme that uses parameterized exponential-power decays to capture both short-term and long-term dependencies.
• It integrates into sequential transformers and neuromorphic systems by employing efficient tensor operations and analog mappings, achieving up to 11× faster computation in some tasks.
• Parameter tuning through hyperparameters like α, β, and γ enables domain-specific adaptation and precision in modeling temporal interactions.

An Exponential-Power Temporal Encoder refers to a class of temporal coding schemes in which, for a set of temporally ordered events or signals, the pairwise relations—or the encoding of features or inter-event intervals—are governed by generalized exponential or exponential-power laws. This mechanism enables efficient, flexible encoding and modeling of temporal dependencies in diverse contexts, most prominently in sequential recommendation transformers and neuromorphic or spiking neural network systems. The approach is characterized by learnable parameters that govern the nonlinearity and decay of temporal dependencies, allowing precise tuning of both short-term and long-term interactions. Notable instantiations include the matrix-based temporal attention module in FuXi-$\gamma$ for sequential recommendation (Yi et al., 14 Dec 2025), the analog exponential mapping of inter-spike intervals in neuromorphic encoders (VS et al., 2022), and logarithmic/spike-efficient schemes in SNNs (Zhang et al., 2018).

1. Mathematical Formulation and Model Definition

The Exponential-Power Temporal Encoder formalizes temporal decay using parameterized, smooth functions of temporal intervals. In the context of sequential recommendation, for a normalized interaction sequence of length $n$ with ascending timestamps $\{t_1,\ldots,t_n\}$, the temporal interval matrix is defined as

$T^{i,j} = |t_i - t_j|,\quad 1\leq i,j\leq n.$

The temporal attention (or decay) matrix is given by

$$A_{ts}^{i,j} = \alpha \cdot \gamma^{\,|t_i - t_j|^\beta}$$

where

• $\alpha \in \mathbb{R}$: base-intensity (scaling),
• $\gamma \in (0,1)$: decay-rate,
• $\beta \in \mathbb{R}$: power-exponent.

This formulation produces a continuous, tunable decay curve that generalizes both pure exponential and power-law decays (Yi et al., 14 Dec 2025).

In neuromorphic coding, the exponential-power relationship appears at the analog level: $T(p) = A\cdot\exp(Bp)$ where $p$ is an input signal (e.g., pixel value), $A,B$ combine device parameters, and $T(p)$ is the encoding interval (e.g., inter-spike interval) (VS et al., 2022).

Spiking SNN approaches, notably Logarithmic Temporal Coding (LTC), discretize a real-valued activation $a \geq 0$ into powers-of-two and encode them into spike times in an exponential/logarithmic scheme—the exponentiated (power-law) time bins directly represent the analog value (Zhang et al., 2018).

2. Parameter Definitions and Learning

Each variant’s parameters operationalize both efficiency and adaptability.

Sequential Transformers (Yi et al., 14 Dec 2025):

• $\alpha$: Learns overall attention scaling.
• $\beta$: Modulates non-linear stretching/compression of the interval axis; typically initialized near $1$.
• $\gamma$: Controls decay; typical initializations are $\gamma=0.8$ (movies), $\gamma=0.9$ (music).
• All are updated end-to-end via standard gradient-based optimizers (AdamW).

Neuromorphic Encoders (VS et al., 2022):

• $A$, $B$: Determined by circuit-level constants (membrane capacitance steps, threshold voltage, current mirror gains, thermal voltage).
• $B=\frac{1}{4sU_T}$: Slope, where $s$ is subthreshold slope factor, $U_T$ is thermal voltage.
• Circuit properties ensure $A,B > 0$; tuning is performed at design-time.

Spiking SNNs (Zhang et al., 2018):

• Exponent range ($e_{min}$, $e_{max}$): Sets the precision and temporal span.
• Implicit power-law through bit encoding and spike timing.
• No continuous parameters per se, but exponent ranges are chosen and regularized during ANN-SNN conversion training.

3. Integration into Model Architectures

Decoder-only Transformers (Yi et al., 14 Dec 2025):

The encoder replaces traditional query-key self-attention with a dual-channel self-attention comprising:

• Temporal channel: Uses $A_{ts}$ for direct sequence-wide mixing, leveraging matrix multiplication.
• Positional channel: Employs Toeplitz-structured weight matrix $W_{pos}$. After an SiLU+RMSNorm projection, both attention channels share the projected value tensor, operate in parallel, and merge via concatenation and normalization. The Exponential-Power temporal matrix is recomputed at each layer input and supports pure matrix/tensor operations—enabling contiguous memory access and eschewing expensive lookup-based implementations.

Neuromorphic Encoders (VS et al., 2022):

Pixel input voltages are mapped via subthreshold PMOS circuits to exponentially related currents, integrated by Successive Leaky Integrate-and-Fire (SLIF) neuron branches with incrementally stepped capacitances. The resulting output, the inter-spike interval, is a direct exponential map of the input. The overall temporal encoding is realized through analog circuit dynamics—critical parameters are set at design and confirmed by simulation.

Exponentiate-and-Fire SNNs (Zhang et al., 2018):

Activations are logarithmically approximated, with each retained power-of-two mapped to a spike at a time bin determined by the exponent. The Exponentiate-and-Fire (EF) neuron integrates inputs with exponentially growing post-synaptic potentials, spikes once its membrane potential crosses a threshold, and resets by subtracting this threshold. The process only uses bit-shifts and additions.

4. Computational Efficiency and Implementation

The Exponential-Power mechanism is motivated by the need for high efficiency—both arithmetic and memory bandwidth.

Bucket-based methods entail $O(n^2)$ random memory accesses (lookups), leading to sub-optimal cache use. In contrast, power-decay approaches such as FuXi-$\gamma$’s encoder perform only element-wise power and exponential operations, which are hardware-amenable:

• Pure tensor kernels (no scattering, no lookups)
• Potential kernel fusion (power+exp+multiply)
• Empirically up to $11\times$ faster than bucket-based temporal encoding over long sequences ($n\geq1000$), and $2.52\times$ over simple inverse proportional baselines (Yi et al., 14 Dec 2025).

In neuromorphic hardware, temporally coded spike intervals leverage the analog circuit’s natural exponential current-voltage relationships for sub-microsecond, low-power operation ($\approx0.4–0.7\,\mu$W per neuron) (VS et al., 2022).

Spiking SNNs using LTC achieve $>75\%$ reduction in spike-related events versus classical rate-coding methods and do so with exact ReLU representational equivalence and O(log a) spike count (Zhang et al., 2018).

5. Effectiveness, Ablation, and Short-vs-Long-Term Dynamics

Ablation studies on sequential recommendation tasks (MovieLens ML-1M/20M) reveal that removal of the Exponential-Power Temporal Encoder incurs the largest drop in HR@10 ($-8.8\%$, $-9.5\%$) among all component ablations (Yi et al., 14 Dec 2025). Visualization of decay curves demonstrates that bucket-based encoding yields stepped functions, and inverse-proportion decays suppress long-term signals. The exponential-power variant supports smooth, continuously tunable decay, allowing practitioners to interpolate between short-term and long-term emphasis through $\beta$ (stretch) and $\gamma$ (decay) hyperparameters. Empirically, domains like movies/videos prefer $\gamma=0.8$ (sharper forgetting), while music domains prefer $\gamma=0.9$ (longer-range reminiscence).

6. Algorithmic Workflow and Pseudocode

The Exponential-Power encoding forward pass typically comprises:

1. Compute absolute temporal interval matrix $T$.
2. Apply nonlinearity $T^\beta$ (generic tensor power).
3. Apply exponential decay $\gamma^{T^\beta}$, scaling by $\alpha$:

Tβ = T ** β A_ts = α * (γ ** Tβ) return A_ts

(Yi et al., 14 Dec 2025)

In LTC SNNs, analog value $a$ is first quantized into powers-of-two, which are then mapped to spike bins; neurons update membrane by bit-shifts and summing in spike-weighted contributions, emitting and resetting spikes as threshold is reached (Zhang et al., 2018).

7. Applications, Advantages, and Limitations

Applications:

• Long-sequence sequential recommendation systems (user-item interaction modeling) (Yi et al., 14 Dec 2025)
• Neuromorphic sensory encoding for SNNs (temporal vision/audio feature encoders) (VS et al., 2022)
• Low-latency spiking inference for efficient SNNs in classification, with exact equivalence to ANN computation (Zhang et al., 2018)

Advantages:

• Smooth, adaptable decay with analytic form, simple to implement as tensor primitives (matrix multiplies).
• Hardware efficiency due to contiguous memory access, absence of lookup/bucket random accesses.
• Smooth hyperparameterization enables domain-dependent adaptation.
• Analytic and circuit-validated correspondence in neuromorphic hardware.
• In SNNs, logarithmic spike efficiency and shift/add-only computation.

Limitations:

• Exponential mappings in analog circuits may compress high-valued intervals, potentially losing dynamic range at input extremes (VS et al., 2022).
• Parameter and hyperparameter selection may require domain-specific tuning for optimal short- vs. long-range trade-offs (Yi et al., 14 Dec 2025).
• In SNNs, quantization error and minimal latency are governed by fixed exponent range and time window (Zhang et al., 2018).
• Analog implementations susceptible to process/temperature variations, requiring calibration in practical systems.

Context	Encoding Formula	Salient Properties
Sequential Rec. Transformer	$A_{ts}^{i,j}=\alpha\gamma^{|t_i-t_j|^\beta}$	Pure tensor ops, smooth decay
Neuromorphic Image Encoding	$T(p)=A\exp(Bp)$	Ultra-low power, analytical model
Spiking SNN, LTC+EF neuron	Quantized powers-of-2, exp bins	$O(\log{a})$ spikes, bit-wise efficiency

The Exponential-Power Temporal Encoder framework unifies a class of efficient, parameterized, and hardware-aligned temporal encodings across sequential learning and neuromorphic domains (Yi et al., 14 Dec 2025, VS et al., 2022, Zhang et al., 2018).