Phased LSTM: Time-Gated RNN for Sparse Data

• Phased LSTM is a recurrent neural network architecture that integrates a learnable oscillatory time gate for precise, asynchronous sequence updates.
• It reduces computational load by activating updates only during open phases, thus preserving cell state and enhancing long-term memory retention.
• Empirical results on tasks such as event-based vision and audio–visual lip reading demonstrate faster convergence and superior efficiency compared to standard LSTM models.

Phased LSTM is a recurrent neural network architecture that extends the conventional Long Short-Term Memory (LSTM) unit with a learnable, oscillatory time gate, enabling efficient and robust modeling of asynchronous, event-driven, and long temporal sequences. By leveraging this time gate, Phased LSTM networks achieve faster convergence, reduced compute, and enhanced long-term memory retention, particularly in settings with sparse or irregularly sampled input streams, such as those produced by neuromorphic sensors or heterogeneous multi-sensor systems (Neil et al., 2016).

1. Standard LSTM Architecture

The baseline LSTM cell maintains a cell state vector $c_t$, a hidden state $h_t$, and three gating mechanisms: input ($i_t$), forget ($f_t$), and output ($o_t$) gates. For discrete time step $t$, the updates are:

$$\begin{aligned} i_t &= \sigma\bigl(x_t W_{xi} + h_{t-1} W_{hi} + w_{ci} \odot c_{t-1} + b_i\bigr) \ f_t &= \sigma\bigl(x_t W_{xf} + h_{t-1} W_{hf} + w_{cf} \odot c_{t-1} + b_f\bigr) \ \widetilde c_t &= f_t \odot c_{t-1} + i_t \odot \tanh\bigl(x_t W_{xc} + h_{t-1} W_{hc} + b_c\bigr) \ o_t &= \sigma\bigl(x_t W_{xo} + h_{t-1} W_{ho} + w_{co} \odot \widetilde c_t + b_o\bigr) \ h_t &= o_t \odot \tanh(\widetilde c_t) \end{aligned}$$

where $\sigma$ denotes the logistic sigmoid, $\odot$ denotes element-wise multiplication, and $W_*$, $h_t$0, $h_t$1 are learnable parameters. Standard LSTM cells update their states at every step, regardless of the input's temporal structure.

2. Phased LSTM Cell: Oscillatory Time Gate Mechanism

Phased LSTM enhances the LSTM unit with a parametric time gate $h_t$2 that regulates memory and hidden state updates as a function of real-valued timestamps $h_t$3, rather than fixed step indices. The gate oscillates according to a learned period $h_t$4, open ratio $h_t$5 (fraction of time spent “open”), and phase shift $h_t$6:

• Phase calculation:

$h_t$7

• Time-gate function:

$h_t$8

where $h_t$9 (e.g., $i_t$0) ensures a nonzero gradient even in the “closed” phase.

• Modified cell and hidden updates (at event time $i_t$1):

$i_t$2

When the time gate $i_t$3, both $i_t$4 and $i_t$5 are essentially copied (with small leak), facilitating sparse and temporally precise updates (Neil et al., 2016).

3. Algorithmic Operation and Computation

The Phased LSTM algorithm processes input pairs $i_t$6 ordered by ascending $i_t$7, permitting naturally asynchronous or event-driven data. At each step:

1. Compute $i_t$8 and $i_t$9 per the parametrized oscillatory cycle.
2. Evaluate standard LSTM gate activations.
3. Propose updated cell/hidden states $f_t$0.
4. Blend updates with previous states, weighted by time gate $f_t$1.
5. In closed phases ($f_t$2), states are virtually unchanged except for a minuscule leak that maintains gradient flow.

This phased computation allows each neuron to update its states only when its gate is open—a fraction $f_t$3 of real time—resulting in a substantial reduction in runtime operations. For $f_t$4, only 5% of inputs result in state changes; other steps carry forward existing state (Neil et al., 2016).

4. Theoretical Implications of Phase-Gated Updates

Sparse, phase-driven gating directly impacts computational efficiency, memory retention, and training dynamics:

• Reduced runtime compute: Many inputs can skip hidden/cell update multiplications when $f_t$5 is closed, enabling order-of-magnitude compute reduction compared to vanilla LSTM, particularly in dense real-time streams.
• Long-term memory: When $f_t$6, the cell state is preserved exactly, preventing the exponential decay mediated by repeated forget-gate multiplication.

$f_t$7

This suggests Phased LSTM can retain information stably across long temporal spans in real time, even as the number of input events grows.

• Gradient propagation shortcut: Backpropagation through time need only traverse open-phase steps, avoiding vanishing gradient over long sequences—accelerating convergence for long-range dependencies.

5. Benchmark Experiments and Empirical Results

Evaluation on sequence modeling tasks demonstrates Phased LSTM's robustness and efficiency across asynchronous, sparse, and long-duration data:

Task	Baseline Models	Key Phased LSTM Outcome
Frequency discrimination	LSTM, BN-LSTM	Maintains high accuracy at oversampled/irregular sampling; converges dramatically faster with lower variance across seeds
Long-range “adding” memory task	LSTM variants	Larger period $f_t$8 (longer gate cycles) yield even faster convergence—enabling gradient shortcuts across hundreds of steps
Event-based vision (N-MNIST)	CNN, BN-LSTM	Classifies after one epoch at ≈90% vs. ≈40% (BN-LSTM) and ≈74% (CNN); final accuracy ≈97.3%; ≈20× fewer updates/neuron (~159 vs. ~3153)
Audio–visual lip reading (GRID)	LSTM, BN-LSTM	Video-only decoding with Phased LSTM converges much faster, achieving >81% accuracy at lower latency and lower compute footprint

In particular, on N-MNIST, Phased LSTM matches or surpasses frame-based and batch-normalized LSTM baselines while requiring an order-of-magnitude fewer per-neuron updates (Neil et al., 2016).

6. Integration with Arbitrary Sensor Rates and Asynchronous Input Streams

Phased LSTM's architecture is inherently compatible with real-valued, irregularly timed inputs. Each unit's time gate oscillator—parametrized by distinct $f_t$9, $o_t$0, and $o_t$1—enables the fusion of multi-rate sensor signals. In multimodal settings, such as combining video (40 ms), audio (10 ms), and event-driven tactile or neuromorphic data (microsecond scale), the cell updates only on new events, maintaining dormant state otherwise.

• A plausible implication is that Phased LSTM supports highly efficient, precision-timed sequence processing for neuromorphic and asynchronous sensor contexts, matching the operational regime of biological spiking networks.

7. Summary and Implications

Phased LSTM introduces a learnable, oscillatory time gate atop conventional LSTM cells, resulting in phase-gated, sparse updates that:

• Reduce computational overhead by limiting active updates to a fraction $o_t$2 of real time,
• Preserve cell state with no exponential decay during gate closings,
• Enable a gradient “shortcut” for accelerated convergence over long or irregular sequences,
• Seamlessly handle asynchronous, multi-rate inputs.

Empirical validation—spanning sine-wave discrimination, long-range “adding” memory, neuromorphic N-MNIST vision, and audio–visual lip reading—supports Phased LSTM's effectiveness on sparse and temporally complex streams, yielding superior accuracy and training speed with notably fewer inference updates than batch-normalized or baseline LSTMs (Neil et al., 2016).