Bayesian Activation-based Recurrence

• Activation-based recurrence is a class of RNN architectures derived from Bayesian updating, yielding additive feedback in the log-odds domain.
• It employs probabilistic gating mechanisms for context retention and input updates, aligning with but advancing beyond conventional LSTM/GRU designs.
• This approach supports bidirectional inference via two-pass smoothing, achieving parameter-efficient improvements in tasks like speech recognition.

Activation-based recurrence refers to a class of recurrent neural network (RNN) architectures in which the recurrence relation at each timestep is explicitly derived from Bayesian updating principles, yielding an additive feedback term in the log-odds domain of the activation. This contrasts with conventional RNNs, where recurrence is often engineered via heuristic gating mechanisms. In activation-based recurrence, the update to a hidden state is grounded in a prescribed probabilistic interpretation, enabling principled modeling of information retention, context resetting, and bidirectional inference. The central formalism is exemplified by the Bayesian Recurrent Unit (BRU), which systematically derives its recurrence, forget, and input-update mechanisms from the application of Bayes’s theorem to latent binary features over sequences (Garner et al., 2019).

1. Bayesian Derivation of Activation-Based Recurrence

The foundation of activation-based recurrence lies in the Bayesian update of a binary latent variable $f$ given observations $X_{1:t}$. At each timestep $t$, the posterior is updated by:

$$P(f\mid X_{1:t}) \propto P(X_t\mid f) P(f\mid X_{1:t-1})$$

Letting $h_t = P(f\mid X_{1:t})$ and $h_{t-1} = P(f\mid X_{1:t-1})$, the recurrence becomes:

$$h_t = \frac{P(X_t \mid f) h_{t-1}}{P(X_t \mid f) h_{t-1} + P(X_t \mid \neg f) (1-h_{t-1})}$$

Expressed as an odds ratio and assuming a logistic likelihood ratio:

$$h_t = \sigma\big(w^\top x_t + b + \text{logit}(h_{t-1})\big)$$

where $\text{logit}(h) = \ln \frac{h}{1-h}$.

Thus, activation-based recurrence yields additive feedback in the log-odds, integrating current evidence $x_t$ and prior hidden state $h_{t-1}$. This additive formulation arises directly from Bayes’s theorem without ad hoc parameterization (Garner et al., 2019).

2. Contextualization via Forget and Input Gates

Activation-based recurrence generalizes to incorporate context indicators reminiscent of gating mechanisms in engineered architectures. By introducing a binary context indicator $\zeta_t$, the prior at time $t$ becomes a convex combination of an unconditional prior $p$ and the previous posterior:

$\hat h_{t-1} = (1-z_{t-1}) p + z_{t-1} h_{t-1}$

with $z_t = P(\zeta_t = 1 | X_{1:t})$ denoting the probability of retaining context.

Similarly, a probabilistic input gate is defined via another indicator $\xi_t$, interpolating between a new-likelihood posterior and the previous state:

$h_t = r_t P(f | X_t) + (1 - r_t) h_{t-1}$

where $r_t = P(\xi_t = 1 | X_{1:t})$.

These mechanisms are functionally and mathematically analogous to the forget gate in Long Short-Term Memory (LSTM) units and the update gate in Gated Recurrent Units (GRU), but arise directly from probabilistic modeling.

3. The Bayesian Recurrent Unit: Formal Structure

The Bayesian Recurrent Unit (BRU) formalizes activation-based recurrence using vectorized update equations, gates, and nonlinearities. For hidden dimension $D$:

• Gates:

$z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$

$r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$

• Forward Pass:

$\hat h_{t-1} = (1 - z_{t-1}) p + z_{t-1} h_{t-1}$

$$n_t = \sigma(W_h x_t + b_h + \text{logit}(\hat h_{t-1}))$$

$h_t = (1 - r_t) \circ n_t + r_t \circ h_{t-1}$

In implementation, the logit nonlinearity can be approximated or absorbed into trainable parameters for stability, yielding:

$$n_t = \sigma(W_h x_t + b_h + z_{t-1} \circ (U_h h_{t-1} + b_{hh}))$$

This principled structure replaces heuristic recurrent computations with explicit probabilistic terms (Garner et al., 2019).

4. Two-Pass, Forward–Backward Inference

Activation-based recurrence enables bidirectional inference via two-pass smoothing, analogous to the Kalman smoother or Hidden Markov Model (HMM) forward–backward algorithm. The backward recursion for unit-wise BRU is:

$$h'_{t-1} = (1 - z_{t-1}) h_{t-1} + z_{t-1} h'_t, \quad h'_T = h_T$$

For layer-wise BRU, a backward gate $s_t$ modulates future influence:

$$h'_{t-1} = s_t \circ (U'_h h'_t + b'_h) + (1-s_t) \circ h_{t-1}, \quad h'_T = h_T$$

Unlike conventional bidirectional RNNs, bidirectional inference here emerges as a direct consequence of Bayesian smoothing rather than being an architectural add-on, allowing future observations to inform prior estimates in a principled manner (Garner et al., 2019).

5. Architectural Parameters and Empirical Evidence

A comparative summary of model variants is as follows:

Model	Parameter Count	Performance
Uni-BRU	$D$ hidden units; same as Uni-GRU	Outperforms Uni-GRU
Layer-wise BRU	Uni-BRU + $U_h$	Comparable to Bi-GRU
Bi-BRU (explicitly bidirectional)	2 × Uni-BRU	Bi-BRU ≳ Bi-GRU ≳ Bi-LSTM

Empirical results on TIMIT, WSJ, and AMI demonstrate that:

• Uni-BRU outperforms Uni-GRU of equivalent dimensionality.
• Layer-wise BRU matches Bi-GRU performance without explicit backward recurrence.
• Bi-BRU yields slightly lower phone error rates (e.g., 14.6% for Bi-BRU vs. 14.9% for state-of-the-art Bi-GRU on TIMIT).

This suggests that activation-based recurrence not only subsumes engineered architectures but offers parameter-efficient improvements (Garner et al., 2019).

6. Generalization and Theoretical Implications

The derivation of activation-based recurrence generalizes across choices of likelihood $P(X|f)$ and prior $P(f)$. Alternative conditional distributions (e.g., Gaussian $\to$ sigmoid, Beta $\to$ log-softplus) yield diverse link functions beyond the sigmoid/ReLU family. Extending to multi-class or continuous latent context variables enables richer gating behaviors. The backward pass, often engineered, here emerges as an intrinsic component of inference, not as an ancillary mechanism.

A plausible implication is that future RNN designs can be systematically founded on generative probabilistic models with recurrence, gates, and bidirectionality arising naturally from inference, moving beyond architecture-specific heuristics. This establishes a framework in which neural network operations can be tailored by the statistical properties of the data and task, grounding architectural decisions in principled probabilistic reasoning (Garner et al., 2019).