Horizontal Recurrence: Hidden State Methods

• Horizontal recurrence is a mechanism that updates internal hidden states sequentially to encode memory and sustain contextual information.
• Hidden state–based methods apply nonlinear transformations and state-space models for stable, efficient propagation of information over time.
• These methods drive applications in sequence modeling, video processing, reinforcement learning, and latent reasoning in modern neural architectures.

Horizontal recurrence refers to the propagation and evolution of information through a sequence by recurrent calculation over hidden states, typically structured along the “horizontal” (time or sequence) axis of a neural or dynamical system. Hidden state–based methods exploit internal states to aggregate, refine, and relay context through time (or layers), enabling systems to capture long-range dependencies, encode memory, and support structured reasoning. Such methods form a foundational substrate in neural network architectures, state-space models, probabilistic dynamical systems, and emerging latent reasoning paradigms. The following sections present a comprehensive survey of formal principles, architectures, methodologies, applications, and current research frontiers for horizontal recurrence and hidden state–based methods.

1. Mathematical Formulation of Horizontal Recurrence

Horizontal recurrence is classically instantiated in recurrent neural networks (RNNs), where a state variable $x_k$ is updated as a function of itself and new inputs at each time step $k$:

$x_{k+1} = A x_k + U h_k + W s_k + b$

$h_k = o(x_k)$

$y_k = V h_k + D s_k + c$

Here, $x_k$ is the state vector propagated horizontally, $h_k$ is a nonlinear hidden state, $s_k$ represents the current external input, and $A$ is often chosen to ensure stable (bounded) state evolution (1612.09022). This explicit state-space formulation illuminates the direct dependence of each state on its predecessor, formalizing horizontal recurrence.

In alternative settings such as master equation dynamics, the horizontal recurrence is encoded in the transition of the probability distribution over (hidden) states via a stochastic matrix $P$, with additional “hidden” states $\tilde{X}$ and timesteps introduced to implement otherwise intractable dynamics (1708.08494).

LLMs and advanced transformers now incorporate horizontal recurrence by maintaining and updating a compressed state $S_t$ across positions or timesteps:

$S_t = \gamma \cdot S_{t-1} + k_t v_t^\top$

where $\gamma$ is a decay factor and $k_t, v_t$ are projections of the current input (2507.06203).

2. Hidden State–Based Mechanisms: Dynamics and Recursion

Hidden state–based methods are characterized by the transformation and propagation of internal states that serve as reservoirs of context or “memory.” In bRNNs, the hidden state $h_k = o(x_k)$ provides a nonlinear summary that influences subsequent state updates via $U h_k$ (1612.09022). The stability of this system is assured by the matrix $A$, chosen such that its eigenvalues satisfy $|a| \leq 1$, preventing unbounded escalation of the state and ensuring fast dynamic response.

Extensions include layered or nested forms of recurrence, such as “recurrence-in-recurrence” architectures that introduce secondary recurrent operations over the sequence of hidden states $\{h_t\}$:

$\tau h_t = \text{IRM}(h_t, \tau h_{t-1})$

This enables the model to retain and process long-range dependencies beyond local temporal neighborhoods (2203.06418).

Particle filtering approaches model the hidden state as a distribution over multiple particles $\{h_t^i\}_{i=1}^K$, each following recurrent dynamics with added noise. The set of particles is updated via transition and measurement steps and resampled to approximate a posterior, yielding a richer uncertainty-aware hidden state representation:

$$\hat{h}_t^i = o_t^i \odot \tanh(C_t^i) + \varepsilon_t^i$$

(2212.09008).

3. Horizontal Recurrence in Structured Reasoning and Latent Computation

Recent research on latent reasoning positions horizontal recurrence as a mechanism for multi-step inference within the model’s hidden representations rather than overt token-level outputs. In these frameworks, state propagation occurs across time, layers, or “depth” in the hidden space, enabling the internalization of complex reasoning traces:

$$x_t^{l+1} = f(x_t^l, g((S_t^l, S_{t-1}^l, ..., S_{t-n}^l), x_t^l))$$

where the current state is a function of both the present and a context built from several prior hidden states (2507.06203). This approach supports the compression of explicit chain-of-thought traces into latent variables, improving representational efficiency and reasoning capacity.

Diffusion-based models further extend horizontal recurrence by iteratively refining the entire output via hidden states and masked noise removal, achieving globally consistent and reversible computation:

$x_t^{(l+1)} = f_t(x_t^l, S_t^l)$

(2507.06203).

4. Hidden State Mixing and State Space Models

State space models (SSMs) generalize horizontal recurrence by introducing continuous or discrete hidden states with prescribed transition and emission dynamics. The EfficientViM architecture, for example, operates by projecting input sequences into a reduced hidden state space and executing channel mixing operations within this space for efficiency:

$\mathbf{h} = (\mathbf{a} \mathbbm{1}_N^\top \odot \mathbf{B})^\top (\mathbf{x}_{\text{in}} \mathbf{W}_{\text{in}})$

$$\mathbf{x}_{\text{out}} \approx \mathbf{C}\left(\mathbf{h} \odot \sigma(\mathbf{h} \mathbf{W}_{\mathbf{z}})\right)\mathbf{W}_{\text{out}}$$

This approach allows successive HSM-SSD (Hidden State Mixer–State Space Duality) layers to refine hidden representations across network depth, a form of horizontal recurrence that optimizes for computational efficiency (2411.15241). EfficientViM’s multi-stage hidden state fusion integrates intermediate states for improved generalization.

5. Probabilistic State Transitions, Hidden Timesteps, and Space-Time Tradeoffs

Horizontal recurrence is not confined to deterministic or learning-based systems. In stochastic dynamical systems, such as those governed by master equations, hidden states and timesteps facilitate the realization of functional transitions that are otherwise unattainable. The embedding of auxiliary (hidden) states and the segmenting of a process into subintervals—where transition probabilities change—address the non-embeddability of certain functions:

• Hidden states $\tilde{X}$ enable indirect transitions in the master equation.
• A tradeoff is established: implementing a given function $f$ may require increasing the number of hidden states if the number of hidden timesteps is minimized, and vice versa (1708.08494).

These tradeoffs are quantified in terms of the function’s fixed points, image sizes, and orbit structures, providing theoretical lower bounds on system complexity.

6. Applications and Empirical Outcomes

Hidden state–based horizontal recurrence underpins a wide range of practical and theoretical advances:

• Time-series and sequence modeling: bRNNs demonstrate robust regression tracking with explicitly bounded trajectory evolution (1612.09022).
• Video deblurring: Recurrence-in-recurrence networks, incorporating inner-recurrence modules and attention-based temporal blending, achieve improved restoration of temporally dependent visual information (2203.06418).
• Meta-reinforcement learning: Hierarchical state-space models with disentangled global (task) and local (state) latent variables facilitate more efficient and adaptive policy training in partially observed environments (2105.06660).
• Uncertainty-aware prediction: Particle-filtered RNNs improve adaptability and accuracy in sequential forecasting tasks by maintaining and updating a latent state distribution (2212.09008).
• Resource-constrained inference: EfficientViM and related SSMs attain favorable speed-accuracy tradeoffs for vision tasks by confining complex operations to low-dimensional hidden state spaces (2411.15241).
• Latent reasoning in LLMs: Horizontal recurrence and propagation of continuous hidden states permit deep, token-free reasoning and scalable memory without linear growth in cache size (2507.06203).

7. Current Trends and Research Directions

Active research explores hybridizations of horizontal recurrence, such as combining “vertical” activation-based recurrence with explicit hidden state propagation; development of infinite-depth latent reasoning via masked diffusion; novel optimization approaches for managing and updating hidden states; and interpretability studies of multistage, multiscale information flow in deep architectures (2507.06203). Additional areas of interest include:

• Mechanistic analysis of gradient and state “leakage,” vanishing/exploding gradients, and their correction via co-state dynamics (1612.09022).
• Design of architectures that accumulate and fuse hidden state information across stages or layers to balance efficiency, expressiveness, and generalization (2411.15241).
• Application of space-time complexity theory to constrain and optimize dynamical system design when implementing sequential computations (1708.08494).

Overall, horizontal recurrence and hidden state–based methods form a unifying thread through both classical and contemporary models of sequence processing, adaptive reasoning, and dynamical system implementation, providing a principled framework for temporal context propagation, memory, and structured computation in artificial and natural learning systems.