Recurrence-Complete Architecture

• Recurrence-complete architecture is a neural model design that serially computes arbitrary recurrent functions, integrating non-associative dependencies over long sequences.
• It leverages dynamic cell composition, explicit memory integration, and recursive processing to handle tasks that require true depth and long-horizon state tracking.
• Empirical findings reveal that these architectures scale linearly with sequence length and outperform parallel models in persistent agentic and planning scenarios.

A recurrence-complete architecture is a computational framework or neural model capable of representing and computing arbitrary recurrent update functions over sequential data, where each update may depend non-associatively on the entire chain of previous states and inputs. This property enables such architectures to achieve the “true depth” of sequence modeling—scaling serial computation linearly with sequence length and ensuring the capacity to aggregate and transform information across arbitrarily long time horizons. Recurrence-completeness is distinguished from architectures relying fully on parallel (attention or scan-like) mechanisms, which are theoretically and empirically shown to lack the required serial dependency to solve certain classes of long-running agentic and planning tasks (Keiblinger, 8 Oct 2025). Key approaches to recurrence-complete architectures leverage Turing-complete computation schemes, dynamic cell composition, explicit memory integration, and carefully designed recursive processing and training schedules.

1. Fundamental Properties and Theoretical Criteria

A recurrence-complete architecture must realize recurrent updates of the form:

$h_t = g(h_{t-1}, h_{t-2}, \ldots, h_{t-k}, x_t)$

where $g(\cdot)$ represents an arbitrary, potentially non-associative transformation. The defining characteristic is that computation proceeds strictly serially across time, requiring each state $h_t$ to be computed after all prior states, resulting in a dependency chain of depth $\Omega(n)$ for sequence length $n$. As a consequence, full forward and reverse (backpropagation) passes incur at least linear computational depth with respect to the input sequence.

Parallel or scan-based architectures (including attention models and state-space models) perform input aggregation or state computation in sublinear or constant depth: such structures provably cannot capture the full class of recurrent functions. Formal proofs establish that any architecture allowing fully parallel forward or backward evaluation lacks recurrence-completeness, as non-associative dependencies cannot be correctly reconstructed in such cases (Keiblinger, 8 Oct 2025).

2. Critical Time and Limitations of Non-Recurrence-Complete Models

A central conjecture posits the existence of a “critical time” $t_{crit}$ beyond which non-recurrence-complete models fail to aggregate sequential inputs correctly (Keiblinger, 8 Oct 2025). Explicitly, for tasks requiring $n_{taskops}$ true sequential operations and a model providing only $c \cdot L$ sequential computations (for constants $c, L$), failure occurs when $n_{taskops} > c \cdot L$. This failure manifests in agentic tasks where continual side-effect aggregation or long-horizon state tracking is essential; for example, software engineering agents tracking codebase evolution or shell interaction sequences. Once the capacity threshold is exceeded, attention-based aggregation mechanisms misrepresent or forget critical state information, limiting utility in persistent or planning scenarios.

3. Architectural Realization: Frame-Based Action Models

The recurrence-complete frame-based action model integrates both parallel and serial modules:

• Frame-head module: Each input frame (e.g., a 2D terminal screen) is embedded by a transformer augmented with intra-frame pooling. This design exploits parallel computation for spatial encoding.
• Sequential integration via LSTM stack: Temporal aggregation is carried out by a stack of LSTM (or equivalent strictly recurrent) cells operating token-by-token over frame embeddings. The LSTM module is essential for recurrence-completeness, ensuring that each temporal update serially incorporates all prior state information.

Training utilizes full backpropagation through time (BPTT), often with recompute-on-the-fly or streaming schedules to limit activation memory to $O(1)$ per step. The resulting architecture is capable of recursively processing arbitrary action sequences over long horizons, as shown by empirical application to GitHub-derived session data (Keiblinger, 8 Oct 2025).

Data Pipeline for Agentic Tasks

• Automatic session reconstruction from version control diffs
• Lossless rendering into compressed “termstreamxz” format for frames
• Action association by labeling keystrokes or terminal commands per frame
• BPTT optimization over action prediction with sequence streaming

4. Empirical Characterization and Scaling Laws

Empirically, training loss as a function of sequence length $L$ obeys a power-law relationship for recurrence-complete architectures:

$$\text{loss}(L | s) \approx A(s) \cdot L^{-\alpha(s)}$$

where $A(s)$ and exponent $\alpha(s)$ evolve dynamically with training steps $s$, often saturating as $\alpha(s) = \alpha_\infty (1 - e^{-s/\tau})$. Notably, at fixed parameter count, longer sequences always amortize their linearly-increasing wall-time cost: even though update time grows with $L$, overall loss as a function of wall-time decreases, making the use of recurrence-complete architectures preferable for tasks demanding ever-longer context integration (Keiblinger, 8 Oct 2025).

5. Contrast with Attention-Based and Parallel Models

Transformer architectures and state-space models rely on parallel aggregation and associative computations across sequence elements. These models, while achieving scalability and successful application to many NLP tasks, are demonstrated—via both theoretical proof and empirical evidence—to be incapable of representing general non-associative recurrent update functions (Keiblinger, 8 Oct 2025). Their “true depth” is bounded by constant or sublinear scaling, limiting their effectiveness for long-running agentic, persistent memory, or planning-based problems. In agentic domains, attention-only aggregation reaches $t_{crit}$, beyond which latent states are incorrectly aggregated, and sequential dependencies required for correct decision-making are lost.

6. Application Domains and Implications

Recurrence-complete architectures are essential for:

• Software engineering agents and domains requiring long-horizon latent state tracking
• Persistent stateful control tasks (e.g., shell or browser automation, navigation in structured environments)
• Planning and search in domains with side effects where non-associative sequential integration of inputs is required

The architectural insights highlight the necessity of strictly serial recurrent computation for tasks where input aggregation and latent state depend on a dynamic, input-length proportional sequence of operations. Models lacking recurrence completeness may succeed on tasks with short context windows but will systematically fail as required temporal depth increases.

7. Future Directions and Theoretical Significance

The theory underlying recurrence-complete architectures prompts reevaluation of dominant paradigms in sequential modeling. Future research may explore:

• Hybrid architectures: Integrating parallel frame-level modules with strictly serial recurrences for efficient long-horizon modeling.
• Improved training protocols: Efficient recompute-on-the-fly schedules to balance memory use and serial depth.
• Algorithmic amortization studies: Quantifying trade-offs between wall-time, context length, and information aggregation in persistent agents.

Work in this domain provides foundational understanding of the algorithmic limits and design principles required for agentic systems, persistent memory architectures, and deep sequential reasoning in neural models. The necessity for recurrence-completeness reframes the limitations of current attention-centric models and establishes a blueprint for architectures capable of truly long-term sequential integration (Keiblinger, 8 Oct 2025).