Continuous-Depth EvoFormer

• The paper introduces a continuous-depth reformulation of the Evoformer via Neural ODEs, preserving core attention operations while significantly reducing memory and computational costs.
• It employs high-order numerical integration methods like RK4 and the adjoint sensitivity method to balance runtime-accuracy trade-offs during training and inference.
• Experimental results demonstrate linear scaling with protein size and competitive structural predictions, though with some limitations in fine-grained features such as loop packing.

The Continuous-Depth EvoFormer is a neural network architecture that reformulates the discrete, multi-block Evoformer—originally the backbone of AlphaFold—for protein structure prediction as a neural ordinary differential equation (Neural ODE). By replacing the stack of discrete blocks with a continuous-depth parameterization, this model preserves the core attention-based operations of the Evoformer while offering significant resource efficiency, adaptive computational trade-offs, and memory benefits. The adoption of continuous-in-depth principles from Neural ODEs and ContinuousNet allows the model to maintain structural plausibility in its predictions while enabling new avenues for biomolecular modeling and interpretation (Sanford et al., 17 Oct 2025, Queiruga et al., 2020).

1. Discretized Evoformer and the ODE Formulation

The standard Evoformer in AlphaFold comprises 48 stacked residual blocks each updating multi-sequence alignment (MSA) and pairwise representations. Each block applies operations such as row and column attention, outer-product-mean, triangle updates, and feed-forwards. Formally, the update at block $k$ is given by

$$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$

where $m_k$ and $z_k$ are the MSA and pairwise states.

This discrete stack, as the block index $k \to \infty$ with infinitesimal step size, converges to the ODE limit:

$$\frac{d s(t)}{dt} = f(s(t), t; \theta), \quad s(0) = (m_0, z_0), \quad s(1) \approx (m_{48}, z_{48}),$$

with continuous state $s(t) = (m(t), z(t))$ and vector field $f$ encapsulating Evoformer-style updates and depth-dependent learned gates $\sigma_m(t), \sigma_z(t) \in (0,1)$. This construction is a concrete example of ODE-ification as proposed by ContinuousNet (Queiruga et al., 2020), translating the blockwise residual dynamics into continuous flows.

2. Core Operations and Numerical Integration

The vector field $f(s(t), t; \theta)$ implements Evoformer-style module updates:

• Attention modules (e.g., MSA-row-attention with pair-bias): $$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$0, with $$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$1 and $$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$2.
• Feed-forward networks: standard 2-layer MLPs, $$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$3.
• Scalar gating functions $$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$4, parameterized as small MLPs in $$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$5, regulate the dynamic rates of the respective updates.

To obtain the output, the ODE is integrated:

$$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$6

using an explicit numerical solver. Experiments primarily employ fixed-step RK4, though adaptive solvers (e.g., Dormand–Prince) are supported, enabling runtime-accuracy trade-offs.

3. Training, Backpropagation, and Memory Efficiency

One of the principal advantages of the continuous-depth Evoformer is the use of the adjoint sensitivity method for backpropagation:

• Let $$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$7 be the loss; the adjoint state $$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$8 satisfies a backward ODE:

$$s_{k+1} = s_k + F_{\theta_k}(s_k), \quad s_k = (m_k, z_k),$$9

with boundary condition $m_k$0.

• Parameter gradients are accumulated as:

$m_k$1

• The forward states $m_k$2 need not be stored for every depth $m_k$3 due to recomputation by reintegration, yielding $m_k$4 memory in "depth," a substantial resource savings over discrete stacks, which scale memory with block count (Sanford et al., 17 Oct 2025, Queiruga et al., 2020).

4. Computational and Empirical Performance

Empirical results demonstrate resource and runtime advantages of the continuous-depth model:

• On a benchmark of 50 proteins (lengths 21–852), the Neural ODE Evoformer requires about 4.85 s per protein ($m_k$5 s/residue), compared to 65.06 s per protein ($m_k$6 s/residue) for OpenFold's 48-block trunk.
• Scaling laws for runtime as a function of residue count are approximately linear for the ODE model ($m_k$7), as opposed to quadratic for the discrete stack ($m_k$8).
• Structural prediction accuracy: the ODE Evoformer, trained in 17.5 hours on a single 8GB GPU, recovers $m_k$9-helices and global topology comparably to a 24-block discrete Evoformer with hidden size 64 (compared to (384,128) in AlphaFold). However, fine loop packing and certain $z_k$0-sheet features are degraded. Secondary structure confidence (pLDDT) remains high for major elements (Sanford et al., 17 Oct 2025).

5. Model Invariance, Adaptive Compute, and Manifestation Flexibility

The continuous-depth construction introduces several architectural and functional properties:

• Weight sharing: Depthwise parameterization is replaced by a single set of shared weights $z_k$1 for the entire flow, reducing parameter count and enforcing smooth transformations through depth.
• Adaptive integration: Switching numerical solvers (e.g., RK4 vs. Dormand–Prince) or modifying error tolerances ($z_k$2, $z_k$3) allows redistribution of computational effort, e.g., allocating more steps for harder proteins and fewer for easier ones, trading runtime for solution accuracy.
• Manifestation invariance: The trained model is invariant to the numerical scheme and depth discretization, i.e., one can trade between compute and memory at inference or training time by varying step size $z_k$4, number of steps $z_k$5, or integration method (Euler, RK2, RK4).
• Incremental-in-depth training: A mesh refinement schedule starts with coarse depth discretizations (small $z_k$6) and iteratively refines both step size and parameter basis, accelerating convergence and reducing training time with no deterioration in final accuracy (Queiruga et al., 2020).

6. Methodological Details and Practical Recommendations

Continuous-Depth Evoformer construction follows the ODEBlock recipe:

• Represent state $z_k$7 and parameterize weights $z_k$8 with a low-dimensional basis $z_k$9 and basis functions $k \to \infty$0, ensuring the mapping to the discrete case as a limit.
• Integrate $k \to \infty$1 using explicit schemes. For stability, select high-order integrators (e.g., RK4), scale residuals by a small $k \to \infty$2 to ensure $k \to \infty$3, and select initial $k \to \infty$4–$k \to \infty$5, refining to $k \to \infty$6–64.
• Adjoint-based backpropagation is recommended for memory efficiency; checkpointing and coarse refinement are fallback options when sensitivity equations become unstable (Queiruga et al., 2020).

Practical benefits include rapid training schedules, controlled accuracy-computation trade-offs, and the ability to inspect the learned vector field $k \to \infty$7 for interpretability in terms of evolutionary-geometric refinement stages.

7. Impact and Implications

The Continuous-Depth Evoformer demonstrates that the evolutionary and geometric feature refinement in protein structure prediction can be recast as integration under a smooth vector field, leveraging Neural ODEs for flexible, memory-efficient, and adaptive computation. Computational efficiency is notably improved: the ODE-based model achieves dramatic resource reduction compared to discrete stacks, making deployment feasible on limited hardware without a severe penalty to overall representing capability in key secondary structure elements (Sanford et al., 17 Oct 2025).

Architecturally, the model invites avenues for interpretability, e.g., by examining time-dependent generator dynamics, and offers explicit mechanisms to adapt inference cost via solver choices—a feature infeasible in rigidly stacked discrete networks. A plausible implication is that continuous-depth paradigms could generalize to other attention-based architectures and domains where compute-adaptive, memory-limited, or interpretable deep learning pipelines are desirable. This establishes Continuous-Depth Evoformer as a foundational contribution to the interface of structured biological modeling and continuous deep dynamical systems (Sanford et al., 17 Oct 2025, Queiruga et al., 2020).