Evolving WKV Attention in DRWKV

• Evolving WKV Attention is a geometry-aware modification of RWKV that replaces conventional raster scanning with an Archimedean spiral to preserve edge continuity.
• It integrates receptance gating with a topology-preserving scanning mechanism to efficiently capture spatial features during low-light detail mining.
• Empirical results show significant improvements in SSIM and PSNR, underlining its effectiveness in edge preservation and fine-detail recovery.

Searching arXiv for the cited RWKV-family papers and closely related work to ground the article in the current literature. Evolving WKV Attention denotes a geometry-aware modification of the RWKV family’s weighted key-value accumulation in which the sequence order itself is altered to better preserve spatial structure. In the specific usage introduced by DRWKV, the mechanism replaces ordinary visual RWKV scanning with an Archimedean-spiral traversal so that receptance-gated WKV aggregation follows edge continuity rather than a generic raster order (Bai et al., 24 Jul 2025). More broadly, it belongs to a larger line of RWKV research in which the original WKV recurrence is progressively extended through retrospective summaries, matrix-valued state updates, bidirectional aggregation, recurrent visual scanning, and sparse long-range retrieval, while retaining the efficiency objective that differentiates RWKV from quadratic self-attention (Datta, 2024).

1. WKV attention as the substrate of RWKV evolution

RWKV, short for Receptance Weighted Key Value, is designed to combine parallelizable training with recurrent inference by reformulating attention into a linear-time weighted key-value mechanism rather than an explicit $QK^\top$ similarity matrix (Datta, 2024). In the review formulation, the parallel WKV operator over a sequence of length $T$ is

$$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$

and the sequential inference form is

$a_t = \exp(w)a_{t-1} + \exp(k_t)$

$b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$

$\text{WKV}_t = \frac{b_t}{a_t}.$

These equations encode a normalized exponentially decayed weighted average rather than all-pairs token interaction. The recurrent state therefore stores compressed history in constant-size accumulators, which gives constant-time update per token at inference and linear complexity in sequence length during training (Datta, 2024).

Receptance is the complementary gating component. In the review’s channel-mixing formulation,

$$\text{ChannelMix}(x_t) = \sigma(W_r x_t) \odot (W_v \phi(W_k x_t)),$$

where $\sigma$ is a sigmoid gate and $\phi$ is often $\text{ReLU}(z)^2$ (Datta, 2024). This gate determines how much of the accumulated key-value information is exposed downstream. Token shifting further injects local context through

$T$0

which gives RWKV blocks an explicitly local temporal branch alongside the recurrent WKV accumulation (Datta, 2024).

A useful theoretical interpretation is that WKV can be rewritten as an implicit causal self-attention operator rather than treated as a completely separate primitive. In the unified implicit-attention formulation, RWKV’s time-mixing block becomes

$T$1

where $T$2 is a lower-triangular, data-dependent attention matrix induced by exponential decay, token content, token shift, and gating rather than by explicit query-key dot products (Zimerman et al., 2024). This interpretation is central to later “evolutions” of WKV, because it clarifies that changing scan order, state dynamics, or auxiliary retrieval pathways changes the induced attention structure even when the model remains recurrent.

2. Early architectural evolutions: longer paths, richer states, stronger retrieval

Several RWKV-family papers modify WKV by changing how historical information is routed rather than by reverting to full Transformer attention. RRWKV is a direct example. It keeps RWKV’s recurrent backbone but inserts intermediate tokens called mediums, $T$3, at regular intervals:

$T$4

with $T$5 initialized as a zero-like sentry token (Wang, 2023). Each medium summarizes a chunk of preceding tokens and is recalibrated through

$T$6

RRWKV then modifies the channel-mix pathway so that the model interpolates between the current token state and a corresponding medium:

$T$7

$T$8

$T$9

The paper’s explicit claim is that mediums shorten the maximum path length and provide a more fluent route to distant information while keeping complexity near-linear at $$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$0 (Wang, 2023).

Later language-model variants modify the recurrent state itself. ARWKV presents RWKV-7 as “native RWKV attention,” with a matrix-valued update

$$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$1

contrasted with the simpler RWKV-6 recurrence

$$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$2

The paper interprets the RWKV-7 state as a matrix-valued attention state and states that $$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$3 is the in-context learning rate (Yueyu et al., 26 Jan 2025). It further claims wider eigenvalues, stronger state tracking than Transformers, and perfect 16k passkey retrieval in a 0.1B model (Yueyu et al., 26 Jan 2025). These claims are presented as ongoing work, but they mark an important shift from scalar or vector decay toward more structured state dynamics.

RWKV-X evolves WKV in a different direction. Rather than only altering the recurrence, it keeps RWKV-7 blocks for short-range modeling and periodically inserts sparse-attention blocks for long-range retrieval (Hou et al., 30 Apr 2025). The recurrent backbone is still

$$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$4

but the model adds Top-$$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$5 Chunk Sparse Attention over chunk-pooled keys:

$$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$6

$$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$7

$$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$8

RWKV-X therefore supplements recurrent compression with explicit content-based retrieval, while claiming training complexity $$\text{WKV}(K, V, W) = \frac{\sum_{i=1}^{T} \exp(k_i - (T-i)w) v_i}{\sum_{i=1}^{T} \exp(k_i - (T-i)w)}$$9 and decoding complexity $a_t = \exp(w)a_{t-1} + \exp(k_t)$0 (Hou et al., 30 Apr 2025).

Variant	Mechanism added to WKV/RWKV	Stated objective
RRWKV	Medium tokens with squeeze and recalibration	Shorten path length and improve long-range dependency capture
ARWKV / RWKV-7	Matrix-valued recurrent state with correction term	Increase expressiveness and state tracking
RWKV-X	Periodically inserted Top-$a_t = \exp(w)a_{t-1} + \exp(k_t)$1 chunk sparse attention	Recover explicit long-range retrieval while keeping linear/constant-time efficiency

Taken together, these developments show that “evolution” in WKV research often means altering the information path around the recurrent accumulation rather than discarding the recurrent accumulation itself.

3. From 1D causality to 2D global context: bidirectional and recurrent visual WKV

Adapting WKV to images requires more than applying a text-oriented causal scan to flattened pixels or patches. The RWKV review notes that visual variants extend token shifting to 2D and introduce bidirectional attention of the form

$a_t = \exp(w)a_{t-1} + \exp(k_t)$2

because spatial context is not naturally one-directional (Datta, 2024). The same review also identifies Restore-RWKV as introducing Re-WKV through repeated bidirectional application,

$a_t = \exp(w)a_{t-1} + \exp(k_t)$3

alongside Omni-Shift for 2D neighborhood aggregation (Datta, 2024). This establishes a recurrent bidirectional template for later visual RWKV variants.

StyleRWKV develops this line explicitly for image style transfer. Its central modification, Re-WKV, is described as a recurrently applied bidirectional attention operator embedded in Skip Scanning, with base bidirectional aggregation

$a_t = \exp(w)a_{t-1} + \exp(k_t)$4

and recurrent update

$a_t = \exp(w)a_{t-1} + \exp(k_t)$5

with $a_t = \exp(w)a_{t-1} + \exp(k_t)$6 (Dai et al., 2024). The stated purpose is to create a global receptive field on images while preserving linear time and memory.

StyleRWKV couples Re-WKV to S-Scanning, which reorganizes image patches using step size $a_t = \exp(w)a_{t-1} + \exp(k_t)$7 and an atrous-like traversal:

$a_t = \exp(w)a_{t-1} + \exp(k_t)$8

with

$a_t = \exp(w)a_{t-1} + \exp(k_t)$9

The paper reports an ablation on recurrence number: $b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$0 gives ArtFID 27.783, FID 18.128, LPIPS 0.593, Time 0.205 s; $b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$1 gives ArtFID 26.370, FID 16.362, LPIPS 0.451, Time 0.213 s; and $b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$2 gives ArtFID 25.639, FID 15.442, LPIPS 0.448, Time 0.236 s (Dai et al., 2024). The paper therefore selects $b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$3 as the best trade-off.

These visual adaptations show a recurring design pattern. WKV is preserved as a recurrent weighted key-value operator, but the sequence order, directionality, and recurrence schedule are redesigned so that the induced attention better matches 2D structure. Evolving WKV Attention in DRWKV extends exactly this pattern, but with an edge-focused topology rather than the global stylization objective of StyleRWKV.

4. Evolving WKV Attention in DRWKV

In DRWKV, Evolving WKV Attention is the spatial modeling mechanism of the Deep Detail Mining stage, introduced to preserve object boundaries, continuous edges, and fine structures under severe low-light degradation (Bai et al., 24 Jul 2025). The paper argues that prior VRWKV-style architectures suffer from mismatch with hierarchical edge features and poor geometric modeling of edges, because edges are not well represented as uniformly distributed Euclidean features when low-light contours are better viewed as curved manifolds in a Riemannian space (Bai et al., 24 Jul 2025).

The mechanism is built on Evolving Scanning, a spiral sequence construction defined by the Archimedean spiral

$b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$4

Here $b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$5 is the initial radius, $b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$6 is the spiral expansion rate, and $b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$7 is the angular parameter (Bai et al., 24 Jul 2025). The scan expands gradually from local to global regions. The paper further extends this to a four-directional spiral system starting from the four corners of the feature map and supporting both clockwise and counterclockwise traversal, with the stated purpose of reducing directional bias and adapting to different edge layouts (Bai et al., 24 Jul 2025).

A topology-preserving constraint formalizes the continuity objective. For any two edge points $b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$8, if

$b_t = \exp(w)b_{t-1} + \exp(k_t)v_t$9

then their 1D sequence positions under spiral traversal should satisfy

$\text{WKV}_t = \frac{b_t}{a_t}.$0

The intended meaning is that nearby points on the same edge should remain nearby after serialization, so that RWKV’s sequential accumulation can follow contour continuity rather than destroy it through an unsuitable scan order (Bai et al., 24 Jul 2025).

Within the ES-RWKV block, the spatial mix first applies Q-Shift to produce receptance, key, and value projections:

$\text{WKV}_t = \frac{b_t}{a_t}.$1

with channel-partitioned shifted feature

$\text{WKV}_t = \frac{b_t}{a_t}.$2

The output of the evolving spatial mix is then

$\text{WKV}_t = \frac{b_t}{a_t}.$3

This retains the RWKV principle of receptance-gated weighted key-value accumulation, but the accumulation is performed over the spiral sequence rather than a conventional linear order (Bai et al., 24 Jul 2025). In that narrow technical sense, Evolving WKV Attention is not a new all-pairs attention primitive; it is an evolution of WKV in which the scan topology is redesigned to better align recurrent aggregation with edge geometry.

5. Position within the DRWKV pipeline and reported empirical effects

DRWKV organizes its method around three components: Global Edge Retinex (GER), Evolving WKV Attention, and Bilateral Spectrum Aligner with MS²-Loss (Bai et al., 24 Jul 2025). GER provides the decomposition model

$\text{WKV}_t = \frac{b_t}{a_t}.$4

with refined reflectance

$\text{WKV}_t = \frac{b_t}{a_t}.$5

Here $\text{WKV}_t = \frac{b_t}{a_t}.$6 is reflectance, $\text{WKV}_t = \frac{b_t}{a_t}.$7 is the edge feature term, $\text{WKV}_t = \frac{b_t}{a_t}.$8 is illumination, $\text{WKV}_t = \frac{b_t}{a_t}.$9 is spatially heterogeneous noise, $$\text{ChannelMix}(x_t) = \sigma(W_r x_t) \odot (W_v \phi(W_k x_t)),$$0 is artifact residual, and $$\text{ChannelMix}(x_t) = \sigma(W_r x_t) \odot (W_v \phi(W_k x_t)),$$1 are weights (Bai et al., 24 Jul 2025). Evolving WKV Attention is the spatial operator that the paper uses to extract and preserve the edge term $$\text{ChannelMix}(x_t) = \sigma(W_r x_t) \odot (W_v \phi(W_k x_t)),$$2 during detail mining.

The overall architecture has a Light Preprocessing stage and a Deep Detail Mining stage. Light Preprocessing estimates noise, illumination, and reflectance through

$$\text{ChannelMix}(x_t) = \sigma(W_r x_t) \odot (W_v \phi(W_k x_t)),$$3

while Deep Detail Mining contains ES-RWKV blocks, channel mix, Bi-SAB, and SEE (Bai et al., 24 Jul 2025). The paper states that Evolving WKV is used “for the first time” with wavelet downsampling to extract edge gradient features in this stage (Bai et al., 24 Jul 2025).

The reported computational profile is lightweight by the standards discussed in the paper. DRWKV uses only 1.67 GFLOPs in the main benchmark table, and 8.28M parameters in the ablation table (Bai et al., 24 Jul 2025). The paper attributes the efficiency to retaining RWKV’s linear-time, recurrent-style computation while improving the scan order for edge structures.

The ablation evidence is reported at the level of ES-RWKV rather than a fully isolated EV-WKV row. On LOLv2-Real, the baseline gives SSIM = 0.415 and PSNR = 12.57 dB; adding ES-RWKV gives SSIM = 0.591 and PSNR = 16.27 dB; further additions eventually yield SSIM = 0.832 and PSNR = 24.12 dB (Bai et al., 24 Jul 2025). The paper explicitly states that after embedding Evolving Scanning / ES-RWKV, SSIM improves by 42.4% and PSNR improves by 29.4% (Bai et al., 24 Jul 2025). Qualitatively, the paper reports more continuous object edges, more stable contours, better preservation of fine details, and less blurring and fewer false edge breaks.

These results do not separate the spiral sequence from every other component in the block, but they do provide direct support for the integrated ES-RWKV design. A plausible implication is that the paper’s main empirical claim concerns the compatibility of spiral scanning with edge topology rather than a claim that scan order alone explains all of DRWKV’s gains.

6. Interpretation, misconceptions, and relation to adjacent RWKV research

A common misconception is that Evolving WKV Attention replaces RWKV with Transformer-style self-attention. The available formulations do not support that interpretation. In DRWKV, the aggregation remains receptance-gated weighted key-value accumulation,

$$\text{ChannelMix}(x_t) = \sigma(W_r x_t) \odot (W_v \phi(W_k x_t)),$$4

and the paper explicitly contrasts this with quadratic self-attention

$$\text{ChannelMix}(x_t) = \sigma(W_r x_t) \odot (W_v \phi(W_k x_t)),$$5

to emphasize lower computational cost (Bai et al., 24 Jul 2025). The modification is in sequence topology, not a return to dense all-pairs similarity.

Another misconception is that “evolving” refers only to a better recurrence equation. Across the RWKV literature, evolution has taken several distinct forms. RRWKV evolves context routing through inserted mediums (Wang, 2023); ARWKV evolves the recurrent state into a more expressive matrix-valued attention memory (Yueyu et al., 26 Jan 2025); StyleRWKV evolves WKV into recurrent bidirectional image aggregation under structured scan paths (Dai et al., 2024); RWKV-X evolves the architecture by adding sparse long-range retrieval blocks (Hou et al., 30 Apr 2025). Evolving WKV Attention in DRWKV belongs to this family of modifications, but its specific novelty is geometric serialization of edge structure rather than sparse retrieval or teacher-distilled state tracking.

The implicit-attention perspective provides a useful unifying lens. If RWKV-style models can be rewritten as $$\text{ChannelMix}(x_t) = \sigma(W_r x_t) \odot (W_v \phi(W_k x_t)),$$6 with a data-controlled lower-triangular operator, then changing scan order, introducing bidirectionality, or adding chunk selection effectively changes the induced attention structure without abandoning recurrence (Zimerman et al., 2024). This suggests that WKV evolution can be understood as the progressive design of better implicit attention operators under linear or near-linear constraints.

The broader RWKV review also frames several open problems that remain relevant here: theoretical understanding is incomplete; scaling limits are unclear; domain-specific tuning is needed; and robustness, interpretability, and hardware support remain open research areas (Datta, 2024). For Evolving WKV Attention specifically, this suggests two immediate questions. First, whether topology-aware scan design can be formalized beyond the heuristic constraints already given. Second, whether edge-aware serialization generalizes to other visual tasks involving irregular structures, such as restoration or segmentation. The existing papers do not answer these questions directly, but they place Evolving WKV Attention within an increasingly diverse family of recurrent implicit-attention mechanisms that trade quadratic connectivity for carefully engineered memory paths.