---
title: 'Bi-RWKV Layers: Bidirectional RWKV Extensions'
url: https://www.emergentmind.com/topics/bi-rwkv-layers
type: topic
---

# Bi-RWKV Layers: Bidirectional RWKV Extensions

Bi-RWKV layers are architectural extensions of the RWKV model family that integrate bidirectional or pseudo-bidirectional context aggregation into the original recurrent-weighted key-value (RWKV) paradigm. These modifications appear predominantly in two orthogonal forms: (1) Bi-WKV blocks as proposed for sequence modeling in audio ("AudioRWKV: Efficient and Stable Bidirectional RWKV for Audio Pattern Recognition" [2509.02167]), and (2) the triplet-block layout for pseudo-bidirectional discrete diffusion ("Triplet-Block Diffusion RWKV" [2605.25969]). Both enable various forms of global or blockwise conditioning in a linear-time, stable, and memory-efficient manner.

## 1. Mathematical Foundations of RWKV and Bidirectionalization

RWKV models construct O(L)-time sequence processors via recurrent key-value recurrences, parameterized by dynamic decay factors, keys, and values computed from per-token projections, plus a receptance gating vector. In the original formulation, the sequence is scanned left-to-right only, enforcing causal dependency.

**Bidirectional RWKV** introduces two variants:

- **Bi-WKV Operator**: Two full WKV recurrences—one forward ($t = 1 \dots L$) and one backward ($t = L \dots 1$)—are computed. The outputs are fused per position using a dynamic gate $G_t$ derived from local features. This yields a layer output $p^{\text{fused}}_t = G_t \circ p^{\rightarrow}_t + (1 - G_t) \circ p^{\leftarrow}_t$, where $\circ$ denotes elementwise multiplication, and $G_t \in [0,1]^d$ [2509.02167].

- **Triplet-Block Pseudo-Biaccess ("Biaccess")**: The input sequence is chunked into logical blocks. For each, three physical blocks are created: (1) masked, (2) masked (predictive; loss applied), (3) clean (hidden state reset). The causal model processes these left-to-right, but because unmasked tokens from block 1 are already absorbed into the hidden state before block 2 is encountered (where loss is computed), the effective context for each predicted token within block 2 is bidirectional relative to the logical block, despite preservation of strict causality at the model level [2605.25969].

## 2. Bi-WKV Layers: Formulation and Forward/Backward Recurrence

The Bi-WKV operator is formalized as follows [2509.02167]:

- For each time step $t$ and LayerNorm-normalized input $x_t$,
  1. Compute dynamic vectors: $w_t$ (decay log-rate), $b_t$ (time offset), $k_t$ (key), $v_t$ (value), $r_t$ (receptance gate).
  2. Forward pass ($t=1\dots L$):
     - Maintain $S^{(K)}_t = \beta_t \circ S^{(K)}_{t-1} + k_t$
     - $S^{(V)}_t = \beta_t \circ S^{(V)}_{t-1} + k_t \circ v_t$
     - $wkv^\rightarrow_t = S^{(V)}_t \circ (S^{(K)}_t)^{-1}$
     - $p^\rightarrow_t = r_t \circ wkv^\rightarrow_t$
  3. Backward pass ($t=L\dots 1$) with identical recurrences, yielding $p^\leftarrow_t$.
  4. Fuse with $G_t$ from a depthwise separable convolutional "convshift" local residual: $p^{\text{fused}}_t = G_t \circ p^\rightarrow_t + (1 - G_t) \circ p^\leftarrow_t$, where $G_t = \sigma(W^g x_{\text{res},t} + b^g)$.

This mechanism maintains O(L·d) time complexity and two d-dimensional recurrent states per scan, making the bidirectional extension computationally efficient and stable. The use of double-exponential decay and S_V/S_K normalization ensures numerical stability at long time scales.

## 3. Triplet-Block Layout for Pseudo-Bidirectional Diffusion

The triplet-block mechanism ({*editor's term*: pseudo-biaccess}) was introduced to unify linear-time causal backbones with the parallel, bidirectional context required for discrete diffusion modeling [2605.25969]. Given a sequence of length $L$:

- Divide input into $N = L/B$ logical blocks of length $B$.
- For each logical block $g^{(i)}$, create three physical blocks:
  - $b_1^{(i)}$: Masked (no loss)
  - $b_2^{(i)}$: Same masked copy (loss computed where not masked)
  - $b_3^{(i)}$: Clean (resets the hidden state)
- Apply randomly sampled mask pattern $m^{(i)}$ per block, including full-mask and forced EOS/PAD positions.
- Because the model operates strictly left-to-right, by the start of $b_2^{(i)}$ the hidden state has already integrated unmasked tokens from $b_1^{(i)}$, providing the lossable block with access to both left and right unmasked context within $g^{(i)}$.
- At inference, a block-level iterative token commitment process initializes all B positions as masked and commits tokens by confidence threshold $\tau$, in a MaskGIT-inspired regime.

There are no modifications to the RWKV cell, hidden size, or weights; only the sequencing of physical input adapts, resulting in bidirectional conditioning at the block level while maintaining O(L) computational complexity.

## 4. Comparison of Approaches: Bi-WKV Versus Triplet-Block Biaccess

| Aspect                     | Bi-WKV (AudioRWKV)                              | Triplet-Block (B³D-RWKV)                    |
|----------------------------|------------------------------------------------|---------------------------------------------|
| Directionality             | True bidirectional (full scan both directions)  | Pseudo-bidirectional within blocks          |
| Computational Cost (per L) | O(2L·d), still linear in L                      | O(L), data-level overhead (3x at training)  |
| Model Parameters           | Adds fusion gate, convshift residual            | No change (all weights untouched)           |
| Context Aggregation        | Global, per-layer                               | Block-local, per training arrangement       |
| Primary Application        | Audio sequence modeling                         | Discrete diffusion language modeling        |

Bi-WKV provides explicit bidirectional context by running separate forward and backward passes and fusing results, while the triplet-block layout arranges inputs such that the causal model’s hidden state delivers blockwise bidirectionality implicitly.

## 5. Empirical Performance and Complexity

The Bi-WKV approach, including fusion via a learnable sigmoid gate and local convshift residual, yields the following results on the AudioSet-2M mAP task [2509.02167]:

- Causal RWKV7 baseline: 34.5 mAP
- Adding bidirectional scan (average fusion): 38.4 mAP
- Weighted gate fusion: 39.0 mAP
- Full ConvShift + Bi-WKV: 40.9 mAP

AudioRWKV’s RWKV7 operator with Bi-WKV in spatial mixing provides up to a 13.3× speedup over FlashAttention on a 4090 GPU with long ($\sim$5 min) audio, with throughput remaining flat as sequence length increases.

For B³D-RWKV [2605.25969], throughput experiments on H100 with a 32-layer d=4096 RWKV-7-g1f-7.2B backbone in 8-GPU training demonstrate:

- B³D-RWKV-7.2B achieves decoding throughput at $\sim$1.6× that of a standard causal RWKV-7-7.2B at comparable quality, reaching up to 2.0× with fewer inference steps or lower confidence thresholds.

Both methods attain substantial gains in bidirectional context efficiency and accuracy compared to their causal RWKV counterparts.

## 6. Implementation Notes and Stability

For Bi-WKV [2509.02167]:

- All bidirectional extensions follow numerically stable RWKV7 recurrences, leveraging double-exponential decay ($\alpha_t = \exp(-\exp(w_t))$) for boundedness.
- The fusion gate $G_t$ is obtained from a convolutional residual, introducing minimal instability. No new unstable operators are introduced.
- Training and inference memory requirements remain linear in sequence length, with minor constants due to dual scans and gating.

For triplet-block B³D-RWKV [2605.25969]:

- The approach is parameter-free at the architectural level. No new layers, parameters, or hidden state representations are involved.
- All weight matrices and state transitions remain as in the vanilla RWKV backbone.
- The only modification occurs at data layout: token sequencing and masking regime.
- During training, $3L$ physical tokens per sequence yields a 3× overhead, still linear in $L$; at inference amortized cost per block is $≪3B$ due to token commitment shortcuts.

## 7. Applications and Broader Impact

Bi-RWKV layers extend the applicability of the RWKV architecture to tasks requiring global or block-local bidirectional context. Bi-WKV is primarily used for audio pattern recognition, where full-sequence context is critical and efficient long-range modeling is required. The triplet-block system enables linear-time, parallelizable, discrete diffusion for language modeling, closing the gap between diffusion-based bidirectional inference and causally trained backbone models, while preserving the efficiency of RWKV.

These innovations enable high-throughput, stable, and generalizable modeling for both language and audio domains, leveraging bidirectional context without incurring quadratic attention cost or destabilizing recurrent recurrences. The Bi-WKV and triplet-block pseudobiaccess approaches thus represent significant advances in efficient sequence modeling architectures [2509.02167, 2605.25969].

Source: https://www.emergentmind.com/topics/bi-rwkv-layers