Papers
Topics
Authors
Recent
Search
2000 character limit reached

TreeWY: Speculative Verification for Gated DeltaNet Hybrids

Published 21 Aug 2026 in cs.AI, cs.CL, cs.DC, cs.LG, and cs.PF | (2608.20961v1)

Abstract: Modern open models are hybrids: most layers are linear-attention (Gated DeltaNet, GDN) layers carrying a small fixed-size recurrent state instead of a growing key-value (KV) cache. This makes ordinary decoding memory-efficient, but hurts speculative decoding. To verify a batch of draft tokens and then roll back the rejected ones, today's systems snapshot the full recurrent state at every draft position for GDN layers, and those snapshots cannot be shared across branches of a draft tree, so a wide, high-acceptance tree becomes memory-infeasible. We remove the snapshots. Using a tree-structured WY transform of the gated delta rule, we compute every draft node's output with a single triangular solve and reconstruct only the one accepted state on commit, storing a small pseudo-value matrix instead of per-node states; the derivation depends only on the gated delta rule, not on any other architectural detail. In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B) this cuts speculative recurrent-state memory and KV-cache pressure at identical acceptance length, turning the freed HBM into higher throughput and much lower time-to-first-token (TTFT) wherever memory binds, and costing a few percent where it does not. For tree width the same memory buys affordability: a wider, higher-acceptance draft becomes possible, though not yet a throughput win.

Authors (1)
Citations (3)

Summary

  • The paper introduces TreeWY, a technique that eliminates per-draft-position recurrent-state snapshots in speculative verification for Gated DeltaNet (GDN) hybrids.
  • Evaluated on two configurations of Qwen, TreeWY reduces peak KV memory usage by 2-3 times.
  • Depending on memory constraint level,Time between first and last token of a sequence for message processing tries at reduced the time by 15%–60% up to

The problem: recurrent-state snapshots during speculative verification

Hybrid architectures such as Qwen3.5 interleave softmax-attention layers, which carry a growing KV cache, with Gated DeltaNet (GDN) linear-attention layers that maintain a fixed-size recurrent state S∈Rdv×dkS \in \mathbb{R}^{d_v \times d_k} per head. In ordinary decoding this fixed state is the hybrid's principal advantage: at 32K context the softmax KV cache costs 0.625 GiB/sequence on Qwen3.5-35B-A3B and 0.94 GiB on 397B-A17B, while GDN states cost only 30 and 90 MiB/sequence respectively. Speculative decoding inverts this picture. Verifying kk draft tokens is a single parallel forward pass in which the GDN state advances past all draft positions before acceptance is decided, so rollback requires either a full-state snapshot per draft position (k+1k{+}1 blocks per sequence per layer for a chain, N+1N{+}1 for an NN-node tree) or some equivalent. For k=3k=3 this quadruples committed state memory to 120 MiB/seq (35B) and 360 MiB/seq (397B), and because snapshots cannot be shared across tree branches, wide draft trees become memory-infeasible. The paper's contribution, TreeWY, removes these snapshots entirely.

Method: a tree-structured WY transform of the gated delta rule

The key algebraic observation is that the gated delta rule,

St=αtSt−1(I−βtktkt⊤)+βtvtkt⊤,S_t = \alpha_t S_{t-1}(I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top,

is decayed additive attention with a corrected value: writing v~t=βt(vt−αtSt−1kt)\tilde v_t = \beta_t(v_t - \alpha_t S_{t-1}k_t) makes both the state and every output a decay-weighted sum over preceding tokens. If all pseudo-values V~∈RN×dv\tilde V \in \mathbb{R}^{N\times d_v} can be obtained without walking the recurrence, no intermediate states are needed. Laying out the draft's NN nodes in DFS pre-order from the committed state kk0, and chaining the per-token rewrite along every root-to-node path, yields one strictly lower-triangular system solved by forward substitution:

kk1

where kk2 is the cumulative decay from kk3. This generalizes DeltaNet's chain WY/UT transform (Yang et al., 2024) to a strict-ancestor tree form; the ancestor-mask structure follows STree (Wu et al., 20 May 2025), which handles only Mamba2's commuting scalar-decay transition. TreeWY's addition is carrying GDN's non-commuting rank-1 delta correction through the same mask. On commit of accepted node kk4, the continuation state is reconstructed directly as kk5, so storage drops from kk6 full state blocks to a single pseudo-value matrix — a 128× smaller object per head at kk7. The derivation depends only on the gated delta rule, not other architectural details.

The implementation lives in vLLM behind two speculative-config options (mamba_state_commit="reconstruct", draft_tree_widths). Chains verify and commit in one fused, CUDA-graph-capturable Triton kernel; real trees require a non-causal ancestor mask that forces piecewise capture and evicts the GDN mixer from graphs. Correctness is checked against the per-node recurrence to kk8 in fp64 and kk9 in fp32, with token streams gated against a shared no-speculation reference rather than claimed bit-identical.

Serving results

Evaluation covers Qwen3.5-35B-A3B (TP1) and 397B-A17B (TP8) on B200 GPUs against vLLM's default snapshotting baseline ("storeall"), sweeping GPU memory utilization over six workloads, with a depth-3 MTP drafter under greedy verification. Across 175 matched points, acceptance length matches the baseline essentially exactly (mean k+1k{+}10, max k+1k{+}11), confirming the closed form does not perturb the output distribution.

Memory: peak KV usage falls k+1k{+}12–k+1k{+}13 at identical load, since TreeWY holds one state block per layer versus k+1k{+}14. Under pressure this translates into far fewer preemptions (1365 vs. 2531 requests preempted).

Throughput and latency: where memory binds, TreeWY wins outright — up to k+1k{+}15 throughput and roughly k+1k{+}16 lower p99 TTFT (683 vs. 27489 ms at 128 concurrency, 35B, gmu 0.6). Where memory is slack, throughput trails by a few percent (k+1k{+}17–k+1k{+}18). A regime decomposition shows the win is conditional, not a fixed multiplier: across memory-bound points TreeWY reaches k+1k{+}19 throughput and N+1N{+}10 lower p99 TTFT at 35B, but the paper is explicit that averaging over both regimes would obscure this conditionality. Notably, the TPOT penalty tracks batch growth almost exactly (Pearson N+1N{+}11), identifying it as the price of admitted concurrency rather than a slower kernel.

Tree width: because stored state stays flat at one block regardless of width, wide trees become affordable where snapshotting cannot host them at all (a store-all baseline's per-request cost grows from 4 to 40 blocks from shape N+1N{+}12 to N+1N{+}13, while TreeWY stays at one). Acceptance length rises monotonically with width (3.23 → 3.58), though it is bounded by draft depth rather than width. The paper concedes plainly that trees are not yet a throughput win: wider trees push more tokens through the target per step and run unfused, so they are reported as enabled and correct, not faster.

Relation to concurrent work

Four prior approaches frame the contribution. Full-state snapshotting is the default in vLLM and SGLang. ReplaySSM [dao2026replayssm] defers state materialization via a rank-1 cache and solves the same triangular system inside the verify window, but ships chain-only. STree verifies trees only for Mamba2, whose scalar transition admits cumulative-sum tricks unavailable to GDN. Bole (Wang et al., 3 Aug 2026), concurrent and independent, derives a similar tree-structured closed form on SGLang reporting N+1N{+}14–N+1N{+}15 transient-memory reduction; no head-to-head was possible because Bole links no code.

The ReplaySSM comparison is the paper's most candid section. On the identical 35B sweep, ReplaySSM converts its freed headroom into more chain throughput than TreeWY (N+1N{+}16–N+1N{+}17 vs. N+1N{+}18–N+1N{+}19 own-baseline), leading on per-token speedup at every load even where neither method admits extra work — which localizes the gap to per-step efficiency, not the memory mechanism (both free equivalent headroom; peak-KV reduction converges within a few percent past the knee). The authors hypothesize the difference lies in ReplaySSM's deferred state writes versus TreeWY's materialization on every commit, but explicitly flag this as attribution, not a validated claim, since no deferred-write prototype was built.

Limitations and open questions

Several limitations are stated by the authors themselves. The tree path cannot be CUDA-graph captured, so its cost exceeds what the mask alone would imply, and trees must be scheduled atomically (requests whose tree exceeds the per-step budget skip speculation entirely). All results use greedy drafting and verification, prefix caching is disabled throughout, and evaluation covers a single model family. The ReplaySSM throughput gap remains unexplained mechanistically, and the deferred-write lever it suggests is untested. No comparison against Bole exists. Whether fusing the tree verify path into a graph-capturable kernel would convert width into a throughput win, and whether deferred commit writes close the per-step gap, are the concrete questions left open.

Conclusion

TreeWY replaces per-draft-position recurrent-state snapshots in GDN hybrids with a single triangular solve over a tree-structured WY transform plus reconstruction of only the accepted state, reducing speculative state memory to one small pseudo-value matrix for chains and trees alike. On chains this yields NN0–NN1 lower KV pressure and, wherever memory binds, up to NN2 throughput and order-of-magnitude TTFT improvements at unchanged acceptance length; on trees it makes wide drafts affordable without yet making them faster. The residual per-step cost relative to ReplaySSM and the uncaptured tree kernel are identified implementation artifacts, and closing them is the stated next step.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.