- The paper introduces TreeWY, a technique that eliminates per-draft-position recurrent-state snapshots in speculative verification for Gated DeltaNet (GDN) hybrids.
- Evaluated on two configurations of Qwen, TreeWY reduces peak KV memory usage by 2-3 times.
- Depending on memory constraint level,Time between first and last token of a sequence for message processing tries at reduced the time by 15%–60% up to
The problem: recurrent-state snapshots during speculative verification
Hybrid architectures such as Qwen3.5 interleave softmax-attention layers, which carry a growing KV cache, with Gated DeltaNet (GDN) linear-attention layers that maintain a fixed-size recurrent state S∈Rdv×dk per head. In ordinary decoding this fixed state is the hybrid's principal advantage: at 32K context the softmax KV cache costs 0.625 GiB/sequence on Qwen3.5-35B-A3B and 0.94 GiB on 397B-A17B, while GDN states cost only 30 and 90 MiB/sequence respectively. Speculative decoding inverts this picture. Verifying k draft tokens is a single parallel forward pass in which the GDN state advances past all draft positions before acceptance is decided, so rollback requires either a full-state snapshot per draft position (k+1 blocks per sequence per layer for a chain, N+1 for an N-node tree) or some equivalent. For k=3 this quadruples committed state memory to 120 MiB/seq (35B) and 360 MiB/seq (397B), and because snapshots cannot be shared across tree branches, wide draft trees become memory-infeasible. The paper's contribution, TreeWY, removes these snapshots entirely.
The key algebraic observation is that the gated delta rule,
St=αtSt−1(I−βtktkt⊤)+βtvtkt⊤,
is decayed additive attention with a corrected value: writing v~t=βt(vt−αtSt−1kt) makes both the state and every output a decay-weighted sum over preceding tokens. If all pseudo-values V~∈RN×dv can be obtained without walking the recurrence, no intermediate states are needed. Laying out the draft's N nodes in DFS pre-order from the committed state k0, and chaining the per-token rewrite along every root-to-node path, yields one strictly lower-triangular system solved by forward substitution:
k1
where k2 is the cumulative decay from k3. This generalizes DeltaNet's chain WY/UT transform (Yang et al., 2024) to a strict-ancestor tree form; the ancestor-mask structure follows STree (Wu et al., 20 May 2025), which handles only Mamba2's commuting scalar-decay transition. TreeWY's addition is carrying GDN's non-commuting rank-1 delta correction through the same mask. On commit of accepted node k4, the continuation state is reconstructed directly as k5, so storage drops from k6 full state blocks to a single pseudo-value matrix — a 128× smaller object per head at k7. The derivation depends only on the gated delta rule, not other architectural details.
The implementation lives in vLLM behind two speculative-config options (mamba_state_commit="reconstruct", draft_tree_widths). Chains verify and commit in one fused, CUDA-graph-capturable Triton kernel; real trees require a non-causal ancestor mask that forces piecewise capture and evicts the GDN mixer from graphs. Correctness is checked against the per-node recurrence to k8 in fp64 and k9 in fp32, with token streams gated against a shared no-speculation reference rather than claimed bit-identical.
Serving results
Evaluation covers Qwen3.5-35B-A3B (TP1) and 397B-A17B (TP8) on B200 GPUs against vLLM's default snapshotting baseline ("storeall"), sweeping GPU memory utilization over six workloads, with a depth-3 MTP drafter under greedy verification. Across 175 matched points, acceptance length matches the baseline essentially exactly (mean k+10, max k+11), confirming the closed form does not perturb the output distribution.
Memory: peak KV usage falls k+12–k+13 at identical load, since TreeWY holds one state block per layer versus k+14. Under pressure this translates into far fewer preemptions (1365 vs. 2531 requests preempted).
Throughput and latency: where memory binds, TreeWY wins outright — up to k+15 throughput and roughly k+16 lower p99 TTFT (683 vs. 27489 ms at 128 concurrency, 35B, gmu 0.6). Where memory is slack, throughput trails by a few percent (k+17–k+18). A regime decomposition shows the win is conditional, not a fixed multiplier: across memory-bound points TreeWY reaches k+19 throughput and N+10 lower p99 TTFT at 35B, but the paper is explicit that averaging over both regimes would obscure this conditionality. Notably, the TPOT penalty tracks batch growth almost exactly (Pearson N+11), identifying it as the price of admitted concurrency rather than a slower kernel.
Tree width: because stored state stays flat at one block regardless of width, wide trees become affordable where snapshotting cannot host them at all (a store-all baseline's per-request cost grows from 4 to 40 blocks from shape N+12 to N+13, while TreeWY stays at one). Acceptance length rises monotonically with width (3.23 → 3.58), though it is bounded by draft depth rather than width. The paper concedes plainly that trees are not yet a throughput win: wider trees push more tokens through the target per step and run unfused, so they are reported as enabled and correct, not faster.
Relation to concurrent work
Four prior approaches frame the contribution. Full-state snapshotting is the default in vLLM and SGLang. ReplaySSM [dao2026replayssm] defers state materialization via a rank-1 cache and solves the same triangular system inside the verify window, but ships chain-only. STree verifies trees only for Mamba2, whose scalar transition admits cumulative-sum tricks unavailable to GDN. Bole (Wang et al., 3 Aug 2026), concurrent and independent, derives a similar tree-structured closed form on SGLang reporting N+14–N+15 transient-memory reduction; no head-to-head was possible because Bole links no code.
The ReplaySSM comparison is the paper's most candid section. On the identical 35B sweep, ReplaySSM converts its freed headroom into more chain throughput than TreeWY (N+16–N+17 vs. N+18–N+19 own-baseline), leading on per-token speedup at every load even where neither method admits extra work — which localizes the gap to per-step efficiency, not the memory mechanism (both free equivalent headroom; peak-KV reduction converges within a few percent past the knee). The authors hypothesize the difference lies in ReplaySSM's deferred state writes versus TreeWY's materialization on every commit, but explicitly flag this as attribution, not a validated claim, since no deferred-write prototype was built.
Limitations and open questions
Several limitations are stated by the authors themselves. The tree path cannot be CUDA-graph captured, so its cost exceeds what the mask alone would imply, and trees must be scheduled atomically (requests whose tree exceeds the per-step budget skip speculation entirely). All results use greedy drafting and verification, prefix caching is disabled throughout, and evaluation covers a single model family. The ReplaySSM throughput gap remains unexplained mechanistically, and the deferred-write lever it suggests is untested. No comparison against Bole exists. Whether fusing the tree verify path into a graph-capturable kernel would convert width into a throughput win, and whether deferred commit writes close the per-step gap, are the concrete questions left open.
Conclusion
TreeWY replaces per-draft-position recurrent-state snapshots in GDN hybrids with a single triangular solve over a tree-structured WY transform plus reconstruction of only the accepted state, reducing speculative state memory to one small pseudo-value matrix for chains and trees alike. On chains this yields N0–N1 lower KV pressure and, wherever memory binds, up to N2 throughput and order-of-magnitude TTFT improvements at unchanged acceptance length; on trees it makes wide drafts affordable without yet making them faster. The residual per-step cost relative to ReplaySSM and the uncaptured tree kernel are identified implementation artifacts, and closing them is the stated next step.