Delta Tokens in ML, LLM, & DeFi

• Delta tokens are specialized representations that encode first-order differences between states, events, or price exposures for efficient modeling and transmission.
• They are integral in generative world modeling and multi-agent LLM communication, enabling significant reductions in model parameters and computational cost while improving predictive accuracy.
• In decentralized finance, delta tokens act as synthetic derivatives that isolate first-order price risk, supporting dynamic hedging strategies and risk modularization.

Delta tokens are specialized representations or instruments that encode first-order differences between consecutive states, events, or price exposures, arising across machine learning, generative modeling, and decentralized finance. Although the term “delta token” manifests differently according to domain, the unifying principle involves efficient transmission, modeling, or hedging of incremental changes—whether as compressed representations of feature or hidden-state transitions, or as synthetic financial primitives isolating instantaneous risk exposure. Recent research formalizes delta tokens through learned encoders for world modeling (Kerssies et al., 6 Apr 2026), state transition deltas in multi-agent LLMs (Tang et al., 24 Jun 2025), and as derivative products in decentralized finance protocols (Bichuch et al., 2024).

1. Delta Tokens in Generative World Modeling

In the context of video generative modeling, delta tokens are compressed, learned representations of the interframe difference in a deep feature space. Given raw frames $v_t$, a frozen Vision Foundation Model (VFM) produces patchwise embeddings $x_t \in \mathbb{R}^{H \times W \times D}$. The raw delta is $\delta^0_t = x_t - x_{t-1}$, and a learned DeltaTok encoder $g$ further compresses this difference into a token $z_t \in \mathbb{R}^D$: $z_t = g(x_{t-1}, x_t, z_{\text{init}})$. Here, $z_t$ captures only structured changes, discarding static scene content.

Each video is thus translated from a high-dimensional spatial-temporal array to a one-dimensional sequence of delta tokens. This enables the design of DeltaWorld, a temporally autoregressive generative model that predicts sequences of plausible futures using delta tokens rather than full-frame or patchwise reconstructions. The approach allows tractable multi-hypothesis training, wherein many futures are generated in parallel and only the best is supervised using a Best-of-Many (BoM) loss on the delta-token sequence. This results in over 1,000$\times$ token reduction (e.g., from 1,024 spatial tokens per frame to one token/frame), over 35$\times$ reduction in model parameters, and approximately 2,000$\times$ lower FLOPs than previous generative baselines, with state-of-the-art performance on forecasting tasks such as Cityscapes, VSPW, and KITTI (Kerssies et al., 6 Apr 2026).

2. State Delta Tokens and Multi-Agent Communication

In multi-agent LLM protocols, delta tokens are the differences in hidden-state vectors across layers and tokens during autoregressive generation. Let an LLM generate tokens $x_t \in \mathbb{R}^{H \times W \times D}$0, with hidden states $x_t \in \mathbb{R}^{H \times W \times D}$1 at layer $x_t \in \mathbb{R}^{H \times W \times D}$2 after generating token $x_t \in \mathbb{R}^{H \times W \times D}$3. The state delta at token $x_t \in \mathbb{R}^{H \times W \times D}$4 and layer $x_t \in \mathbb{R}^{H \times W \times D}$5 is $x_t \in \mathbb{R}^{H \times W \times D}$6.

State Delta Encoding (SDE) pairs each output token $x_t \in \mathbb{R}^{H \times W \times D}$7 with its corresponding vector of deltas $x_t \in \mathbb{R}^{H \times W \times D}$8, with injection layers $x_t \in \mathbb{R}^{H \times W \times D}$9 chosen by protocol. These delta tokens are then transmitted alongside the natural language token sequence to downstream agents, which use them to update their own internal representation $\delta^0_t = x_t - x_{t-1}$0. This mechanism exposes more of the latent reasoning trajectory, substantially mitigating information loss associated with token-level communication.

Empirical evidence shows that SDE-equipped systems (i.e., those using state delta tokens) outperform natural-language-only and soft-probability-vector communication on multi-agent QA, debate, and workflow tasks. For example, SDE improves absolute performance by up to 17.3 percentage points on agent workflow tasks (e.g., HotpotQA), and up to 13.7 percentage points in mathematical multi-agent debates (Tang et al., 24 Jun 2025).

3. Delta Tokens as Synthetic Financial Derivatives

In decentralized finance (DeFi), delta tokens are constructed as synthetic instruments that provide pure first-order (“delta”) exposure to state variables, most notably the mid-price of assets in Automated Market Maker (AMM) liquidity pools (Bichuch et al., 2024). Given a CPMM liquidity token with mark-to-market value $\delta^0_t = x_t - x_{t-1}$1 (where $\delta^0_t = x_t - x_{t-1}$2 is the mid-price), its delta is $\delta^0_t = x_t - x_{t-1}$3.

Risk-neutral pricing incorporating swap fees yields $\delta^0_t = x_t - x_{t-1}$4, so the delta token is $\delta^0_t = x_t - x_{t-1}$5. A “delta token” is thus a claim whose payoff tracks this instantaneous price sensitivity, which can be constructed by dynamically hedging a liquidity token using the corresponding number of underlying asset units. The existence and on-chain minting of such delta tokens would modularize risk—liquidity tokens carry full Greek exposure (delta, gamma, vega), while delta tokens isolate first-order price risk for arbitrage or hedging strategies (Bichuch et al., 2024).

4. Mathematical Formulations and Architectures

Domain	Delta Token Definition	Key Equation / Map
Video World Modeling	Compressed feature difference	$\delta^0_t = x_t - x_{t-1}$6
LLM Multi-Agent Communication	Hidden-state vector difference	$\delta^0_t = x_t - x_{t-1}$7
DeFi Derivative (AMM Delta)	Instantaneous price sensitivity	$\delta^0_t = x_t - x_{t-1}$8

In world modeling, the encoder $\delta^0_t = x_t - x_{t-1}$9 is a ViT-B transformer mapping frame pairs and an init token to a compressed delta token (Kerssies et al., 6 Apr 2026). For LLMs, SDE protocols specify which hidden state layers $g$0 are used, and protocol logic for extraction and injection is formalized by pseudocode (Tang et al., 24 Jun 2025). In DeFi, explicit hedging strategies realize delta tokens as tradable, dynamically replicable claims (Bichuch et al., 2024).

5. Empirical Performance and Efficiency Gains

Delta tokens deliver substantial empirical gains across domains. In generative world modeling, token reduction exceeds 1,000$g$1 per frame, with 35$g$2 model parameter reduction and approximately 2,000$g$3 lower FLOPs, while maintaining or improving accuracy on dense forecasting benchmarks (Kerssies et al., 6 Apr 2026). In LLM multi-agent settings, SDE-based communication surpasses alternatives such as CIPHER and natural-language-only transmission in accuracy (e.g., GSM8K: NL 0.9061 vs. SDE 0.9178; HotpotQA EM: NL 21.0% vs. SDE 22.67%) (Tang et al., 24 Jun 2025). In DeFi, properly designed delta tokens eliminate systematic arbitrage opportunities by isolating first-order risk, as demonstrated in empirical studies of Uniswap v3 pools where hedged strategies drift under naïve pricing but are neutralized once delta tokens are properly constructed (Bichuch et al., 2024).

6. Critical Insights and Limitations

The commonality across all manifestations is that delta tokens efficiently capture or isolate “just the change”—either as compressed temporal/feature updates, hidden reasoning transitions, or first-order financial price exposure. In generative modeling, this enables tractable multi-hypothesis forecasting and models capable of expressing event diversity with reduced capacity and compute, due to the low-dimensional support of video dynamics (Kerssies et al., 6 Apr 2026). In multi-agent LLM protocols, delta tokens mitigate loss of abstraction-specific reasoning information otherwise discarded in tokenization (Tang et al., 24 Jun 2025). In DeFi, pure delta tokens modularize risk and theoretically eliminate arbitrage that arises from mispricing liquidity claims (Bichuch et al., 2024).

However, practical considerations exist. In modeling, the expressiveness of the delta token depends on the quality of the encoder and task; in communication, injection of state deltas assumes architectural compatibility across agents; in DeFi, construction of delta tokens relies on market access and execution, and is sensitive to discretization, transaction fees, and market frictions (this suggests deployment details will matter in practice).

7. Connections to Related Methods and Future Directions

Delta tokens generalize or refine prior methods for efficient representation (compressed state changes vs. full states), protocol augmentation in LLM communication (state trajectory transmission vs. output probability vectors or embeddings), and modularization of risk (synthetic Greeks in DeFi). There is substantial potential for further cross-pollination: improved delta encoders for multimodal data, integration of delta-token protocols in large-scale distributed multi-agent systems, and on-chain composability of financial building blocks beyond first-order “delta” exposure (e.g., for gamma/vega tokens) (Kerssies et al., 6 Apr 2026, Tang et al., 24 Jun 2025, Bichuch et al., 2024).

As the orthogonality and transferability of the delta-token abstraction become clearer, research may focus on optimizing encoders, establishing formal guarantees of informativeness or arbitrage-freeness, and scaling architectures to new domains.