TDSE: Dynamic Transformer Sequence Encoding

• TDSE is a dynamic sequence encoding principle that leverages learned positional dynamics or adaptive segmentation to replace fixed token structures.
• The FLOATER variant uses Neural ODEs to generate continuous-time positional signals, ensuring smooth extrapolation and parameter efficiency in Transformer blocks.
• TimeSqueeze employs dynamic patching with a state-space model for efficient long-horizon forecasting, reducing computational overhead while preserving temporal details.

Searching arXiv for the cited TDSE/FLOATER and TimeSqueeze papers to ground the article in current records. Transformer Dynamic Sequence Encoder (TDSE) denotes a class of Transformer-compatible sequence encoding mechanisms that replace static, manually fixed sequence representations with adaptive dynamics. In the literature provided here, the term appears in two closely related but non-identical senses. In “Learning to Encode Position for Transformer with Continuous Dynamical Model” (Liu et al., 2020), it refers to FLOATER, a continuous dynamical position encoder in which positional signals are generated by a learned ordinary differential equation. In “TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting” (Ankireddy et al., 11 Mar 2026), it denotes the dynamic sequence encoding pipeline that combines a state-space encoder, content-aware segmentation, compressed Transformer tokens, and causal unpatching for long-context forecasting. This suggests that TDSE is best understood as a dynamic-sequence-design principle: the sequence representation is learned as a function of underlying dynamics rather than imposed by a fixed sinusoid or a uniform patch grid.

1. Terminological scope and primary variants

The two uses of TDSE share a common objective: to make Transformer sequence handling adaptive while preserving compatibility with standard self-attention. In FLOATER, the adaptive object is the positional signal; in TimeSqueeze, it is the tokenization and compression pattern of the input series. Both formulations are presented as alternatives to fixed design choices that can limit flexibility, extrapolation, or efficiency (Liu et al., 2020, Ankireddy et al., 11 Mar 2026).

Source	TDSE object	Core mechanism
“Learning to Encode Position for Transformer with Continuous Dynamical Model” (Liu et al., 2020)	Positional encoding	Learned continuous-time latent trajectory $z(t)$ via Neural ODE
“TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting” (Ankireddy et al., 11 Mar 2026)	Dynamic sequence compression	SSM encoder, thresholded segmentation, compressed Transformer tokens, causal unpatching

In the first formulation, the central problem is that non-recurrent models are permutation equivalent unless position information is explicitly injected, whereas sinusoidal encoding is manually designed and learned position embeddings restrict maximum sequence length. In the second, the central problem is the trade-off between point-wise embeddings, which preserve temporal fidelity but scale poorly, and fixed-length patching, which improves efficiency but can disrupt natural transitions and blur informative local dynamics. The shared theme is not architectural identity, but the replacement of static sequence structure by data-adaptive dynamics.

2. Continuous dynamical position encoding in FLOATER

FLOATER models position encodings as samples from a continuous-time latent trajectory $z(t)\in\mathbb{R}^d$ governed by a learnable ODE (Liu et al., 2020):

$\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$

with equivalent integral form

$$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$

The discrete positional encoding for the $i$-th token is obtained by sampling this trajectory at positions $t_i$, typically with $t_i = i\times \Delta t$, so that $z_i \coloneqq z(t_i)$. In all experiments, $\Delta t = 0.1$. The formulation is explicitly designed so that the encoding can be arbitrarily extrapolated to longer lengths.

Each Transformer block $n$ maintains its own initial vector $z(t)\in\mathbb{R}^d$0, while the same $z(t)\in\mathbb{R}^d$1 and therefore the same parameter set $z(t)\in\mathbb{R}^d$2 are shared across all blocks. The result is a block-specific trajectory family with shared dynamics and distinct initial conditions. In the “warm-start” setting, these initial states are initialized so that the flow is initially zero and matches a pretrained sinusoidal model.

The vector field $z(t)\in\mathbb{R}^d$3 is implemented in all experiments as a two-layer feed-forward network with one nonlinear activation, such as ReLU, between layers. Its input is the concatenation $z(t)\in\mathbb{R}^d$4 of dimension $z(t)\in\mathbb{R}^d$5, linearly projected to a hidden size $z(t)\in\mathbb{R}^d$6, often set equal to $z(t)\in\mathbb{R}^d$7, then projected back to $z(t)\in\mathbb{R}^d$8. The paper reports about $z(t)\in\mathbb{R}^d$9 million extra parameters in $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$0 for the Transformer-Base setting, roughly half the size of a single-layer sinusoidal embedding table of fixed length, and a fraction of the cost of $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$1-sized learned embeddings in every layer. No special regularization was required beyond standard weight-decay and small initial weight scale for $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$2.

A key structural claim is that FLOATER replaces the usual position embedding $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$3 by $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$4, and that adding $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$5 inside query, key, and value projections is equivalent to adding a position-dependent bias inside each self-attention. If the usual query is $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$6, FLOATER uses $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$7 with the same $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$8. Because $\frac{dz(t)}{dt} = f(t, z(t); \theta_h),$9 is zero if and only if $$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$0 and $$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$1, FLOATER strictly contains sinusoidal-only Transformers as a special case.

3. Optimization, numerical integration, and computational profile

FLOATER is trained end-to-end by integrating $$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$2 inside the Transformer computational graph, with the overall loss depending on all sampled states $$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$3 (Liu et al., 2020). Gradients with respect to $$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$4 and $$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$5 are computed using the adjoint sensitivity method of Neural ODEs. The description in the paper defines the adjoint $$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$6 and backpropagates by solving the augmented ODE

$$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$7

backward in time, while accumulating

$$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$8

The implementation relies on the publicly available torchdiffeq package. In practice, the ODE is solved on subintervals of length $$z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau.$$9. For machine translation, a high-order Runge–Kutta solver is used with step size $i$0 and therefore five substeps. For GLUE and RACE, the method switches to a simple midpoint rule with the same substep size to reduce overhead.

Several engineering choices are reported for limiting training cost: warm-starting from a pretrained Transformer, initializing $i$1 small so the dynamic begins near zero, and caching positional biases for many training iterations when $i$2 changes little. These optimizations limit the added training cost to approximately $i$3–$i$4 relative to a vanilla Transformer, with zero overhead at inference. At inference time, all $i$5 may be precomputed up to the maximum needed length at no additional overhead.

The parameter-efficiency argument is explicit. Injecting position signals into every block, as in universal Transformers, usually leads to a per-layer positional cost; FLOATER instead adds only $i$6 extra parameters rather than $i$7. A full learned embedding matrix of size $i$8 per layer would cost $i$9, reported as about $t_i$0 million parameters for six layers, whereas FLOATER adds only $t_i$1 million.

4. Empirical behavior in machine translation and language understanding

The empirical evaluation of FLOATER covers neural machine translation, language understanding, question answering, and ablation against RNN-based alternatives (Liu et al., 2020). On WMT’14 En–De and En–Fr, for both Transformer-Base and Transformer-Large, FLOATER injected at every block raises En–De BLEU by approximately $t_i$2–$t_i$3 points over fixed sinusoidal encodings and by similar margins over fixed-length learned embeddings. Even when used only at the input layer, it still outperforms baselines by approximately $t_i$4–$t_i$5 BLEU.

A central result concerns inductive generalization to longer sentences. When training only on sentences shorter than $t_i$6 tokens, FLOATER maintains BLEU on much longer sentences up to $t_i$7 tokens, whereas sinusoidal and learned embeddings degrade sharply. The reported visualization complements this finding: unlike fixed sinusoidals, described as perfectly smooth, or learned embeddings, described as chaotic for unseen positions, the FLOATER trajectory exhibits smooth, data-adapted curves that generalize to unseen positions without manual design.

On GLUE, with RoBERTa-Base warm-start, FLOATER matches or improves upon $t_i$8 of $t_i$9 tasks, often by $t_i = i\times \Delta t$0–$t_i = i\times \Delta t$1 points, especially on low-data tasks such as RTE. On RoBERTa-Large, gains of $t_i = i\times \Delta t$2–$t_i = i\times \Delta t$3 points are described as common. On RACE, accuracy improves from $t_i = i\times \Delta t$4 to $t_i = i\times \Delta t$5 on the large model, while SQuAD 1.1 and 2.0 F1/EM are modestly higher or matched.

The RNN ablation is especially relevant to the interpretation of the method. Replacing the flow-based ODE with a $t_i = i\times \Delta t$6- or $t_i = i\times \Delta t$7-layer standard RNN, using either scalar or vectorized inputs, yields strictly lower BLEU, reported as $t_i = i\times \Delta t$8–$t_i = i\times \Delta t$9 versus FLOATER’s $z_i \coloneqq z(t_i)$0, for comparable parameter counts. This does not show that recurrent structure is universally ineffective; it shows, more narrowly, that the reported flow-based parameterization is superior to those specific RNN substitutes in the tested setting.

5. TDSE in TimeSqueeze: dynamic patching for time-series forecasting

In TimeSqueeze, TDSE denotes a dynamic patching architecture for time series forecasting rather than a continuous positional ODE (Ankireddy et al., 11 Mar 2026). The high-level pipeline is: input series $z_i \coloneqq z(t_i)$1; full-resolution SSM encoding; dynamic patching; Transformer processing on compressed tokens with preserved absolute positions; causal unpatching; SSM decoding and forecasting. The SSM encoder uses Mamba layers and produces point-wise features $z_i \coloneqq z(t_i)$2.

The underlying state-space formulation is given as a discretized linear state-space model. With internal state $z_i \coloneqq z(t_i)$3 and input $z_i \coloneqq z(t_i)$4, the continuous system

$z_i \coloneqq z(t_i)$5

is discretized to

$z_i \coloneqq z(t_i)$6

The output $z_i \coloneqq z(t_i)$7 is then mixed by a point-wise nonlinearity and residual connection to give the embedding $z_i \coloneqq z(t_i)$8, and stacking $z_i \coloneqq z(t_i)$9 such layers yields $\Delta t = 0.1$0.

Dynamic segmentation is driven by a local signal-complexity rule. The sliding power estimate over a window of length $\Delta t = 0.1$1 is

$\Delta t = 0.1$2

and the complexity score is

$\Delta t = 0.1$3

A patch boundary is declared at index $\Delta t = 0.1$4 if

$\Delta t = 0.1$5

equivalently,

$\Delta t = 0.1$6

All indices with $\Delta t = 0.1$7 mark starts of new patches. The description states that this allocates shorter patches in high-variation regions and larger patches in smooth segments.

After segmentation, the boundary sequence is $\Delta t = 0.1$8. Compressed tokens are formed as $\Delta t = 0.1$9 for $n$0, and the original index $n$1 is retained as the absolute position ID. The Transformer backbone, including examples such as an MoE Transformer, processes $n$2 together with their absolute position identifiers $n$3. Unpatching simply repeats or upsamples each transformed token $n$4 across its original patch length $n$5, producing $n$6 while ensuring causality: the final representation at time $n$7 depends only on tokens with original positions less than or equal to $n$8.

The complexity analysis is framed in terms of full sequence length $n$9 and compressed token count $z(t)\in\mathbb{R}^d$00. A point-wise baseline has Transformer complexity approximately $z(t)\in\mathbb{R}^d$01 per layer and memory approximately $z(t)\in\mathbb{R}^d$02. Fixed-size patching with patch length $z(t)\in\mathbb{R}^d$03 yields $z(t)\in\mathbb{R}^d$04 and complexity approximately $z(t)\in\mathbb{R}^d$05. Dynamic patching yields $z(t)\in\mathbb{R}^d$06 tokens and complexity approximately $z(t)\in\mathbb{R}^d$07. The token reduction ratio is defined as $z(t)\in\mathbb{R}^d$08, empirically reported as about $z(t)\in\mathbb{R}^d$09 on pretraining data, implying an attention-cost speedup of approximately $z(t)\in\mathbb{R}^d$10, so $z(t)\in\mathbb{R}^d$11 corresponds to approximately $z(t)\in\mathbb{R}^d$12 smaller quadratic cost.

The reported empirical results emphasize both efficiency and forecasting quality. TimeSqueeze attains up to $z(t)\in\mathbb{R}^d$13 faster convergence and $z(t)\in\mathbb{R}^d$14 higher data efficiency than equivalent point-token baselines in large-scale pretraining. It reports peak GPU memory reduction of approximately $z(t)\in\mathbb{R}^d$15 at $z(t)\in\mathbb{R}^d$16 and up to $z(t)\in\mathbb{R}^d$17 throughput gain for long forecasting horizons such as horizon $z(t)\in\mathbb{R}^d$18. On long-term benchmarks including ETTh1/2, ETTm1/2, and Weather, it matches or slightly outperforms the point-wise Time-MoE model and outperforms fixed-size patching and pure linear encoders by $z(t)\in\mathbb{R}^d$19–$z(t)\in\mathbb{R}^d$20 in average MSE. The ablations state that dynamic patching is better than fixed patching of size $z(t)\in\mathbb{R}^d$21 or $z(t)\in\mathbb{R}^d$22 by approximately $z(t)\in\mathbb{R}^d$23–$z(t)\in\mathbb{R}^d$24 MSE reduction, that the SSM encoder is much better than simple linear embeddings, and that removing fine-grained residuals or absolute position IDs degrades performance by $z(t)\in\mathbb{R}^d$25–$z(t)\in\mathbb{R}^d$26.

6. Conceptual significance, misconceptions, and limitations

The two TDSE formulations address different failure modes of standard Transformer sequence handling. FLOATER addresses the fact that non-recurrent models are less sensitive to position because position information among input units is not inherently encoded, while TimeSqueeze addresses the inefficiency and representational rigidity of point-wise tokenization and fixed-length patching in long-horizon forecasting (Liu et al., 2020, Ankireddy et al., 11 Mar 2026). The commonality is architectural adaptivity: sequence structure is produced by a learned dynamical process or a content-aware segmentation rule rather than by a fixed table or a uniform patch schedule.

A frequent misconception would be to treat TDSE as synonymous with a learned embedding table. That characterization is inaccurate for both variants. In FLOATER, the encoding is the discretization of a continuous latent trajectory $z(t)\in\mathbb{R}^d$27 and can be extrapolated to longer lengths. In TimeSqueeze, the compressed tokens are selected from full-resolution SSM features according to dynamic boundaries, and the original indices $z(t)\in\mathbb{R}^d$28 are explicitly preserved as absolute position IDs. Another misconception would be to regard dynamic encoding as eliminating positional structure; in both cases, positional information remains central, but it is modeled dynamically rather than statically.

The limitations are also specific to each instantiation. FLOATER adds approximately $z(t)\in\mathbb{R}^d$29–$z(t)\in\mathbb{R}^d$30 training overhead relative to a vanilla Transformer, even though inference incurs zero overhead after precomputation. TimeSqueeze requires choosing the threshold $z(t)\in\mathbb{R}^d$31 to hit a target compression rate, and it notes possible performance degradation if patches become too coarse in highly volatile signals. Potential extensions named in the TimeSqueeze description include learning boundary placements end-to-end in embedding space rather than thresholding raw $z(t)\in\mathbb{R}^d$32, multi-level or hierarchical patching, and extension to irregularly sampled series. For FLOATER, the reported evidence supports three stated desiderata—inductiveness, data-driven flexibility, and parameter efficiency—while the measured gains are consistently described as small but reliable rather than large.

Taken together, these works position TDSE as a research direction rather than a closed recipe. One branch treats positional encoding itself as a dynamical system sampled along sequence index; another treats the input sequence as an object to be dynamically segmented before attention. A plausible implication is that future uses of the term will continue to refer to Transformer encoders in which sequence structure is learned through explicit dynamics, whether those dynamics govern position, segmentation, compression, or their combination.