--- title: 'TDSE: Dynamic Transformer Sequence Encoding' url: https://www.emergentmind.com/topics/transformer-dynamic-sequence-encoder-tdse type: topic --- # TDSE: Dynamic Transformer Sequence Encoding Searching arXiv for the cited TDSE/FLOATER and TimeSqueeze papers to ground the article in current records. Transformer Dynamic Sequence Encoder (TDSE) denotes a class of Transformer-compatible sequence encoding mechanisms that replace static, manually fixed sequence representations with adaptive dynamics. In the literature provided here, the term appears in two closely related but non-identical senses. In “Learning to Encode Position for Transformer with Continuous Dynamical Model” [2003.09229], it refers to FLOATER, a continuous dynamical position encoder in which positional signals are generated by a learned ordinary differential equation. In “TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting” [2603.11352], it denotes the dynamic sequence encoding pipeline that combines a state-space encoder, content-aware segmentation, compressed Transformer tokens, and causal unpatching for long-context forecasting. This suggests that TDSE is best understood as a dynamic-sequence-design principle: the sequence representation is learned as a function of underlying dynamics rather than imposed by a fixed sinusoid or a uniform patch grid. ## 1. Terminological scope and primary variants The two uses of TDSE share a common objective: to make Transformer sequence handling adaptive while preserving compatibility with standard self-attention. In FLOATER, the adaptive object is the positional signal; in TimeSqueeze, it is the tokenization and compression pattern of the input series. Both formulations are presented as alternatives to fixed design choices that can limit flexibility, extrapolation, or efficiency [2003.09229] [2603.11352]. | Source | TDSE object | Core mechanism | |---|---|---| | “Learning to Encode Position for Transformer with Continuous Dynamical Model” [2003.09229] | Positional encoding | Learned continuous-time latent trajectory $z(t)$ via Neural ODE | | “TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting” [2603.11352] | Dynamic sequence compression | SSM encoder, thresholded segmentation, compressed Transformer tokens, causal unpatching | In the first formulation, the central problem is that non-recurrent models are permutation equivalent unless position information is explicitly injected, whereas sinusoidal encoding is manually designed and learned position embeddings restrict maximum sequence length. In the second, the central problem is the trade-off between point-wise embeddings, which preserve temporal fidelity but scale poorly, and fixed-length patching, which improves efficiency but can disrupt natural transitions and blur informative local dynamics. The shared theme is not architectural identity, but the replacement of static sequence structure by data-adaptive dynamics. ## 2. Continuous dynamical position encoding in FLOATER FLOATER models position encodings as samples from a continuous-time latent trajectory $z(t)\in\mathbb{R}^d$ governed by a learnable ODE [2003.09229]: $$ \frac{dz(t)}{dt} = f(t, z(t); \theta_h), $$ with equivalent integral form $$ z(t) = z(s) + \int_s^t f(\tau, z(\tau); \theta_h)\, d\tau. $$ The discrete positional encoding for the $i$-th token is obtained by sampling this trajectory at positions $t_i$, typically with $t_i = i\times \Delta t$, so that $z_i \coloneqq z(t_i)$. In all experiments, $\Delta t = 0.1$. The formulation is explicitly designed so that the encoding can be arbitrarily extrapolated to longer lengths. Each Transformer block $n$ maintains its own initial vector $z^{(n)}(0)\in\mathbb{R}^d$, while the same $f$ and therefore the same parameter set $\theta_h$ are shared across all blocks. The result is a block-specific trajectory family with shared dynamics and distinct initial conditions. In the “warm-start” setting, these initial states are initialized so that the flow is initially zero and matches a pretrained sinusoidal model. The vector field $f(t,z;\theta_h)$ is implemented in all experiments as a two-layer feed-forward network with one nonlinear activation, such as ReLU, between layers. Its input is the concatenation $[z;t]$ of dimension $d+1$, linearly projected to a hidden size $h$, often set equal to $d$, then projected back to $\mathbb{R}^d$. The paper reports about $0.53$ million extra parameters in $\theta_h$ for the Transformer-Base setting, roughly half the size of a single-layer sinusoidal embedding table of fixed length, and a fraction of the cost of $L_{\max}$-sized learned embeddings in every layer. No special regularization was required beyond standard weight-decay and small initial weight scale for $\theta_h$. A key structural claim is that FLOATER replaces the usual position embedding $p_i$ by $z_i:=z(t_i)$, and that adding $z_i$ inside query, key, and value projections is equivalent to adding a position-dependent bias inside each self-attention. If the usual query is $W_q(x_i+p_i)+b_q$, FLOATER uses $W_q(x_i+z_i)+b_q$ with the same $W_q,b_q$. Because $z_i$ is zero if and only if $f\equiv 0$ and $z(0)=0$, FLOATER strictly contains sinusoidal-only Transformers as a special case. ## 3. Optimization, numerical integration, and computational profile FLOATER is trained end-to-end by integrating $z(t)$ inside the Transformer computational graph, with the overall loss depending on all sampled states $z_i=z(t_i)$ [2003.09229]. Gradients with respect to $\theta_h$ and $z(0)$ are computed using the adjoint sensitivity method of Neural ODEs. The description in the paper defines the adjoint $a(t)=\partial L/\partial z(t)$ and backpropagates by solving the augmented ODE $$ \frac{da}{dt} = -\, a(t)^\top \frac{\partial f}{\partial z} $$ backward in time, while accumulating $$ \frac{\partial L}{\partial \theta_h} = - \int a(t)^\top \frac{\partial f}{\partial \theta_h}\, dt. $$ The implementation relies on the publicly available `torchdiffeq` package. In practice, the ODE is solved on subintervals of length $\Delta t=0.1$. For machine translation, a high-order Runge–Kutta solver is used with step size $\Delta t/5=0.02$ and therefore five substeps. For GLUE and RACE, the method switches to a simple midpoint rule with the same substep size to reduce overhead. Several engineering choices are reported for limiting training cost: warm-starting from a pretrained Transformer, initializing $\theta_h$ small so the dynamic begins near zero, and caching positional biases for many training iterations when $f$ changes little. These optimizations limit the added training cost to approximately $20$–$30\%$ relative to a vanilla Transformer, with zero overhead at inference. At inference time, all $z_i$ may be precomputed up to the maximum needed length at no additional overhead. The parameter-efficiency argument is explicit. Injecting position signals into every block, as in universal Transformers, usually leads to a per-layer positional cost; FLOATER instead adds only $O(\theta_h)$ extra parameters rather than $N\cdot L_{\max}\cdot d$. A full learned embedding matrix of size $L_{\max}\times d$ per layer would cost $O(N\cdot L_{\max}\cdot d)$, reported as about $1.6$ million parameters for six layers, whereas FLOATER adds only $O(\theta_h)\approx 0.53$ million. ## 4. Empirical behavior in machine translation and language understanding The empirical evaluation of FLOATER covers neural machine translation, language understanding, question answering, and ablation against RNN-based alternatives [2003.09229]. On WMT’14 En–De and En–Fr, for both Transformer-Base and Transformer-Large, FLOATER injected at every block raises En–De BLEU by approximately $0.4$–$0.8$ points over fixed sinusoidal encodings and by similar margins over fixed-length learned embeddings. Even when used only at the input layer, it still outperforms baselines by approximately $0.3$–$0.6$ BLEU. A central result concerns inductive generalization to longer sentences. When training only on sentences shorter than $80$ tokens, FLOATER maintains BLEU on much longer sentences up to $140+$ tokens, whereas sinusoidal and learned embeddings degrade sharply. The reported visualization complements this finding: unlike fixed sinusoidals, described as perfectly smooth, or learned embeddings, described as chaotic for unseen positions, the FLOATER trajectory exhibits smooth, data-adapted curves that generalize to unseen positions without manual design. On GLUE, with RoBERTa-Base warm-start, FLOATER matches or improves upon $7$ of $8$ tasks, often by $0.2$–$1.8$ points, especially on low-data tasks such as RTE. On RoBERTa-Large, gains of $0.5$–$1.0$ points are described as common. On RACE, accuracy improves from $82.8$ to $83.3$ on the large model, while SQuAD 1.1 and 2.0 F1/EM are modestly higher or matched. The RNN ablation is especially relevant to the interpretation of the method. Replacing the flow-based ODE with a $1$- or $2$-layer standard RNN, using either scalar or vectorized inputs, yields strictly lower BLEU, reported as $27.99$–$28.16$ versus FLOATER’s $28.57$, for comparable parameter counts. This does not show that recurrent structure is universally ineffective; it shows, more narrowly, that the reported flow-based parameterization is superior to those specific RNN substitutes in the tested setting. ## 5. TDSE in TimeSqueeze: dynamic patching for time-series forecasting In TimeSqueeze, TDSE denotes a dynamic patching architecture for time series forecasting rather than a continuous positional ODE [2603.11352]. The high-level pipeline is: input series $x_1,\dots,x_T$; full-resolution SSM encoding; dynamic patching; Transformer processing on compressed tokens with preserved absolute positions; causal unpatching; SSM decoding and forecasting. The SSM encoder uses Mamba layers and produces point-wise features $h_{1:T}\in\mathbb{R}^{T\times D}$. The underlying state-space formulation is given as a discretized linear state-space model. With internal state $s_t\in\mathbb{R}^n$ and input $u_t=x_t$, the continuous system $$ \frac{ds(t)}{dt} = A s(t) + B u(t), \qquad y(t) = C s(t) + D u(t) $$ is discretized to $$ s_t = \bar A s_{t-1} + \bar B u_t, \qquad o_t = \bar C s_t + \bar D u_t. $$ The output $o_t$ is then mixed by a point-wise nonlinearity and residual connection to give the embedding $h_t$, and stacking $L$ such layers yields $h_{1:T}$. Dynamic segmentation is driven by a local signal-complexity rule. The sliding power estimate over a window of length $L$ is $$ P_i = \frac{1}{L}\sum_{j=i-L}^{i-1} x_j^2, $$ and the complexity score is $$ c(i) = \frac{|x_i - x_{i-1}|}{\sqrt{P_i}}. $$ A patch boundary is declared at index $i$ if $$ b_i = 1 \iff c(i) > \tau, $$ equivalently, $$ b_i = \begin{cases} 1, & |x_i - x_{i-1}| > \tau\,\sqrt{P_i},\\ 0, & \text{otherwise.} \end{cases} $$ All indices with $b_i=1$ mark starts of new patches. The description states that this allocates shorter patches in high-variation regions and larger patches in smooth segments. After segmentation, the boundary sequence is $p_0=1