Derivative Delay Embedding (DDE) Overview

• Derivative Delay Embedding (DDE) is a method that converts raw time series into a structured embedding space using finite differences and delay stacking.
• It employs a discretized grid-mapping approach to maintain fixed memory usage and operate in constant time for online, real-time scenarios.
• The Markov Geographic Model (MGM) augments DDE through probabilistic classification by combining geographic state distributions with transition statistics for enhanced noise robustness.

Derivative Delay Embedding (DDE) provides a principled approach for online modeling and classification of streaming time series, characterized by invariance to input length, phase, and baseline, and by computational efficiency suitable for real-time settings. Unlike classical fixed-length or batch-oriented techniques, DDE incrementally transforms raw time-series into a structured embedding space using finite differences and delay embeddings, enabling memory-efficient modeling regardless of stream length or alignment. The Markov Geographic Model (MGM) augments DDE with a nonparametric mechanism for probabilistic classification by leveraging both steady-state distribution and transition statistics in the discretized embedding space.

1. Mathematical Formulation of Derivative Delay Embedding

DDE operates on a discrete time-series $y_t \in \mathbb{R}$ (or $\mathbb{R}^n$), leveraging a finite difference to estimate derivatives: $\dot{y}(t) \approx \frac{y_t - y_{t-\tau}}{\tau}$ where $\tau$ is a positive integer lag (often $\tau=1$ for maximal temporal resolution). In practice, for each timestep $t$, the derivative signal is: $y'_t = (y_t - y_{t-\tau}) / \tau$ To form the DDE vector, stack $d$ such derivatives spaced by delay step $s$: $$\mathbf{v}_t = [y'_t, \; y'_{t-s},\; y'_{t-2s}, \; \ldots, \; y'_{t-(d-1)s}]^T \in \mathbb{R}^d$$ This vector retains latent dynamical information in the time series via recursive patterns.

To enable fixed memory usage, a $d$-dimensional grid overlays the continuous embedding space. The grid-mapping

$G : \mathbb{R}^d \rightarrow \{1, \ldots, N\}^d$

rounds each coordinate of $\mathbf{v}_t$ to its nearest cell index. The resulting discrete DDE state is: $$\mathbf{x}_t = G(\mathbf{v}_t) \in \{1, \ldots, N\}^d$$ Empirical evidence suggests $N \approx 50$ bins per axis provides adequate resolution and tractable statistics.

2. Online Computation and Streaming Efficiency

The incremental nature of DDE is realized by maintaining a circular buffer of length $(d-1)s + 1$ and updating as new data arrives:

• Upon receiving $y_t$, push to buffer; discard oldest sample.
• Compute $y'_t$ via finite difference.
• Construct $\mathbf{v}_t$ by gathering the required delayed derivatives.
• Discretize via grid-mapping $G$ to obtain $\mathbf{x}_t$.

Each update requires only $O(1)$ time and memory (plus a lightweight lookup for cell indexing), without regard for the total stream length. Thus, DDE is suitable for online, real-time systems and continuous data streams. This property sharply distinguishes DDE from batch/segmentation-based approaches requiring repeated normalization or windowing.

3. Theoretical Properties: Invariance and Embedding

Takens' theorem provides the foundational basis: delay embedding of a generic observable produces a diffeomorphic reconstruction of the latent dynamics when $d > 2 \dim X$. DDE inherits and strengthens this property by operating on derivatives rather than raw signals, leading to several invariance properties:

• Additive shift invariance: Finite differences remove constant baseline drift, making the model robust to offset.
• Phase and length invariance: The embedding’s geography and attractor occupancy are determined by intrinsic, recurrent structure, not by stream alignment or segment length.
• Memory efficiency in infinite streams: Once the attractor region is reached, only a bounded finite set of grid-cells are revisited, allowing memory use to remain constant in unbounded streaming scenarios.

Replacing the observation function $\psi(x_t)$ in Takens' framework with the derivative $\dot{y}(t)$ is theoretically justified, with $\dot{y}$ acting as a generic observable.

4. Markov Geographic Model (MGM) for Nonparametric Classification

MGM augments DDE for online, probabilistic classification by maintaining:

• Geographic state distribution: For each class $c$, the frequency count $n_c(u)$ for each cell $u$ traversed by the training trajectory is aggregated. The normalized log-scaled probability for cell $u$ is:

$$P_c(u) = \frac{\log[n_c(u) + 1]}{\sum_v \log[n_c(v) + 1]}$$

The logarithmic form softens the peak dominance of high-count cells typical near zero-crossings.

• Transition counts: MGM maintains a sparse count $T_c(u \rightarrow v)$ for jumps from cell $u$ (at $t-1$) to $v$ (at $t$), converting to conditional probabilities:

$$P_c(v \mid u) = \frac{T_c(u \rightarrow v)}{\sum_w T_c(u \rightarrow w)}$$

Test trajectories $(\mathbf{x}_1, ..., \mathbf{x}_t)$ are then scored by cumulative sum and product: $$S_c(\mathbf{x}_{1:t}) = \left[\sum_{j=1}^t P_c(\mathbf{x}_j)\right] \times \left[\prod_{i=2}^t P_c(\mathbf{x}_i \mid \mathbf{x}_{i-1})\right]$$ Online updates maintain running statistics, $O(1)$ per timestep, enabling immediate classification via $c^* = \arg\max_c S_c(\mathbf{x}_{1:t})$.

5. Robustness Enhancement via Neighborhood Matching

Exact cell transitions may prove brittle due to noise or quantization artifacts. Robust classification incorporates local spatial neighborhoods in the $d$-dimensional grid. For neighborhood radius $r$ (typically a single cell), transition counts and state frequencies are aggregated over $r$-neighborhoods: $$\widehat{T}_c(u\rightarrow v) = \sum_{\substack {u' \in N_r(u) \ v' \in N_r(v)}} T_c(u' \rightarrow v')$$ State probability $P_c(u)$ is similarly softened. Empirically, this approach significantly enhances noise tolerance.

6. Parameter Selection and Practical Heuristics

Critical parameters for effective DDE-MGM implementation include:

Parameter	Heuristic/Default	Comment
Delay step $s$	$s \approx N/(2d n)$, $n$ = dominant FFT index	Ensures sufficient coverage
Embedding $d$	Use false-nearest-neighbor criterion	Increase until stability
Grid size $N$	$\sim$50 bins/axis	Balances resolution/memory
Neighborhood $r$	One grid cell	For robust matching

A plausible implication is that FFT-based estimation of $s$ and nearest-neighbor checks for $d$ streamline parameter selection without ad hoc tuning.

7. Computational Complexity and Memory Requirements

Each streaming sample requires:

• One finite-difference operation,
• One $d$-vector shift,
• One grid assignment ($O(1)$),
• Two counter updates ($O(1)$ in practice via hashing/sparse tables).

Per-class memory is bounded by $N^d$ state counts and a manageable set of active transitions. For $d=2, N=50$ typical, $\sim$2,500 floats plus a few thousand sparse transitions, enabling deployment in both embedded and high-throughput systems.

8. Illustrative Example in Embedding Space

Consider a 1-dimensional sinusoid with phase and offset drift. In raw space, different trials vary. After finite-difference, offset is removed and trials align in derivative space. Delay embedding into $d=2$ yields $(\dot y(t), \dot y(t-s))$ pairs tracing out planar loops. Overlaying a $50 \times 50$ grid, each loop visits a subset of cells, producing observable statistics for both geographic (cell occupancy) and Markov (transition) models. During deployment, streaming input is continuously scored and classified with immediate feedback.

---

Classical delay embedding as formulated by Takens (“Detecting strange attractors in turbulence,” 1981) underpins the validity of DDE. The DDE-MGM scheme defined above achieves real-time, fully online classification of arbitrary-length time series and is invariant to offset, misalignment, and duration, with state-of-the-art empirical performance and computational efficiency (Zhang et al., 2016).