SOFA: Symbolic Fourier Approximation Index

• SOFA is an exact similarity search index for large-scale data series that uses frequency-domain discretization and adaptive quantization to ensure tight lower bounds.
• It integrates a block-structured in-memory tree inspired by MESSI with a learned Symbolic Fourier Approximation, enabling efficient k-NN and range queries.
• Empirical evaluations show that SOFA outperforms SAX-based methods, achieving significant speedups and scalability improvements on high-frequency, noisy datasets.

The Symbolic Fourier Approximation Index (SOFA) is an exact, high-throughput similarity search index developed for large-scale data series (DS), which are ordered sequences of real values. Designed to address the limitations of Symbolic Aggregate approXimation (SAX)-based methods on high-frequency or noisy signals, SOFA combines a block-structured in-memory tree index inspired by MESSI and a learned Symbolic Fourier Approximation (SFA) summarization. With data-adaptive, frequency-domain discretization and tight lower-bounding of Euclidean distances, SOFA provides state-of-the-art performance for exact similarity queries on terascale DS collections (Schäfer et al., 2024).

1. Architecture: Block-Structured Tree and Symbolic Fourier Summarization

SOFA integrates two core innovations:

• Tree Index Inspired by MESSI: The index is structured as an in-memory tree with:
  • A root node holding up to $2w$ child pointers ($w$ = symbol bits across segments).
  • Binary inner nodes, each storing an SFA word summarizing its subtree.
  • Leaf nodes containing up to $L$ series (SFA words plus pointers to raw DS).
  • When a leaf exceeds $L$, it is split along a symbol-bit dimension, increasing the bit cardinality for that segment, analogous to the iSAX bit-split operator.
• Symbolic Fourier Approximation (SFA): SFA transforms each $z$-normalized series $A=(a_1, \dots, a_n)$ into a symbolic word $A'=(\alpha_0, \dots, \alpha_{l-1})$ of length $l \ll n$, using an alphabet $\Sigma$ of size $|\Sigma| \ll n$. SFA operates as follows:
  1. Converts the input series into the frequency domain using the Discrete Fourier Transform (DFT).
  2. Selects $l$ real or imaginary Fourier coefficients with the largest variance across the dataset.
  3. Quantizes each coefficient using learned equi-width breakpoints, converting continuous values into discrete symbols.

This synergy enables fast, exact similarity search even in the presence of high-frequency signal content (Schäfer et al., 2024).

2. Symbolic Fourier Approximation: Real-to-Symbolic Encoding

SFA's pathway to symbolic summarization involves several phases:

• DFT Transformation: For z-normalized series $A$, compute $A'_{full} = DFT(A) = (X_0, X_1, \dots, X_{n-1})$, storing real and imaginary parts as $2n$ real values.

• Variance-Based Feature Selection: For each real or imaginary component $j \in [0, n-1]$, compute $VAR_j$ (the across-series variance of that DFT component). Select indices $best\_l$ corresponding to the top-$l$ variances; coefficients with larger variance improve quantization quality and lower-bounding tightness.

• Adaptive Quantization: For each selected $j$, partition $[min_j, max_j]$ into $|\Sigma|$ equi-width intervals

$$\beta_j(0) = min_j,\quad \beta_j(a) = min_j + a\Delta_j,\quad \Delta_j = \frac{max_j-min_j}{|\Sigma|},\quad a=0 \ldots |\Sigma|$$

assigning symbol $\sigma_a$ if $X_j(A) \in [\beta_j(a-1), \beta_j(a))$. This transforms $A$ into an SFA word $SFA(A)=(\alpha_{b_0}, \dots, \alpha_{b_{l-1}})$ where $b_0, \dots, b_{l-1}$ are the indices of the selected DFT coefficients.

This design enables wide quantization bins for high-variance coefficients, supporting tight lower bounds and efficient indexing.

3. Index Construction and Query Workflow

The SOFA index construction proceeds as follows:

1. Learning SFA Binning (MCB Quantization):

   • Sample $1\%$ of series to estimate per-coefficient variance.
   • Compute DFT on these samples.
   • Select $l$ DFT dimensions by highest variance.
   • Fit equi-width bins for each chosen coefficient.
2. SFA Word Representation:
   • For each series, apply DFT.
   • Extract values at $best\_l$ indices.
   • Quantize these with precomputed bins to obtain the SFA word.
   • Insert the SFA word with a pointer to raw DS into the tree index, splitting leaves as needed.
3. Exact Query (k-NN and Range Search):
   • Transform the query series into an SFA word.
   • Best-first tree traversal: each thread maintains a priority queue ordered by lower-bounding distance (LBD).
   • Leaves or series with $LBD \geq$ current best-so-far are pruned.
   • Otherwise, full Euclidean distances on raw data are computed to possibly update the solution set.
   • Process continues until the priority queues are empty, guaranteeing exact retrieval.

Pseudocode for all main components is explicitly presented in (Schäfer et al., 2024): Learn-Bins with MCB_Quantization, SFA_Transform, index build, and exact k-NN query.

4. Lower Bounding Techniques and Computational Optimization

SOFA employs the GEMINI framework to support exactness via lower-bounding:

• DFT-Based Lower Bound: For $l < n/2$, the squared DFT-distance between compressed representations,

$$d^2_{DFT}(A', B') = (a'_0-b'_0)^2 + 2\sum_{i=1}^{l-1} (a'_i-b'_i)^2 \leq d^2_{ED}(A,B)$$

allows for safe pruning in the index.

• Symbolic-Numerical Distance: In the SFA context,

$$mind_i(\alpha_i, b'_i) = \begin{cases} 0 & \text{if}\ b'_i \in [\beta_i(a-1), \beta_i(a))\ \beta_i(a-1) - b'_i & \text{if}\ b'_i < \beta_i(a-1)\ b'_i - \beta_i(a) & \text{if}\ b'_i > \beta_i(a) \end{cases}$$

The total SFA lower-bounding distance is

$$d^2_{SFA}(A', B') = 2\sum_{i=1}^{l-1} mind_i^2(\alpha_i, b'_i) \leq d^2_{ED}(A,B)$$

ensuring safe pruning during multi-threaded best-first traversal.

• SIMD Acceleration and Early Abandonment: All quantization and distance computations are vectorized with SIMD instructions. Chunks of $8$ or $16$ floats are processed together, with early termination if the running sum exceeds the best-so-far threshold. This maximizes core utilization and minimizes wasted compute on non-promising candidates (Schäfer et al., 2024).

5. Empirical Evaluation: Datasets, Performance, and Lower-Bound Tightness

The evaluation of SOFA covers a novel benchmark of $17$ datasets ($\sim 1$ billion DS, $\sim 1$ TB):

• Datasets: Astrophysics (Astro, $100$M series, $ℓ=256$) and $12$ seismic datasets (including LenDB, SCEDC, STEAD, etc., with up to $100$M series per collection), and five computer vision datasets (BigANN, Deep1B, SALD, SIFT1b).
• Competitors: SOFA is compared to FAISS (IndexFlatL2, CPU multi-threaded), UCR Suite-P (SIMD parallel sequential scan), and MESSI (iSAX-based method). Hardware: $2\times$Intel Xeon 6254, $36$ cores, $756$ GB RAM.
• Index Build Times: SOFA requires $15$–$60$ s per dataset (DFT overhead), compared to MESSI’s $\sim 15$ s. Parallel scaling is sublinear at $36$ cores due to synchronization.
• 1-NN Query Performance: On $36$ cores, SOFA achieves median query times down to $58$ ms, averaging $2$–$4\times$ faster than FAISS, $2$–$3\times$ faster than MESSI, and up to $10\times$ versus a parallel scan. On high-frequency datasets (LenDB), SOFA obtains a $38\times$ speedup over MESSI.
• Scalability and Parameter Insights: SOFA exhibits linear scaling with $k$ in $k$-NN and with core count up to $18$–$36$ threads. Increasing leaf size to $20$k yields improved throughput, with plateau between $10$k–$20$k.
• Lower Bound Tightness (TLB): Tightness is quantified as $TLB:{=}LBD/\text{true}_\text{ED}$ ($\in[0,1]$), with higher $TLB$ indicating better pruning. On UCR and SOFA’s $17$ datasets, SFA (equi-width + variance) consistently outperforms iSAX, especially for small $|\Sigma|$ (e.g., TLB: $0.48$ vs $0.62$ for $|\Sigma|=4$; $0.76$ vs $0.82$ for $|\Sigma|=256$). Critical-difference tests confirm statistically significant superiority of SFA_EW+Var over iSAX.

Parameter recommendations are $l=16$ (word length), $|\Sigma|=256$ (symbol bits), and variance-based coefficient selection for best balance of index size ($\sim 16$ bytes/series) and tight pruning (Schäfer et al., 2024).

6. Applications and Significance

SOFA’s design reflects the requirements of contemporary scientific domains managing large, high-frequency time series—such as astronomy and seismology—where dimensionality reduction must preserve exact distances for rigorous similarity search. Empirical results show that SOFA’s combination of variance-based frequency-domain representation and multi-threaded, SIMD-accelerated search obviates the need for lossy or approximate solutions in these data regimes, providing substantial performance gains without sacrificing exactness under the GEMINI framework.

7. Related Methods and Broader Context

SOFA extends the paradigm established by SAX/iSAX and MESSI by integrating frequency-domain adaptive symbolic representations and advanced index organization. Unlike standard SAX (piecewise aggregate approximation in time domain with fixed bins), SOFA’s SFA leverages the data’s spectral structure, selecting and quantizing frequency components for both tightness and efficiency. This enables effectiveness in high-frequency and noisy settings where time-domain symbolic methods underperform. The highly optimized nature of SOFA—including SIMD-accelerated lower-bounding and multi-core, multi-threaded query processing—positions it as a primary reference for exact large-scale DS similarity search in scientific data management (Schäfer et al., 2024).