Normalized Substring Complexity

• Normalized substring complexity is a combinatorial measure that computes the maximum ratio of distinct substrings of length k divided by k, revealing text richness.
• Efficient algorithms, in both online and offline settings, achieve O(n) and polylogarithmic time per character, enabling real-time compressibility estimations.
• The measure underlies practical compression methods like Lempel–Ziv and BWT, bridging theoretical insights with applications in data streaming and genomic indexing.

Normalized substring complexity is a formally defined combinatorial measure of a string’s diversity of substrings, normalized by their lengths. Given a string $w$ of length $n$, the normalized substring complexity $\delta(w)$ is

$$\delta(w) = \max_{1 \leq k \leq n} \frac{c_w[k]}{k}$$

where $c_w[k]$ is the number of distinct substrings of length $k$ in $w$. This measure is quantifiable online and offline and is closely related—within polylogarithmic factors—to practical dictionary-based compression algorithms such as Lempel–Ziv and Burrows–Wheeler transform based methods. Multiple efficient algorithms exist for tracking and evaluating $\delta$ in streaming and compressed contexts, under both amortized and worst-case guarantees (Kucherov et al., 18 Oct 2025, Bernardini et al., 2020, Kawamoto et al., 2022).

1. Formal Definition and Basic Properties

Normalized substring complexity is defined for any finite string $w$ as

$$\delta(w) = \max_{1 \leq k \leq n} \frac{c_w[k]}{k}$$

where $c_w[k]$ counts the distinct substrings of length $k$. The function $c_w[k]$ typically exhibits three phases in its profile:

• An initial strictly increasing part,
• A plateau,
• A strictly decreasing segment to $c_w[n] = 1$.

The maximum $\delta$ is usually attained for small $k$, especially in highly repetitive strings.

Normalized complexity provides a lower bound for the string attractor measure $\gamma$, as shown in (Bernardini et al., 2020). Monotonicity under appending and prepending is established (Kawamoto et al., 2022):

• Adding characters does not decrease $\delta$;
• The measure reflects sensitively local edits as well as bulk changes.

2. Computation: Online and Sublinear Algorithms

Efficient algorithms exist for both batch and streaming computation of $\delta$.

• Offline, Linear Space: $\delta$ can be computed in $O(n)$ time and space via suffix tree traversal. The substring complexity at each depth $k$ corresponds to the number of internal nodes at that depth (Bernardini et al., 2020).
• Online, Polylogarithmic Time (Kucherov et al., 18 Oct 2025):
  • Amortized $O(\log n)$ per character: The algorithm maintains $c_w[k]$ using geometric insights (convex hulls), updating counts in bulk for each new character. It computes the length $\alpha_i$ of the shortest unique suffix via online suffix tree techniques, and updates $c_w$ accordingly. The maximum is found by maintaining tangents from the origin to the convex hull of $(k, c_w[k])$.
  • Worst-case $O(\log^3 n)$ per character: This version uses balanced trees and lazy propagation for dynamic convex hull maintenance, guaranteeing worst-case bounds for bulk updates to $c_w[k]$.
• Sublinear Space (Bernardini et al., 2020, Kawamoto et al., 2022):
  • For a space budget $b \in [1, n]$, $\delta$ can be computed in $O(n^3 \log b / b^2)$ time and $O(b)$ space in comparison models.
  • In the Word-RAM model, for $b \in [\sqrt{n}, n]$ the time is $\tilde{O}(n^2 / b)$.
• Run-Length Compressed Input: Given $T$ with $r$ runs, $\delta$ is computed in $O(\min(r \lg\lg r, r \lg_r n))$ time and $O(r)$ space by operating directly on the $r$-suffix tree and augmenting for LCA and associated queries (Kawamoto et al., 2022).

Table: Algorithmic Methods for Computing $\delta(w)$

Setting	Time Complexity	Space Complexity
Offline	$O(n)$	$O(n)$
Online (amortized)	$O(\log n)$ per character	$O(n)$
Online (worst-case)	$O(\log^3 n)$ per character	$O(n)$
Sublinear (RAM)	$\tilde{O}(n^2/b)$	$\tilde{O}(b)$
Run-length compressed	$O(r\log\log r)$ or $O(r\log_r n)$	$O(r)$

3. Structural and Geometric Insights

A geometric perspective is key: $\delta$ is the maximal slope among the lines from the origin to $(k, c[k])$. Algorithms (in particular, Brewer et al. for dynamic convex hulls) enable the maintenance of this maximization efficiently under bulk updates, crucial for streaming and batch settings (Kucherov et al., 18 Oct 2025).

The plateau and monotonic segments in $c[k]$ lead to efficient division of updates; when a new unique suffix increases the diversity for multiple $k$ simultaneously, a contiguous interval of $c[k]$ is incremented.

The run-length compressed setting uses the concept of "deepest matching nodes" (DMNs) in the $r$-suffix tree. The count $|D_k|$ determines $|S_T(k)|$ (the number of distinct $k$-length substrings), so

$\delta = \max_k \frac{|D_k|}{k}$

There are only $O(r)$ breakpoints in $k$ where $|D_k|$ changes, enabling efficient event enumeration and maximization.

4. Relation to Compression and Attractors

$\delta$ serves as a proxy for the compressibility of repetitive strings. It is established that

$\delta \leq \gamma$

where $\gamma$ is the minimum string attractor size (Bernardini et al., 2020, Kawamoto et al., 2022). Many well-studied string compression algorithms (such as Lempel–Ziv and BWT-based methods) yield space bounds that are polylogarithmic in $\delta$; thus, for highly compressible strings, $\delta \ll n$, so compressed data structures and indexes have $O(\delta \log(n/\delta))$ size, supporting efficient random access and pattern matching queries.

In streaming scenarios, maintaining $\delta$ with each new symbol supports real-time estimation of compressibility and adaptively choosing optimal compressed representations.

5. Comparison with Related Complexity Measures

Normalized substring complexity differs from (but lower bounds) many classical repetitiveness metrics:

• Shannon Entropy is optimal for statistical compression but does not address high-repetition regimes.
• String Attractors ($\gamma$) are more general but NP-hard to compute exactly (Bernardini et al., 2020).
• Measures based on LZ77/LZ78 parses, equal-letter runs (BWT), or SUS/MUS counts have different normalizations; e.g., the maximal number of minimal unique substrings (MUS) and shortest unique substrings (SUS) are linear in $n$ (Mieno et al., 2016).

In applications—such as genome indexing, version repository management, and document archiving—$\delta$ provides a direct, algorithmically tractable lens on structural complexity.

6. Applications and Future Directions

Normalized substring complexity is essential for:

• Efficient construction of compressed indexes (especially run-length and repetitive cases).
• Adaptive streaming compression (real-time selection of representation as data arrives).
• Comparison and benchmarking of redundancy across data sources.
• Online compressibility monitoring—important in systems for live data acquisition and genomics.
• Algorithmic study of string attractors and fine-grained repetitiveness properties.

A plausible implication is that continued improvement in online algorithms for $\delta$ may yield further practical speedups and more refined applications in large-scale computational biology and real-time data compression, as string streams and huge repetitive datasets become ubiquitous.

7. Summary

Normalized substring complexity $\delta$ quantifies the "richness" of substrings in a text relative to substring lengths and underpins the space complexity of modern compressed text indexes. Polylogarithmic-time and sublinear-space algorithms for its computation (Kucherov et al., 18 Oct 2025, Bernardini et al., 2020, Kawamoto et al., 2022) make it both theoretically robust and practically deployable, especially in highly repetitive settings. Its algorithmic tractability, connection to string attractors, and role as a proxy for compressibility solidify its significance in the combinatorial and algorithmic study of string complexity.