Linear-Time Diversity Maintenance

• Linear-time diversity maintenance is a set of algorithmic techniques that efficiently extract diverse, representative subsets from large-scale or dynamic data.
• Methods such as core-set construction, incremental trees, and fairness-aware selection enable scalable clustering, streaming updates, and evolutionary optimization in linear time.
• These approaches are applied in clustering, particle filtering, and population dynamics, balancing approximation quality, runtime efficiency, and fairness guarantees.

Linear-time diversity maintenance refers to algorithmic strategies and data structures that enable efficient (space-/pass-/round-optimal) extraction and sustainment of highly diverse solutions or representative subsets from massive or dynamic data, with overall computational cost scaling linearly with the size of the input, working set, or population. This class of methods is critical for scalable summarization, clustering, data selection, multimodal optimization, particle filtering, and population dynamics in neutral evolution, especially where the underlying space exhibits bounded geometric or combinatorial complexity (e.g., bounded doubling dimension, structured populations, or incremental/streaming update models).

1. Core-set Construction and Streaming Algorithms

Linear-time diversity maximization in metric spaces of bounded doubling dimension employs core-set strategies in both streaming and MapReduce frameworks (Ceccarello et al., 2016). A “core-set” is a small subset $T \subseteq S$ such that the maximum diversity attainable on $T$ is within a constant factor $(\alpha+\epsilon)$ of the optimum on $S$, for any diversity objective (e.g., min pairwise distance, remote-clique, remote-star, etc.). Key algorithmic variants include:

• Streaming Multi-Merge (SMM): Maintains a set $T$ with at most $k'+1$ representatives using geometric thresholding on incoming points.
• Composable Core-sets: In MapReduce, local core-sets are computed on distributed partitions and merged.

For streaming, per-point updates and merge steps involve only $O(1)$ distance computations per new item, yielding pass-efficient, linear-space algorithms. In doubling metrics, the core-set cardinality grows as $(1/\epsilon)^D k$ ($D$ is the doubling dimension). Sequential diversity maximization algorithms applied to the core-set provide overall $(\alpha+\epsilon)$-approximations.

Table: Core-set and Diversity Maintenance in Streaming Models

Model	Core-set Size	Update/Pass Complexity	Approximation Ratio
Streaming	$(32/\epsilon)^D k$	O(1) per update	$\alpha+\epsilon$
MapReduce	Merge local core-sets	2 rounds	$\alpha+\epsilon$

2. Incremental Trees for Dynamic Diversification

Incremental Cover Trees (ICT) exploit hierarchical tree structures for maintaining diversity in dynamic pools, supporting insertions/deletions in $O(\log \Delta)$ time (with $\Delta$ aspect ratio, exponential in doubling dimension) (Marienwald et al., 2018, Pellizzoni et al., 2023). Each tree layer encodes a cover; diverse subsets are extracted from the “termination layer” where node separation and covering guarantees provably bound the diversity loss. Theoretical analysis yields worst-case approximation factors ($2 + 2b/(b-1)$, with $b$ base; e.g. $6$ for $b=2$), matching empirical performance and proving tightness.

The augmented cover tree data structure is extended for matroid and fair diversity constraints by storing additional summaries (weights, maximal independent sets) at each node. This enables coreset extraction and diversity queries in linear time with respect to the number of points maintained at each tree level, independent of $\epsilon$ or $k$.

3. Diversity Maintenance in Evolutionary Computation

Population diversity in evolutionary algorithms is classically measured by the sum of pairwise Hamming distances. Analytical frameworks yield the drift formula for the expected diversity in $(\mu+1)$-EAs (Lengler et al., 2023):

$E[S(P_{t+1})] = (1-\delta) S(P_t) + \alpha$

$$S_0 = \frac{\alpha}{\delta} = \frac{(\mu-1)\mu^2 \chi n}{2(\mu-1)\chi + n}$$

where $\chi$ is the expected number of bit flips and $n$ is the string length. Convergence to equilibrium occurs in expected $O(\mu^2 \ln n)$ or $O(n \ln n)$ time, indicating that diversity can be updated and controlled in $O(n)$ time per generation by summarizing bit position statistics.

Structural diversity mechanisms, such as maintaining best solutions at every Hamming distance from an original seed (Doerr et al., 2019), enable linear-time re-optimization in response to dynamic changes in optimization problems, reducing runtime from $\Omega(n^2)$ to $O(y n)$ for problems like LeadingOnes and minimum spanning tree recovery.

Objective decomposition and lexicase selection (Boldi et al., 2023, Boldi et al., 2023) further show that diversity (as QD-score or Coverage) can often be maintained implicitly in linear time by optimizing many sub-objectives through randomized filtering rather than explicit diversity measures.

4. Fairness and Diversity in Large-scale Selection

Linear-time algorithms balance diversity (max-min distance) and fairness (representation constraints) in data selection and streaming (Moumoulidou et al., 2020, Addanki et al., 2022, Kurkure et al., 6 Apr 2024, Hu et al., 14 Apr 2025). Representative methods include:

• Fair-Swap and Fair-Flow: Swap-based and network-flow algorithms yield $1/4$ or $1/(3m-1)$ approximations in $O(n)$ time (where $m$ is the number of groups).
• Multiplicative Weights Update (MWU) with BBD-trees: Efficient linear-time LP relaxation and rounding techniques achieve constant-approximation for FairDiv in near-linear time using only $O(n)$ space, even for streaming and distributed environments (Kurkure et al., 6 Apr 2024).
• Bilevel randomized online policies: Hierarchical greedy and fractional-selection followed by online rounding enable max-min fairness objectives for diversity across $d$ dimensions in recruitment and crowdsourcing, with competitive ratio guarantees scaling as $1/(4\sqrt{d}\lceil \log_2 d \rceil)$ or $\Omega(1/d^{3/4})$ (Hu et al., 14 Apr 2025).

These frameworks generalize to overlapping groups and matroid constraints; streaming and composable coreset construction ensures the scalability of fair diversity maintenance in massive or online settings.

5. Particle Filtering and Ancestry-based Clustering

In particle filters, linear-time diversity maintenance is realized by leveraging the ancestry tree topology, avoiding explicit similarity computations. Particles are clustered based on shared ancestors in recursively-defined subtrees; clusters (niches) of size $\approx k$ are formed from maximal subtrees where the leaf count surpasses $k$ (Vallivaara et al., 28 Sep 2025). Intra-cluster fitness sharing normalizes weights, and cluster-dependent selection boosts unassigned particles, preventing premature convergence while maintaining compactness of estimate distributions. Clustering is performed in $O(P)$ time (for $P$ particles), with guaranteed robustness across multimodal data and challenging initial conditions.

6. Population Structure and Consensus Time

Diversity maintenance in structured populations, modeled by evolutionary graphs under birth-death or death-birth updating, exhibits absorption times sensitive to network topology (Brewster et al., 12 Mar 2025). For $N$ individuals:

• Complete graphs: $T_N = N^2-N$ (quadratic)
• Cycle: $T_N = \Theta(N^3)$ (cubic)
• Star (bd): $T_N = \Theta(N^3)$; Star (db): $T_N = \Theta(N\ln N)$ (quasilinear)
• Double star (bd): $T_N = \Theta(N^4)$ (quartic)
• Directed contracting star: $T_N \ge 2^{\Theta(N\ln N)}$ (superexponential)

Graph structure and updating rules yield a Pareto front: diversity can be sustained for superexponential times in designed graphs, but rapid consensus (loss of diversity) is attainable in nearly linear time for specific update protocols. This characterizes fundamental limits and optimal strategies for temporal diversity retention in evolutionary and agent-based models.

7. Concluding Synthesis

Linear-time diversity maintenance across diverse domains—clustering, streaming, evolutionary computation, fairness-aware selection, particle filtering, and population dynamics—is driven by algorithmic advances in core-set construction, incremental data structures, proxy and fitness-sharing mechanisms, objective decomposition, online rounding, and topology-aware clustering. These approaches rigorously balance approximation quality with scalability, with theoretical guarantees dependent on geometric or combinatorial properties (doubling dimension, population structure, attribute dimensionality). Research continues to probe trade-offs (e.g., approximation factors vs. running time vs. fairness guarantees), application in non-Euclidean and high-dimensional settings, and robustness in adversarial or dynamic environments.