Quantile Tracking via P Control

• Quantile tracking is a method employing proportional-feedback control to estimate quantiles with rigorous error guarantees in dynamic and distributed data streams.
• The approach includes the QEWA method for single streams and centralized or decentralized protocols, which adjust estimates rapidly using adaptive gains.
• Practical implementations demonstrate low computational cost and robust error control, making these methods essential for real-time analytics and drift detection.

Quantile tracking (“P control”) refers to a class of algorithms, models, and distributed protocols designed for online estimation and tracking of quantiles in data streams and distributed systems. These methods guarantee accuracy in the presence of dynamism, nonstationarity, and partial observability. P control terminology highlights the proportional-feedback structure of some quantile tracking updates, where the estimator is adjusted at each time step by an amount proportional to a quantile-focused error. Quantile tracking is a foundational primitive in real-time analytics, distributed monitoring, and adaptive control.

1. Mathematical Principles and P-Control Analogy

Quantile tracking algorithms seek to estimate the $q$-th quantile $Q_n$ of a dynamic or distributed data stream with rigorous error guarantees. The core mechanism of P-control-based methods is the application of discrete-time proportional feedback, analogous to a P-loop in control theory.

In the sequential P-control setting, the update for the current quantile estimate $Q_n$ is given by

$Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$

where the mixing weight $b_n$ is dynamically computed from discrepancies between $x_n$ (the current observation) and $Q_n$ (the estimate), with $b_n$ increasing as $|x_n - Q_n|$ grows. This constitutes a proportional-feedback controller on the quantile estimation error:

$e_n = q - F(Q_n),$

where $Q_n$0 is the true CDF. The expected update has the form

$Q_n$1

with loop gain $Q_n$2 and normalization $Q_n$3. This structure ensures that the estimator promptly corrects for drift or sudden changes in the underlying data stream, providing rapid adaptation to nonstationarity while maintaining quantile-specific guarantees (Hammer et al., 2019).

2. Single-Stream Quantile Tracking via Generalized EWAs

The quantile estimation in dynamically varying, single-stream environments is addressed with the Generalized Exponentially Weighted Average (QEWA) approach (Hammer et al., 2019). This method adjusts its step size online based on the conditional means above and below the current estimate:

$Q_n$4

with a normalization

$Q_n$5

For $Q_n$6:

$Q_n$7

and for $Q_n$8:

$Q_n$9

The adaptive gain $Q_n$0 ensures that larger deviations invoke larger corrections, yielding fast recovery in the presence of concept drift or abrupt distributional shifts.

Theoretical convergence is established via stochastic approximation, with guarantees that $Q_n$1 as $Q_n$2, $Q_n$3, and $Q_n$4, provided stability criteria on conditional mean sequence and step-sizes (Hammer et al., 2019).

3. Distributed Quantile Tracking: Centralized and Decentralized Settings

Distributed quantile tracking addresses the estimation of quantiles over data generated at multiple, geographically separated sites, under communication constraints. Two dominant approaches are represented in the literature:

3.1 Centralized Coordinator Model (Additive Error)

In (0812.0209), $Q_n$5 remote sites and a central coordinator collaboratively track a $Q_n$6-quantile under the guarantee

$Q_n$7

with $Q_n$8 being the global multiset. The protocol leverages local triggering via counters for arrivals $Q_n$9 or $Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$0 (the current estimate), sending messages to the coordinator only when local discrepancies surpass $Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$1 within a round of size $Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$2. The coordinator invokes a “recenter” procedure to reselect $Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$3 when the accumulated discrepancy between left and right counts exceeds a threshold.

Multi-quantile support is provided through a binary interval tree maintained by the coordinator, enabling simultaneous tracking of all quantiles at comparable cost. The protocol achieves a worst-case communication complexity of $Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$4, with a matching lower bound (0812.0209).

3.2 Decentralized Gossip-Based Protocols (Relative Error)

In the unstructured P2P setting, DUDDSketch (Pulimeno et al., 23 Jul 2025) deploys a fully decentralized, gossip-based protocol for collaborative tracking of quantiles with relative-value error:

$Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$5

across a network of $Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$6 peers, each maintaining an $Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$7-space UDDSketch summary [UDDSketch: Epicoco et al. 2020]. The system iteratively merges and averages sketch summaries and scalars ($Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$8, $Q_{n+1} = (1 - b_n) Q_n + b_n x_n,$9) with their neighbors using push–pull gossip. The space and communication are both independent of input cardinality, and convergence to a globally consistent sketch is guaranteed with error decaying exponentially in the number of gossip rounds.

UDDSketch employs logarithmic bucketing for quantile estimation, with collapses when buckets exceed $b_n$0 in number. The inflation of relative error due to collapse is tightly controlled, converging to the same summary as the sequential UDDSketch that would be computed on the global dataset (Pulimeno et al., 23 Jul 2025).

4. Error Guarantees and Theoretical Properties

A summary of key error guarantees for representative quantile tracking protocols appears below.

Approach	Error Type	Guarantee	Asymptotics
QEWA (P control, single stream)	Online RMSE, convergence	$b_n$1; rapid error reduction after concept drift	$b_n$2 time, memory
Central coordinator (additive) (0812.0209)	Additive rank	$b_n$3	$b_n$4 comm
DUDDSketch (Pulimeno et al., 23 Jul 2025)	Relative-value	$b_n$5	$b_n$6 space; $b_n$7 rounds

DUDDSketch preserves permutation invariance and mergeability, ensuring the distributed sum of local sketches reduces to the same quantile summary as the centralized algorithm. Gossip-based consensus converges exponentially fast due to the spectral properties of the averaging protocol, with bucket estimate errors decaying as $b_n$8 (convergence factor $b_n$9) (Pulimeno et al., 23 Jul 2025).

In central-coordinator protocols, the communication lower bound of $x_n$0 is shown to be tight via adversarial constructions that maximize quantile flips, necessitating coordinated communication throughout the event sequence (0812.0209).

5. Implementation, Algorithm Design, and Practical Issues

In QEWA and related single-stream P-control schemes, per-sample computational and storage costs are $x_n$1, controlled by efficient recursions for quantile and conditional mean estimates. Step-size $x_n$2 determines the trade-off between adaptation speed and variance; $x_n$3 is recommended, where $x_n$4 is the characteristic timescale of drift (Hammer et al., 2019). Forgetting factors for conditional means $x_n$5 ensure locality. Initialization of $x_n$6 and conditional means critically impacts early performance, with empirical quantile or trimmed mean suggested.

For distributed protocols, batching and quantile interval sketching (separator maintenance) enable amortization of communication overhead. DUDDSketch offers resilience to network churn and node failures, with convergence holding robustly under various real-world failure and churn models, provided distribution is not adversarially skewed to exploit network partitioning (Pulimeno et al., 23 Jul 2025). Centralized approaches are sensitive to the accuracy of local counter triggers and to communication delays, though quantile-tracking error remains bounded at all times.

Randomization and sampling can relax communication requirements but may lose determinism and worst-case optimality for all parameter regimes (0812.0209).

6. Experimental Evaluation and Applications

Extensive synthetic and real-world experimental evaluations confirm the superior adaptivity and tracking accuracy of P-control-based quantile tracking. QEWA demonstrates the lowest RMSE and quickest adaptation among tested estimators on data streams subject to both gradual and abrupt concept drift; the self-tuning gain mechanism leads to improved "snap-back" after quantile-crossing outliers (Hammer et al., 2019). In applications such as real-time concept drift detection for indoor climate control, QEWA enables rapid retraining of prediction models, keeping forecast error quantiles within desired thresholds.

DUDDSketch achieves fast convergence (approximately 10–25 gossip rounds on standard topologies) for global quantile tracking in large-scale P2P settings. Quantile tracking error uniformly approaches zero compared to the centralized ground-truth summary. Experiments on adversarial, uniform, exponential, and real energy consumption data validate theoretical error bounds and system robustness under churn (Pulimeno et al., 23 Jul 2025).

The central-coordinator additive-error algorithm maintains bounded error under arbitrary data arrival sequences and supports practical quantile errors in the 1–5% range, with computational and communication costs aligned with theoretical asymptotics (0812.0209).

7. Context and Connections to Related Methodology

Quantile tracking with P-control unifies insights from stochastic approximation, streaming sketch algorithms, distributed aggregation, and feedback control theory. Its application ranges from adaptive machine learning pipelines—where continuous monitoring and drift-detection are essential—to streaming analytics and large-scale distributed monitoring (SLAs, risk, and health monitoring).

Connections to mergeable sketch structures (e.g., UDDSketch, Greenwald-Khanna summaries) underline the importance of compositionality and permutation invariance for distributed systems. P-control interprets quantile estimation as control of a stochastic dynamical system, where adaptation targets distributional statistics, leading to provably convergent and robust algorithms that generalize classical mean-tracking EWAs and stochastic gradient estimators.

A plausible implication is that as quantile-based SLAs and real-time monitoring proliferate, proportional-feedback quantile trackers will remain central to both centralized and decentralized approaches, given their rigorous error control, efficiency, and theoretic underpinnings (Hammer et al., 2019, 0812.0209, Pulimeno et al., 23 Jul 2025).