Weighted Aggregation Method

• Weighted aggregation methods are statistical techniques that use coordinated sampling to create accurate, low-variance summaries of multi-weight data.
• The approach leverages shared randomness across weight assignments to minimize variance and reduce storage compared to independent sampling.
• Practical applications include network monitoring, financial analytics, and IoT data streams, enabling scalable and efficient query processing.

Weighted aggregation methods are a family of statistical and algorithmic techniques that enable the estimation or summarization of aggregates in large-scale data settings where each key (or unit) is assigned multiple sets of weights. This paradigm arises in numerous scenarios, including temporally evolving datasets, multi-attribute records, resource usage logs at various sites, and other cases requiring joint consideration of several weight assignments per item. Coordinated weighted sampling, the primary focus of this domain, constructs sample-based summaries that allow accurate, low-variance estimation of aggregates across one or more weight assignments with stringent space and computational guarantees (0906.4560).

1. Motivation and Context

Many big data sources are naturally modeled as a set of keys $I$, each carrying a vector of weights $w^{(b)}(i)$ for $b$ in some set $B$ (weight assignments). Applications driving this demand include:

• Snapshots of a database at multiple time points (evolving data)
• Telemetry, where each measurement is a vector (e.g., bytes, packets, features per IP flow)
• Aggregates over multiple weight sets (e.g., $L_1$ differences, maxima, minima)
• Queries determined "after the fact" requiring retrospective summarization

Classic sampling or sketching designs—such as weighted sampling, min-wise hashing, and bottom-$k$ sketches—were largely developed for the scalar case, i.e., each key has a single weight. In the face of vector-weighted data, naïve designs (independent samples per weight assignment) yield poor overlap and extremely high variance for multi-assignment aggregations.

Coordinated weighted sampling overcomes these limitations, allowing a single unified sketch to support accurate, low-variance, and storage-efficient estimation of diverse aggregates across all weight assignments.

2. Core Principles of Coordinated Weighted Sampling

The principle of coordinated weighted sampling is to "share randomness" between sampling operations for different weight assignments. Rather than draw independent random samples for each weight assignment $b \in B$, the approach generates a single uniform random value $u(i) \sim U[0,1]$ for each key $i$. This shared-seed is then used for all assignments, inducing strong positive correlation between the sampling decisions for each assignment.

For each assignment $b$ and key $i$, the rank is determined by a transformation of $u(i)$ and the weight:

• For exponential distribution (common in weighted sampling):

$r^{(b)}(i) = -\ln(u(i)) / w^{(b)}(i)$

• For priority/order sampling (using uniform $[0,1/w]$):

$r^{(b)}(i) = u(i) / w^{(b)}(i)$

A bottom-$k$ sample for each assignment is then extracted by selecting the $k$ keys with the smallest ranks $r^{(b)}(i)$. Crucially, all these samples are embedded in an overall summary, and the total number of distinct keys sampled across all assignments is much smaller than independent per-assignment sketches.

The inclusion probability of key $i$ with respect to assignment $b$ (conditionally on the $(k+1)$-st rank) is:

$$p(i, r^{-i}) = \min\{ 1, w^{(b)}(i) \cdot r^{(b)}_{(k+1)}(I) \}$$

or, when considering joint inclusion over multiple assignments,

$$p(i, r^{-i}) = \max_{b \in B} \{ F_{w^{(b)}(i)}( r^{(b)}_{(k+1)}(I) ) \}$$

where $F_w(\tau) = 1 - \exp(-w \tau)$ for exponential ranks.

Because the sampling is coordinated, a key “heavy” in one assignment (with high $w^{(b)}(i)$) is much more likely to be included in samples for other assignments, even if its weight under those is smaller. This dramatically reduces the variance of estimators involving multiple assignments.

3. Algorithmic Description and Estimation

The coordinated sketch enables design of unbiased estimators for a broad suite of aggregate queries, both within and across weight assignments.

A generic estimator for an aggregate of the form

$\sum_{i \in J} f(i)$

(where $f(i)$ may depend on one or multiple assignments; e.g., $f(i) = w^{(b)}(i)$, $f(i) = \max_b w^{(b)}(i)$, or $f(i) = L_1(\{ w^{(b)}(i) \}_b )$) is constructed by assigning each sampled key an “adjusted weight” $a(i)$ such that

$E[a(i)] = f(i)$

For example, for the Horvitz–Thompson estimator:

$a(i) = f(i) / p(i)$

with $p(i)$ the appropriate (possibly conditional) inclusion probability.

The approach further allows "inclusive estimators," and more nuanced “l-set” and “s-set” estimators, to exploit overlaps across assignment-specific samples for variance reduction.

Theoretical analysis in the paper proves (Theorem~\ref{sharing:lemma}) that coordinated samples are optimal in minimizing the number of distinct keys. The variance reduction for aggregates involving multiple assignments is shown empirically and theoretically to be orders-of-magnitude better than using independent samples.

4. Performance Characterization and Empirical Results

Empirical evaluation targets both IP packet trace data (keys: e.g., IP addresses/tuples; weight assignments: bytes, packets, time subsequences) and stock quotes (keys: tickers; assignments: price types, volumes, days). The following are observed:

• Variance: For $L_1$ difference queries (e.g., between two time periods), the coordinated method yields normalized estimator variances orders of magnitude lower than independent sampling.
• Storage: With coordinated sampling, the number of distinct sampled keys required is substantially reduced for the same target variance, demonstrating superior space efficiency.
• Flexibility: The same compact summary supports diverse queries (single assignment sums, maxima, minima, $L_1$ differences, subpopulation totals).

Empirical plots presented in the paper (e.g., variance vs. storage) make clear the magnitude of the improvement.

5. Implementation Characteristics and Scalability

Coordinated weighted sampling is streaming-friendly and compatible with distributed data collection. The summary is built via a single pass (or distributed passes) over data, with per-key sketches and a single, global randomness source. Bottom-$k$ sketches are space-efficient (size $O(k)$) and require per-key storage proportional to the number of keys that enter the union of per-assignment samples.

Key benefits:

• Processing and storage scale with $k$ (not dataset size), suitable for massive datasets and fast data streams.
• Sketches support a posteriori querying: new aggregate queries can be answered after the summary has been constructed, provided they are aggregations expressible in terms of sampled keys and associated weights.
• Direct application to network monitoring, stream analytics, and financial time-series, particularly where multiple metrics per entity must be summarized approximately under space or bandwidth constraints.

6. Theoretical Distinctions and Optimality

Key theoretical advances over previous (non-coordinated) designs include:

• Statistical Optimality: For any given total sample size, the variance of multi-assignment aggregates using coordinated samples is minimized. Independent samples produce weak estimators for quantities like max, min, and $L_1$ difference across assignments.
• Reusability: The set union of per-assignment samples is significantly smaller due to overlap, leveraging the statistical “importance” correlation induced by shared randomness.
• Generality: The approach generalizes to arbitrary injective weight assignments, holds for exponential and other common rank distributions, and is easily implemented in streaming or distributed contexts.

7. Practical Applications and Broader Impact

The coordinated weighted aggregation methodology directly addresses core problems in:

Application Domain	Aggregate(s) Enabled	Impact
IP Traffic Monitoring	$L_1$ difference, sum/by key	Efficient anomaly detection, bandwidth estimation
Financial Data (Stocks)	High/Low/Close/Max-L1 diff	Compact time-series analytics, flexible subqueries
Streaming Telemetry (IoT)	Multi-attribute aggregation	Scalable statistical summarization
Data Warehousing	Cross-snapshot difference	Efficient resource usage and change tracking

By enabling robust, low-variance, a posteriori aggregate computation under stringent space or bandwidth budgets, coordinated weighted sampling is foundational for scalable network measurement, financial monitoring, time-evolving databases, and big data analytics. Its generality and statistical optimality represent a significant advance in streaming summarization and approximate query processing (0906.4560).