Sparse Probability of Agreement (SPA)

• SPA is a measure that quantifies agreement in sparsely observed settings, generalizing pairwise agreement for both annotation tasks and random CSP solutions.
• It employs various weighting schemes—including flat, annotations_m1, and inverse-variance—to minimize variance while preserving unbiasedness under random missingness.
• Efficient computation and empirical validation demonstrate that SPA scales well and offers precise overlap estimation in large-scale, incomplete data environments.

Sparse Probability of Agreement (SPA) quantifies agreement or overlap rates in settings where observations or labels are only sparsely and incompletely available. SPA generalizes the notion of pairwise agreement in two important domains: inter-annotator agreement in annotation tasks, and empirical overlap in solutions of high-dimensional linear systems with random structure. In both cases, SPA provides a principled estimator or limiting value for the probability that two randomly chosen elements (annotators, solutions) agree on a random instance (item, variable), under conditions of sparse observation or structural optimization constraints.

1. SPA in Annotation Tasks: Formal Definition and Motivation

Given $I$ items, each labeled by a varying subset ($n_i$) of annotators and $C$ possible labels, let $n_{i c}$ denote the count of annotators assigning label $c$ to item $i$. The item-level agreement probability is defined as

$$P_i = \frac{\sum_{c=1}^C n_{i\,c} (n_{i\,c} - 1)}{n_i(n_i-1)}$$

which is the probability that two distinct, randomly selected annotators agree on item $i$.

Traditional agreement metrics assume all annotators label all items ($n_i = n\,\,\forall i$), so mean item agreement gives the standard “joint probability of agreement.” SPA generalizes this to arbitrary sparsity by introducing a nonnegative weight $k_i$ per item and defining

$$P_\text{SPA} = \frac{1}{\sum_{i=1}^I k_i} \sum_{i=1}^I k_i \frac{\sum_{c=1}^C n_{i\,c} (n_{i\,c}-1)}{n_i(n_i-1)}$$

$P_\text{SPA}$ represents the probability that two randomly chosen annotators agree on a randomly chosen item, where both draw and item weighting are precisely specified to accommodate arbitrarily missing labels (Nørregaard et al., 2022).

2. Unbiasedness Under Random Missingness

SPA is constructed so that, when annotations are missing completely at random (MCAR)—that is, the probability any annotation is missing may depend on $i$ (the item index) but not on the true agreement or label—then $\mathbb{E}[P_\text{SPA}]$ equals the fully observed agreement.

For a single item, the expected agreement probability is preserved as annotations are dropped randomly down to as few as two per item, via

$\mathbb{E}[P_i'] = P_i$

where $P_i$ and $P_i'$ denote the pre- and post-removal probabilities, respectively. At the dataset level, the weighted sum over items is also preserved in expectation. Therefore, under the MCAR assumption, SPA is an unbiased estimator for overall agreement even in highly incomplete annotation matrices (Nørregaard et al., 2022).

3. Weighting Schemes and Variance Considerations

While unbiasedness of SPA is invariant to the choice of weights $k_i$, its variance is sensitive to this choice. The following weighting schemes are provided:

Name	Weight Expression	Notes
flat	$k_i = 1$	All items equally weighted
annotations	$k_i = n_i$	Proportional to number of annotations
annotations_m1	$k_i = n_i - 1$	Proportional to number of annotator pairs; $0$ if singleton
edges	$k_i = \binom{n_i}{2}$	Number of annotation pairs per item
inv_var	$k_i \propto 1/\operatorname{Var}[P_i]$	Minimizes variance, does not require class prior
inv_var_class	$k_i \propto 1/\operatorname{Var}[P_i]$ (with class prior)	Refines variance under known/estimated label distribution

Simple weighting schemes offer interpretability and ease of computation; “annotations_m1” delivers the greatest variance reduction among basic choices. Inverse-variance weighting schemes—either with or without class prior—minimize $\operatorname{Var}[P_\text{SPA}]$ and behave similarly to the “edges” scheme in empirical studies (Nørregaard et al., 2022).

4. Algorithmic Computation and Complexity

Computation of SPA proceeds as follows:

1. For each item $i$, compute $n_i = \sum_{c} n_{i\,c}$. Exclude items with $n_i < 2$.
2. Calculate $P_i$ via the agreement formula.
3. Assign $k_i$ according to the chosen weighting scheme.
4. Normalize by $K = \sum_i k_i$.
5. Output $P_\text{SPA} = \frac{1}{K}\sum_i k_i\,P_i$.

Per-item computation is $O(n_i + C)$, with overall computational complexity $O(N + I C)$ (where $N = \sum_i n_i$). Flat, annotations, annotations_m1, and edges impose trivial $O(1)$ per-item overhead. Inverse-variance schemes may require up to $O(n_\text{max}^2)$ precomputation (Nørregaard et al., 2022).

5. Empirical Behavior and Practical Guidance

Empirical evaluations on crowdsourced annotation datasets demonstrate two central findings:

• Random removal of annotations leaves the expected SPA unchanged, empirically confirming unbiasedness.
• Variance reductions are observed as either the number of annotators per item or the number of annotated items increases. The annotations_m1 scheme delivers the greatest variance reduction among simple schemes; inverse-variance schemes (especially without a class prior) perform comparably to the edges weighting (Nørregaard et al., 2022).

These results support the practical recommendation of pairing simplicity (flat, annotations_m1) in general scenarios with theoretically grounded inverse-variance weighting where optimality in uncertainty is desired.

6. SPA as Overlap in Sparse Random Systems

Beyond annotation, SPA also arises as the “overlap” in random constraint satisfaction problems. Notably, the analysis of the sparse parity (XORSAT) model over $\mathbb{F}_2$ explores SPA as the empirical agreement fraction between two independent solutions $x, x'$ of a sparse linear system $A x = y$, where $A$ is a random $n \times n$ matrix with each entry $A_{ij} = 1$ independently with probability $d/n$.

The overlap is defined as

$$\alpha(x,x') = \frac{1}{n} \left| \{ i : x_i = x'_i \} \right|$$

For $d < e$, this overlap concentrates around a deterministic value $(1+\alpha_*)/2$, where $\alpha_*$ solves a certain fixed-point equation depending on $d$. For $d > e$, the overlap, conditioned on $A$, is sharply concentrated but, when averaged over matrices, splits between two values $(1+\alpha_*)/2$ and $(1+\alpha^*)/2$ with asymptotic probabilities $1/2$ each. These regimes reflect critical phenomena in random CSPs and connections to replica symmetry and phase transitions (Coja-Oghlan et al., 2021).

7. Broader Context and Theoretical Significance

SPA provides a flexible, unbiased measure of agreement in sparse, arbitrarily incomplete settings. In annotation, it subsumes the joint probability of agreement and addresses realistic requirements in crowdsourcing, where full label matrices are unattainable. In random combinatorial optimization, SPA (as solution overlap) offers insight into structural transitions, concentration properties, and symmetry breaking phenomena.

SPA’s weighting flexibility supports practical and theoretical requirements, trading variance minimization for interpretability, and its computational properties ensure scalability to large datasets and systems (Nørregaard et al., 2022, Coja-Oghlan et al., 2021). Its applicability across domains underscores its significance in modern large-scale data, learning, and inference problems.