Set-Overlap Scoring Rule

• Set-Overlap Scoring Rule is a mathematically rigorous method that quantifies similarity in discrete sets via maximal-weight bipartite matching and continuous level-set forecasts using proper scoring functions.
• It leverages semantic similarity measures, such as cosine similarity, to go beyond exact matching, enabling scalable top‑k retrieval and improved evaluation of multivariate forecasts.
• The framework employs filtering strategies and strict propriety to ensure computational efficiency and theoretical consistency in both search and probabilistic assessment.

The set-overlap scoring rule provides a principled and mathematically rigorous approach for quantifying the similarity or accuracy of sets in both discrete and continuous contexts. In the discrete setting, as utilized in set similarity search, the set-overlap score implements a maximal-weight bipartite matching between two sets, allowing for semantic similarity in addition to exact matches. In the continuous and probabilistic context, set-overlap scoring rules underpin the consistent assessment of level set forecasts—such as those for density or cumulative probability—by providing strictly proper scoring functions derived from $L^2$-type decompositions. Both paradigms have found critical applications: in scalable and exact top-$k$ semantic overlap search problems, as in the KOIOS framework, and in forecast evaluation for multivariate distributions. The resulting frameworks ensure both computational tractability and theoretical consistency, forming a foundation for advanced search, retrieval, and probabilistic assessment methodologies (Mundra et al., 2023, Meng et al., 2020).

1. Formal Definition: Discrete and Continuous Set-Overlap Scoring Rules

Discrete (Finite Set) Semantic Overlap

For two finite sets $A$ and $B$, the semantic overlap is defined as the maximum weight of a matching in a bipartite graph $G = (V, E)$ with $V = A \uplus B$ and $E \subseteq A \times B$, where the edge $(a, b)$ is assigned a nonnegative weight $w(a, b)$ reflecting the semantic similarity between $a \in A$ and $k$0. The set-overlap score is

$k$1

If $k$2, $k$3, the smaller side is padded with dummy nodes (zero weight edges) for use with standard assignment algorithms (Mundra et al., 2023).

Continuous (Level Set) Overlap

In the evaluation of forecasts for multivariate distributions, let $k$4 be an estimated distribution with smoothed density $k$5, and $k$6 an arbitrary Borel set (typically a level set of $k$7). The (level-set) set-overlap scoring function is

$k$8

where $k$9 is a smoothing kernel, $A$0 is a weight/density, $A$1 the realized outcome, and $A$2 the level set threshold (Meng et al., 2020).

2. Construction of Edge Weights and Similarity Functions

In the discrete semantic overlap paradigm, KOIOS generalizes overlap measurement by allowing $A$3 to arise from any symmetric similarity function $A$4. A canonical choice is cosine similarity between pre-trained embeddings: $A$5 An optional threshold $A$6 is applied: $A$7 Edges with $A$8 are omitted. This allows semantic relationships, not just exact matches, to be incorporated (Mundra et al., 2023).

In the continuous/scoring rule setting, $A$9 operates as a smoothing kernel determining the spatial penalty in the integral. For example, $B$0 for density level sets, or $B$1 for CDF level sets (Meng et al., 2020).

3. Thresholding, Normalization, and Properties

In KOIOS, the threshold $B$2 enforces a minimum semantic similarity, controlling which element pairs can participate in the matching. The resulting overlap score ranges in $B$3. Optional normalization by $B$4 is possible to map scores into $B$5, though no further normalization is applied by default (Mundra et al., 2023).

For level-set scoring functions, the integral is evaluated at fixed $B$6 (level set threshold), with proper weighting $B$7. These scoring rules are shown to be strictly proper: for each $B$8, the associated $B$9 has a unique minimum for the true level set $G = (V, E)$0 under reasonable regularity conditions (Meng et al., 2020).

4. Relationship to Exact Overlap and Classical Scoring Rules

If the similarity function is chosen as $G = (V, E)$1 if $G = (V, E)$2, $G = (V, E)$3 otherwise, the semantic overlap collapses to vanilla set intersection, i.e.,

$G = (V, E)$4

Generally, $G = (V, E)$5, since all identical elements contribute with maximum possible edge weight (Mundra et al., 2023).

In the continuous case, classical excess-mass and isoprobability scoring rules emerge as special cases of the general set-overlap scoring function for density and CDF level sets, respectively:

• Density: $G = (V, E)$6.
• CDF: $G = (V, E)$7 (Meng et al., 2020).

5. Computational Strategies and Efficient Algorithms

A naive implementation of maximum-weight matching has $G = (V, E)$8 complexity; therefore, KOIOS employs a filter-and-verify strategy for scalable top-$G = (V, E)$9 retrieval:

• Refinement: Uses an inverted index and semantic token stream to maintain and update upper/lower bounds (UB/LB) on overlap scores for candidates, rapidly pruning low-potential sets using Lemma-derived bounds.
• Post-processing: For the remaining candidates, exact overlap computation is selectively applied. Filtering techniques include the No-EM filter (skip matching when $V = A \uplus B$0), early termination based on dual variables during matching, and prioritization via max-heaps to optimize resource allocation and convergence (Mundra et al., 2023).

In continuous scoring, $V = A \uplus B$1 is computed via simple Monte Carlo integration: sample $V = A \uplus B$2 and average $V = A \uplus B$3 over those $V = A \uplus B$4, yielding convergence rate $V = A \uplus B$5 (Meng et al., 2020).

6. Applications and Empirical Evaluations

KOIOS applies the semantic overlap scoring rule for efficient and exact top-$V = A \uplus B$6 set similarity search in large repositories, demonstrating that for medium to large sets, less than $V = A \uplus B$7 of candidates require expensive bipartite matching. Pruning via filtering yields substantial computational savings without sacrificing exactness, and results favor semantic overlap over vanilla overlap for quality of retrieval (Mundra et al., 2023).

For set-overlap scoring rules in multivariate settings, simulation studies (e.g., mixtures of bivariate normals with varying parameters) confirm that the true data-generating process achieves the lowest average scores under density-level-set, CDF-level-set, and LPM-based set scores, establishing both ranking consistency and strict propriety. These results validate the theoretical framework for forecast evaluation and risk assessment (Meng et al., 2020).

7. Theoretical Properties and Consistency

For both discrete and continuous forms, set-overlap scoring rules inherit strict propriety under natural regularity conditions. In the discrete case, this means that only the sets with the highest actual semantic overlap will maximize the score. In the continuous context, Theorem 2 establishes that for each $V = A \uplus B$8, the expected score $V = A \uplus B$9 is minimized only when the forecasted set coincides (almost everywhere) with the true level set, provided $E \subseteq A \times B$0 almost everywhere and the true level set is appropriately regular (e.g., its boundary has zero $E \subseteq A \times B$1-measure) (Meng et al., 2020). This ensures that both semantic overlap search and probabilistic level-set evaluation incentivize truthful and accurate predictions or retrievals.