RegTop-k: Regret Minimization in Top-k Selections

Updated 27 March 2026

RegTop-k is a framework that unifies k-regret minimization queries with regularized top-k gradient sparsification to minimize worst-case regret for various linear utility functions.
It employs advanced algorithmic approaches—including greedy, sampling-based, and geometric methods—to provide provable performance guarantees in data retrieval and online ranking.
In distributed machine learning, RegTop-k leverages Bayesian regularization to sparsify gradients, enhancing convergence rates and reducing communication overhead.

RegTop-k (Regularized Top-k or k-Regret Minimization) has emerged as a central concept across multiple subfields in data management, information retrieval, online learning to rank, and distributed machine learning. The term denotes either advanced forms of top-k query processing that optimize worst-case user regret over all admissible utility functions (k-regret minimization) or regularized selection in top-k gradient sparsification (regularized Top-k), as well as specific algorithms for efficient partial-information learning in online ranking and in subset selection. The scope of RegTop-k thus encompasses the design, analysis, and application of algorithms that optimize top-k selections under various constraints, focusing on minimizing regret or maximizing statistical efficiency in adversarial, multi-objective, or resource-limited settings.

1. Formal Definitions and Theoretical Foundations

The RegTop-k framework is most rigorously defined in the context of k-regret minimization queries and their associated optimization criteria. Let $D \subseteq \mathbb{R}_+^d$ be a dataset of $n$ points, and let $\mathcal{U}$ denote a family of nonnegative linear utility functions $u: \mathbb{R}_+^d \rightarrow \mathbb{R}_+$ , specifically $u(p) = \sum_iv[i]\cdot p[i]$ for $v[i] \geq 0$ .

Regret-Ratio: For a subset $S \subseteq D$ and a utility $u\in \mathcal{U}$ , the regret ratio is

$R_D(S, u) = 1 - \frac{\max_{p\in S}u(p)}{\max_{p\in D}u(p)} \in [0,1].$

The maximum regret ratio over all utilities is

$R_D(S, \mathcal{U}) = \sup_{u\in \mathcal{U}} R_D(S, u).$

Happiness Ratio: Defined as $H_D(S, u) = 1 - R_D(S, u) = \frac{\max_{p\in S}u(p)}{\max_{p\in D}u(p)}$ , with the minimum over utilities $H_D(S, \mathcal{U}) = \inf_{u\in \mathcal{U}} H_D(S, u)$ .
RegTop-k query: The k-regret minimization query (RegTop-k) is the combinatorial optimization

$S^* = \arg\max_{S \subseteq D,\, |S| \leq k} H_D(S, \mathcal{U}) = \arg\min_{S \subseteq D,\, |S| \leq k} R_D(S, \mathcal{U}),$

i.e., finding a size- $k$ subset that minimizes the worst-case relative regret for all linear users.

In online learning to rank, RegTop-k also refers to achieving minimax regret under top-k feedback, where the learner receives only the relevance of the $k$ highest-ranked items each round and aims to minimize cumulative loss relative to the best fixed permutation in hindsight (Chaudhuri et al., 2016, Zhang et al., 2023).

In distributed gradient sparsification, "RegTop-k" designates the selection of $k$ gradient coordinates per worker via a Bayesian MAP estimate, incorporating past error accumulation to regularize the effective step size and ameliorate adverse convergence phenomena (Bereyhi et al., 10 Jan 2025, Bereyhi et al., 2024).

2. Algorithmic Approaches in RegTop-k

k-Regret Minimization: Greedy, Sampling, and Geometric Algorithms

Greedy (PreGreed): Iteratively builds $S$ by adding the point maximizing the marginal gain in $H_D(S, \mathcal{U})$ , requiring the solution of a LP at each step. This framework is not submodular but admits approximation guarantees via submodularity ratio $\gamma$ and curvature $\alpha$ :

$H_D(S_k, \mathcal{U}) \geq \frac{1}{\alpha}[1 - (1 - \frac{\alpha \gamma}{k})^{k-1}] H_D(S^*, \mathcal{U}),$

which approaches the classic $1-1/e$ bound as $\gamma, \alpha \to 1$ and $k \gg 1$ (Zheng et al., 2021).

Sampling-Based (StocPreGreed): Reduces computational burden by sampling a subset in each greedy iteration, with sample size $s = \frac{n}{k} \ln (\lambda/(\lambda-1+\epsilon))$ , maintaining provable guarantees for regret ratio in expectation.
Geometric Plane Sweep (2D) and Greedy Heuristics (High-D): Dual-space algorithms leverage top-k depth contours: for $d=2$ , an $O(c n^2)$ plane sweep with dynamic programming computes the optimal k-regret minimizer for cardinality budget $c$ ; for $d>2$ a greedy local search iteratively improves the current solution by swapping primal points whose dual hyperplanes cross the worst-case direction, exploiting the structure of distance-to-contour lemmas (Chester et al., 2012).

Regularized Top-k for Distributed Learning

Bayesian Regularized Top-k: Each worker models the global top- $k$ set as a latent variable, inferring the posterior that coordinate $j$ should be selected using a prior proportional to the local accumulated gradient and a likelihood informed by distortion from past aggregate gradients. The MAP mask selects entries maximizing $a_n^t(j)\cdot \tanh\left((1+\Delta_n^t(j))/\mu\right)$ , where $\Delta_n^t(j)$ is the posterior distortion metric and $\mu$ a hyperparameter. RegTop-k controls implicit learning rate scaling and reduces over-accumulation errors of classical Top-k (Bereyhi et al., 10 Jan 2025, Bereyhi et al., 2024).

3. Applications and Empirical Performance

RegTop-k strategies are utilized across diverse domains:

Database Systems: In multi-criteria decision making, RegTop-k queries select $k$ representative records which, for any linear user preference, return results with worst-case regret bounded by a small ratio. These methods mitigate the information overload of skyline queries and the inflexibility of fixed top-k queries (Zheng et al., 2021, Chester et al., 2012).
Hidden Matrix Top-k Retrieval: For approximate retrieval with unknown or expensive attribute access, RegTop-k algorithms use regression or kernel-smoothed Gaussian models on observed prefixes to prune candidates dynamically, thus substantially reducing query cost while controlling the loss in top-k accuracy (Ukkonen, 2010).
Online Learning to Rank: RegTop-k characterizes regret-optimal behavior under top-k feedback, including block-exploration strategies, unbiased gradient estimators for contextual surrogates, and formal lower bounds. Regret rates transition from $\Theta(T^{2/3})$ to $\Theta(T^{1/2})$ as feedback increases or for special loss functions (e.g., Precision@n) (Chaudhuri et al., 2016, Zhang et al., 2023).
Distributed Training: In large-scale SGD, RegTop-k sparsification enables aggressive gradient compression (e.g., 0.1% density) with minimal accuracy loss and eliminates bias effects that stall convergence at high compression, as evidenced in deep network experiments (ResNet-18/CIFAR-10: 8 percentage point gain over Top-k at fixed sparsity) (Bereyhi et al., 10 Jan 2025, Bereyhi et al., 2024).

4. Computational Complexity and Theoretical Guarantees

The principal RegTop-k algorithms exhibit varied complexity profiles and theoretical properties:

Setting	Algorithm	Time Complexity	Approximation/Regret Guarantee
k-Regret Minimization (full)	PreGreed	$O(n k^3 d)$	$(1-1/e)$ -type approx. (weak submodular)
k-Regret Minimization (sampled)	StocPreGreed	$O(k s k^2 d)$ , $s = O(n)$	$1-\exp(-...)$ in expectation
2D Plane Sweep	Sweep-DP	$O(c n^2)$	Exact in $d=2$
High-dimensional Greedy	Greedy-LS	$O(n c^2)$ per iteration	Empirically near-optimal
Distributed RegTop-k	Regularized Top-k	$O(J)$ /worker/round	Full-convergence in convex, smooth cases
Hidden-matrix Retrieval	Regression-pruning	$O(nm)$ worst, $O(nm')$ avg	Empirical, no formal error bound
Online Learning to Rank (ranking)	Blocked + FTPL, OGD	Poly( $m$ , $k$ ), $T$	$O(T^{2/3})$ , $O(T^{1/2})$

Key results include:

NP-hardness of the decision version of k-regret minimization (by reduction from Partition/ $k$ -Set Cover) (Zheng et al., 2021).
Regret-optimality in online ranking only when $k\geq m-1$ (full or nearly full feedback) for generic measures, but for Precision@n, $k=1$ suffices for $\Theta(T^{1/2})$ minimax regret (Zhang et al., 2023).
Formal expectation bounds for StocPreGreed and RegTop-k convergence and quality loss under aggressive compression or subsampling.
In practice, theoretical submodularity ratios and curvature are typically close to 1, and empirical regret ratios are often $\ll 0.01$ for $k\geq 25$ (Zheng et al., 2021).

RegTop-k encompasses and generalizes several classical and modern paradigms:

Skyline and Set Cover: RegTop-k generalizes skyline queries by bounding the "regret" a user experiences for any linear utility, and is related to multi-cover instances in set systems (Zheng et al., 2021).
Partial Monitoring and Bandit Feedback: In online learning, RegTop-k is characterized within the partial monitoring hierarchy; global and local observability properties partition the achievable minimax regret regimes (Chaudhuri et al., 2016, Zhang et al., 2023).
Gradient Compression and Error Feedback: Online Bayesian regularization in RegTop-k sparsification directly addresses the pathological scaling and bias phenomena endemic to simple Top-k error accumulation (Bereyhi et al., 10 Jan 2025, Bereyhi et al., 2024).
Approximate Retrieval: Regression-guided row pruning in hidden-matrix RegTop-k methods leverages statistical prediction to minimize expensive reads, representing a fusion of top-k retrieval, active learning, and probabilistic inference (Ukkonen, 2010).

Research continues on formalizing approximation/cost guarantees for regression-based pruning, extending regret analyses to multi-graded or adaptive adversaries, and adapting geometric regret minimizers to large-scale or streaming architectures.

6. Empirical Findings and Practical Performance

Comprehensive empirical studies across real and synthetic datasets highlight the efficacy and stability of RegTop-k methods:

In multi-objective datasets, PreGreed and StocPreGreed consistently achieve worst-case regret within 1–5% of optimal, with order-of-magnitude speedups on large skylines and highly stable performance across multiple runs. Empirical submodularity ratios ( $\gamma$ ) and curvatures ( $\alpha$ ) are close to 1, mitigating the gap between worst-case and in-practice guarantees (Zheng et al., 2021).
Hidden-matrix RegTop-k achieves drastic cost reductions (down to 20–40% of full cost) while retaining 80–90% inexact top-k accuracy, outperforming branch-and-bound baselines by a wide margin (Ukkonen, 2010).
Distributed RegTop-k provides $8$–$9$ percentage point test accuracy lifts over standard Top-k for deep models, and closes most of the gap to uncompressed training at extreme sparsities (Bereyhi et al., 10 Jan 2025, Bereyhi et al., 2024).
In online ranking, algorithmic instantiations achieve the minimax regret rates dictated by the underlying loss/feedback structure, and block-based strategies for non-contextual and contextual settings yield performance close to information-theoretic limits even under sparse feedback (Zhang et al., 2023, Chaudhuri et al., 2016).

7. Assumptions, Limitations, and Open Problems

The RegTop-k suite is subject to several structural and theoretical constraints:

All theoretical guarantees in k-regret minimization assume nonnegative linear utilities; extending to nonlinear functions remains open.
No formal error bounds for regression-based hidden-matrix retrieval—performance is empirical and dependent on the representativeness and Gaussianity of training data (Ukkonen, 2010).
For online learning to rank, regret rates rely on strict global or local observability of the bandit feedback structure; for normalized ranking measures (e.g., NDCG), no algorithm can achieve sublinear regret with top-1 feedback (Chaudhuri et al., 2016, Zhang et al., 2023).
In distributed learning, convergence guarantees for RegTop-k inherit smoothness and convexity assumptions; theoretical improvement over classical Top-k is predicated on the regularization effect preventing overaccumulation (Bereyhi et al., 10 Jan 2025).

Future research directions include formalizing cost-vs-accuracy trade-offs in probabilistic heuristics, extending geometric algorithms beyond $d=2$ , adapting to adaptive adversarial feedback, and leveraging more sophisticated Bayesian inference or regularization structures for robust sparsification and query processing.