---
title: Min-K% Probability Analysis
url: https://www.emergentmind.com/topics/min-k-probability-analysis
type: topic
---

# Min-K% Probability Analysis

Min-K% Probability Analysis is a statistical framework focused on the analysis and high-confidence quantile estimation of the minimum of a collection of random variables, typically using order statistics such as the minimum of i.i.d. samples or the minima of token-level log-probabilities in autoregressive models. Its central concept is to quantify the most extreme (lowest) fraction—namely, the worst-performing k%—of a sequence of probabilistic outcomes for robust risk and anomaly assessment, pre-training data detection, and reliable high-probability guarantees.

## 1. Mathematical Formulation and Core Principles

The Min-K% criterion evaluates the behavior of the lowest k% order statistics. Given a sequence $\{Z_1, ..., Z_T\}$ (e.g., token-level log-likelihoods), the Min-K% score is the mean of the lowest m values, where $m = \lceil k\% \times T \rceil$ and the order statistics $Z_{(1)} \le \dots \le Z_{(T)}$ are sorted. For log-probabilities in language models:

\[
\text{Min-K\%}_k(x) = \frac{1}{m} \sum_{j=1}^m \ell_{(j)}
\]

where $\ell_{(j)}$ are the sorted log-probabilities. This technique generalizes to other settings, including coverage of quantiles and the minimum across i.i.d. random variables. In probabilistic estimation, "Min-K% Probability Analysis" refers to the explicit control and quantification of the smallest (worst) outcomes, and their deviation probabilities, via information-theoretic bounds.

## 2. Concentration Inequalities for the Minimum

Tight finite-sample bounds on the minimum of i.i.d. random variables, particularly binomial or general discrete/continuous distributions, underpin rigorous Min-K% analysis. Let $X_1, ..., X_m$ be independent $\mathrm{Bin}(n,p)$, and $M = \min_i X_i$. Explicit nonasymptotic high-probability bounds can be derived as follows [2502.18611]:

- **Lower bound:** For threshold $k$, with $t = k/n$,
  \[
  \Pr[M \geq k] \geq 1 - m\,e^{-n\,D(t\|p)}
  \]
- **Upper bound:**
  \[
  \Pr[M < k] \leq 1 - \left(1 - e^{-n\,D(t\|p)-C_n} \right)^m
  \]
  where $D(t\|p)$ is the binary KL divergence and $C_n = 4\log(n+1) + [\log\frac{p}{1-p}]_+$.

For quantile-based selection, identify $k$ corresponding to the $(1-q)$-quantile of $\mathrm{Bin}(n,p)$ via $D(k/n \| p) \approx \frac{\log(1/q)}{n}$. This framework generalizes: for any collection of i.i.d. variables, the Min-K% behavior is tightly controlled by Sanov-type and Chernoff-style exponential bounds in terms of KL divergence.

## 3. Min-K% in Statistical Estimation: High-Confidence Quantiles and Missing Mass

In distribution estimation, Min-K% analysis allows for robust large-deviation quantile control. For the missing mass $U_n$ (the remaining probability of symbols not observed in a sample from a discrete distribution), explicit quantile bounds follow from variance-sensitive large-deviation inequalities [1210.3248]:

\[
m_k = \mathbb{E} U_n + \sqrt{\frac{1}{n} \ln\frac{1}{\delta}}
\]

ensuring $P(U_n \geq m_k) \leq \delta$, thus controlling the upper Min-K% quantiles. This principle extends to other order statistics, quantiles, and risk measures for high-probability guarantees.

## 4. Min-K% for Model Behavior: Memorization and Outlier Detection

The Min-K% statistic has been adopted for pre-training data detection in large language models. For each position $t$, using the model conditional distribution $P(\cdot | x_{<t})$, Min-K% selects the lowest-$k\%$ log-likelihoods across a sequence. Empirically, these measure the model's weakest predictions, which are highly indicative of whether a sample was seen during training [2404.02936].

- **Min-K%++ Extension:** Advances beyond averaging the lowest log-probabilities by normalizing token log-probs against the context’s conditional mean and variance, i.e.,
  \[
  s_t = \frac{\log P(x_t|x_{<t}) - \mu_{x_{<t}}}{\sigma_{x_{<t}}}
  \]
This discrete curvature score sharply identifies local maxima ("memorized" points). The Min-K%++ method shows substantial AUROC improvements in training- vs. non-training data detection benchmarks.

| Method      | Statistic                | Application                       |
|-------------|------------------------- |-----------------------------------|
| Min-K%      | Worst-$k\%$ log-probs    | Non-training data identification  |
| Min-K%++    | Curvature-normalized min | Mode/memorization detection       |

These Min-K% analytics are robust to context variability and reveal sharp boundaries between memorized and non-memorized sequences.

## 5. Universal Lower Bounds and Sharpness in Continuous Laws

In continuous probability contexts, universal Min-K% lower bounds quantify how likely it is that, conditional on an extreme event (sum exceeding a threshold), the minimum of a set takes small values. For i.i.d. $(X, Y)$ with continuous density $f$ and median $\operatorname{med}(X)$,
\[
\sup_{z > 0} \mathbb{P}\bigl(\min(X,Y)\leq z \mid X+Y \geq 2z\bigr)
\geq \frac{1}{24 + 8\log_2(\operatorname{med}(X)\|f\|_{L^\infty})}
\]
This logarithmic denominator is unavoidable—sharpened constructions prove the bound is optimal up to constants [1801.00663].

## 6. High-Probability Minimax Rates in Discrete Estimation

In large-deviation regimes, Min-K% quantile control is essential in establishing high-probability minimax lower bounds. For estimating a discrete distribution $p$ of support size $K$ from $n$ samples, the minimax lower bound for the KL risk at confidence level $\delta$ is
\[
\Omega\left(\frac{\max\{K,\,\ln K\ln(1/\delta)\}}{n}\right)
\]
No estimator can beat this rate for the Min-K% quantile of the KL loss [2507.17316]. Efficient algorithms such as OTB (Online-to-Batch) with suffix averaging achieve matching upper bounds up to additional $\ln\ln K$ factors. The penalty in controlling rare Min-K% probability events accounts for the additional sample complexity compared to expected-risk settings.

| Setting                       | Min-K% Rate                                       | Reference       |
|-------------------------------|---------------------------------------------------|-----------------|
| Expected KL risk              | $\Theta(K/n)$                                     | [2507.17316]    |
| High-prob Min-K% KL quantile  | $\max\{K, \ln K \ln(1/\delta)\}/n$                | [2507.17316]    |

## 7. Connections and Open Problems

Min-K% probability analysis bridges order statistics, large deviations, concentration of measure, and robust estimation. Its methodology elucidates phenomena in both discrete (model quantiles, missing mass, high-confidence tail behavior) and continuous (scale-invariant conditional minima) probabilistic systems. Open questions remain regarding optimal Min-K% bounds for sums of independent random variables, weightings in scale-dependent inequalities, and tightness classes for particular distribution shapes [1801.00663].

The Min-K% paradigm thus delivers a general toolkit for quantifying tail events, calibrating risk, and detecting outlier or memorized behavior in high-dimensional statistical and machine learning settings, supported by sharp probabilistic inequalities and algorithmic adaptivity.

Source: https://www.emergentmind.com/topics/min-k-probability-analysis