Empirical Success Rule: Theory & Applications

• The empirical success rule is a data-dependent decision procedure that selects the option with the highest observed average outcome without relying on preset significance thresholds.
• In clinical trials, it achieves near-optimality with a maximum regret as low as 0.012 compared to 0.071 for 5% hypothesis testing, markedly reducing type II errors.
• It underpins adaptive methods in evolutionary computation and sequential stopping problems, enabling dynamic parameter adjustment and robust performance across settings.

Searching arXiv for recent and relevant papers on "empirical success rule" and related "success rule" usage. In the cited literature, the empirical success rule denotes a family of data-dependent decision procedures that act on observed success rather than on oracle knowledge or rigid significance thresholds. In statistical decision theory, it prescribes the treatment with the highest observed average patient outcome in a randomized trial; in evolutionary computation, closely related success rules adapt step size or mutation rate according to observed improvement frequencies; and in sequential stopping with unknown Bernoulli parameter, a plug-in rule replaces the unknown success probability by an online empirical estimate (Manski et al., 2020, Akimoto et al., 2018, Paindaveine, 8 Apr 2026).

1. Formal statistical-decision meaning

Manski and Tetenov formulate the empirical success rule as a treatment rule for randomized trials. For treatments $t=1,\ldots,L$, let

$\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$

where $u(\cdot)$ is the patient-relevant outcome function. The rule chooses

$t^*=\arg\max_t \hat{u}_t,$

and, if there is a tie, randomizes evenly among maximizers (Manski et al., 2020).

A central property of this formulation is symmetry across trial arms. The rule does not defer to standard care as a status quo and does not require that an innovation pass a statistical-significance threshold before it can be chosen. In the COVID-19 trial setting studied by Manski and Tetenov, this symmetry is presented as especially relevant because standard care itself may be uncertain or suboptimal (Manski et al., 2020).

Evaluation is conducted through near-optimality, defined via maximum regret. In state $s$,

$$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$

where $u_{st}$ is the true mean outcome of treatment $t$, $\mathbb{E}_s[\delta(t,\psi)]$ is the expected fraction assigned to $t$ by rule $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$0, and $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$1 is the observed data. Near-optimality is the maximum regret over all possible states. This criterion jointly accounts for the probability and magnitude of decision errors (Manski et al., 2020).

2. Near-optimality and comparison with hypothesis testing

In the trial framework, the empirical success rule is compared with the conventional practice of choosing an innovation only if the estimated treatment effect is positive and statistically significant. Manski and Tetenov argue that the conventional rule is asymmetric, defers to standard care, and can incur substantial type II error costs when trials are small and estimates are imprecise (Manski et al., 2020).

The reported quantitative comparisons are sharp. In two-arm trials with 100 patients per group and binary outcome, the empirical success rule achieves near-optimality of $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$2, versus $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$3 for $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$4 hypothesis testing; the paper states that the empirical success rule is about six times nearer to optimality. In multi-arm trials, including five-arm settings, the regret of the empirical success rule is reported as more than three times lower than that of multiple-hypothesis-test-based rules. The same summary states that using the empirical success rule with 100 patients per arm confers the same reduction in maximum expected loss as massively increasing sample size for hypothesis testing, with the approximation “ESR with 100/arm $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$5 Hypothesis testing with 4000/arm” (Manski et al., 2020).

The illustrative example given in the same work emphasizes magnitude-sensitive decision error. When mortality under a new treatment is $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$6 and under standard care is $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$7, the hypothesis-test rule fails to adopt the better treatment with probability $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$8, yielding expected loss $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$9; the empirical success rule yields expected loss $u(\cdot)$0 (Manski et al., 2020).

The paper further reports that the empirical success rule achieves the minimal possible near-optimality in binary, balanced trials, citing Stoye and Hirano & Porter. It also states that the good properties extend to multi-arm trials, to heterogeneous patients by applying the rule within covariate strata, and to multiple outcomes by defining composite welfare measures (Manski et al., 2020).

3. Success-based adaptation in evolutionary computation

In evolutionary computation, the most prominent related construct is the one-fifth success rule originating with Rechenberg. In the $u(\cdot)$1-Evolution Strategy analyzed on the sphere function, the algorithm maintains a mean vector $u(\cdot)$2 and a step size $u(\cdot)$3. Each iteration samples

$u(\cdot)$4

If $u(\cdot)$5, the step is a success, $u(\cdot)$6, and the step size is increased according to $u(\cdot)$7. Otherwise, $u(\cdot)$8 and $u(\cdot)$9. The supplied summary states that this empirical success rule directly adapts the step size to maintain a success probability near $t^*=\arg\max_t \hat{u}_t,$0, and that the normalized step size governs the probability of success (Akimoto et al., 2018).

The same idea has been generalized in discrete evolutionary algorithms as a family of success-based update rules parameterized by an update strength $t^*=\arg\max_t \hat{u}_t,$1 and a success ratio $t^*=\arg\max_t \hat{u}_t,$2. In the self-adjusting $t^*=\arg\max_t \hat{u}_t,$3 EA on LeadingOnes, a successful offspring updates the mutation rate to $t^*=\arg\max_t \hat{u}_t,$4, while a failure updates it to $t^*=\arg\max_t \hat{u}_t,$5, with clipping to $t^*=\arg\max_t \hat{u}_t,$6 and $t^*=\arg\max_t \hat{u}_t,$7. The generalized target success rate is $t^*=\arg\max_t \hat{u}_t,$8, so the classical one-fifth rule corresponds to $t^*=\arg\max_t \hat{u}_t,$9 (Doerr et al., 2019).

A misconception addressed by this line of work is that the literal one-fifth target is universally optimal. For LeadingOnes, the best performance is reported for small update strengths $s$0 and success rate $s$1, rather than $s$2. At fitness level $s$3, the target mutation rate is defined by

$s$4

which yields

$s$5

The main runtime bound is

$s$6

and the minimum occurs at $s$7, corresponding to the $s$8 target success rate (Doerr et al., 2019).

4. Drift theory, unbounded potentials, and continuous-domain hitting times

The 2018 analysis of the $s$9-ES with one-fifth success rule establishes the first explicit application of drift analysis to a randomized search heuristic with continuous domain, and the first non-asymptotic analysis for that algorithm. The technical obstacle is that natural potentials such as $$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$0 are unbounded below as the optimum is approached, and rare but extremely good steps can invalidate simple arguments unless controlled (Akimoto et al., 2018).

To address this, the paper introduces new drift theorems for unbounded-below potential functions. The upper-bound theorem uses a truncated process

$$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$1

and if the conditional expected truncated drift satisfies

$$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$2

then for the hitting time $$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$3 of target $$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$4,

$$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$5

A corresponding lower-bound theorem is given for monotone non-increasing and integrable processes with expected decrease at least $$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$6, yielding

$$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$7

These theorems permit drift analysis with potentials such as $$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$8 (Akimoto et al., 2018).

The constructed potential couples optimization and adaptation: $$\text{Regret in state } s = \max_t u_{st} - \sum_t u_{st}\,\mathbb{E}_s[\delta(t,\psi)],$$9 where $u_{st}$0, and $u_{st}$1 defines a well-adapted range for the normalized step size. The first term tracks optimization progress; the max-terms penalize step sizes that are too small or too large, thereby forcing adaptation (Akimoto et al., 2018).

Let

$u_{st}$2

The paper proves

$u_{st}$3

with $u_{st}$4. As summarized in the supplied notes,

$u_{st}$5

so the convergence rate scales as $u_{st}$6. A matching lower-bound ingredient is that even hit-and-run does not allow more than

$u_{st}$7

per iteration (Akimoto et al., 2018).

5. Discrete mutation control and hybrid learning-based rules

The discrete theory sharpens the interpretation of success rules as online parameter control. For LeadingOnes, the self-adjusting $u_{st}$8 EA with small $u_{st}$9 and optimal success ratio $t$0 attains

$t$1

which the supplied summary states matches the asymptotic runtime of the $t$2 EA with the best possible fitness-dependent mutation rate. The one-fifth rule remains competitive but slightly worse, at approximately $t$3, while still outperforming any static mutation rate. The same summary reports that for $t$4, the self-adjusting EA outperforms any static mutation rate (Doerr et al., 2019).

This result clarifies a second misconception: success rules are not merely heuristic surrogates for finely tuned schedules. Under the analyzed conditions, they can be asymptotically equivalent to an oracle fitness-dependent mutation strategy. The paper also states that similar results hold for the resampling variant of the $t$5 EA, which enforces at least one bit flip per iteration (Doerr et al., 2019).

A further extension combines success-based control with reinforcement learning. The hybrid Q-learning mechanism HQL controls the mutation rate of the $t$6 EA by maintaining a Q-table over states defined by the number of successful offspring in a generation and over two actions: multiplying the current mutation rate by $t$7 or decreasing it by $t$8. If the Q-values of the two actions are equal, the action-selection tie is broken by the one-fifth-th success rule: increase the mutation rate if the best offspring is strictly better than the parent, and decrease it otherwise (Buzdalova et al., 2020).

The reported empirical finding is that HQL achieves equal or superior performance to all techniques tested in Rodionova et al. and does so simultaneously for all offspring population sizes $t$9. The same source states that the promising performance is not restricted to OneMax but extends to several other benchmark problems, including LeadingOnes, neutrality, plateau, and ruggedness functions. It also notes that no full theoretical convergence analysis is provided because tracking Q-values and state dependencies is complex (Buzdalova et al., 2020).

6. Plug-in stopping rules, regulatory usage, and limits of oracle-freeness

A structurally related use of empirical success appears in the last-success problem with unknown Bernoulli parameter. With $\mathbb{E}_s[\delta(t,\psi)]$0 i.i.d. $\mathbb{E}_s[\delta(t,\psi)]$1 and $\mathbb{E}_s[\delta(t,\psi)]$2 unknown, the plug-in rule estimates

$\mathbb{E}_s[\delta(t,\psi)]$3

and stops at

$\mathbb{E}_s[\delta(t,\psi)]$4

Its win probability is

$\mathbb{E}_s[\delta(t,\psi)]$5

The paper derives an exact recursion for state probabilities $\mathbb{E}_s[\delta(t,\psi)]$6 and a representation of $\mathbb{E}_s[\delta(t,\psi)]$7 that can be computed in $\mathbb{E}_s[\delta(t,\psi)]$8 time (Paindaveine, 8 Apr 2026).

The same work proves that for $\mathbb{E}_s[\delta(t,\psi)]$9, the plug-in rule is always suboptimal in finite horizon: $t$0 where $t$1 is the oracle value from Bruss’ sum-the-odds theorem. It also establishes a decision-theoretic impossibility result: for any fixed $t$2, the class of $t$3-blind rules has no greatest element under dominance, even allowing randomization (Paindaveine, 8 Apr 2026).

At the same time, the paper identifies regimes in which empirical plug-in performance approaches the oracle benchmark. For any $t$4, finite-horizon oracle bounds are established on $t$5; if $t$6, the worst-case deficit is of order $t$7, and this rate is matched by a minimax lower bound. In sparse regimes where $t$8 with $t$9,

$\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$00

so the plug-in rule is asymptotically oracle-optimal and both values converge to $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$01. Uniform convergence cannot be extended to the critical regime $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$02, and no oracle-free sequence of rules can converge uniformly to the oracle value over all $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$03 (Paindaveine, 8 Apr 2026).

A related but distinct regulatory usage appears in work on alternatives to the two-trials rule for drug approval. There, an empirical success rule is described as inference based on the actual results across studies rather than a rigid, pre-set rule. In that setting, p-value combination methods such as Pearson’s method, Edgington’s method, and the harmonic mean $\hat{u}_t=\frac{1}{n_t}\sum_{i:\, t_i=t}u(y_i),$04-test are studied under constraints of overall and partial Type-I error control, with Edgington’s method eventually recommended because it is easy to implement and communicate, has only moderate partial Type-I error rate inflation, and substantially increased project power (Held, 2023).