Follow-the-Perturbed-Leader (FTPL)

• FTPL is an online learning algorithm that perturbs cumulative losses or rewards to select actions, offering strong theoretical guarantees in various environments.
• Asymmetric and hybrid Fréchet-type perturbations enable best-of-both-worlds performance, while symmetric variants work well in two-armed settings but face challenges in higher dimensions.
• The duality between FTPL and FTRL through heavy-tailed perturbations informs innovative regularization strategies, guiding future research in online learning and combinatorial bandits.

Follow-the-Perturbed-Leader (FTPL) is a class of online learning algorithms in which random perturbations are added to cumulative losses or rewards to determine the next action. Recent research has established new theoretical foundations for FTPL in adversarial, stochastic, and combinatorial settings, with particular focus on the role of perturbation tail behavior. The introduction of Fréchet-type (heavy-tailed) perturbations, asymmetric hybrid constructions, and connections to regularization have led to significant advances in regret guarantees, best-of-both-worlds (BOBW) properties, and a deeper understanding of FTPL’s capabilities and limitations.

1. Analytical Foundations: FTPL Policies and Fréchet-type Perturbations

A distribution is called Fréchet-type if its right tail decays polynomially: for index $\alpha > 0$, $1-F(x) = \tilde{\Theta}(x^{-\alpha})$ as $x \to \infty$, where $\tilde{\Theta}$ hides polylogarithmic factors. This class contains distributions such as Fréchet, Pareto, and heavy-tailed Student-$t$. Such distributions are characterized by regular variation: $g(x)$ is regularly varying with index $\alpha$ if $\lim_{x \to \infty} g(tx)/g(x) = t^\alpha$ for all $t > 0$.

The classical FTRL–FTPL duality extends to unbounded perturbations by matching FTPL policies (with potentially heavy-tailed perturbations) to FTRL policies with convex conjugate regularizers. FTPL choice probabilities are expressed as integrals involving the perturbation density $f$ and cumulative $F$, and, under absolute continuity and support conditions, there is a one-to-one correspondence with FTRL policies (see Lemma 2.1, (Lee et al., 26 Aug 2025)). However, standard monotonicity assumptions (such as $f/F$ decreasing) break down for heavy-tailed laws, and new analysis is required to control regret and stability terms.

2. Asymmetric and Hybrid Perturbations: Achieving BOBW Guarantees

Recent works rigorously established that FTPL with asymmetric Fréchet-type perturbations achieves BOBW performance (minimax optimal $\mathcal{O}(\sqrt{KT})$ regret adversarially, and logarithmic regret stochastically) (Lee et al., 8 Mar 2024, Lee et al., 26 Aug 2025). "Asymmetric Fréchet-type perturbations" (Editor's term) are distributions with a heavy (Fréchet-type) right tail and a lighter, possibly Gumbel-type, left tail, often denoted $F_{\alpha,\beta}$ with indices for right/left tails.

Key results (see Theorem 3.1 in (Lee et al., 26 Aug 2025)) show:

• If the left tail is lighter than the right (e.g., $β \geq α+2$), then key sensitivity ratios controlling stability, such as $-\phi'_i(\lambda)/\phi_i(\lambda)$, are bounded in terms of arm statistics. This enables transfer of the standard regret analysis.
• As a consequence, FTPL with Laplace–Pareto, asymmetric Pareto, or Gumbel–Fréchet hybrid perturbations yields

$$\text{Regret}(T) = \mathcal{O}(\sqrt{KT}) \text{ adversarially,}$$

$$\text{Regret}(T) = \mathcal{O}\left(\sum_{i: \Delta_i > 0}\frac{\log T}{\Delta_i}\right) \text{ stochastically,}$$

provided the optimal arm is unique.

Hybrid (asymmetric) FTPL policies match the best rates previously known only for FTRL with hybrid regularizers.

3. Symmetric Fréchet-type Perturbations: Two-Armed vs Multi-Armed Settings

FTPL with symmetric Fréchet-type perturbations (e.g., symmetric Pareto, $f(x) \asymp 1/(|x|+1)^{\gamma}$) exhibits nuanced behavior, depending on the number of arms:

• For $K=2$, the sensitivity ratio $-\phi'/\phi^{3/2}$ remains bounded (see Proposition 4.1, (Lee et al., 26 Aug 2025)), reflecting correspondence with a $1/2$-Tsallis entropy regularizer and ensuring both minimax and BOBW regret.
• For $K \geq 3$, this ratio can become unbounded as the cumulative loss gap grows (see Proposition 4.2), which invalidates the key condition for standard BOBW analysis.
• This limitation is specific to symmetric heavy-tailed perturbations, and does not occur with asymmetric or nonnegative variants.

These findings highlight a sharp dichotomy: while symmetric heavy tails suffice for BOBW in two-armed bandits, they generally fail in higher dimensions without alternative analysis techniques.

4. FTPL in Combinatorial and Semi-Bandit Settings

FTPL with Fréchet-type perturbations is extended to size-invariant combinatorial semi-bandit problems (Zhan et al., 9 Apr 2025, Chen et al., 14 Jun 2025). In $m$-set semi-bandits, the learner selects $m$ arms from $d$ under adversarial or stochastic losses.

Summary of regret results:

FTPL Variant	Setting	Regret Bound
Fréchet (shape $\alpha=2$)	Adversarial	$\mathcal{O}\left(\sqrt{nmd}\right)$
Pareto	Adversarial	$O\left(\sqrt{mdT}\right)$
Fréchet-type, $\alpha$	Size-invariant	$$O\left(\sqrt{m^2 d^{\frac{1}{\alpha}}T}+\sqrt{mdT}\right)$$
Fréchet (shape $\alpha=2$)	Stochastic	$$O\left(\sum_{i:\Delta_i>0} \frac{\log n}{\Delta_i} + \frac{m^2 d}{\Delta}\right)$$

Geometric resampling (GR) and its conditional variant (CGR) are employed to estimate selection probabilities efficiently, reducing computation from $O(d^2)$ to $O(md(\log(d/m)+1))$ (Chen et al., 14 Jun 2025).

A notable technical correction is provided in (Chen et al., 14 Jun 2025): monotonicity claims used in prior geometric resampling proofs for Fréchet distributions were shown to be incorrect for the decrease case (see their analysis of $h(x)$ and related ratios), invalidating purported monotonicity and necessitating new proof strategies.

5. Connections to FTRL, Regularization, and Duality

FTPL with appropriately designed perturbations can be mapped precisely to FTRL with hybrid or Tsallis entropy regularizers. In (Lee et al., 26 Aug 2025), duality is established even in heavy-tailed settings by leveraging regular variation and extreme value theory:

• For reward (or loss) vector $\nu$,

$$\phi(\nu) = \arg\max_p \left\langle p, \nu \right\rangle - V(p)$$

where $V$ is the convex conjugate of the FTPL potential function induced by the perturbation law.

• For $K=2$ with symmetric Fréchet-type perturbations, numerical evidence suggests an exact correspondence with regularizers of Tsallis entropy type.
• The hybrid Gumbel–Fréchet policies are competitive with those based on hybrid regularization in FTRL, incorporating both exploration incentives (light tail) and heavy-tail robustness.

Explicit connections are established between the existence of Fréchet-type marginals and regularization functions capable of achieving BOBW results, thus clarifying design principles for both FTPL and FTRL.

6. Key Open Problems and Future Directions

• While asymmetric (including hybrid) heavy-tailed FTPL policies now enjoy rigorous BOBW guarantees for general $K$ (Lee et al., 26 Aug 2025), symmetric heavy-tailed designs remain insufficient for $K>2$ in standard analyses. This suggests further investigation is necessary to determine if alternative analysis or new constructions can restore optimality.
• The lack of monotonicity in critical sensitivity ratios identified in (Chen et al., 14 Jun 2025) highlights the technical complexity in extending geometric resampling and regret guarantees to broader combinatorial settings with heavy-tailed perturbations.
• Applications and algorithms exploiting the FTRL–FTPL duality with unbounded perturbations—such as hybrid Gumbel–Fréchet constructions—provide templates for future work in structured bandits and reinforcement learning domains.
• The precise mapping between Tsallis-entropy-based FTRL and FTPL (especially symmetry/asymmetry) invites new hybrid policies and raises foundational questions regarding which regularizers can be realized in FTPL via perturbation mechanisms.

7. Summary and Practical Insights

• Asymmetric and hybrid Fréchet-type perturbations (right tail regularly varying with $\alpha > 1$, left tail lighter) endow FTPL with BOBW optimality in adversarial and stochastic regimes for multi-armed and certain combinatorial problems (Lee et al., 26 Aug 2025, Lee et al., 8 Mar 2024).
• Symmetric Fréchet-type perturbations can match BOBW guarantees in two-armed bandits, but generally break necessary stability properties for $K\geq 3$.
• In combinatorial (m-set) semi-bandits, FTPL with appropriate Fréchet-type perturbations achieves optimal (or near-optimal) regret and maintains computational efficiency via geometric resampling, with technical caveats regarding monotonicity proofs and sensitivity ratios (Zhan et al., 9 Apr 2025, Chen et al., 14 Jun 2025).
• The FTRL–FTPL duality extends to unbounded perturbations, facilitating policy design via regularization insights and offering competitive hybrid regularizer alternatives.
• These developments resolve long-standing conjectures about FTPL’s performance and guide algorithm designers to favor asymmetric heavy-tailed perturbations in practice for robust, efficient, and theoretically optimal online learning in a variety of settings.