---
title: Bernoulli f-Divergence Inequality
url: https://www.emergentmind.com/topics/bernoulli-f-divergence-inequality
type: topic
---

# Bernoulli f-Divergence Inequality

The Bernoulli $f$-divergence inequality provides sharp, explicit relations between various $f$-divergences (of the Csiszár type) for Bernoulli distributions, frequently parameterized in terms of the total variation distance. These inequalities subsume and generalize classical results such as Pinsker’s, and form a kernel for both classical and quantum information theoretic bounds. The foundational results revolve around convexity properties of the generating function $f$ and leverage reduction arguments to two-point supports.

## 1. Definition and Principal Formulation

Let $f:(0,\infty)\to\mathbb{R}$ be convex with $f(1)=0$. For probability measures $P\ll Q$, the $f$-divergence is defined by
\[
D_f(P\|Q) = \int_{q>0} f\left(\frac{p}{q}\right) dQ + f'(\infty) P\{ q=0 \}
\]
where $p=dP/d\lambda$, $q=dQ/d\lambda$ under any dominating measure $\lambda$. For Bernoulli distributions $P=\text{Bern}(p)$, $Q=\text{Bern}(q)$,
\[
D_f(\text{Bern}(p)\|\text{Bern}(q)) = q\,f\left(\frac{p}{q}\right) + (1-q)\,f\left(\frac{1-p}{1-q}\right)
\]
[1302.0336][0903.1765][2601.12027][2501.14340].

## 2. Sharp Lower Bounds via Total Variation

The central inequalities relate $D_f(\text{Bern}(p)\|\text{Bern}(q))$ to the total variation distance $\delta=|p-q|$:

- **Bröcker’s monotonic lower bound** [0903.1765]:
  \[
  D_f(\text{Bern}(p)\|\text{Bern}(q)) \geq f(1+\delta/2) + f(1-\delta/2)
  \]
  This is tight for Bernoulli variables. The bounding function is strictly increasing in $\delta$ under mild regularity assumptions.

- **Sharp minimization via support reduction** [1302.0336]:
  For the minimum $D_f$ at fixed $|p-q|=V$,
  \[
  D_f(\text{Bern}(p)\|\text{Bern}(q)) \geq (1-V) f\left(\frac{1+V}{1-V}\right)
  \]
  attained when $p=(1+V)/2$, $q=(1-V)/2$, i.e., at symmetric pairs.

## 3. Best-Possible Generalized Pinsker Inequalities

The framework in [0906.1244] gives integral representations and tight “Pinsker-type” lower bounds for arbitrary $f$ in terms of total variation:
\[
D_f(\text{Bern}(p)\|\text{Bern}(q)) \geq \Psi_f(\delta) := 2\left[\bar\Gamma_f\left(\frac{1}{2}-\frac{\delta}{2}\right) + \frac{\delta}{2} \Gamma_f\left(\frac{1}{2}\right) - \bar\Gamma_f\left(\frac{1}{2}\right) \right]
\]
where $\Gamma_f(\pi) = \int_0^\pi \gamma_f(t)\,dt$, $\bar\Gamma_f(\pi) = \int_0^\pi \Gamma_f(t)\,dt$, and $\gamma_f(\pi) = \frac{1}{\pi^3} f''(\frac{1-\pi}{\pi})$ for twice-differentiable $f$.

The minimizing, or extremal, Bernoulli pairs for fixed $\delta$ have points at $p=(1+\delta)/2$, $q=(1-\delta)/2$.

## 4. Explicit Algebraic and Sandwich Inequalities

The “binary $f$-divergence inequality” [2501.14340][1502.06428] provides sharp algebraic sandwich bounds between any two Bernoulli $f$-divergences, with formulas involving ratios and the $\chi^2$ divergence:
\[
m D_f(P\|Q) \leq p\,f\left(\frac{p}{q}\right) + (1-p)\,f\left(\frac{1-p}{1-q}\right) - f\left(1 + \frac{(p-q)^2}{q(1-q)}\right) \leq M D_f(P\|Q)
\]
where
\[
m = \min\left\{ \frac{p}{q}, \frac{1-p}{1-q} \right\},\quad
M = \max\left\{ \frac{p}{q}, \frac{1-p}{1-q} \right\}
\]
and the total variation and $\chi^2$ divergence are
\[
\delta = |p-q|,\qquad \chi^2(P,Q) = \frac{\delta^2}{q(1-q)}
\]
This inequality gives explicit control of the $f$-divergence in terms of basic symmetric functions of $p$ and $q$ [1502.06428].

## 5. Optimality, Tightness, and Equality Conditions

The reductions above are maximally tight for Bernoulli laws. Tightness follows from the fact that the relevant functions (Bayes-risk curve, data processing contractions, etc.) achieve their extrema for binary distributions. Equality is attained precisely when $dP/dQ$ takes only two values and $f$ is affine over the critical support points involved in the inequalities.

Cases of equality in the sandwich bound occur only in degenerate cases (i.e., $p=q$ or $f$ affine) or for the aforementioned symmetric extremal pairs.

## 6. Instantiations and Special Cases

The Bernoulli $f$-divergence inequalities specialize to classical divergences:

| $f$ function              | $D_f$ expression                                                              | Lower Bound Example          |
|---------------------------|-------------------------------------------------------------------------------|-----------------------------|
| $t\log t$ (KL)            | $q(p/q)\log(p/q) + (1-q)((1-p)/(1-q))\log((1-p)/(1-q))$                       | $(1-V)\ln\left(\frac{1+V}{1-V}\right)$ |
| $(\sqrt t-1)^2/2$ (Hellinger) | $q(\sqrt{p/q}-1)^2/2 + (1-q)(\sqrt{(1-p)/(1-q)}-1)^2/2$                   | $1 - \sqrt{1-V^2}$           |
| $(t-1)^2$ ($\chi^2$)      | $q((p/q)^2 - 1) + (1-q)(((1-p)/(1-q))^2 - 1)$                                 | $V^2 / (q(1-q))$             |

All these bounds encode sharp relationships that are maximally attained for the extremal Bernoulli pairs [1302.0336][2501.14340][0903.1765][0906.1244].

## 7. Applications and Extensions

The Bernoulli $f$-divergence inequality underpins several advanced methods:
- **Interactive statistical decision making**: The reduction and inversion to two-sided intervals for monotone transforms of risk (e.g., for prior-predictive CVaR and quantile lower bounds) [2601.12027].
- **Transfer to quantum divergences**: The inequalities lift directly to quantum settings by reduction to classical analogues on two-point supports, sidestepping complex matrix analysis [2501.14340].
- **Information-theoretic converse bounds**: Generalization of Fano’s inequality and derivation of tight explicit bounds for loss probabilities, exponential moments, and tail risks.

The Bernoulli $f$-divergence inequality is thus a foundational tool for optimally relating statistical divergences under minimal informativeness constraints, with broad implications for hypothesis testing, risk minimization, and quantum information theory.

Source: https://www.emergentmind.com/topics/bernoulli-f-divergence-inequality