Sharp bounds on aggregate expert error

Abstract: We revisit the classic problem of aggregating binary advice from conditionally independent experts, also known as the Naive Bayes setting. Our quantity of interest is the error probability of the optimal decision rule. In the case of symmetric errors (sensitivity = specificity), reasonably tight bounds on the optimal error probability are known. In the general asymmetric case, we are not aware of any nontrivial estimates on this quantity. Our contribution consists of sharp upper and lower bounds on the optimal error probability in the general case, which recover and sharpen the best known results in the symmetric special case. Since this turns out to be equivalent to estimating the total variation distance between two product distributions, our results also have bearing on this important and challenging problem.

• The paper presents sharp bounds on the optimal error probability for aggregating binary advice from conditionally independent experts.
• It improves results for symmetric cases and pioneers nontrivial bounds for asymmetric settings.
• Advanced techniques, including the Neyman-Pearson lemma and total variation distance analysis, support its rigorous findings.

Aggregation of Expert Advice, Revisited

In this paper, "Aggregation of expert advice, revisited," Aryeh Kontorovich revisits the classic problem of aggregating binary advice from conditionally independent experts, known in literature as the Naive Bayes setting. The primary focus of the research is to determine tight bounds on the optimal error probability of the decision rule under both symmetric and asymmetric cases.

Problem Context and Contributions

The paper approaches the problem of aggregating binary advice by modeling the scenario where a binary random variable $Y$ is observed, followed by a sequence of conditionally independent binary observations given $Y$. The main quantity of interest is the optimal decision rule's error probability, denoted as $P(f(X) \neq Y)$. Specifically, the author provides sharp upper and lower bounds on this error probability, addressing both symmetric (where sensitivity equals specificity) and asymmetric cases.

In the symmetric case, existing reasonably tight bounds are enhanced by this research. However, the real advancement is in the asymmetric case where prior to this work, no nontrivial estimates were well-known. The paper asserts new upper and lower bounds for the optimal error probability, also contributing to the understanding of the total variation distance between two product distributions, a notably challenging problem in probability theory.

Main Results

Upper Bound

The upper bound presented in the paper is articulated in terms of the balanced accuracies $\pi_i = (\psi_i + \eta_i)/2$:

$$P(f \neq Y) \le \frac{1}{2} \sqrt{\prod_{i=1}^n (\psi_i + \eta_i)}[1 - (\psi_i + \eta_i)].$$

This upper bound is significant as it recovers the known bounds for the symmetric case while providing a robust estimate for the asymmetric case. It is highlighted that this bound evaluates correctly at extremities, e.g., when all $\pi_i = 1/2$, the bound yields $P(f \neq Y) = 1/2$, which matches the exact value.

Lower Bound

The lower bound provided is derived rigorously and can be expressed as:

$$P(f \neq Y) \ge \frac{1}{2} \cdot 2^n \sqrt{\prod_{i=1}^n \pi_i (1 - \pi_i)} \cdot \exp \left( -\frac{1}{2} \sum_{i=1}^n \left| \gamma_i \right| \right),$$

where $\gamma_i = \log (\pi_i / (1 - \pi_i))$. This bound offers a significant theoretical improvement over previous works and shows sharpness in specific scenarios, especially when $\pi_i \to 1/2$.

For the symmetric case, a sharper lower bound is presented, improving constants in previously known bounds derived by other researchers.

Proof Techniques and Algorithmic Implications

The paper employs several advanced mathematical techniques and lemmata to establish the bounds, including leveraging the Neyman-Pearson lemma, Scheffé’s identity, and exploiting the properties of total variation distance in product distributions.

Future Implications and Conclusion

The results presented have both theoretical and practical implications. From a theoretical standpoint, these bounds not only enhance the understanding of the error probabilities in aggregation settings but also contribute broadly to the study of total variation distances. Practically, these bounds can improve decision rules in machine learning models that rely on multiple advisory signals, significantly impacting fields like ensemble learning.

Speculatively, future developments might focus on tighter bounds for specific problem instances or developing efficient algorithms to compute these bounds given the observation that exact computation of certain total variation distances can be computationally challenging.

In conclusion, this paper substantiates the lower and upper bounds of error probabilities in the domain of binary advice aggregation with rigor, offering substantial contributions to both theoretical and application-based advancements in machine learning, statistics, and probability.