Realizable Bayes-Consistency for General Metric Losses

Abstract: We study strong universal Bayes-consistency in the realizable setting for learning with general metric losses, extending classical characterizations beyond $0$-$1$ classification \citep{bousquet_theory_2021, hanneke2021universalbayesconsistencymetric} and real-valued regression \citep{attias_universal_2024}. Given an instance space $(\mathcal X,ρ)$, a label space $(\mathcal Y,\ell)$ with possibly unbounded loss, and a hypothesis class $\mathcal H \subseteq \mathcal Y^{\mathcal X}$, we resolve the realizable case of an open problem presented in \citet{pmlr-v178-cohen22a}. Specifically, we find the necessary and sufficient conditions on the hypothesis class $\mathcal H$ under which there exists a distribution-free learning rule whose risk converges almost surely to the best-in-class risk (which is zero) for every realizable data-generating distribution. Our main contribution is this sharp characterization in terms of a combinatorial obstruction: Similarly to \citet{attias2024optimallearnersrealizableregression}, we introduce the notion of an infinite non-decreasing $(γ_k)$-Littlestone tree, where $γ_k \to \infty$. This extends the Littlestone tree structure used in \citet{bousquet_theory_2021} to the metric loss setting.

• The paper demonstrates that universal Bayes-consistency is achievable if and only if there is no infinite non-decreasing Littlestone tree with diverging gaps.
• It introduces rigorous lower and upper bounds through adversarial tree constructions and game-theoretic strategies to address challenges in unbounded loss settings.
• The work reveals that finite Bayes risk alone is insufficient, emphasizing the crucial role of combinatorial structure and loss geometry in ensuring consistency.

Realizable Bayes-Consistency for General Metric Losses

Problem Setting and Motivation

The paper establishes a sharp characterization of strong universal Bayes-consistency in the realizable setting for learning with general metric losses, extending the classical theory beyond the $0$-$1$ classification and real-valued regression frameworks. Given an instance space $(\mathcal X, \rho)$, label space $(\mathcal Y, \ell)$—where $\ell$ may be unbounded—and a hypothesis class $\mathcal H \subseteq \mathcal Y^\mathcal X$, the primary question is to determine when it is possible for a learning rule to achieve almost sure convergence of risk to zero, uniformly over all realizable data-generating distributions.

The challenge arises specifically in the unbounded metric loss setting: unlike bounded losses, rare events with exponentially growing penalties may prevent risk from converging to zero even as error probability vanishes. This phenomenon is unaddressed by distributional finiteness ($R^\star < \infty$), and is not captured by classical learning theory.

Main Characterization

The paper resolves an open problem posed in [pmlr-v178-cohen22a], providing a necessary and sufficient combinatorial condition for strong universal Bayes-consistency in the realizable regime. The decisive obstruction is the existence of an infinite non-decreasing $(\gamma_k)$-Littlestone tree, with gaps $\gamma_k \to \infty$:

• Infinite non-decreasing $(\gamma_k)$-Littlestone tree: A rooted, full binary tree where each depth-$1$0 node $1$1 has two outgoing labels $1$2 separated by at least $1$3, and each finite root-to-leaf labeling can be realized by some $1$4. Under mild compact-parameterization assumptions, every infinite path through such the tree is realized by a single hypothesis.

The main theorem states:

Realizable strong universal Bayes-consistency for general metric losses is possible for $1$5 if and only if $1$6 does not admit an infinite non-decreasing $1$7-Littlestone tree with $1$8.

Furthermore:

• Lower Bound: If such a tree exists, for every learning algorithm $1$9 there is a realizable distribution $(\mathcal X, \rho)$0 where expected risk is infinite for all sample sizes (Theorem~\ref{thm:lower-bound}).
• Upper Bound: If no such tree exists, a constructive distribution-free learning rule achieves almost sure risk convergence to zero for every realizable $(\mathcal X, \rho)$1 (Theorem~\ref{thm:realizable-upper}).

This result establishes a separation showing that finiteness-type distributional conditions (e.g., $(\mathcal X, \rho)$2) do not suffice for consistency in the unbounded metric-loss regime.

Technical Approach and Proof Architecture

Lower Bound Construction

The lower bound adapts the classical Littlestone tree adversarial argument to metric loss, exploiting the fact that in the presence of a tree with diverging gaps, any learner is forced to guess over unseen regions of the instance space where the loss can be made arbitrarily large. The construction behaves as follows:

• For every sample size, infinitely many nodes along the Littlestone tree remain unobserved.
• For each such node, the learner must guess between labels separated by increasing $(\mathcal X, \rho)$3. As $(\mathcal X, \rho)$4, the expected risk diverges due to independent rare events contributing unbounded loss.
• This is proven via a Borel–Cantelli argument, showing almost sure infinite risk for any learner.

Upper Bound Construction

The upper bound employs an infinite Gale-Stewart game framework. Absence of the obstruction implies the learner has a measurable winning strategy. The argument proceeds in four main steps:

1. Gale-Stewart Game Simulation: For each instance, the strategy produces a set of admissible labels with diameter at most $(\mathcal X, \rho)$5, where $(\mathcal X, \rho)$6 is terminal game round determined by stabilization over the sample.
2. Partitioning: The instance space $(\mathcal X, \rho)$7 is partitioned into countably many cells, each associated with a bounded label region.
3. Local Learning: On each cell, the problem reduces to a classical bounded-range metric learning subproblem, handled by existing algorithms (e.g., MedNet [pmlr-v178-cohen22a]).
4. Aggregation: Aggregating cell-wise predictors yields a universally Bayes-consistent predictor.

Strong almost sure convergence is proven via sample splitting, countable partition arguments, and classical consistency guarantees for bounded losses.

Practical and Theoretical Implications

The combinatorial characterization has significant implications:

• Structural Regularity Required: The result implies that structural constraints on hypothesis classes (such as bounded range, Lipschitz continuity on compact domains) guarantee consistency, while classes that permit increasingly separated label choices on disjoint regions (e.g., via disjoint intervals) fail to be learnable.
• Beyond Boundedness: Even classes with unbounded range can be consistent if regularity precludes the tree obstruction (e.g., $(\mathcal X, \rho)$8-Lipschitz functions on $(\mathcal X, \rho)$9).
• Compact-Parameterization Assumption: The bridging lemma exploits compactness and continuity in the parameterization to ensure finite-prefix realizability implies infinite-path realizability. This is essential for technical completeness and covers nearly all practical learning settings.
• Non-sufficiency of $(\mathcal Y, \ell)$0: The paper demonstrates, via explicit counterexamples, that finite Bayes risk is insufficient for consistency when label losses are unbounded—emphasizing the crucial role of combinatorial class geometry.

Comparison to Prior Work

The paper builds upon and generalizes the universal learning and realizable regression frameworks (see (Bousquet et al., 2020, Attias et al., 2023, Hanneke et al., 2023), [pmlr-v178-cohen22a]), introducing a decisive obstruction for metric loss settings absent in bounded-loss theory. The results extend the foundational role of Littlestone trees from online mistake bounds and binary learning to general metric spaces, structured outputs, and cost-sensitive settings.

The use of infinite game-theoretic characterizations continues the line of work from (Bousquet et al., 2020, Hanneke et al., 2023), and is orthogonal to metric-complexity approaches (e.g., doubling dimension) that depend on geometric properties rather than combinatorial class structure.

Limitations and Future Directions

• The characterization is sharp in the realizable setting under the compact-parameterization assumption; extension to non-compact settings remains an open technical problem.
• Extension to the agnostic setting, i.e., when $(\mathcal Y, \ell)$1 need not be realizable by $(\mathcal Y, \ell)$2, is left as an open problem. The upper-bound stabilization crucially relies on realizability for legality of the game simulation.
• The work thus opens avenues for further exploration of the interplay between combinatorial obstructions and loss geometry in both realizable and agnostic universal learning.

Conclusion

This paper provides a complete characterization of strong universal Bayes-consistency for general metric losses in the realizable setting via the existence of infinite non-decreasing Littlestone trees with unbounded gaps. The result demonstrates the fundamental interplay between combinatorial hypothesis class structure and loss geometry, establishing rigorous boundaries for distribution-free learnability that transcend classical finiteness-based criteria. Future work is required to extend this framework to the agnostic regime and to further relax parameterization assumptions.