Curvature Benefits at Vocabulary Scale

Determine whether hyperbolic curvature improves RiLM language-modeling performance at larger vocabulary scales and under full-vocabulary decoding, beyond the controlled 10,000-word experiments reported in the paper.

Background

RiLM removes the vocabulary-sized output matrix by deriving token probabilities from geodesic distances between the contextual state and vocabulary embeddings. Although this reduces parameter costs, decoding still requires evaluating distances to all vocabulary embeddings, creating an O(|V| * d) computational cost. The paper evaluates vocabulary sizes of 2,000 and 10,000 tokens, finding that hyperbolic RiLM is substantially better than tied SSM baselines but that Flat RiLM becomes slightly more stable and competitive with HypRiLM at 10,000 tokens.

The authors therefore leave unresolved whether negative curvature continues to provide a benefit as vocabulary size and embedding width increase, particularly for full-vocabulary settings of approximately 33,000 tokens. This question concerns empirical scaling behavior rather than the already-established performance differences in the controlled experiments.