Robust training of nondeterministic stack language models

Develop a sufficiently robust implementation of nondeterministic stack language models to determine whether nondeterministic stacks can improve generalization and achieve statistically demonstrable effectiveness on realistic artificial languages with cross-serial dependencies.

Background

The paper finds that nondeterministic-stack LLMs exhibit greater training instability and variance than models using superposition stacks. Although nondeterministic stacks have greater theoretical expressive power, this instability may obscure their potential effectiveness on realistic artificial languages containing cross-serial dependencies. The authors therefore leave unresolved whether nondeterministic stacks can be shown to work effectively once a more robust implementation is available.