Asymptotic expected dimension for large activation degree

Establish that for any fixed widths d=(d0,…,dL) with di>1 for all hidden layers i=1,…,L−1, there exists a threshold activation degree r̃(d) such that for all activation degrees r>r̃(d), the neurovariety V_{d,r} attains its expected dimension.

Background

The paper studies the dimension of neurovarieties V_{d,r} associated with polynomial neural networks and uses this dimension as a measure of expressivity. Due to multi-homogeneity symmetries in parameters, an expected dimension edim(V_{d,r}) is defined; architectures are called defective when dim(V_{d,r})<edim(V_{d,r}).

In the shallow single-output case, the Alexander–Hirschowitz Theorem shows non-defectivity except for a short list of exceptional cases. Motivated by this, the authors state an asymptotic conjecture that for sufficiently large activation degree r, deep neurovarieties with all hidden widths greater than one should attain their expected dimension.