Architectural Effects on Superposition

Determine how specific architectural choices in neural networks (e.g., activation functions, normalization, layer design, and inductive biases) influence the prevalence, structure, and properties of superposition and the resulting monosemanticity or polysemanticity.

Background

The paper discusses how architectural and training choices can affect monosemanticity and interpretability, suggesting that design decisions may modulate the extent of superposition in practice.

Systematically characterizing architectural effects would guide the development of intrinsically more interpretable models without sacrificing performance, informing both intrinsic and post-hoc interpretability strategies.