Can architectural equivariance overcome the failure of post-hoc regularization to reduce the geometric alignment tax?

Determine whether implementing architectural equivariance—such as reverse-complement-equivariant layers in DNA sequence models—can successfully mitigate the geometric distortion introduced by discrete-token cross-entropy training in biological foundation models in cases where embedding-level consistency regularization (e.g., reverse-complement consistency regularization, RCCR) fails to preserve population-level manifold geometry.

Background

The paper argues that discrete tokenization with cross-entropy objectives induces geometric distortion in model representations, termed the Geometric Alignment Tax. An embedding-level RCCR intervention on DNABERT-2 achieved perfect per-sequence reverse-complement consistency but degraded population-level manifold geometry, suggesting that post-hoc regularization may redistribute rather than remove the distortion.

Given this failure of regularization to preserve geometry, the authors pose whether architectural mechanisms that build the symmetry directly into the model (e.g., reverse-complement-equivariant layers) could succeed where regularization does not, thus directly addressing the causal bottleneck imposed by discrete objectives.