Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs (2111.15135v2)

Abstract: Coordinate-MLPs are emerging as an effective tool for modeling multidimensional continuous signals, overcoming many drawbacks associated with discrete grid-based approximations. However, coordinate-MLPs with ReLU activations, in their rudimentary form, demonstrate poor performance in representing signals with high fidelity, promoting the need for positional embedding layers. Recently, Sitzmann et al. proposed a sinusoidal activation function that has the capacity to omit positional embedding from coordinate-MLPs while still preserving high signal fidelity. Despite its potential, ReLUs are still dominating the space of coordinate-MLPs; we speculate that this is due to the hyper-sensitivity of networks -- that employ such sinusoidal activations -- to the initialization schemes. In this paper, we attempt to broaden the current understanding of the effect of activations in coordinate-MLPs, and show that there exists a broader class of activations that are suitable for encoding signals. We affirm that sinusoidal activations are only a single example in this class, and propose several non-periodic functions that empirically demonstrate more robust performance against random initializations than sinusoids. Finally, we advocate for a shift towards coordinate-MLPs that employ these non-traditional activation functions due to their high performance and simplicity.

• The paper demonstrates that non-periodic activation functions can robustly encode continuous signals in coordinate-MLPs without using positional embeddings.
• It reveals key insights on Lipschitz smoothness and derivative properties that directly influence MLP performance and reduce initialization sensitivity.
• Empirical results show that activations like Gaussian and Laplacian outperform traditional functions such as ReLU in complex tasks like 3D view synthesis.

Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs

The paper "Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs" addresses the limitations of current activation functions used in coordinate multi-layer perceptrons (MLPs) for modeling multidimensional continuous signals. The authors, Sameera Ramasinghe and Simon Lucey, propose an expanded understanding and framework of activation functions that can potentially enhance the performance of coordinate-MLPs without relying on positional embeddings.

Overview and Motivation

Coordinate-MLPs are increasingly utilized in various fields for encoding multidimensional signals due to their capability to represent continuous functions with potentially unlimited resolution. However, conventional activation functions such as ReLU often fall short in capturing high-frequency details of signals, leading to the widespread use of positional embeddings as a workaround. The reliance on embeddings imposes constraints on architecture complexity.

Recent work by Sitzmann et al. suggested sinusoidal activations as a solution, eliminating the need for positional embeddings while maintaining fidelity. Nevertheless, the sensitivity of these sinusoidal functions to initialization parameters acts as a bottleneck, preventing their widespread adoption. This paper extends the theoretical foundation of activation functions within coordinate-MLPs by introducing a broader class of functions that maintain robustness against initialization sensitivity.

Theoretical Contributions

The authors delve into an in-depth analysis of activation functions, focusing on their effect on signal encoding through coordinate-MLPs. Key highlights include:

1. Role of Lipschitz Smoothness and Singular Value Distribution:
   • The analysis sheds light on the intrinsic properties like Lipschitz smoothness and singular value distribution of hidden-layer representations. These factors are crucial in defining the efficacy of the MLP in encoding signals. The authors derive mathematical relationships connecting these properties to the activation functions.
2. Beyond Periodicity:
   • While periodicity has been considered essential in previous methodologies, this paper challenges the notion by demonstrating that non-periodic functions can also achieve suitable performance levels. This pivotal recognition introduces several non-periodic functions that exhibit robust encoding capabilities despite initialization randomness.
3. Derivation and Implications:
   • The research posits that the effectiveness of an activation function is strongly linked to its derivative properties, with parameterized functions offering flexibility and tuning capacity for different signal characteristics.

Practical Implications

The formulated theoretical insights translate into practical guidelines. Among these, selecting suitable hyper-parameters for activation functions based on the nature of the input signal can significantly influence MLP performance. The proposed framework allows better anticipation of a function's performance characteristics prior to implementation, thus streamlining the design process for new applications.

Empirical Validation

The empirical section validates the theoretical findings by showcasing the performance of the proposed non-periodic activation functions. The results indicate that these functions outperform conventional MLP activations like ReLU in tasks without using positional embeddings. Particularly, Gaussian and Laplacian activations deliver competitive results with fewer parameters and faster convergence rates, achieving advanced performance in complex tasks like 3D view synthesis.

Future Directions

While the paper positions non-periodic functions as viable alternatives to sinusoidal activations, further exploration is warranted to encompass a wider array of signals and more complex applications. Moreover, understanding the intrinsic geometry of output spaces in relation to their hidden representations can offer additional insights into enhancing coordinate-MLP frameworks.

Conclusion

This paper re-evaluates the attributes required for activation functions in coordinate-MLPs, offering a theoretical and empirical rethinking on their role. By contesting the necessity of periodicity and proposing versatile non-periodic functions, it opens avenues for simpler, more efficient models in multidimensional signal processing without the rigor of positional embedding, thus setting a new trajectory for research in neural signal representations.