Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2010.03533v2)

Abstract: Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction of the compute/storage for inference, and also have the potential to enable efficient training. However, naively training unstructured sparse NNs from random initialization results in significantly worse generalization, with the notable exceptions of Lottery Tickets (LTs) and Dynamic Sparse Training (DST). Through our analysis of gradient flow during training we attempt to answer: (1) why training unstructured sparse networks from random initialization performs poorly and; (2) what makes LTs and DST the exceptions? We show that sparse NNs have poor gradient flow at initialization and demonstrate the importance of using sparsity-aware initialization. Furthermore, we find that DST methods significantly improve gradient flow during training over traditional sparse training methods. Finally, we show that LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from - however, this comes at the cost of learning novel solutions.

Overview of "AAAI Press Formatting Instructions for Authors Using LaTeX"

The paper "AAAI Press Formatting Instructions for Authors Using LaTeX — A Guide" serves as a comprehensive manual for authors preparing their manuscripts for submission to the AAAI Press proceedings. This document outlines the explicit formatting requirements that authors must adhere to in order to ensure their papers are properly formatted for electronic publishing and subsequent physical distribution. These guidelines are specifically tailored to the 2022 AAAI Press proceedings and provide detailed instructions to maintain uniformity across all publications.

Key Contributions and Specifications

The paper provides extensive information on using LaTeX to produce manuscripts, which is the preferred method due to its powerful typesetting capabilities. Specific requirements are stipulated to standardize the appearance and format of submissions:

• Template Usage: Authors are required to use the 2022 AAAI Press LaTeX style file (aaai22.sty) and bibliography style files (aaai22.bst). Deviations from these templates, including altering float and margin settings, are strictly prohibited to preserve uniformity.
• Font and Document Settings: Manuscripts must be written in Times or Nimbus font. Type 3 fonts are unacceptable due to resolution issues and their incompatibility with high-resolution outputs. The document should be in a two-column format on US letter-sized paper.
• Metadata and Submission: Authors must include accurate and plain ASCII metadata for author names and titles within the document. Comprehensive instructions are given for preparing and submitting LaTeX source files, figures, and accompanying files in a single compressed archive.
• Graphic and Tabular Representation: All figures should be included in a compatible format such as .png, .jpg, or .pdf, avoiding .gif, .ps, or .eps. Tables and illustrations are to be formatted with specific type requirements and must be free from overflow into margins.

Theoretical and Practical Implications

This guide addresses both theoretical aspects of document preparation and practical submission processes. By setting rigorous formatting standards, AAAI ensures that papers are not only visually consistent but also accessible in digital and print formats. The detailed instructions on metadata and submissions highlight the emphasis on both presentation and archiving, which are vital for the research community's shared knowledge base.

From a theoretical standpoint, the guide implicitly underscores the enduring importance of structured document standards within academic publishing. As research outputs become more digital, the uniformity of formatting facilitates better indexing, retrieval, and dissemination of research findings.

Future Developments in Academic Formatting

Looking forward, the challenges and opportunities presented in the paper anticipate future developments in academic formatting. As the research community increasingly explores open-access and online publishing platforms, the standards outlined could evolve to accommodate interactive elements and multimedia content while preserving essential principles of accessibility and citation integrity.

The rigorous adherence to typesetting standards exemplified in the AAAI guide could serve as a blueprint for future academic publication protocols, ensuring they adapt to technological advancements without compromising on quality or accessibility. Consequently, as technological capabilities expand, future editions of the guide may include instructions for incorporating augmented reality or interactive datasets in publications, further enriching the research landscape.

In conclusion, this document is pivotal in guiding authors through complex formatting requirements, ensuring that their research is presented in a professional and consistent manner, reflective of the high standards expected by the research community.