SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter Pruning (2207.03677v5)

Abstract: Neural architecture search (NAS) has demonstrated amazing success in searching for efficient deep neural networks (DNNs) from a given supernet. In parallel, the lottery ticket hypothesis has shown that DNNs contain small subnetworks that can be trained from scratch to achieve a comparable or higher accuracy than original DNNs. As such, it is currently a common practice to develop efficient DNNs via a pipeline of first search and then prune. Nevertheless, doing so often requires a search-train-prune-retrain process and thus prohibitive computational cost. In this paper, we discover for the first time that both efficient DNNs and their lottery subnetworks (i.e., lottery tickets) can be directly identified from a supernet, which we term as SuperTickets, via a two-in-one training scheme with jointly architecture searching and parameter pruning. Moreover, we develop a progressive and unified SuperTickets identification strategy that allows the connectivity of subnetworks to change during supernet training, achieving better accuracy and efficiency trade-offs than conventional sparse training. Finally, we evaluate whether such identified SuperTickets drawn from one task can transfer well to other tasks, validating their potential of handling multiple tasks simultaneously. Extensive experiments and ablation studies on three tasks and four benchmark datasets validate that our proposed SuperTickets achieve boosted accuracy and efficiency trade-offs than both typical NAS and pruning pipelines, regardless of having retraining or not. Codes and pretrained models are available at https://github.com/RICE-EIC/SuperTickets.

• The paper presents a novel framework that jointly performs architecture searching and parameter pruning to identify efficient subnetworks from supernets.
• It demonstrates that extracted lottery tickets are task-agnostic, offering transferable performance benefits across diverse tasks.
• The method improves computational efficiency and model scalability, providing a new pathway for optimizing neural network design.

Analysis of "Author Guidelines for ECCV Submission"

The paper "Author Guidelines for ECCV Submission," purportedly authored anonymously, serves as a directive text for prospective authors intending to submit their manuscripts to the European Conference on Computer Vision (ECCV). This document is integral in maintaining consistency, anonymity, and quality across submissions, thereby facilitating an efficient review process. As a guide rather than a traditional research paper, it focuses on formatting, submission policies, and the nuances of preparing a manuscript for ECCV consideration.

Overview and Structure

The document is structured to address multiple components necessary for a successful submission. It begins with an introduction to the overall expectations for submissions followed by detailed sections on initial submission requirements, review process policies, manuscript preparation, and conversion to camera-ready copies. Each section is precisely delineated to provide comprehensive guidance.

Key Components and Their Implications

1. Language and Document Formatting: The guidelines stipulate that all submissions must be composed in English and adhere to a strict 14-page limit, excluding references. This restriction underscores the conference's emphasis on conciseness and relevance in academic writing, aligning with standard norms across high-tier academic forums.
2. Submission Anonymity and Review Process: The double-blind review process is highlighted, ensuring that both authors and reviewers remain unidentified to each other, thus promoting unbiased evaluations. This method, crucial for maintaining the integrity and objective assessment of contributions, is meticulously supported by guidelines on the anonymization of submissions.
3. Managing Dual Submissions: A rigorous stance on dual submissions is presented, with explicit definitions of what constitutes a 'publication' and the associated criteria for manuscript overlap. This prevents repeated dissemination of the same research, reinforcing ECCV's commitment to novelty and originality in research contributions.
4. Camera-Ready Submission and Technical Rigor: The transition from initial submission to camera-ready version requires stringent adherence to the provided template and formatting norms. Such meticulous attention to detail enhances the quality of final publications, thus honoring the scholarly standards expected by ECCV and its publishers, Springer.

Future Directions and Considerations

While this paper does not present empirical research data or theoretical models, its implications are profound in the administrative domain of academic publishing, specifically within computer vision. The guidelines reflect a broader trend in academia towards enhancing reproducibility, transparency, and standardization in publishing processes.

In terms of future developments, this document anticipates evolving expectations in manuscript preparation, potentially accommodating emerging trends such as open-review processes or the integration of more collaborative review platforms driven by artificial intelligence. Additionally, reflecting on shifts towards digital and decentralized conference models, the guidelines could adapt to include more comprehensive instructions for virtual and hybrid conference settings.

Conclusion

Overall, the "Author Guidelines for ECCV Submission" encapsulates the critical facets of manuscript submissions, echoing ECCV's dedication to high-quality, original contributions to the field of computer vision. Its clear directives and stringent policies serve not just as a procedural manual but also as a model for other conferences seeking to uphold the integrity and quality of their scholarly discourse. This document remains a pivotal element in the academic ecosystem, contributing to a refined and transparent publication process.