Major TOM: Expandable Datasets for Earth Observation (2402.12095v2)

Abstract: Deep learning models are increasingly data-hungry, requiring significant resources to collect and compile the datasets needed to train them, with Earth Observation (EO) models being no exception. However, the landscape of datasets in EO is relatively atomised, with interoperability made difficult by diverse formats and data structures. If ever larger datasets are to be built, and duplication of effort minimised, then a shared framework that allows users to combine and access multiple datasets is needed. Here, Major TOM (Terrestrial Observation Metaset) is proposed as this extensible framework. Primarily, it consists of a geographical indexing system based on a set of grid points and a metadata structure that allows multiple datasets with different sources to be merged. Besides the specification of Major TOM as a framework, this work also presents a large, open-access dataset, MajorTOM-Core, which covers the vast majority of the Earth's land surface. This dataset provides the community with both an immediately useful resource, as well as acting as a template for future additions to the Major TOM ecosystem. Access: https://huggingface.co/Major-TOM

• The paper presents Major TOM, a framework that unifies fragmented EO datasets via a standardized, index-based geographic grid system.
• It introduces the MajorTOM-Core dataset with over 2.5 trillion Sentinel-2 pixels across 2.25 million samples, offering an extensive resource.
• The framework preserves data integrity by avoiding destructive preprocessing, ensuring alignment with the native resolution of Sentinel-2 imagery.

Expanding Earth Observation Datasets with Major TOM

The paper "Major TOM: Expandable Datasets for Earth Observation" by Alistair Francis and Mikolaj Czerkawski addresses a compelling challenge in the field of Earth Observation (EO): the aggregation and interoperability of fragmented datasets. With a focus on deep learning applications, this work introduces Major TOM (Terrestrial Observation Metaset), a framework designed to facilitate the integration and expansion of EO datasets by employing a standardized, index-based geographic grid system.

Framework Overview and Contribution

Major TOM offers a novel approach to data curation and dissemination in Earth Observation by establishing a unifying grid system that simplifies dataset integration. The framework is underpinned by a geographically-indexed grid, which allows data from various sources to be merged seamlessly. This mechanism is crucial for building extensive datasets without repeating collection efforts, thereby optimizing data utility for deep learning models that require expansive training data.

Moreover, the introduction of MajorTOM-Core, an expansive open-access dataset within this framework, illustrates the potential of Major TOM to serve as both a valuable immediate resource and as a scalable template for future datasets. The MajorTOM-Core dataset comprises over 2.5 trillion pixels of Sentinel-2 imagery, representing one of the largest openly available datasets of its kind, covering a substantial portion of the Earth's land surface.

Numerical Results and Dataset Characteristics

Major TOM sets a benchmark in terms of dataset volume and geographical scope. The MajorTOM-Core dataset, with its Sentinel-2 imagery, encompasses approximately 2,250,000 samples across the globe, summing up to over 2.5 trillion pixels. This dataset is not only extensive but also geospatially inclusive, providing data over nearly all regions observed by the Sentinel-2 mission, notwithstanding sporadic gaps in areas like Greenland's interior and equatorial regions where cloud coverage poses challenges.

The technical design of Major TOM ensures data is retained in its most useful form. By avoiding destructive preprocessing such as downsampling or band reduction, the dataset aligns perfectly with the native resolutions of Sentinel-2 imagery, thus maintaining data integrity. Additionally, the framework facilitates the seamless integration of diverse data types, evidenced by the ongoing expansion to include Sentinel-1 and other remote sensing data modalities.

Implications and Future Developments

This framework has notable implications for both practical applications and theoretical advancements in remote sensing. By providing a standardized method to access and combine diverse datasets, Major TOM allows researchers to develop and test machine learning models with greater efficacy and transparency. Researchers can rapidly prototype innovative models with diverse data inputs, enabling novel insights and applications across various domains such as environmental monitoring, agriculture, and disaster management.

The theoretical underpinnings of Major TOM demonstrate a promising path forward for data scalability in Earth Observation. The ease of integration and standardized access techniques proposed could serve as a foundation for further development in geographically aware deep learning approaches and enhanced data fusion methodologies.

Future iterations of Major TOM may well expand to include even more sophisticated data types and modalities, possibly incorporating real-time data streams and other environmental datasets, which would further bolster the framework’s applicability and utility.

Conclusion

Major TOM stands as a significant contribution toward advancing data curation for Earth Observation applications. Through its innovative grid system and extensive dataset offerings, it provides a robust toolset for researchers in the field, significantly easing the manipulation and integration of large-scale datasets. With its forward-looking design, Major TOM is positioned to become a cornerstone in the development of scalable, interoperable EO datasets, offering enhanced capabilities for both current and future research endeavors.