EarthPT: a time series foundation model for Earth Observation (2309.07207v2)
Abstract: We introduce EarthPT -- an Earth Observation (EO) pretrained transformer. EarthPT is a 700 million parameter decoding transformer foundation model trained in an autoregressive self-supervised manner and developed specifically with EO use-cases in mind. We demonstrate that EarthPT is an effective forecaster that can accurately predict future pixel-level surface reflectances across the 400-2300 nm range well into the future. For example, forecasts of the evolution of the Normalised Difference Vegetation Index (NDVI) have a typical error of approximately 0.05 (over a natural range of -1 -> 1) at the pixel level over a five month test set horizon, out-performing simple phase-folded models based on historical averaging. We also demonstrate that embeddings learnt by EarthPT hold semantically meaningful information and could be exploited for downstream tasks such as highly granular, dynamic land use classification. Excitingly, we note that the abundance of EO data provides us with -- in theory -- quadrillions of training tokens. Therefore, if we assume that EarthPT follows neural scaling laws akin to those derived for LLMs, there is currently no data-imposed limit to scaling EarthPT and other similar `Large Observation Models.'
- “GPT-NeoX-20B: An Open-Source Autoregressive Language Model” In arXiv, 2022 DOI: 10.48550/arXiv.2204.06745
- “Language Models are Few-Shot Learners” In Advances in Neural Information Processing Systems 33 Curran Associates, Inc., 2020, pp. 1877–1901 URL: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
- OpenAI “GPT-4 Technical Report” In OpenAI Whitepaper, 2023 URL: https://openai.com/research/gpt-4
- “Training Compute-Optimal Large Language Models” In arXiv, 2022 DOI: 10.48550/arXiv.2203.15556
- “RWKV: Reinventing RNNs for the Transformer Era” In arXiv, 2023 DOI: 10.48550/arXiv.2305.13048
- “Learning Curves: Asymptotic Values and Rate of Convergence” In Advances in Neural Information Processing Systems 6, 1993 URL: https://proceedings.neurips.cc/paper/1993/hash/1aa48fc4880bb0c9b8a3bf979d3b917e-Abstract.html
- “Scaling Laws for Neural Language Models” In arXiv, 2020 DOI: 10.48550/arXiv.2001.08361
- “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” In arXiv, 2022 DOI: 10.48550/arXiv.2201.11903
- “Astronomia ex machina: a history, primer and outlook on neural networks in astronomy” In R. Soc. Open Sci. 10.5 The Royal Society, 2023, pp. 221454 DOI: 10.1098/rsos.221454
- R. Friel “Chinchilla’s Wild Implications”, 2022 URL: https://www.alignmentforum.org/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications
- “Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning” In arXiv, 2022 DOI: 10.48550/arXiv.2211.04325
- “Satellite Image Time Series Classification With Pixel-Set Encoders and Temporal Self-Attention” In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) IEEE, 2020, pp. 13–19 DOI: 10.1109/CVPR42600.2020.01234
- “Self-attention for raw optical Satellite Time Series Classification” In ISPRS J. Photogramm. Remote Sens. 169 Elsevier, 2020, pp. 421–435 DOI: 10.1016/j.isprsjprs.2020.06.006
- “SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery” In arXiv, 2022 DOI: 10.48550/arXiv.2207.08051
- “Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning” In arXiv, 2022 DOI: 10.48550/arXiv.2212.14532
- “Lightweight, Pre-trained Transformers for Remote Sensing Timeseries” In arXiv, 2023 DOI: 10.48550/arXiv.2304.14065
- “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 4171–4186 DOI: 10.18653/v1/N19-1423
- “An Efficient Global Scale Sentinel-1 Radar Backscatter and Interferometric Processing System” In Remote Sens. 14.15 Multidisciplinary Digital Publishing Institute, 2022, pp. 3524 DOI: 10.3390/rs14153524
- “Array programming with NumPy” In Nature 585 Nature Publishing Group, 2020, pp. 357–362 DOI: 10.1038/s41586-020-2649-2
- “Language Models are Unsupervised Multitask Learners” In OpenAI Whitepaper, 2019 URL: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- “Adam: A Method for Stochastic Optimization” In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015 URL: http://arxiv.org/abs/1412.6980
- P.J. Huber “Robust Estimation of a Location Parameter” In Ann. Math. Stat. 35.1 Institute of Mathematical Statistics, 1964, pp. 73–101 DOI: 10.1214/aoms/1177703732
- “Forecasting vegetation condition with a Bayesian auto-regressive distributed lags (BARDL) model” In Nat. Hazards Earth Syst. Sci. 22.8 Copernicus GmbH, 2022, pp. 2703–2723 DOI: 10.5194/nhess-22-2703-2022
- K. Pearson “LIII. On lines and planes of closest fit to systems of points in space” In London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2.11 Taylor & Francis, 1901, pp. 559–572 DOI: 10.1080/14786440109462720
- “ADS” In Monthly Notices of the Royal Astronomical Society, Volume 441, Issue 2, p.1741-1759 441.2, 2014, pp. 1741 DOI: 10.1093/mnras/stu642
- “A Generalist Agent” In Transactions on Machine Learning Research, 2022 URL: https://openreview.net/forum?id=1ikK0kHjvj
- “Llama 2: Open Foundation and Fine-Tuned Chat Models” In arXiv, 2023 DOI: 10.48550/arXiv.2307.09288
- “Towards an astronomical foundation model for stars with a Transformer-based model” In arXiv, 2023 DOI: 10.48550/arXiv.2308.10944
- “Quantifying the Carbon Emissions of Machine Learning” In arXiv, 2019 DOI: 10.48550/arXiv.1910.09700
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.