2000 character limit reached
GeoLLaVA: Efficient Fine-Tuned Vision-Language Models for Temporal Change Detection in Remote Sensing (2410.19552v2)
Published 25 Oct 2024 in cs.CV
Abstract: Detecting temporal changes in geographical landscapes is critical for applications like environmental monitoring and urban planning. While remote sensing data is abundant, existing vision-LLMs (VLMs) often fail to capture temporal dynamics effectively. This paper addresses these limitations by introducing an annotated dataset of video frame pairs to track evolving geographical patterns over time. Using fine-tuning techniques like Low-Rank Adaptation (LoRA), quantized LoRA (QLoRA), and model pruning on models such as Video-LLaVA and LLaVA-NeXT-Video, we significantly enhance VLM performance in processing remote sensing temporal changes. Results show significant improvements, with the best performance achieving a BERT score of 0.864 and ROUGE-1 score of 0.576, demonstrating superior accuracy in describing land-use transformations.
- Prompt–rsvqa: Prompting visual context to a language model for remote sensing visual question answering. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1371–1380.
- Hao Chen and Zhenwei Shi. 2020. A spatial-temporal attention-based method and a new dataset for remote sensing image change detection. Remote Sensing, 12(10).
- Change detection methods for remote sensing in the last decade: A comprehensive review. Remote Sensing, 16(13).
- Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6172–6180.
- Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. Advances in Neural Information Processing Systems, 35:197–211.
- QLoRA: Efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems.
- Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- Spacenet: A remote sensing dataset and challenge series. Preprint, arXiv:1807.01232.
- LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
- Rsgpt: A remote sensing vision language model and benchmark. Preprint, arXiv:2307.15266.
- Geochat: Grounded large vision-language model for remote sensing. The IEEE/CVF Conference on Computer Vision and Pattern Recognition.
- Vision-language models in remote sensing: Current progress and future trends. IEEE Geoscience and Remote Sensing Magazine, 12(2):32–66.
- Rs-clip: Zero shot remote sensing scene classification via contrastive vision-language supervision. International Journal of Applied Earth Observation and Geoinformation, 124:103497.
- Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
- Remoteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 62:1–16.
- Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306.
- Llava-next: Improved reasoning, ocr, and world knowledge.
- Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems.
- Language models as black-box optimizers for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12687–12697.
- Era: A data set and deep learning benchmark for event recognition in aerial videos [software and data sets]. IEEE Geoscience and Remote Sensing Magazine, 8(4):125–133.
- OpenAI. 2024. Chatgpt: A large language model. https://chat.openai.com/. Accessed: 2024-10-07.
- Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
- Rajvardhan Patil and Venkat Gudivada. 2024. A review of current trends, techniques, and challenges in large language models (llms). Applied Sciences, 14(5):2074.
- Floodnet: A high resolution aerial imagery dataset for post flood scene understanding. IEEE Access, 9:89644–89654.
- A survey of modelling trends in temporal gis. ACM Comput. Surv., 51(2).
- Using historical maps within a gis to analyze two centuries of rural landscape changes in southern italy. Land, 6(3).
- Villa: Fine-grained vision-language representation learning from real-world data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22225–22235.
- EfficientVLM: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13899–13913, Toronto, Canada. Association for Computational Linguistics.
- Skyscript: A large and semantically diverse vision-language dataset for remote sensing. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6):5805–5813.
- Gis and remote sensing application for vegetation mapping. In T. Choudhury, B. Koley, A. Nath, JS. Um, and A.K. Patidar, editors, Geo-Environmental Hazards using AI-enabled Geospatial Techniques and Earth Observation Systems, Advances in Geographic Information Science. Springer, Cham.
- From easy to hard: Learning language-guided curriculum for visual question answering on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 60.
- Vision-language models for vision tasks: A survey. Preprint, arXiv:2304.00685.
- Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
- Msnet: A multilevel instance segmentation network for natural disaster damage assessment in aerial videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2710–2720. IEEE.
- Transforming remote sensing images to textual descriptions. International Journal of Applied Earth Observation and Geoinformation, 108:102741.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
