Few-Shot Hyperspectral Image Classification With Unknown Classes Using Multitask Deep Learning (2009.03508v1)

Abstract: Current hyperspectral image classification assumes that a predefined classification system is closed and complete, and there are no unknown or novel classes in the unseen data. However, this assumption may be too strict for the real world. Often, novel classes are overlooked when the classification system is constructed. The closed nature forces a model to assign a label given a new sample and may lead to overestimation of known land covers (e.g., crop area). To tackle this issue, we propose a multitask deep learning method that simultaneously conducts classification and reconstruction in the open world (named MDL4OW) where unknown classes may exist. The reconstructed data are compared with the original data; those failing to be reconstructed are considered unknown, based on the assumption that they are not well represented in the latent features due to the lack of labels. A threshold needs to be defined to separate the unknown and known classes; we propose two strategies based on the extreme value theory for few-shot and many-shot scenarios. The proposed method was tested on real-world hyperspectral images; state-of-the-art results were achieved, e.g., improving the overall accuracy by 4.94% for the Salinas data. By considering the existence of unknown classes in the open world, our method achieved more accurate hyperspectral image classification, especially under the few-shot context.

• The paper proposes MDL4OW, a multitask deep learning method using CNNs, reconstruction, and Extreme Value Theory to classify known and detect unknown classes in hyperspectral images.
• Experimental results show MDL4OW outperforms traditional methods on multiple datasets, accurately identifying novel classes and improving classification metrics under few-shot settings.
• Addressing the open-world assumption is crucial for accurate HSI classification, offering practical benefits for environmental monitoring and resource management.

Few-Shot Hyperspectral Image Classification With Unknown Classes Using Multitask Deep Learning

The research paper addresses a notable challenge in the domain of hyperspectral image classification: the assumption that classification systems are closed and complete, ignoring the potential existence of unknown or novel classes in unseen data. The paper proposes a multitask deep learning approach, termed MDL4OW (Multitask Deep Learning for Open World), to reconcile this limitation by incorporating the classification of unknown classes alongside known classes.

Methodology

The proposed MDL4OW approach employs a convolutional neural network (CNN) structured to perform dual tasks: classification and reconstruction of hyperspectral images. The workflow involves:

• Encoder and Classifier: Utilizes CNN architectures like residual units for feature extraction, followed by a Softmax function for assigning probabilities to known classes.
• Reconstruction Branch: A network decoder attempts to reconstruct the input hyperspectral data. The underlying rationale is that known classes, due to training exposure, will show minimal reconstruction error, while unknown classes, lacking representation in training, will exhibit significant reconstruction errors.
• Extreme Value Theory (EVT): Implemented to model the distribution of reconstruction losses, enabling the identification of unknown classes. EVT provides a statistical basis for setting thresholds that distinguish between known and unknown classes based on reconstruction errors.

Experimental Evaluation

Empirical evaluations were conducted on three hyperspectral datasets: the University of Pavia, Salinas Valley, and Indian Pines. These datasets were chosen to reflect varying degrees of complexity and openness (percentage of unknown classes):

1. University of Pavia: The method achieved notable improvements in overall accuracy (OA) and F1 scores compared to baseline and other state-of-the-art methods, particularly under few-shot settings (20 samples per class).
2. Salinas Valley: MDL4OW provided significant gains in F1 scores and reduced mapping errors, underscoring its utility in precise crop mapping scenarios.
3. Indian Pines: Demonstrated effective open-world classification, identifying novel classes while preserving the accuracy of known classes.

Across all datasets, the proposed MDL4OW outperformed conventional methods, actively rejecting novel classes, which traditional classification systems incorrectly labeled within existing categories, thus mitigating the risk of overestimating certain land cover types.

Implications and Future Directions

The paper highlights the criticality of addressing the open-world assumption in hyperspectral image classification, especially given the often oversimplified close-world assumption prevalent in existing methodologies. The improved accuracy of land cover area estimates posits practical benefits for environmental monitoring and resource management.

Looking forward, enhancements to the MDL4OW approach could involve refining the loss estimation process, possibly integrating alternative statistical models to improve EVT analysis, and exploring extensions into real-world applications such as urban planning, environmental management, and agricultural precision. Advancements in handling hyperspectral data with better representational learning can further augment the classification accuracy across diverse, and increasingly complex, open-world scenarios.