Task-Guided Multi-Annotation Triplet Learning for Remote Sensing Representations

Abstract: Prior multi-task triplet loss methods relied on static weights to balance supervision between various types of annotation. However, static weighting requires tuning and does not account for how tasks interact when shaping a shared representation. To address this, the proposed task-guided multi-annotation triplet loss removes this dependency by selecting triplets through a mutual-information criteria that identifies triplets most informative across tasks. This strategy modifies which samples influence the representation rather than adjusting loss magnitudes. Experiments on an aerial wildlife dataset compare the proposed task-guided selection against several triplet loss setups for shaping a representation in an effective multi-task manner. The results show improved classification and regression performance and demonstrate that task-aware triplet selection produces a more effective shared representation for downstream tasks.

• The paper introduces TG-MATL, a mutual information-based approach that dynamically selects informative triplets to build unified embedding spaces.
• It employs a two-stage training pipeline with frozen Vision Transformer embeddings to balance semantic and geometric supervision effectively.
• Experiments on the AWIR dataset show TG-MATL outperforms static-weighting methods, achieving superior results in both classification and geometric regression.

Task-Guided Multi-Annotation Triplet Learning for Remote Sensing Representations

Introduction and Motivation

Multi-task learning in remote sensing mandates the formation of a unified embedding space that can simultaneously encode heterogeneous supervision sources, most commonly semantic (class) labels and geometric (box) attributes. Conventional multi-annotation triplet loss (MATL) methods fuse these different annotation signals using static scalar weights, controlling the impact of each annotation type during representation learning. However, static weighting suffers from several deficiencies: it requires extensive hyperparameter tuning, fails to adapt to the statistical dependencies between annotation types, and may be suboptimal for datasets with class imbalance or annotation noise. Critically, such approaches do not consider the interaction between tasks in shaping the geometry of the learned representation, often leading to over-emphasis on either semantic or geometric structure.

This work introduces a task-guided multi-annotation triplet loss (TG-MATL) that eschews static weighting in favor of a dynamic sample selection process guided by mutual information between annotation types. Rather than tuning loss magnitudes, this approach selects the most informative triplets, effectively controlling which sample relationships sculpt the embedding space. The proposed method demonstrates improved performance on both classification and regression tasks in remote sensing, indicating that mutual information-guided triplet selection provides a more effective mechanism for multi-task representation learning.

Methodology

Two-Stage Training Architecture

The proposed approach is implemented as a two-stage pipeline. In the first stage, frozen Vision Transformer–based embeddings are extracted. Different representation learning losses, including traditional and task-guided variants, are used to supervise the embedding space without fine-tuning the base encoder. In the second stage, linear task heads are trained atop the learned embeddings to measure the effect of upstream loss formulation on downstream classification and regression tasks.

Figure 1: The two-stage architecture leverages frozen ViT-based embeddings, shapes the latent space using representation losses, and evaluates with linear heads for each task.

Multi-Annotation Triplet Loss Recap

The baseline MATL formulation extends ordinary triplet loss by incorporating triplets from multiple annotation types. Specifically, anchor-positive pairs can be selected via semantic class agreement or via geometric similarity derived from discretized bounding box attributes. The combined objective is a weighted sum:

$$\mathcal{L}_{\mathrm{MATL}} = (1-\lambda)\, \mathcal{L}_{\mathrm{class}} + \lambda\, \mathcal{L}_{\mathrm{box}}$$

where $\lambda$ balances semantic and geometric supervision. This design introduces a crucial hyperparameter and still processes all possible triplets, regardless of informativeness.

Mutual Information Guided Triplet Selection

TG-MATL introduces task-driven sampling by leveraging mutual information (MI) between geometric features and class labels. For each sample, a relevance score is computed as the sum over geometric features, weighted by the MI between that feature and the class label. Per-class, a top fraction of high-relevance samples is retained, potentially augmented by a uniformly random subset to maintain within-class diversity.

The mutual information metric exposes the statistical dependence structure between annotation types, ensuring only those anchor-positive pairs exhibiting strong cross-task relationships guide embedding construction. Anchor and positive samples are thus filtered using this MI-driven relevance mask, while negatives are sampled according to the multi-annotation protocol.

Experimental Results

Dataset and Implementation

Experiments are conducted on the Animal Wildlife Image Repository (AWIR) dataset, which provides both class labels and detailed bounding boxes for animals in aerial remote sensing imagery. Features such as area and symmetric squareness, derived from box annotations, capture geometric variance within and across classes. Frozen ViT-based models (DINOv2, CLIP, and MAE) are employed as feature extractors, and their output is used across all comparative loss regimes.

Downstream Task Performance

Results show that task-guided triplet selection yields notable improvements on downstream tasks. When evaluated on classification accuracy and box regression $R^2$, TG-MATL matches or outperforms baseline MATL for both tasks. Remarkably, hard-sampled class-label-only triplet loss leads to the highest classification accuracy, but substantially degrades geometric regression $R^2$ due to lack of geometric structural alignment. TG-MATL, in contrast, achieves superior geometric regression while retaining high classification accuracy, demonstrating a more balanced joint-embedding structure.

Figure 2: Heatmaps demonstrate how varying the proportion of top-relevance (MI-guided) samples and random samples modulates downstream classification and regression performance for TG-MATL with CLIP embeddings.

Analysis of Sampling Strategies

A comprehensive study on the effect of hyperparameter tuning—specifically, the fractions of top-relevance and random samples per class—reveals substantive dynamics. Increasing the random sampling fraction within each class enhances semantic discrimination (classification accuracy), though it may dilute geometrically-informative supervision, thereby decreasing geometric regression performance. Conversely, focusing on high-relevance samples sharpens geometric reasoning but risks overfitting or reduced class diversity. This illustrates the necessity of balancing sample informativeness against intra-class diversity—a trade-off that the mutual information-guided framework seeks to manage systematically.

Practical and Theoretical Implications

By directly intervening at the level of sample selection (as opposed to loss weighting), task-guided triplet methods exert a more targeted influence over the structure of the latent space. Mutual information provides a principled measure for evaluating cross-annotation relevance, enabling informed sample inclusion that optimally supports both semantic and geometric reasoning tasks.

This paradigm is theoretically extensible. In principle, continuous-valued geometric labels could be used directly (without discretization) for even more granular control—suggesting applications in unimodal and multimodal domains where supervision emerges from heterogeneous and interdependent annotation sources. Furthermore, the generality of the MI-guided selection mechanism positions the approach as a template for multi-task learning beyond remote sensing (e.g., general vision-language tasks, multi-modal fusion in Earth science, or robotics).

Conclusion

This study demonstrates that mutual information-guided triplet selection within the multi-annotation framework generates more effective and balanced representations for multi-task remote sensing analysis. The method surpasses static loss-weighting in aligning embedding structure to both semantic and geometric objectives, as validated on a benchmark aerial wildlife dataset. Future extensions should explore MI-guided sampling in a continuous-label regime and across even more heterogeneous, multi-modal annotation sources.