2000 character limit reached
Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery
Published 26 Feb 2026 in cs.CV | (2602.22613v1)
Abstract: Vision-language foundation models (VLFMs) promise zero-shot and retrieval understanding for Earth observation. While operational satellite systems often lack full multi-spectral coverage, making RGB-only inference highly desirable for scalable deployment, the adoption of VLFMs for satellite imagery remains hindered by two factors: (1) multi-spectral inputs are informative but difficult to exploit consistently due to band redundancy and misalignment; and (2) CLIP-style text encoders limit semantic expressiveness and weaken fine-grained alignment. We present SATtxt, a spectrum-aware VLFM that operates with RGB inputs only at inference while retaining spectral cues learned during training. Our framework comprises two stages. First, Spectral Representation Distillation transfers spectral priors from a frozen multi-spectral teacher to an RGB student via a lightweight projector. Second, Spectrally Grounded Alignment with Instruction-Augmented LLMs bridges the distilled visual space and an expressive LLM embedding space. Across EuroSAT, BigEarthNet, and ForestNet, SATtxt improves zero-shot classification on average by 4.2%, retrieval by 5.9%, and linear probing by 2.7% over baselines, showing an efficient path toward spectrum-aware vision-language learning for Earth observation. Project page: https://ikhado.github.io/sattxt/
Summary
- The paper presents a novel two-stage pre-training method (SATtxt) that distills spectral cues from multi-spectral data to improve RGB-only satellite imagery analysis.
- It leverages spectral representation distillation and alignment with instruction-augmented LLMs to bridge the gap between visual and textual modalities.
- Experimental results indicate up to 4.2% improvement in zero-shot classification and 5.9% in retrieval, highlighting efficient inference performance.
Spectrally Distilled Representations in Satellite Imagery
Introduction
The paper "Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery" (2602.22613) addresses key challenges in deploying Vision-Language Foundation Models (VLFMs) for Earth observation using satellite imagery. Despite the promise of robust zero-shot classification and retrieval capabilities, current VLFMs are hindered by the lack of full multi-spectral coverage and limitations in semantic expressiveness when employing CLIP-style text encoders. The proposed method, SATtxt, introduces a novel approach to leverage spectral cues during training while enabling RGB-only inference, thus enhancing scalability and operational efficiency.
Methodology
Two-Stage Pre-Training Pipeline
The methodology is anchored on a two-stage pre-training pipeline designed to exploit spectral cues from multi-spectral data while ensuring efficient RGB-based inference.
- Spectral Representation Distillation (SRD): The first stage involves transferring spectral priors from a frozen multi-spectral teacher model to an RGB student model through a lightweight projector. This projector reconstructs multi-spectral representations from RGB inputs, allowing the RGB encoder to capture spectral knowledge indispensably.
- Spectrally Grounded Alignment with Instruction-Augmented LLMs (SGI-LLM): The second stage aligns the spectrally distilled visual representations with a robust text space generated by instruction-augmented LLMs. This alignment enhances the semantic richness of the LLM's embeddings and bridges the gap between text and visual modalities.
Figure 1: Two-stage pre-training pipeline for SATtxt.
Advantages and Implementation Details
The approach not only preserves the spectral knowledge within a lightweight framework but also leverages LLMs to encapsulate richer text semantics. Training involves spectral alignment and contrastive learning between visual and text encoders without the need to fine-tune the underlying frozen models, showcasing efficient bridge establishment between distinct modalities.
Experimental Results
The performance of SATtxt was rigorously evaluated across several satellite benchmarks, including EuroSAT, BigEarthNet, and ForestNet. The model demonstrated significant improvements over existing baselines, achieving an average enhancement of 4.2% in zero-shot classification and 5.9% in retrieval.
Despite operating solely with RGB data at inference, SATtxt achieved robust semantic alignment comparable to multi-spectral baselines. This highlights the efficacy of spectral distillation and text alignment via LLMs in satellite imagery applications.
Discussion and Implications
The research underscores the practical advancements in integrating spectral awareness within RGB constraints, aligning high-domain specificity in satellite imagery analysis with the expressive semantic capacity of LLMs. The proposed SATtxt framework offers a scalable solution for deploying spectrum-aware VLFMs in Earth observation scenarios, facilitating zero-shot and retrieval tasks without extensive spectral data dependency.
Looking forward, potential expansions include enhancing coverage to sensors beyond optical imagery, such as SAR or thermal imaging, and exploring model compression techniques for increased efficiency. These avenues will further pave the way for versatile applications in real-time satellite data analysis and broader environmental monitoring frameworks.
Conclusion
The paper presents SATtxt as a novel spectrum-aware vision-LLM addressing significant challenges in deploying VLFMs for satellite imagery analysis. By retaining spectral cues during training while operating exclusively on RGB inputs, SATtxt sets a precedent for efficient and scalable satellite data exploitation, fostering advancements in Earth observation through zero-shot and retrieval capabilities. The structured approach highlights significant strides in aligning textual expressiveness with visual semantics, marking a substantial progression in remote sensing and satellite imagery interpretation.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about (in simple terms)
This paper shows how to teach a computer to understand satellite photos using only regular color images (the kind with red, green, and blue—RGB) when it’s actually used, while still learning helpful “extra colors” during training. It also connects those images to a smart LLM so the computer understands richer, clearer text descriptions. The result is a system, called SATtxt, that can recognize places and find images from text without needing lots of labeled examples.
The big questions the researchers asked
- How can we get the benefits of many satellite “colors” (called spectral bands—some are invisible to human eyes) without needing those extra bands at test time, when they’re often missing or messy?
- Can we replace simpler text encoders with a stronger LLM to better match images to words and phrases?
- Will this make “zero-shot” tasks better—like recognizing or retrieving things the model wasn’t directly trained on?
How the method works, with everyday analogies
The training happens in two stages.
Stage 1: Learning extra “colors” without seeing them later
- Analogy: Imagine a teacher who can see many more colors than you (like infrared and other bands), and a student who only sees normal RGB. The teacher already knows a lot about how those extra colors reveal useful details (like vegetation health, water, or buildings).
- The researchers keep both the teacher and the student “frozen” (unchanged). They train a tiny “translator” between them:
- The teacher looks at the full multi-spectral image (many bands).
- The student looks at the RGB version.
- The translator learns to map the student’s RGB features into the teacher’s “extra-color” feature space.
- Outcome: After this, the RGB student has “absorbed” the teacher’s spectral knowledge. At test time, the model only needs RGB, but it behaves as if it remembers those extra bands.
In technical terms:
- “Distillation” = transferring knowledge from a strong teacher to a student.
- “Projector/translator” = a small neural layer that reshapes one set of features to match another.
- “Frozen encoders” = the big models don’t change; only the small translator is trained. This saves time and avoids overfitting.
Stage 2: Matching images to smarter text
- Analogy: Now that the student “thinks” with extra-color knowledge, we want it to speak the same language as a very smart writer who can understand detailed instructions—an LLM.
- They connect the RGB image features (now enriched with spectral cues) to text features produced by an instruction-augmented LLM. Both image and text backbones are kept frozen. They only train small “projectors” on each side to align the spaces.
- They use contrastive learning:
- “Pull together” the image and its matching text description.
- “Push apart” mismatched pairs.
- Because the LLM can handle longer, richer text (not just short labels), the alignment becomes more detailed and accurate.
In technical terms:
- “Instruction-augmented LLM” = a LLM that handles prompts with helpful guidance, not just short labels. This makes the text side more expressive than standard CLIP-style text encoders.
- “Contrastive learning” = a way to make matching pairs similar and non-matching pairs dissimilar in the model’s feature space.
- Caching: Text features can be computed once and reused, so using a big LLM doesn’t slow down real-time use.
What they found (and why it matters)
Across three common satellite benchmarks (EuroSAT, BigEarthNet, ForestNet), SATtxt:
- Improved zero-shot classification by about 4.2% on average over strong baselines.
- Improved text-to-image retrieval by about 5.9% on average.
- Improved linear probing (a simple test of feature quality) by about 2.7% on average.
- Beat some models that use full multi-spectral inputs—even though SATtxt uses RGB only at test time.
- Produced sharper, more focused similarity maps (for example, better highlighting rivers or residential areas), which helps explain its better accuracy.
- In open-vocabulary segmentation (labeling pixels without a special decoder), it reached 31.23 mIoU, better than a strong multi-spectral baseline.
Why this matters:
- In the real world, full multi-spectral stacks are often missing, misaligned, or expensive to process. Getting strong performance with RGB alone makes deployment easier and cheaper.
- Using a smarter LLM brings more precise matching between what the image shows and how we describe it in words.
What this could change going forward
- Practical monitoring: Agencies can classify land cover, find deforestation, or search for floods using widely available RGB images, while still benefiting from “remembered” spectral knowledge learned during training.
- Less labeling: Stronger zero-shot performance means you can do more with fewer labeled examples.
- Flexible retrieval: You can search for images using richer, more natural descriptions, thanks to the LLM-based text side.
Limitations and next steps:
- Right now, the method focuses on optical imagery (like Sentinel-2). Extending it to other sensors, such as radar (SAR) or thermal, would make it even more broadly useful.
- Using a LLM for text features can require more memory, though caching helps reduce the cost during use.
In short: SATtxt cleverly learns from many-band satellite images and from a smart LLM, then runs with just RGB and still performs great. This makes satellite understanding more accurate, accessible, and ready for real-world scenarios.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a consolidated list of concrete limitations and open questions that the paper leaves unresolved, framed to guide actionable future research.
- Modalities beyond optical RGB remain unsupported at inference; extend SRD and alignment to non-optical sensors (e.g., SAR, thermal, nighttime lights) and evaluate on multi-sensor datasets.
- Cross-sensor generalization is untested; assess transfer to sensors with different spectral/spatial characteristics (e.g., Landsat, PlanetScope, WorldView, aerial imagery, panchromatic) and mixed-resolution fusion.
- Temporal reasoning is absent; incorporate multi-temporal inputs and evaluate on change detection, crop phenology, flood mapping, and seasonally variable scenes.
- SRD assumes paired and well-registered RGB–MS samples; develop and test unpaired/weakly aligned distillation (e.g., cycle-consistency, contrastive matching) to handle common archive scenarios without exact pairs.
- Robustness to spectral band dropouts, sensor drift, and atmospheric artifacts is not analyzed; simulate missing bands, cloud/haze, and misregistration during SRD and measure downstream degradation.
- Distillation granularity is global (pooled MS tokens); investigate patch-level or region-level spectral distillation to better preserve spatial–spectral correspondences, especially for segmentation.
- The RGB vision backbone is frozen during SRD; study partial or low-rank adaptation of the backbone (vs projector-only) to absorb spectral priors and quantify overfitting vs generalization trade-offs.
- Projector architecture is fixed and shallow; ablate projector depth/width, nonlinearity, cross-attention vs MLP/Transformer, and weight sharing to balance performance, compute, and stability.
- Teacher model dependence is only lightly probed; systematically compare different MS teachers (architectures, pretraining objectives, datasets) and quantify how teacher capacity/quality affects SRD gains.
- Domain shift beyond Sentinel-2-derived benchmarks is underexplored; evaluate on additional domains (urban high-res, forestry LiDAR-fused RGB, coastal/aquatic imagery) and geographically/seasonally distinct test sets.
- Prompt and instruction design is not deeply analyzed; perform prompt-sensitivity and instruction-ablation studies (length, task hints, domain jargon), prompt ensembling, and automatic instruction generation.
- Text encoder choices are limited; explore smaller multilingual LLMs, domain-adapted language encoders, and distilling LLM embeddings into compact text adapters to reduce memory while retaining expressiveness.
- Multilingual and cross-lingual capabilities are untested; evaluate zero-shot classification/retrieval for prompts in multiple languages and mixed-language settings common in geospatial workflows.
- Free-form, compositional, and relational queries are not evaluated; test complex natural-language prompts (multi-concept, spatial relations, negation) beyond class labels for retrieval and OV tasks.
- Broader downstream tasks are missing; benchmark on detection (e.g., ships, buildings), instance segmentation, road/river extraction, counting, and weakly supervised/low-shot settings.
- Open-set recognition and OOD robustness are not assessed; measure false-positive rates, confidence calibration, and OOD detection when queried with unseen classes or out-of-domain imagery.
- Training/data scaling laws are unknown; quantify how performance scales with (i) pretraining image–text corpus size/quality, (ii) LLM size, (iii) SRD training duration, and (iv) projector capacity.
- Caption quality/noise sensitivity is not studied; analyze robustness to noisy, hallucinated, or misaligned LLM captions and investigate filtering or self-correction strategies.
- Computational footprint is only briefly stated; provide detailed benchmarks of training/inference latency, memory, and throughput across GPUs/edge devices, and compare against CLIP-sized alternatives.
- Caching assumptions are idealized; evaluate latency and memory when handling large, dynamic query sets (not just cached label sets) to reflect real-world retrieval workloads.
- Spatial resolution and tiling strategies are not discussed; study the effect of input resolution, tiling/windowing schemes, and multi-scale pooling on both global and pixel-level tasks.
- Failure modes of spectral hallucination are not examined; characterize when SRD induces misleading “spectral” cues from RGB (e.g., snow/ice, turbid water, aerosols) and design mitigations.
- Integration of geospatial metadata is absent; explore adding coordinates, time, season, and ancillary layers (DEM, land-cover priors) into the alignment to improve context-aware reasoning.
Practical Applications
Immediate Applications
The following applications can be deployed now using the paper’s SATtxt method (RGB-only inference with spectral priors) and training recipe (SRD + LLM-aligned projectors), leveraging the reported gains in zero-shot classification, retrieval, open-vocabulary segmentation, and linear probing.
- Bold application name: Zero-shot land-cover tagging for RGB imagery
- Sectors: geospatial/GIS, environment, urban planning
- Tools/products/workflows: GIS plugin or API that auto-tags tiles with labels (e.g., “forest,” “urban,” “water,” “permanent crops”) using prompt templates; batch processing of STAC catalogs
- Assumptions/dependencies: RGB optical imagery with sufficient resolution; prompt library for consistent class definitions; domain shift mitigation for new regions/sensors; human QA for critical uses
- Bold application name: Rapid disaster mapping when multispectral is unavailable
- Sectors: emergency management, insurance, humanitarian aid
- Tools/products/workflows: Post-event triage via text-to-image retrieval (e.g., “collapsed buildings,” “flooded neighborhoods”), open-vocabulary segmentation overlays for inundation/damage footprints; dashboard to prioritize field visits
- Assumptions/dependencies: Cloud-free RGB coverage; seasonal appearance variability; human-in-the-loop validation for high-stakes decisions
- Bold application name: Deforestation and encroachment alerts from RGB
- Sectors: forestry, conservation, biodiversity
- Tools/products/workflows: Change monitoring pipeline that periodically runs zero-shot queries (e.g., “new logging roads,” “cleared forest patches”) and flags hotspots; alerting system integrated with ranger apps
- Assumptions/dependencies: Revisit frequency and resolution adequate to detect small changes; some phenomena (e.g., species-level disturbances) require MS/SAR for reliability
- Bold application name: Water bodies and river corridor monitoring
- Sectors: environmental monitoring, water management
- Tools/products/workflows: Zero-shot retrieval/segmentation for “river,” “reservoir,” “coastal lagoon,” with patch-level similarity maps to delineate boundaries; GIS overlays for change tracking
- Assumptions/dependencies: RGB provides shape/extent cues but not physico-chemical properties; turbidity/algal blooms better with MS; careful thresholding and QA needed
- Bold application name: Infrastructure and energy asset mapping
- Sectors: energy/utilities, telecom, urban planning
- Tools/products/workflows: Text queries like “solar farm,” “wind turbines,” “transmission substation,” “new road” to index assets; embedding-based deduplication; rapid updates for asset registries
- Assumptions/dependencies: Distinctive visual signatures in RGB; regional design differences; confirmation workflows to reduce false positives
- Bold application name: Agricultural pre-labeling and field delineation from drones/satellites (RGB)
- Sectors: ag-tech, precision agriculture
- Tools/products/workflows: Open-vocabulary segmentation for field boundaries; zero-shot class candidates (e.g., “orchard,” “vineyard,” “rice paddy”) to bootstrap labeling and scouting
- Assumptions/dependencies: Crop health/biochemical traits are not reliably captured in RGB; local agronomy expertise and in-field verification recommended
- Bold application name: Portfolio and ESG geospatial screening
- Sectors: finance/ESG, corporate risk
- Tools/products/workflows: Monitor potential expansions or new constructions (“new open-pit mine,” “new solar installation”) around known assets using text-to-image retrieval; periodic screening reports
- Assumptions/dependencies: Legal/ethical use of imagery; reliance on RGB-visible cues; clear escalation/verification processes for compliance
- Bold application name: Natural-language search over image archives
- Sectors: software/geoportals, data providers
- Tools/products/workflows: “Text-to-EO search” API for end-users to describe scenes (“dense residential by a river”) and retrieve matching tiles; caches of label embeddings for low-latency query
- Assumptions/dependencies: Prompt quality and consistent vocabulary; guardrails for ambiguous/unsafe queries; indexing at archive scale
- Bold application name: Open-vocabulary segmentation overlays in GIS
- Sectors: government planning, utilities, academia
- Tools/products/workflows: Decoder-free segmentation masks from patch-wise image–text similarities for planning layers (e.g., “wetlands,” “industrial zones”)
- Assumptions/dependencies: Class boundary accuracy depends on spatial resolution and scene complexity; recommended for pre-annotation and exploratory analysis
- Bold application name: Dataset curation and active learning for remote sensing
- Sectors: industry R&D, academia
- Tools/products/workflows: Use text-to-image retrieval to surface candidate tiles for underrepresented classes; pre-seed labels to reduce annotation burden; active-learning loops using similarity scores
- Assumptions/dependencies: Curated prompt sets and review interfaces; bias management for long-tail categories
- Bold application name: Lightweight training and adaptation for low-resource teams
- Sectors: public sector, NGOs, startups
- Tools/products/workflows: Replicate Stage 2 locally with cached LLM embeddings and frozen backbones; adapt to regional vocabularies or priority tasks with minimal compute
- Assumptions/dependencies: Access to captions/prompts in local context; memory budget for LLM embeddings (can be cached and pruned)
- Bold application name: Onboard RGB tagging on small UAVs
- Sectors: robotics, conservation, agriculture
- Tools/products/workflows: Edge inference using cached text embeddings and quantized SATtxt vision+projectors to tag frames (“erosion,” “illegal dumping,” “crop lodging”) during flight
- Assumptions/dependencies: On-device memory/compute constraints; model compression/quantization; safety approvals for operational use
Long-Term Applications
These applications require additional research, scaling, or integration (e.g., broader modality coverage, larger deployments, standardization, or regulatory validation).
- Bold application name: Distillation from non-optical sensors (SAR, thermal) into RGB models
- Sectors: disaster response, defense, maritime, wildfire management
- Tools/products/workflows: Extend SRD to transfer priors from SAR/thermal teachers to RGB students for all-weather/night-time proxies; hybrid models that fall back to SAR when RGB is unavailable
- Assumptions/dependencies: Paired or co-registered multi-sensor datasets; robust cross-modality alignment; careful evaluation to avoid hallucinating non-visible properties
- Bold application name: Global, open-vocabulary land-cover and object maps on demand
- Sectors: environment, urban planning, NGOs
- Tools/products/workflows: Prompt-driven mapping services (“map all ‘wetlands’ within X region”), producing dynamic, class-on-demand layers backed by uncertainty estimates
- Assumptions/dependencies: Scalable tiling/inference infrastructure; standardized prompts and class definitions; continuous validation with ground truth
- Bold application name: Regulatory compliance monitoring at scale
- Sectors: public sector, environmental law, coastal/forestry agencies
- Tools/products/workflows: Automated detection of setbacks, buffer violations, protected area encroachment, and illegal extraction activities with audit trails and evidence management
- Assumptions/dependencies: Jurisdiction-specific rules encoded as prompts/policies; rigorous QA and appeals processes; multi-sensor corroboration for legal robustness
- Bold application name: Multilingual, domain-calibrated prompt libraries
- Sectors: government, NGOs, education
- Tools/products/workflows: Local-language prompt packs aligned to regional ontologies and SDG indicators; LLM-assisted prompt validation against geospatial standards
- Assumptions/dependencies: Terminology harmonization (e.g., CORINE/LCZ/FAO taxonomies); cultural and linguistic nuances; continual updates
- Bold application name: Human-in-the-loop geospatial assistants
- Sectors: software, public sector, enterprises
- Tools/products/workflows: Interactive LLM+VLM copilots in GIS that suggest prompts, explain model rationales via similarity maps, and recommend verification data sources
- Assumptions/dependencies: UX design for explainability; governance for model suggestions; audit logs and traceability
- Bold application name: Continuous learning across new sensors and seasons
- Sectors: data providers, platform vendors
- Tools/products/workflows: Periodic SRD refreshes as new satellites launch; domain adaptation pipelines to handle seasonal/atmospheric shifts; MLOps for foundation models
- Assumptions/dependencies: Access to new teacher models and co-registered datasets; monitoring to avoid catastrophic forgetting
- Bold application name: OGC/STAC-compliant text-to-imagery services
- Sectors: geospatial software, cloud platforms
- Tools/products/workflows: Standardized APIs for “text-to-retrieval” and “text-to-segmentation” over STAC catalogs; integration with map tiles and vector services
- Assumptions/dependencies: Community standards and governance; indexing costs; security for multi-tenant deployments
- Bold application name: Parametric insurance and risk analytics
- Sectors: insurance, reinsurance, catastrophe modeling
- Tools/products/workflows: Triggers based on prompt-defined phenomena (e.g., “flood extent exceeding X% of insured area”) with adjudication interfaces linking evidence tiles
- Assumptions/dependencies: Regulatory acceptance; calibration with ground measurements; multi-sensor fusion for higher confidence
- Bold application name: Urban digital twins with prompt-driven updates
- Sectors: smart cities, infrastructure management
- Tools/products/workflows: Automatically flag “new construction,” “informal settlements,” or “landfill expansion” in digital twins; tasking workflows for targeted high-res collection
- Assumptions/dependencies: High-resolution imagery access; alignment with cadastral data; privacy and governance frameworks
- Bold application name: Education and workforce upskilling in geospatial AI
- Sectors: academia, vocational training
- Tools/products/workflows: Curricula leveraging SATtxt’s zero-shot/segmentation capabilities to teach open-vocabulary EO analysis; hands-on labs with prompt engineering for RS
- Assumptions/dependencies: Open weights, datasets, and compute credits for learners; accessible tooling and GUIs
Cross-cutting dependencies and considerations
- Model scope and limits: Current method is tailored to optical imagery; some tasks fundamentally require multi-spectral/thermal/SAR for scientific reliability.
- Data availability: SRD relies on a capable multi-spectral teacher and paired MS–RGB data during training; performance outside similar domains may degrade.
- Prompt design and bias: Instruction-augmented LLM embeddings improve expressiveness but can encode biases; curated prompt libraries and governance are necessary.
- Operational readiness: For high-stakes decisions (disaster relief, compliance), human-in-the-loop workflows, uncertainty estimation, and multi-sensor corroboration are recommended.
- Compute and memory: Inference uses RGB-only vision encoders with cached text embeddings (low latency); training requires LLM embeddings but can be precomputed; edge deployments may need quantization/distillation.
- Licensing and ethics: Imagery usage must comply with licenses; sensitive-use cases require privacy and ethical safeguards.
Glossary
- band redundancy: Overlapping information across spectral bands that can reduce efficiency or cause instability when using multi-spectral inputs. "due to band redundancy and misalignment"
- centering and temperature sharpening: A stabilization technique in self-distillation that centers teacher outputs and sharpens distributions with temperature parameters. "centering and temperature sharpening strategy of DINO"
- CLIP-style text encoder: The lightweight text encoder used in CLIP-like models, often limited in expressiveness for fine-grained alignment. "CLIP-style text encoders limit semantic expressiveness and weaken fine-grained alignment."
- contrastive learning objective: A loss that brings matched image–text pairs together and pushes mismatched pairs apart in embedding space. "trained under a contrastive learning objective."
- cosine similarity: A similarity measure based on the cosine of the angle between two vectors, commonly used to compare embeddings. "we compute cosine similarity between patch-level vision embeddings and the text embedding"
- cross-modal knowledge distillation: Transferring knowledge from one modality (e.g., multi-spectral) to another (e.g., RGB) via a teacher–student setup. "a cross-modal knowledge distillation framework"
- decoder-free approach: A segmentation or prediction method that avoids a dedicated decoder by leveraging alignment between patch features and text. "we follow the decoder-free approach from DINOtxt"
- DINO: A self-distillation method that learns visual representations without labels using teacher–student training with centering and temperature sharpening. "centering and temperature sharpening strategy of DINO"
- exponential moving average (EMA): A running average that gives more weight to recent values, used here to stabilize teacher targets. "an exponential moving average of the teacher's pre-softmax outputs"
- InfoNCE: A widely used contrastive loss that maximizes agreement of positive pairs against negatives within a batch. "using a symmetric InfoNCE objective"
- instruction-augmented LLMs: LLMs whose embeddings are enhanced by prompts with task instructions for richer semantics. "Spectrally Grounded Alignment with Instruction-Augmented LLMs bridges the distilled visual space and an expressive LLM embedding space."
- JEPA (Joint-Embedding Predictive Architecture): A framework where models learn by predicting relationships between embeddings rather than reconstructing inputs. "within the JEPA framework"
- LiT (Locked-image Tuning): A training paradigm that freezes the image encoder and tunes only the text side for alignment. "LiT: Locked-image Tuning"
- linear probing: Evaluating representation quality by training a single linear layer on frozen features for downstream tasks. "For linear probing, we follow the PANGAEA setup"
- mean average precision (mAP@100): The average precision metric computed over ranked retrieval results, truncated at the top 100 items. "We report the class-averaged mAP@100."
- mean pooling: Aggregating token embeddings by averaging them to obtain a sentence-level representation. "Mean pooling over all token embeddings produces a sentence-level representation"
- modality-aware knowledge aggregation: A mechanism to integrate information across different input modalities in a way that respects their differences. "introduces modality-aware knowledge aggregation together with wavelength-based patch embeddings"
- multi-crop strategy: An augmentation technique using multiple crops (global and local) to create diverse views for training. "The RGB set follows a multi-crop strategy with local crops"
- multi-spectral (MS) imagery: Images with multiple spectral bands beyond RGB, including non-visible wavelengths. "Multi-spectral imagery includes both visible (RGB) and non-visible bands"
- open vocabulary segmentation: Segmenting images into categories specified by arbitrary text labels without training on fixed classes. "open vocabulary segmentation and linear probing."
- projector: A lightweight neural module that maps features from one embedding space into another for alignment. "via a lightweight projector."
- self-distillation: A method where a model learns from its own predictions (teacher–student framework) using augmented views. "self-distillation approaches~\cite{anysat, dinov1} match representations across augmented views"
- Spectral Representation Distillation (SRD): Transferring multi-spectral knowledge into an RGB model by reconstructing MS representations from RGB features. "Spectral Representation Distillation transfers spectral priors from a frozen multi-spectral teacher to an RGB student via a lightweight projector."
- spectral priors: Prior knowledge about how scene materials behave across wavelengths, useful for interpreting multi-spectral data. "transfers spectral priors from a frozen multi-spectral teacher"
- Spectrally Grounded Alignment with Instruction-Augmented LLMs (SGI-LLM): Aligning spectrally enriched visual features with instruction-augmented LLM embeddings using contrastive training. "Spectrally Grounded Alignment with Instruction-Augmented LLMs bridges the distilled visual space and an expressive LLM embedding space."
- stop-gradient operator: An operation that prevents gradients from flowing through certain tensors during backpropagation. "the stop-gradient operator"
- UMAP: A non-linear dimensionality reduction technique for visualizing high-dimensional embeddings. "UMAP ($n_{\text{neighbors} = 50$, , cosine distance) visualization of feature embeddings"
- wavelength-based patch embeddings: Patch representations that incorporate information about spectral wavelengths to handle variable band inputs. "wavelength-based patch embeddings"
- zero-shot classification: Classifying images into categories without seeing labeled examples of those categories during training. "SATtxt improves zero-shot classification on average by 4.2%"
Collections
Sign up for free to add this paper to one or more collections.