GLAC: Guiding LSN via Adapted CLIP

• The paper introduces GLAC, which leverages an adapted CLIP to provide semantic language constraints for enhanced data efficiency and robust performance in pansharpening, few-shot video action recognition, and semantic segmentation.
• It details a two-phase adaptation process that fine-tunes both visual and textual encoders for domain-specific inputs, employing cross-attention and distribution matching to bridge vision and language modalities.
• Empirical results demonstrate GLAC's effectiveness through measurable improvements in spectral distortion, mIoU, and overall classification metrics across diverse architectures and datasets.

Guiding LSN with Adapted CLIP (GLAC) encompasses a family of modules and training protocols for augmenting Ladder Side Networks (LSN) with semantic supervision distilled from adapted CLIP models. GLAC is designed to inject robust language-grounded priors and transfer powerful vision–language features into lightweight or domain-specific networks, thereby improving generalization, alignment, and sample efficiency across tasks such as pansharpening, few-shot video action recognition, and lightweight semantic segmentation.

1. Conceptual Foundation and Motivation

GLAC formalizes multi-modal supervision whereby a pre-trained, lightly adapted CLIP backbone acts as a "teacher" to regularize the LSN-based backbone through semantic constraints or distribution matching. The theoretical premise is that deep visual backbones trained on limited, domain-specific data—especially under label scarcity or modality mismatch—benefit significantly from externally mined semantic cues. The adapted CLIP provides structured guidance, leveraging textual prompts and multimodal representation alignment both at pixel and sequence levels.

In pansharpening, GLAC (as codified in CLIPPan (Jian et al., 14 Nov 2025)) enables unsupervised training at full resolution by aligning fusion processes to protocol-text prompts. In semantic segmentation, GLAC bridges feature spaces between lightweight visual backbones and CLIP text embeddings for more effective class token alignment (Jin et al., 2023). For few-shot action recognition, GLAC regularizes the estimation of $\alpha$-distance–based inter-frame dependencies (Long et al., 12 Dec 2025).

2. Adaptation of CLIP for Downstream Domains

The first phase of GLAC entails fine-tuning CLIP to represent non-natural image inputs (e.g., multispectral, panchromatic, or video frames) and to understand the target fusion or matching process.

• Visual Encoder Adaptation: Input stems are replaced with domain-specific convolutions (e.g., $C_{MS} \to 3$ maps for multispectral bands), followed by bottleneck CLIP-Adapters post-encoder blocks. In video, a lightweight Multi-Head Self-Attention (MHSA) adapter is appended to model temporal dependencies.
• Text Encoder Adaptation: Bottleneck adapters are placed after the transformer layers. Prompts are fixed and protocol-aligned; tokenization remains standard.
• Fine-Tuning Protocol: In CLIPPan, full-resolution remote sensing data is organized into triplets for supervised adaptation; inter-modal and intra-modal contrastive losses are used, along with fusion-aware regression. In TS-FSAR, frames are encoded in parallel by CLIP and LSN branches, with only adapters and side networks updated (Jian et al., 14 Nov 2025, Long et al., 12 Dec 2025).

3. Fusion and Constraint Mechanisms in GLAC

GLAC's operational core involves pushing backbone outputs to align with protocol-aligned semantics or match teacher distributions.

a) Semantic Language Constraint Losses (SLCL)

• Pansharpening:

$$L_d = 1 - \frac{1}{2}[\langle\Delta V^I_{MS},\Delta V^T_{MS}\rangle + \langle\Delta V^I_{PAN},\Delta V^T_{PAN}\rangle]$$

where difference vectors $\Delta V$ are computed as outputs minus reference features (visual and text), and prompt-based fusion transitions serve as semantic anchors (Jian et al., 14 Nov 2025).

b) Distribution Matching (KL/CE) in Video

• Action Recognition:

For each video, the LSN produces an $\alpha$-distance–based representation and softmax prediction $p$; CLIP+MHSA adapter produces $q$. The GLAC objective regularizes $p$ to match $q$ via

$$\mathcal{L}_{GLAC} = \sum_i p_i \log(p_i / q_i) - \sum_i y_i (\log p_i + \log q_i)$$

where $y_i$ is the ground truth one-hot vector (Long et al., 12 Dec 2025).

c) Bidirectional Fusion (Conv-Former) in Segmentation

• Lightweight Segmentation:

GLAC employs parallel CNN and Transformer branches, bridged by dual cross-attention modules (visual-to-text and text-to-visual), enabling class token reweighting and pixel-wise textual context (Jin et al., 2023). The fusion optimizes feature alignment in a shared embedding space.

4. Integrated Training Objectives and Gradient Structure

GLAC domains exhibit composite losses:

• Pansharpening:

$L_{S2} = L_{unsup} + \alpha\cdot L_d$, where $L_{unsup}$ includes spectral, spatial, QNR, and pseudo-supervision losses, and $\alpha\sim 0.1-0.5$ (Jian et al., 14 Nov 2025).

• Semantic Segmentation:

Pixelwise cross-entropy on the score map $S$ suffices due to the implicit alignment enforced by cross-attention bridges (Jin et al., 2023).

• Few-Shot Video Classification:

$$\mathcal{L} = \mathcal{L}_{LSN} + \lambda_1 \mathcal{L}_{TS-DCM} + \lambda_2 (\mathcal{L}_{GLAC-KL} + \mathcal{L}_{GLAC-CE}),$$

with gradient flow restricted to non-CLIP parameters (Long et al., 12 Dec 2025).

The CLIP backbone and text encoder are universally frozen; adapters, fusion modules, and side networks are trainable. Optimizer and learning rates are task-specific (e.g., Adam, SGD, AdamW).

5. Algorithmic Workflow and Implementation Characteristics

A unified operational outline for GLAC modules comprises:

Phase/Branch	Input	Trainable Components	GLAC-Specific Output
CLIP Adaptation	RGB/MS/video triplets	Adapters, fusion heads	Semantic and protocol-aligned features
LSN/Backbone	Task domain data	LSN, decoder, fusion modules	Prediction and features for regularization
CLIP Teacher Branch	Same as above	MHSA adapter (video)	Reference distribution or features

• Batch sizes, augmentation, and normalization are empirically tuned per domain; e.g., CLIPPan uses batch size $32$ and histogram-matching for MS bands (Jian et al., 14 Nov 2025), Conv-Former adopts $L=6$ blocks for ADE20K (Jin et al., 2023).
• GLAC modules are plug-and-play, model-agnostic, and compatible with a broad array of visual backbones including MobileNetV2, Xception, EfficientFormer, ResNet, ViT-B, Swin-T (Jin et al., 2023).
• Pseudocode and workflow details explicitly delineate which parameters are frozen (CLIP encoders) and which receive gradients (adapters, fusion modules, side networks).

6. Empirical Performance and Ablation Analysis

GLAC consistently delivers state-of-the-art results, verified by systematic experiments:

• Unsupervised Pansharpening (CLIPPan):

QNR increased by $\sim0.008$–$0.015$ with up to $66\%$ reduction in spectral distortion on WorldView-3, and improved MPSNR, ERGAS, and SAM ($\sim$1.8 dB, 2.5, 1.2° gains) (Jian et al., 14 Nov 2025).

• Few-Shot Action Recognition (TS-FSAR):

GLAC regularization yields superior $\alpha$-distance correlation estimation under limited supervision and improved benchmark performance (Long et al., 12 Dec 2025).

• Semantic Segmentation:

On ADE20K (validation): GLAC+FPN (MobileNetV2) achieves 32.2 mIoU (vs baseline 25.1; DenseCLIP 22.3), Xception 40.3 mIoU, EfficientFormer 45.9 mIoU; improvements of 3.8–9.9 points over prior methods (Jin et al., 2023). Ablations confirm learned cross-attention is essential, with inner-product fusion resulting in 3–5 mIoU drops, and $L=6$ blocks optimal.

7. Domain Significance, Transferability, and Limitations

GLAC paradigms unify vision–language signals for dense prediction, fusion, and set matching tasks. A plausible implication is extensibility to other structured modalities (e.g., medical imaging, multi-sensor fusion), provided CLIP adaptation is sufficient to mitigate domain bias. The plug-and-play design allows rapid transferability to new architectures and datasets. Limitations include reliance on CLIP's representational coverage and the necessity for both semantic prompts and well-designed adapters for each new domain.

GLAC robustly links lightweight and task-specific networks to language priors and semantic regularization, setting new standards in unsupervised, sample-efficient, and modality-bridged learning (Jian et al., 14 Nov 2025, Long et al., 12 Dec 2025, Jin et al., 2023).