Concept-Shared Instruction Aligning (CSIA)

• CSIA is a pretraining strategy that maps diverse sensor modalities (e.g., RGB, SAR, Infrared) into a shared linguistic space via causal language modeling.
• It decouples semantic alignment from task-specific detection training, utilizing a shared ViT-Large encoder, a learnable projection head, and a frozen Qwen2 language model.
• Integrated in BabelRS with LVSA for multi-scale feature fusion, CSIA enhances detection mAP and optimization stability by reducing gradient conflicts.

Concept-Shared Instruction Aligning (CSIA) is a component of BabelRS, a unified language-pivoted pretraining framework for heterogeneous multi-modal remote sensing object detection. Within this framework, CSIA aligns each sensor modality to a shared set of linguistic concepts, using language as a semantic pivot to bridge heterogeneous visual representations. It is designed for settings in which diverse sensors such as RGB, SAR, and Infrared must be brought into a common semantic space before downstream detection. In BabelRS, this alignment stage is explicitly decoupled from downstream task learning, and is implemented with a shared ViT-Large image encoder, a learnable linear projection head, and a frozen pretrained LLM, Qwen2, which serves as the semantic pivot (Li et al., 2 Mar 2026).

1. Problem setting and role within BabelRS

CSIA is formulated for heterogeneous multi-modal remote sensing object detection, where the objective is to accurately detect objects from diverse sensors. Existing approaches are described as largely adopting a late alignment paradigm, in which modality alignment and task-specific optimization are entangled during downstream fine-tuning. BabelRS addresses this by separating semantic alignment from detection training, and CSIA is the mechanism responsible for the alignment stage (Li et al., 2 Mar 2026).

The module operates over a set of sensor modalities,

$\mathcal{M}=\{m_1,\dots,m_K\},$

with a modality-specific dataset

$\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$

where $x_i^m$ is an image, $q_i^m$ is a natural-language instruction or question, and $r_i^m$ is the corresponding text response. This construction places heterogeneous visual inputs into a shared instruction-response format. A plausible implication is that the heterogeneity of the sensor domain is handled upstream, at the level of concept normalization and language supervision, rather than being deferred to detector-specific fine-tuning.

BabelRS comprises two key components: CSIA and Layerwise Visual-Semantic Annealing (LVSA). CSIA provides the alignment objective, while LVSA progressively aggregates multi-scale visual features to provide fine-grained semantic guidance. The paper states that CSIA and LVSA are fully completed before any detection loss is introduced, and that fine-tuning subsequently focuses only on detection, with no auxiliary alignment losses (Li et al., 2 Mar 2026).

2. Shared concepts, instructions, and linguistic supervision

The semantic substrate of CSIA is a shared vocabulary or ontology. The object and scene categories appearing across all sub-datasets, including Million-AID, GAIA, SARLang, RSVQA, and GeoChat, are merged into a canonical set of concepts $C_{\text{shared}}$. Synonyms are harmonized, with the example “vehicle” $\rightarrow$ “car,” and the resulting concept set contains approximately 150–200 common terms such as “bridge,” “harbor,” “ship,” “car,” “road,” and “crop field” (Li et al., 2 Mar 2026).

This concept set is crafted by hand-curating the union of category labels, question templates, and caption tokens across the 12 pretraining sub-datasets. The paper is explicit that concepts are not mined automatically; instead, all sub-dataset annotations are normalized to the unified ontology. This directly constrains the scope of CSIA: its concept sharing is annotation-driven rather than discovered through unsupervised concept induction. A common misunderstanding would be to treat CSIA as an automatic concept mining method; the reported design does not do this.

Each concept $c \in C_{\text{shared}}$ is embedded in natural-language instructions. Examples given include “Select the correct category for this image: $\langle$categories$\rangle$,” “What is the object in the boxed region? Classes: …,” and “Describe the scene in a short sentence.” Instructions $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$0 and responses $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$1 are tokenized and embedded using the LLM’s own tokenizer and token embeddings. Because $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$2 remains frozen during CSIA and only the visual encoder $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$3 and projection head $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$4 are learned, the linguistic space is fixed during pretraining. This suggests that semantic consistency is enforced by adapting the visual side to a stable language-defined target space rather than by co-adapting both modalities.

3. Mathematical formulation

For a single sample $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$5, CSIA uses a shared ViT-Large encoder $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$6 for all modalities:

$\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$7

where $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$8 is the number of visual tokens. A learnable linear projection head $\mathcal{D}^m=\{(x_i^m,q_i^m,r_i^m)\},$9 maps the ViT output dimension $x_i^m$0 to the language-model hidden dimension $x_i^m$1:

$x_i^m$2

The projected visual tokens are concatenated with the frozen text embeddings of the instruction and response:

$x_i^m$3

Here, $x_i^m$4 denotes the frozen text embeddings of $x_i^m$5, and $x_i^m$6 is a pretrained LLM, specifically Qwen2, serving as a frozen semantic pivot (Li et al., 2 Mar 2026).

The training objective is the standard causal language-modeling cross-entropy loss over the response tokens:

$x_i^m$7

In practice, optimization is performed over the entire multi-modal pretraining corpus

$x_i^m$8

through

$x_i^m$9

No additional contrastive or regularization terms are introduced in CSIA. Standard weight decay on $q_i^m$0 and $q_i^m$1 is applied via AdamW. This is a notable design constraint: the alignment objective is entirely language-modeling-based. A plausible implication is that the method avoids auxiliary cross-modal objectives in order to keep modality alignment semantically anchored to the shared instruction-response space.

4. Alignment architecture and interaction with LVSA

Although BabelRS performs CSIA jointly with LVSA, the alignment mechanism follows a specific sequence. From the ViT backbone, multi-scale feature maps

$q_i^m$2

are extracted at a selected subset of layers

$q_i^m$3

In the reproduction notes, LVSA is specified to use layers $q_i^m$4 of ViT-Large, which identifies $q_i^m$5 for the reported configuration (Li et al., 2 Mar 2026).

At training step $q_i^m$6 with anneal duration $q_i^m$7, LVSA computes

$q_i^m$8

and fuses features by

$q_i^m$9

The fused representation is then flattened into

$r_i^m$0

projected to $r_i^m$1 via $r_i^m$2, concatenated with tokenized instruction $r_i^m$3 and response prefix $r_i^m$4, and fed into $r_i^m$5. The loss $r_i^m$6 is then computed and back-propagated into $r_i^m$7 and $r_i^m$8.

The pseudocode given in the paper initializes $r_i^m$9 from ViT-Large in InternVL-2.5, initializes $C_{\text{shared}}$0 randomly, freezes $C_{\text{shared}}$1, samples batches across modalities, extracts $C_{\text{shared}}$2 from layers $C_{\text{shared}}$3, computes the annealed fusion, forms the sequence $C_{\text{shared}}$4, evaluates next-token logits with $C_{\text{shared}}$5, averages the per-sample losses, and updates $C_{\text{shared}}$6 with AdamW. The reported tensor shapes are also explicit: $C_{\text{shared}}$7, while the final sequence length is $C_{\text{shared}}$8.

The interaction between CSIA and LVSA addresses what the paper calls the granularity mismatch between high-level language representations and dense detection objectives. LVSA progressively aggregates multi-scale visual features to provide fine-grained semantic guidance. In this arrangement, CSIA defines the cross-modal semantic target, and LVSA modulates the visual features that are exposed to that target.

5. Training recipe and transfer to detection

The pretraining configuration uses 8$C_{\text{shared}}$9 NVIDIA A40 (48 GB) GPUs, global batch 128, AdamW, learning rate $\rightarrow$0, and weight decay $\rightarrow$1. Fine-tuning for detection uses the same GPUs, per-GPU batch 4, learning rate $\rightarrow$2, and weight decay $\rightarrow$3. Automatic mixed precision is used. For optimization scheduling, the reproduction notes specify AdamW with linear warmup of 500 steps plus cosine decay (Li et al., 2 Mar 2026).

CSIA pretraining is reported to run for approximately $\rightarrow$4k steps, with LVSA anneal duration $\rightarrow$5k steps, described as balanced by Fig. 7. After CSIA, nothing is frozen: $\rightarrow$6 and $\rightarrow$7 are loaded into the unified detection model, modality-specific heads are attached, and the model is fine-tuned until convergence, approximately 12–15 epochs over SOI-Det. The sequencing is explicit: CSIA and LVSA are fully completed before any detection loss is introduced, and fine-tuning focuses only on detection.

This training order is central to the method’s interpretation. The paper summarizes CSIA as reducing cross-modal gradient conflict by decoupling semantic alignment via language modeling from detection training. A plausible implication is that optimization stability is improved because the shared representation is already shaped by cross-modal semantic supervision before detector-specific gradients are applied.

6. Empirical results, ablations, and interpretive boundaries

The main SOI-Det results reported in Table 1 compare BabelRS with SM3Det, identified as a late-alignment state of the art. SM3Det achieves $\rightarrow$8 and $\rightarrow$9, whereas BabelRS achieves $c \in C_{\text{shared}}$0 and $c \in C_{\text{shared}}$1, corresponding to gains of $c \in C_{\text{shared}}$2 and $c \in C_{\text{shared}}$3, respectively. On individual modalities or benchmarks, the reported gains are $c \in C_{\text{shared}}$4 AP on SARDet-100K, $c \in C_{\text{shared}}$5 AP on DroneVehicle IR, and $c \in C_{\text{shared}}$6 AP on DOTA RGB (Li et al., 2 Mar 2026).

The comparison with other pretraining strategies in Table 2 reports CLIP at $c \in C_{\text{shared}}$7 and BabelRS at $c \in C_{\text{shared}}$8, a gain of $c \in C_{\text{shared}}$9. MAE, BEiT, and ScaleMAE are reported in the $\langle$0–$\langle$1 range, while BabelRS reaches $\langle$2. These comparisons position CSIA-based language-pivoted pretraining as materially different from pretraining strategies that do not use the same shared instruction-response alignment regime.

Optimization stability is documented in Fig. 3 and Table 4. Under automatic mixed precision, several late-alignment methods diverge with NaN, while BabelRS converges stably, achieving AMP $\langle$3, $\langle$4, and $\langle$5. The paper therefore links the decoupled pretraining paradigm not only to final accuracy but also to stability under mixed-precision optimization.

The ablation on feature-merge strategies in Table 6 further clarifies the role of LVSA in the CSIA pipeline. The reported variants are: concat with $\langle$6, sum with $\langle$7, per-layer heads with $\langle$8, and LVSA shared head with $\langle$9, which is the best setting. Figure 1 reports that performance dips if $\rangle$0 or $\rangle$1k, and peaks around $\rangle$2k and approximately $\rangle$3k steps of pretraining.

Several boundaries of interpretation are explicit in the design. CSIA does not introduce additional contrastive or regularization terms; it does not mine concepts automatically; and it does not continue to apply auxiliary alignment losses during detection fine-tuning. Accordingly, CSIA should not be conflated with contrastive vision-language pretraining, automatic ontology induction, or joint end-to-end alignment-and-detection optimization. Within BabelRS, it is a pretraining-stage alignment mechanism that maps heterogeneous images into a shared linguistic space through standard causal language modeling, and its reported effectiveness depends on that decoupled formulation (Li et al., 2 Mar 2026).