BabelRS: Language-Pivoted Remote Sensing

• BabelRS is a language-pivoted framework that unifies RGB, SAR, infrared, and other sensor modalities without requiring spatially paired images.
• It employs a two-stage approach by decoupling early semantic alignment from downstream detection, which enhances training stability and performance.
• The framework integrates Concept-Shared Instruction Aligning and Layerwise Visual-Semantic Annealing to mitigate gradient conflicts and improve multi-modal detection.

BabelRS is a language-pivoted pretraining framework for heterogeneous multi-modal remote sensing object detection. It is designed to unify RGB, SAR, infrared, and other sensor modalities under a single detector without requiring spatially paired multi-sensor images. Its central design choice is to decouple modality alignment from downstream detection: a pretraining stage aligns heterogeneous visual representations through language supervision, and a fine-tuning stage performs detection without auxiliary alignment losses. In the cited formulation, this decoupling is intended to address instability associated with late alignment, including slow or unstable convergence, gradient explosions, NaN failures under AMP, and poorer generalization, especially for SAR and IR (Li et al., 2 Mar 2026).

1. Nomenclature and scope

In the arXiv literature, BabelRS denotes the remote-sensing framework introduced in "Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining" (Li et al., 2 Mar 2026). The term should be distinguished from several unrelated or only superficially similar uses of "Babel" and "RS" in other papers.

Term	Meaning in the cited paper	Relation to BabelRS
BabelRS	Language-pivoted pretraining for heterogeneous multi-modal remote sensing detection	Direct topic (Li et al., 2 Mar 2026)
"Babel Tower"	Hypothesis about multilingual capability emergence in multilingual code LLMs	Distinct concept (Chen et al., 2024)
Babel-17	Programming language for purely functional structured programming	Unrelated; “BabelRS” does not appear as its name (Obua, 2010)
KP RS-ABE	Key-policy revocable-storage attribute-based encryption	Unrelated cryptographic use of “RS” (Giacon et al., 2016)

A common source of confusion is the proximity of names. The "Babel Tower" paper studies multilingual code LLMs and proposes a dynamic theory in which multiple languages initially share a single knowledge system dominated by a primary language and then gradually shift toward language-specific systems (Chen et al., 2024). By contrast, BabelRS addresses heterogeneous remote sensing detection. Likewise, Babel-17 is a dynamically typed language with a formal syntax and semantics specification for purely functional structured programming, not a remote sensing framework (Obua, 2010). The cryptographic paper on key-policy RS-ABE concerns revocable-storage attribute-based encryption and is unrelated to language-pivoted vision pretraining (Giacon et al., 2016).

2. Problem setting: late alignment in heterogeneous remote sensing detection

BabelRS is motivated by the dominant late alignment paradigm in unified multi-modal remote sensing detection. Under that paradigm, models are initialized from generic backbones and are then required during downstream fine-tuning to align heterogeneous modality feature spaces while simultaneously learning the detection task. The cited argument is that this tight coupling is problematic because the modalities are physically different: RGB depends on reflectance and texture, SAR depends on scattering, and infrared depends on thermal emission (Li et al., 2 Mar 2026).

The appendix formalizes the late-alignment setting as a joint objective:

$$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \left[ \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y) + \lambda \mathcal{L}_{\text{align}}(E_\theta(x), m) \right].$$

The stated criticism is twofold. First, the same shared backbone must satisfy detection loss and alignment or regularization loss at the same time, creating a difficult loss landscape with interfering gradients from different modalities. Second, because alignment occurs only during the detection stage, fine-tuning must repair modality mismatch while also learning object localization and classification. The appendix characterizes this regime by gradient conflict and condition-number explosion when modality distributions are far apart (Li et al., 2 Mar 2026).

Within this formulation, BabelRS redefines the task as an early semantic alignment problem. A plausible implication is that the framework treats modality mismatch as a representation-learning issue to be solved upstream, rather than as a side constraint inside the detector optimization itself.

3. Two-stage learning and language as the semantic pivot

BabelRS separates training into a pretraining stage and a fine-tuning stage. In the pretraining stage, modality alignment is learned through language supervision only; in the fine-tuning stage, detection is optimized without auxiliary alignment losses. The appendix writes this decomposition as:

$$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$

for pretraining, and

$$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$

for fine-tuning (Li et al., 2 Mar 2026).

The semantic pivot is a pretrained LLM, denoted $\Phi$, used as a shared semantic anchor. The framework’s premise is that semantically equivalent scenes from different modalities can be described using the same linguistic concepts. The example given is that an RGB image of a car and a SAR image of a car both map to the linguistic concept “car.” Instead of forcing modalities to match each other directly in feature space, BabelRS makes them match through the same language space (Li et al., 2 Mar 2026).

This pivot is operational rather than metaphorical. The vision encoder is trained so that visual tokens from distinct modalities support the same instruction-following response distribution under the pretrained LLM. This suggests that semantic agreement is imposed at the level of conditional generation, not via an explicit pairwise metric between modality embeddings.

4. Concept-Shared Instruction Aligning

The first core component of BabelRS is Concept-Shared Instruction Aligning (CSIA). It aligns heterogeneous modalities into a shared linguistic concept space using instruction-following supervision. The paper assumes a set of modalities

$\mathcal{M} = \{m_1, m_2, \dots, m_K\}$

and disjoint modality-specific datasets

$$\mathcal{D} = \{\mathcal{D}^{m_1}, \mathcal{D}^{m_2}, \dots, \mathcal{D}^{m_K}\},$$

with each sample in modality $m_k$ written as

$$\mathcal{D}^{m_k} = \{(x_i^{m_k}, q_i^{m_k}, r_i^{m_k})\}.$$

Here, $x_i^{m_k}$ is an image, $q_i^{m_k}$ is an instruction or question, and $$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$0 is a response or answer. The cited tasks include VQA, captioning, visual grounding, and classification-style instructions (Li et al., 2 Mar 2026).

A modality-shared vision encoder $$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$1 extracts visual tokens, which are then projected into the input embedding space of the pretrained LLM $$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$2. Training uses a causal language modeling objective:

$$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$3

In the implementation described in the paper, visual tokens are concatenated with textual tokens, and the language modeling loss is applied only to the response tokens (Li et al., 2 Mar 2026).

The stated reason this aligns modalities is that different modalities must produce embeddings that lead the LLM to generate the same semantic response. The appendix summarizes this as implicit semantic equivalence via conditional generation:

$$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$4

for semantically equivalent observations from modalities $$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$5 and $$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$6 (Li et al., 2 Mar 2026).

Three advantages are emphasized. CSIA requires no paired multi-sensor images; it does not force direct feature collapse because alignment is achieved through conditional generation rather than explicit embedding matching; and it performs stable pre-alignment before detection fine-tuning. In the paper’s terminology, this is the mechanism by which language serves as a bridge across heterogeneous visual representations (Li et al., 2 Mar 2026).

5. Layerwise Visual-Semantic Annealing and detector adaptation

The second core component is Layerwise Visual-Semantic Annealing (LVSA). Its purpose is to address the mismatch between high-level language supervision and dense detection objectives. The cited observation is that aligning only the final ViT layer, as in many vision-LLMs, is insufficient for detection because detection requires multiple scales, fine-grained localization, and intermediate spatial detail. At the same time, naively fusing all intermediate layers can disturb pretrained feature distributions and create instability (Li et al., 2 Mar 2026).

Let the ViT encoder produce features from $$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$7 blocks:

$$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$8

A subset of layers is selected for fusion:

$$\min_{\theta} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,q,r)\sim P_m} \mathcal{L}_{\text{lang}}(E_\theta(x), q, r)$$9

Fusion is controlled by the annealing coefficient

$$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$0

where $$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$1 is the training step and $$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$2 is the annealing duration. The fused representation is

$$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$3

Early in training, the representation is dominated by $$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$4, preserving the pretrained semantic space; later, intermediate multi-scale features are progressively incorporated (Li et al., 2 Mar 2026).

For LVSA, the paper uses the 3rd, 9th, 18th, and last ViT layers as the multi-scale feature set. BabelRS is initialized from InternVL-2.5 1B, which includes a ViT-Large visual backbone and a Qwen2 LLM. After pretraining, fine-tuning uses a shared backbone, modality-specific detection heads, and random sampling across datasets. No extra alignment losses are used during fine-tuning, and the total detection objective is written simply as

$$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$5

The architecture therefore treats semantic alignment as a precondition for detection rather than a concurrent auxiliary task (Li et al., 2 Mar 2026).

6. Evaluation, metrics, and empirical profile

BabelRS is pretrained on a curated multi-source remote sensing vision-language corpus including Million-AID, LevirCC, VHM, RSVQA, FIT_RS, GAIA, SARLang, MMRS-1M, GeoChat, DIOR-RSVG, VRSBench, and Mini-InternVL. These sources cover VQA, captioning, visual grounding, and classification, and the paper states that naming conventions are standardized so that concepts such as “bridge,” “harbor,” and “ship” map consistently. Fine-tuning is evaluated on SOI-Det, which combines SARDet-100K for SAR, DOTA-v1.0 for optical, and DroneVehicle for infrared (Li et al., 2 Mar 2026).

The reported evaluation metrics are AP@50, mAP averaged over IoU thresholds $$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$6 to $$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$7, and H-mAP, introduced to balance performance across modalities. The modality-specific mean average precision is defined as

$$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$8

and the harmonic modality score as

$$\min_{\theta,\psi} \sum_{m \in \mathcal{M}} \mathbb{E}_{(x,y)\sim P_m} \mathcal{L}_{\text{det}}(D_\psi(E_\theta(x)), y)$$9

The stated motivation is that ordinary global mAP can be biased toward modalities with more categories, especially RGB, whereas the harmonic mean penalizes failure on any modality (Li et al., 2 Mar 2026).

In Table 1, BabelRS reports the best overall SOI-Det performance with AP@50: 81.32, mAP: 51.57, and H-mAP: 53.02. Against the strongest fine-tuning-stage baseline cited there, SM3Det reports 80.68 / 50.20 / 51.31, while BabelRS reports 81.32 / 51.57 / 53.02. Per modality, BabelRS is reported to improve notably on SARDet-100K: 91.70 / 63.30 and DroneVehicle: 79.63 / 51.32, while DOTA remains competitive (Li et al., 2 Mar 2026).

Table 2 compares BabelRS with pretraining methods including CLIP, MAE, BEiT, BEiTv2, DINOv2, RemoteCLIP, SatMAE, and ScaleMAE. The reported BabelRS result is again 81.32 AP@50, 51.57 mAP, and 53.02 H-mAP. The paper interprets the gap versus CLIP-style methods as evidence that global semantic alignment alone is insufficient for dense detection, and that LVSA helps bridge that gap (Li et al., 2 Mar 2026).

Optimization stability is a prominent empirical theme. The paper reports that late-alignment methods exhibit slower convergence, unstable loss trajectories, and occasional divergence, while BabelRS starts from a lower initial loss and converges smoothly. Under Automatic Mixed Precision, MAE, BEiTv2, DINOv2, and ScaleMAE are reported as NaN, whereas BabelRS remains stable and achieves AP@50 79.13, mAP 50.17, and H-mAP 51.52 (Li et al., 2 Mar 2026). This supports the paper’s claim that decoupling alignment from detection improves numerical robustness.

The ablations are consistent with that interpretation. Among feature merge strategies, the proposed LVSA merge gives the best result, with mAP: 51.57 and H-mAP: 53.02. For the annealing schedule, performance is reported to dip around 6k steps, then improve steadily, and saturate around 20k steps. As the annealing duration $\Phi$0 increases from $\Phi$1 to 6k, performance improves; peak performance is reported at $\Phi$2, and larger $\Phi$3 gives no additional gain (Li et al., 2 Mar 2026). This suggests that controlled integration of intermediate visual layers is not merely an implementation detail but part of the framework’s central optimization logic.