LVSA: Layerwise Visual-Semantic Annealing

• The paper introduces LVSA, a progressive pretraining mechanism that fuses intermediate visual features with language-aligned representations to enhance dense detection.
• LVSA addresses the granularity mismatch by gradually integrating multi-scale visual details, leading to improved spatial precision and training stability.
• Empirical results show that LVSA boosts performance by over 2 mAP points compared to naïve fusion methods on heterogeneous remote sensing benchmarks.

Layerwise Visual-Semantic Annealing (LVSA) is a pretraining mechanism introduced in BabelRS for heterogeneous multi-modal remote sensing object detection, where it serves to make language-pivoted pretraining usable for dense detection by addressing the mismatch between high-level language alignment and the spatially resolved, multi-scale representations required for localization across RGB, SAR, and infrared imagery (Li et al., 2 Mar 2026). In BabelRS, LVSA progressively aggregates intermediate visual features into the language-aligned representation during pretraining, so that the encoder exposed to downstream fine-tuning preserves both semantically normalized cross-modal structure and the spatial detail needed by dense detectors. More broadly, later work and related analyses connect LVSA-like behavior to a general transition across network depth from fine visual detail toward semantically aligned representations, but the explicit term and mechanism originate in BabelRS (Li et al., 2 Mar 2026).

1. Origin, definition, and motivating mismatch

LVSA was introduced as one of the two key components of BabelRS, alongside Concept-Shared Instruction Aligning (CSIA), in the paper "Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining" (Li et al., 2 Mar 2026). Its stated purpose is to resolve a core granularity mismatch: language-aligned encoders typically align semantics at a single final layer, whereas heterogeneous remote sensing detection requires spatially resolved, multi-scale features. The mechanism therefore progressively and gently pulls intermediate visual features into the language-aligned space during pretraining.

The motivating problem is twofold. First, remote sensing modalities are heterogeneous because their physics differ, including optical reflectance, SAR scattering, and thermal emission, so their raw feature distributions and optimal representational subspaces diverge. Second, traditional late alignment optimizes modality alignment and detection objectives jointly during downstream fine-tuning. The paper states that this coupling causes gradient conflicts and ill-conditioned loss geometry: modality-specific detection gradients can be highly misaligned, and alignment losses impose additional sharp curvature along alignment directions. As model capacity grows, including ViT-Large and MoE, these conflicts frequently manifest as unstable training, gradient explosions, and NaNs (Li et al., 2 Mar 2026).

Within this framing, LVSA addresses the granularity gap rather than the cross-modal semantic gap alone. CSIA maps all modalities to a shared linguistic concept space through an instruction-following LLM, while LVSA gradually injects multi-scale visual features into that language-aligned representation. This suggests that LVSA is best understood not as a standalone fusion block, but as a curriculum over feature granularity inside language-pivoted pretraining.

2. Role inside BabelRS and system interfaces

BabelRS explicitly decouples modality alignment from downstream task learning. In this design, semantic alignment is performed during pretraining through language, and object detection is optimized only during fine-tuning. LVSA sits in the pretraining stage rather than in the detector, and its output is the fused visual feature presented to the LLM during CSIA pretraining (Li et al., 2 Mar 2026).

Its inputs are a set of backbone feature maps at different depths and an annealing schedule controlling how much intermediate features contribute over training steps. Its output is a fused, gradually multi-scale visual feature that is fed to the LLM via a shared projector. The paper states that this fused feature produces more spatially sensitive visual tokens for the LLM, improving dense detection transfer.

The interface with CSIA is direct. CSIA provides the alignment objective through causal language modeling in an instruction-following setup, while LVSA controls the feature content exposed to the LLM over time. The interface with the detector is indirect: there is no special detector-side LVSA module. After pretraining, fine-tuning attaches standard detection heads to the shared backbone and optimizes only detection losses. The benefit of LVSA is therefore carried by the pretrained encoder’s multi-scale, semantically normalized features rather than by a persistent architectural module at detection time (Li et al., 2 Mar 2026).

This placement is central to the method’s stated stability advantages. Because modality alignment is completed before downstream detection optimization, the detector does not need to solve semantic normalization and localization simultaneously. A plausible implication is that LVSA’s contribution depends not only on its fusion rule, but also on its location before detector fine-tuning.

3. Architecture, fusion rule, and annealing schedule

BabelRS initializes from InternVL-2.5 1B, specifically a ViT-Large variant paired with a Qwen2 LLM. For LVSA, the multi-scale feature set uses the 3rd, 9th, 18th, and last ViT layers, following guidance from recent multi-layer perception works (Li et al., 2 Mar 2026). Let the backbone feature maps be

$$V=\{F_l\}_{l=1\ldots L}, \qquad F_l \in \mathbb{R}^{H \times W \times C},$$

with a selected subset $S \subset \{1,\ldots,L\}$ such that $L \in S$.

The fusion is deliberately simple. The paper states that it uses no attention and no deformable sampling, choosing stability over complexity. The annealing coefficient increases linearly:

$\alpha(t)=\min(t/\tau, 1),$

where $t$ is the training step and $\tau$ is the annealing duration. The fused feature is defined as a convex interpolation between the final-layer feature and the mean of the selected layers:

$$\tilde{F} = (1-\alpha(t))F_L + \alpha(t)\left(\frac{1}{|S|}\sum_{l\in S}F_l\right).$$

Early in pretraining, when $t$ is small, $\tilde{F} \approx F_L$, preserving the original language-aligned distribution. As $t$ approaches $S \subset \{1,\ldots,L\}$0, intermediate layers contribute more strongly, adding finer spatial detail and scale diversity. The paper characterizes this as an annealing process that avoids destabilizing the pretrained backbone while achieving multi-scale guidance (Li et al., 2 Mar 2026).

The schedule study identifies $S \subset \{1,\ldots,L\}$1k steps as the best balance between preserving the pretrained distribution and adding spatial detail. Performance improved with longer $S \subset \{1,\ldots,L\}$2 up to 6k steps, and pretraining gains saturated around 20k steps; very long pretraining mildly overfits. Thus $S \subset \{1,\ldots,L\}$3 increases linearly from 0 to 1 across the first 6k steps and remains at 1 thereafter (Li et al., 2 Mar 2026).

A defining negative statement is also part of the mechanism’s identity: there is no cross-modal attention term and no InfoNCE in LVSA. Guidance is provided implicitly by the CSIA instruction-following loss on the fused feature tokens.

4. Training dynamics and optimization pathway

CSIA supplies the supervision under which LVSA operates. For any modality $S \subset \{1,\ldots,L\}$4, such as RGB, SAR, or IR, a shared vision encoder $S \subset \{1,\ldots,L\}$5 produces visual tokens that are concatenated with textual instruction tokens and fed into the LLM $S \subset \{1,\ldots,L\}$6. Given image $S \subset \{1,\ldots,L\}$7, instruction $S \subset \{1,\ldots,L\}$8, and response $S \subset \{1,\ldots,L\}$9, the alignment objective is standard causal language modeling:

$L \in S$0

In LVSA-equipped pretraining, the visual input to $L \in S$1 is the fused feature $L \in S$2 rather than only the final backbone layer (Li et al., 2 Mar 2026). This is the precise point at which linguistic supervision reaches intermediate visual representations.

The pretraining configuration reported for BabelRS uses InternVL-2.5 1B with Qwen2, selected ViT layers 3rd, 9th, 18th, and last, hardware of 8 × NVIDIA A40 with 48 GB, a global batch size of 128, AdamW, and a learning rate of $L \in S$3 (Li et al., 2 Mar 2026). The instruction data mixes VQA, grounding, captioning, and classification formats, including RSVQA, SARLang, VRSBench, and DIOR-RSVG. Prompts include category lists, referential boxes, and counting. The paper states that this breadth encourages semantically rich, spatially aware features.

During fine-tuning for detection, the model uses a shared backbone, namely the LVSA+CSIA-pretrained encoder, together with modality-specific detection heads. Training randomly samples across SOI-Det datasets. No auxiliary alignment objectives are used at this stage; the total loss is the sum of task-specific detection losses,

$L \in S$4

The fine-tuning optimizer remains AdamW with learning rate $L \in S$5, weight decay $L \in S$6, and per-GPU batch size $L \in S$7 (Li et al., 2 Mar 2026). This training division makes LVSA a pretraining-time mechanism whose effect persists through the encoder rather than through explicit downstream supervision.

5. Empirical evidence, ablations, and stability

The principal downstream benchmark is SOI-Det, which comprises SARDet-100K for SAR, DOTA-v1.0 for optical aerial imagery, and DroneVehicle for infrared data. Reported metrics are AP@50, mAP $L \in S$8, and Harmonic Modality mAP (H-mAP), the last intended to penalize imbalanced cross-modality performance (Li et al., 2 Mar 2026).

BabelRS, which includes both LVSA and CSIA, achieves overall AP@50 of 81.32, mAP of 51.57, and H-mAP of 53.02. By modality, the reported scores are 91.70 AP@50 and 63.30 mAP on SARDet-100K, 77.73 AP@50 and 46.96 mAP on DOTA, and 79.63 AP@50 and 51.32 mAP on DroneVehicle. Compared to late-alignment state-of-the-art methods, including SM3Det with mAP 50.20 and H-mAP approximately 51.31, BabelRS is reported to achieve stronger cross-modal balance (Li et al., 2 Mar 2026).

The most direct evidence for LVSA comes from the merge-strategy ablation. The baseline that passes only the final layer to the projector reaches mAP 49.33 and H-mAP 50.67. Naïve concat, sum, and per-layer projectors reach approximately 50.25 to 50.31 mAP. LVSA-based progressive merge reaches mAP 51.57 and H-mAP 53.02, yielding +2.24 mAP and +2.35 H-mAP over the final-layer-only baseline (Li et al., 2 Mar 2026). This establishes that the gain is not merely from using multiple layers, but from using a progressive fusion schedule.

The paper also attributes stability improvements to the early alignment strategy combined with LVSA’s gentle multi-scale infusion. Loss curves show smoother convergence and lower initial loss for BabelRS than for late alignment. Under Automatic Mixed Precision, several late-alignment baselines diverge with NaNs, whereas BabelRS remains stable and reports AP@50 79.13, mAP 50.17, and H-mAP 51.52 (Li et al., 2 Mar 2026). This result is consistent with the motivation that gradual exposure of intermediate features improves conditioning.

Qualitative analysis is described in restrained terms. The paper does not include attention maps, but it reports that progressive multi-scale integration via LVSA improves spatial precision and cross-modal consistency, particularly in SAR and IR, where single-layer language alignment underperforms. It further states that abrupt or naïve multi-layer mixing harms stability and yields limited benefits, whereas gradual fusion leads to cleaner localization and better detection transfer (Li et al., 2 Mar 2026).

6. Relation to adjacent multi-layer VLM research

LVSA belongs to a broader line of work examining how different visual layers encode distinct mixtures of detail and semantics, but related papers differ substantially in whether they actually instantiate annealing. "Rethinking Visual Layer Selection in Multimodal LLMs" does not introduce, define, or reference LVSA by name; instead, it studies CLIP-ViT layer behavior through Layer-wise Representation Similarity and shows that shallow and middle layers often outperform deep layers on counting, positioning, and localization, while deep layers are essential for OCR (Chen et al., 30 Apr 2025). Its simple fusion of shallow, middle, and deep layers improves performance on 9 out of 10 datasets, which supports the intuition behind LVSA, but the paper explicitly does not employ a curriculum or layer-emphasis annealing schedule.

A different but related direction appears in "Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-LLMs," which introduces an instruction-guided vision aggregator rather than LVSA itself (Li et al., 2024). There, visual layers are grouped into low, low-to-mid, mid-to-high, and high bands, and task instructions dynamically determine fusion weights. The paper states that no explicit annealing schedule is used; instead, the allocator dynamically shifts emphasis toward deeper groups for semantic-rich tasks and toward shallower or mid-level groups for fine-grained perception. This suggests an instruction-conditioned analogue to LVSA’s layerwise emphasis, but it remains inference- or task-conditioned rather than time-conditioned pretraining.

"The Hidden Evolution of Disguised Visual Context inside the VLM" uses the term LVSA in a broader interpretive sense, describing the progressive reshaping of visual tokens within an LLM from raw, nonlinguistic inputs into language-compatible representations as depth increases (Suharitdamrong et al., 18 Jun 2026). In that controlled comparison, in-context injection shows smooth token-level CKA continuity, a frequency trajectory in which high-frequency content rises in early and middle layers and drops in final layers, and deeper-layer convergence of visual tokens toward language space. The paper presents these as hallmarks of a layerwise annealing process inside the LLM rather than as the specific pretraining fusion rule introduced in BabelRS.

"Reading Images Like Texts: Sequential Image Understanding in Vision-LLMs" likewise does not use the term LVSA, but it documents a two-stage progression from attribute recognition in shallow-to-middle layers to semantic disambiguation in middle-to-deep layers (Li et al., 23 Sep 2025). That analysis is based on multimodal logit-lens token maps, ratios of attribute and representative words, and spatial reasoning geometry under 2D RoPE. The connection to LVSA is conceptual: both describe a depth-dependent transformation from local visual evidence to semantically organized representations, but the paper does not propose BabelRS’s annealed fusion mechanism.

Taken together, these papers indicate that LVSA sits at the intersection of three themes: multi-layer visual complementarity, task dependence of layer utility, and progressive semantic normalization. What distinguishes LVSA proper is that it operationalizes these themes as a pretraining-time curriculum over feature granularity for heterogeneous remote sensing detection (Li et al., 2 Mar 2026).

7. Limitations, misconceptions, and future directions

The main limitation stated for LVSA in BabelRS is its dependence on the quality and coverage of the concept vocabulary and instruction corpus. The paper notes that very fine-grained or physics-specific SAR and IR phenomena might be under-described by natural language. It also focuses on static image detection rather than temporal dependencies, and it acknowledges the computational overhead of running a VLM during pretraining, though this is mitigated by using InternVL-2.5 1B and moderate pretraining duration (Li et al., 2 Mar 2026).

A common misconception is to equate any multi-layer visual fusion with LVSA. The reported ablations argue against that equivalence: naïve concat, sum, and per-layer projectors improve only marginally over the final-layer baseline, whereas LVSA’s progressive merge yields the largest gains. Another misconception is to treat LVSA as a detector-side architectural add-on. In BabelRS, it is explicitly not used during downstream detection; fine-tuning relies on conventional detection losses and standard detection heads, while LVSA’s contribution persists through the pretrained encoder (Li et al., 2 Mar 2026).

The paper identifies several future directions: physics-aware or expert-augmented prompts for subtle sensor-specific semantics, hybrid designs with lightweight modality-aware adapters atop the language-aligned core, and extensions to segmentation, instance segmentation, and time-series detection with temporal modeling and multi-pass sensors (Li et al., 2 Mar 2026). Related work suggests additional possibilities. The broader evidence that different layers favor OCR, spatial reasoning, or semantic disambiguation implies that future variants of LVSA could become more task-conditional, more architecture-sensitive, or more explicit about internal representational diagnostics, but such extensions remain prospective rather than part of the original method (Chen et al., 30 Apr 2025).

In summary, LVSA is a progressive fusion mechanism that resolves the mismatch between language-aligned semantics and dense detection requirements by annealing intermediate visual features into the language-pivoted alignment process. Its significance lies less in architectural complexity than in where and how it intervenes: before downstream detection, under linguistic supervision, and with a linear schedule that preserves the final-layer distribution early and broadens it into a multi-scale semantic representation later (Li et al., 2 Mar 2026).