Bi-Cross-Modal Curriculum Transfer Learning

• The paper introduces a dual curriculum (IR-VIS and IR-T) that progressively bridges the gap between visible pretraining and real-world infrared imagery.
• It employs dynamic loss variation weighting to suppress misaligned examples and enhance reliable infrared-text pairs, improving training stability.
• Ablation studies demonstrate that an Ascending-Stratified Random schedule boosts both perceptual accuracy and semantic grounding in the IRGPT framework.

Searching arXiv for the specified paper to ground the article and citations. Bi-cross-modal curriculum transfer learning is the difficulty-aware transfer strategy introduced in IRGPT to adapt a visible-pretrained multimodal LLM to real-world infrared imagery using authentic infrared-text supervision rather than synthetic infrared generation (Cao et al., 19 Jul 2025). In the IRGPT formulation, the method addresses two bottlenecks simultaneously: the absence of large-scale authentic infrared-text supervision in the wild and the large modality mismatch between real infrared imagery and the visible-image knowledge embedded in pretrained MLLMs. Its central premise is that real infrared images are not merely “dark visible images”: they exhibit different statistics, weaker semantic cues, and thermal saliency patterns that vary with wavelength and sensor type. Accordingly, the transfer process is organized as a curriculum over two complementary cross-modal directions—infrared-visible and infrared-text—so that training proceeds from easier, more learnable correspondences to harder, more infrared-specific ones (Cao et al., 19 Jul 2025).

1. Problem formulation and rationale

IRGPT situates bi-cross-modal curriculum transfer learning within real-world infrared multimodal learning, where direct reuse of visible-image pretraining is inadequate (Cao et al., 19 Jul 2025). The paper states that pretrained multimodal models already possess strong visible-image understanding, but that directly applying them to infrared often causes hallucinations or poor grounding. It also contrasts prior infrared-VLM approaches that rely on style-transferred synthetic IR, arguing that such methods are constrained by generator quality and fixed generative distributions and therefore preserve neither infrared authenticity nor true infrared-specific variability.

Within this setting, curriculum transfer learning is motivated by the difficulty of optimizing on infrared data even after constructing the large-scale InfraRed-Text Dataset (IR-TD). The paper characterizes infrared images as semantically sparse, low-contrast, and sensor-dependent, which makes training from scratch difficult and makes naive transfer unstable. The curriculum is therefore designed to expose the model first to samples that are easier in two distinct senses: visually closer to the visible domain and semantically better aligned with text (Cao et al., 19 Jul 2025).

The intended effect is stabilization of optimization and improvement of generalization. The paper explicitly frames the method as a gradual bridge from familiar visible-image structure to unfamiliar infrared structure, rather than a direct domain jump. A plausible implication is that the curriculum is not merely a data-ordering heuristic but a mechanism for controlling the geometry of cross-modal transfer under severe domain shift.

2. Dual structure: IR-VIS and IR-T

The method is termed bi-cross-modal because it defines two curricula that target different aspects of adaptation (Cao et al., 19 Jul 2025).

Curriculum	Primary role	Easier samples
IR-VIS curriculum	Visual domain transfer between infrared and visible images	Infrared samples visually closer to visible patterns
IR-T curriculum	Semantic grounding between infrared images and text	Infrared-text pairs that are well aligned and clean

The IR-VIS curriculum addresses the domain gap between infrared and visible imagery. The paper’s intuition is that near-infrared images are closer to visible images, whereas thermal infrared images are more dissimilar. Since the base model is already strong on visible images, infrared samples whose appearance is more similar to visible imagery should be easier for the vision encoder to absorb first. To obtain discriminative features, the paper uses a feature extractor from InfMAE, retrained with masking on both infrared and visible grayscale images. Because many infrared samples do not have one-to-one visible counterparts, the method does not depend on directly paired samples; instead, it estimates a cross-modal sample distance by projecting intra-domain distances onto the inter-domain direction (Cao et al., 19 Jul 2025).

The IR-T curriculum addresses semantic alignment between infrared images and text. Here the premise is that some infrared-text pairs are well aligned and easy to learn, whereas others are noisy, weakly aligned, or semantically complex. A pretrained CLIP model is used as a “warm” scorer, and its loss on an image-text pair is interpreted as a proxy for alignment difficulty. The paper further argues that raw CLIP loss alone is insufficient because its distribution is close to Gaussian and does not clearly separate truly hard-but-useful samples from misaligned ones, which motivates a dynamic metric based on loss variation during warm-up (Cao et al., 19 Jul 2025).

Taken together, the two curricula partition the transfer problem into a visual-domain lesson and a semantic-grounding lesson. The paper’s ablations later indicate that these lessons are not redundant: Lesson 1 mainly influences perceptual accuracy, while Lesson 2 better supports semantic correctness (Cao et al., 19 Jul 2025).

3. Difficulty estimation and mathematical formulation

The IR-VIS branch defines difficulty through a combination of global domain discrepancy and sample-level relative position in feature space (Cao et al., 19 Jul 2025). The global discrepancy is computed with Maximum Mean Discrepancy (MMD):

$$\text{MMD}^2(\mathcal{X}^{ir}, \mathcal{X}^{vis}) = \left\| \mathbb{E}_{x^{ir}\sim \mathcal{X}^{ir}}[\phi(x^{ir})] - \mathbb{E}_{x^{vis}\sim \mathcal{X}^{vis}}[\phi(x^{vis})] \right\|_{\mathcal{H}}^2$$

In the paper’s notation, this is given in RKHS form with a Gaussian kernel feature map $\phi:\mathcal{X}\to\mathcal{H}$. The corresponding domain centers are

$$\mu^{ir}=\frac{1}{|\mathcal{X}^{ir}|}\sum_{x_i^{ir}\in \mathcal{X}^{ir}}\phi(x_i^{ir}), \qquad \mu^{vis}=\frac{1}{|\mathcal{X}^{vis}|}\sum_{x_j^{vis}\in \mathcal{X}^{vis}}\phi(x_j^{vis})$$

and the infrared sample distance score is

$$s_i^{ir\rightarrow vis} = \left\langle \phi(x_i^{ir})-\mu^{ir}, \mu^{vis}-\mu^{ir}\right\rangle + \text{MMD}^2(\mathcal{X}^{ir}, \mathcal{X}^{vis})$$

The paper emphasizes that this score is an approximate ranking signal rather than an exact metric. For the curriculum, exact distances are unnecessary; relative difficulty ordering is sufficient (Cao et al., 19 Jul 2025).

The IR-T branch defines difficulty through a loss variation rate computed from a pre-warmed CLIP model. If the pairwise loss before and after warm-up is denoted by $l$ and $l'$, the dynamic signal is

$\alpha = \frac{l' - l}{l}$

The interpretation given in the paper is explicit. If $\alpha > 0$, the loss has increased after warm-up, suggesting possible misalignment or instability. If $\alpha \le 0$, the sample is more learnable or becoming better aligned (Cao et al., 19 Jul 2025). This is used not only for ranking but also for weighting, because the method aims to suppress bad or noisy pairs while emphasizing hard-but-useful ones.

A plausible implication is that the two scores encode different failure modes of infrared transfer. The IR-VIS score quantifies how far a sample sits from visible-domain priors, whereas the IR-T score quantifies whether semantic supervision is stabilizing or destabilizing under warm-up.

4. Ordering, sampling, and weighting

The curriculum uses a baby-step style progression in which data are divided into $M$ tiers by difficulty and traversed from easier to harder (Cao et al., 19 Jul 2025). The two rankings from IR-VIS and IR-T are combined into a comprehensive ordering over samples. However, the method does not enforce a fully deterministic easy-to-hard pass. Instead, it couples tiered progression with random sampling within each tier, thereby preserving the global direction of the curriculum while avoiding overfitting to a rigid sequence.

The paper names the preferred schedule Ascending-Stratified Random and reports that it is the best schedule in ablation (Cao et al., 19 Jul 2025). It also evaluates several alternatives: Difficulty-Descending, Difficulty-Ascending, Bidirectional Ordering, Random, Descending-Stratified Random, and Ascending-Stratified Random. Reverse curricula are reported as harmful, while ascending and stratified-random curricula are beneficial.

Bi-cross-modal curriculum transfer learning also modifies the optimization objective through per-sample weights derived from the loss variation rate $\phi:\mathcal{X}\to\mathcal{H}$0:

$\phi:\mathcal{X}\to\mathcal{H}$1

where $\phi:\mathcal{X}\to\mathcal{H}$2 is the sigmoid function. The behavior is asymmetric by design. For positively varying losses, the weight is reduced, suppressing likely misaligned samples. For non-positive variation, the weight is increased, emphasizing samples that are useful and difficult (Cao et al., 19 Jul 2025).

The final training objective is weighted cross-entropy:

$\phi:\mathcal{X}\to\mathcal{H}$3

where $\phi:\mathcal{X}\to\mathcal{H}$4 is the one-hot label and $\phi:\mathcal{X}\to\mathcal{H}$5 is the predicted class probability. In this formulation, the curriculum acts along two axes simultaneously: it determines when a sample is seen and how strongly it influences learning (Cao et al., 19 Jul 2025).

5. Position within IRGPT and IR-TD

Within IRGPT, bi-cross-modal curriculum transfer learning is confined primarily to the first stage of a two-stage training pipeline (Cao et al., 19 Jul 2025). The pipeline consists of:

1. Incremental pre-training: only the vision encoder and projection layer are trained, with the goal of adapting the visible-pretrained model toward infrared understanding and image-text alignment.
2. Supervised instruction fine-tuning: the projector and LLM are trained, and the LLM uses LoRA for efficient adaptation.

The curriculum is applied during incremental pre-training, where the model must first absorb infrared-visible and infrared-text alignment before instruction tuning. The second stage then benefits from the better-aligned infrared representations established during pretraining (Cao et al., 19 Jul 2025).

This stage-specific placement is closely tied to the structure of IR-TD. The dataset is divided into a pre-training subset, an instruction subset, and a benchmark, containing 190k, 33k, and 37k image samples respectively. The paper also states that the full IR-TD corpus comprises over 260K authentic image-text pairs, with texts derived from two complementary sources: LLM-generated descriptions of visible images and rule-based descriptions of annotations (Cao et al., 19 Jul 2025).

The resulting training design reflects the paper’s broader claim that infrared transfer should not depend on synthetic infrared generation. Instead, the model is adapted directly on authentic real-world infrared data, using curriculum order and weighting to negotiate the gap between visible pretraining and infrared understanding (Cao et al., 19 Jul 2025).

6. Ablation evidence and empirical effects

The paper provides ablations intended to isolate the contribution of the curriculum itself (Cao et al., 19 Jul 2025). In the curriculum component ablation, the full baseline without curriculum yields psum 464.49 and nsum 26.85. Adding curriculum components improves performance, and the full bi-cross-modal curriculum with $\phi:\mathcal{X}\to\mathcal{H}$6 weighting achieves psum 485.79 and nsum 4.39, which is the best result reported in that ablation.

The ablation also attributes distinct effects to the two lessons. Removing curriculum or reversing it degrades performance. Lesson 2 is reported as more influential for positive-sum tasks than Lesson 1. Lesson 1 mainly affects perceptual accuracy, while Lesson 2 better supports semantic correctness (Cao et al., 19 Jul 2025). This partition is consistent with the design logic of IR-VIS as a visual transfer mechanism and IR-T as a semantic alignment mechanism.

In the schedule ablation, Difficulty-Descending is the worst among structured curricula, Difficulty-Ascending improves performance, and Ascending-Stratified Random is best overall (Cao et al., 19 Jul 2025). The paper reports for Ascending-Stratified Random:

• zero-shot psum 328.65
• zero-shot nsum 93.87
• fine-tune psum 485.79
• fine-tune nsum 4.39

These results support the paper’s claim that easy-to-hard progression with within-tier randomness is preferable to either rigid sorting or reverse-order exposure.

On the full 9-task benchmark, IRGPT with curriculum learning achieves the best overall results and, in zero-shot evaluation, outperforms the baseline InternVL2-8B by 76.35 positive sum points while reducing negative sum by 31.56 (Cao et al., 19 Jul 2025). The paper identifies strong gains on recognition and grounding, improved ReID, and better counting and location metrics, especially after fine-tuning. After instruction tuning, IRGPT reaches the highest reported overall performance with psum 485.79 and nsum 4.39, and it even surpasses InternVL2-26B. The paper interprets this as evidence that data quality, realism, and curriculum design matter more than simply scaling model size (Cao et al., 19 Jul 2025).

7. Interpretation, scope, and methodological significance

Bi-cross-modal curriculum transfer learning is presented not as a generic curriculum-learning template but as a transfer mechanism specialized for infrared multimodal learning under authentic data scarcity and pronounced modality shift (Cao et al., 19 Jul 2025). Its specificity lies in the fact that the “easy” dimension is defined twice: once by closeness to visible-domain structure and once by infrared-text semantic learnability. This dual definition distinguishes it from curricula that rank samples solely by model loss or by visual complexity.

The principal methodological contrast in the paper is with synthetic-style-transfer-based infrared training. According to the reported argument, synthetic infrared data creates a modality gap because generated images are constrained by the generator’s distribution and often fail to reflect genuine infrared sensor characteristics, thermal patterns, or real-world nuisances (Cao et al., 19 Jul 2025). In that framing, synthetic data can lead the model to learn artifacts of synthesis rather than the infrared modality itself. By contrast, IRGPT trains on authentic real-world infrared data and uses difficulty-aware ordering plus weighting to moderate the transfer from visible pretraining.

A common misconception addressed by the paper is the assumption that infrared adaptation can be solved by scaling or by direct reuse of visible-pretrained multimodal models. The reported comparison with larger-scale baselines suggests otherwise: a plausible implication is that infrared transfer quality depends critically on the alignment path taken during optimization, not only on backbone scale. Another misconception is that a curriculum should be a strict easy-to-hard sort. The ablation on Ascending-Stratified Random suggests that stochasticity within difficulty tiers is beneficial, implying that robustness requires both ordered progression and controlled randomness (Cao et al., 19 Jul 2025).

In summary, the method functions as a difficulty-aware bridge between visible-image multimodal pretraining and authentic infrared multimodal understanding. It ranks samples along infrared-visible and infrared-text axes, weights them by dynamic loss variation, and trains in an ascending stratified manner. Within IRGPT, this design is associated with improved recognition, grounding, ReID, counting, and location performance on a 9-task benchmark, while avoiding the synthetic-data dependence criticized in earlier infrared-VLM pipelines (Cao et al., 19 Jul 2025).