CO-ISD: Implicit Self-Distillation for SIRST

• The paper introduces CO-ISD, a method that enables implicit self-distillation by sharing encoder-decoder parameters to transfer VFM semantic benefits to a lightweight SIRST detector.
• It employs collaborative optimization by synchronously training a VFM-guided main branch and a lightweight branch, achieving significant runtime reductions while maintaining detection performance.
• Empirical results on SIRST3 and MSDA-Net demonstrate that CO-ISD recovers most semantic gains from VFM-assisted training, outperforming explicit distillation methods in efficiency.

Searching arXiv for the exact CO-ISD paper and closely related distillation literature to ground the article in current research. {"query":"arXiv CO-ISD Collaborative Optimization-based Implicit Self-Distillation FDEP SIRST (Yu et al., 5 Dec 2025)", "max_results": 10} {"query":"(Yu et al., 5 Dec 2025) Collaborative Optimization-based Implicit Self-Distillation", "max_results": 5} Collaborative Optimization-based Implicit Self-Distillation (CO-ISD) is a training strategy introduced within the “Foundation-Driven Efficient Paradigm (FDEP)” for single-frame infrared small target (SIRST) detection. It is designed to exploit the global semantic priors of visual foundation models (VFMs), specifically frozen DINO/DINOv2 features, while avoiding the deployment-time cost of retaining a VFM-assisted branch during inference. In FDEP, CO-ISD couples a semantically stronger main branch and an original lightweight branch through parameter sharing and synchronized backpropagation, so that semantic transfer occurs implicitly through shared optimization rather than through an explicit distillation loss (Yu et al., 5 Dec 2025).

1. Problem setting and motivation

CO-ISD addresses a specific deployment and optimization problem. In FDEP, frozen VFM features are fused into a detector to improve SIRST performance, but a naive design would require keeping the VFM-related path at inference time, increasing latency and model cost. The method is therefore intended to preserve the semantic gains of VFM-assisted training while allowing only the lightweight detector branch to remain at test time. The same paper also states that conventional explicit distillation is unstable in SIRST because the task exhibits extreme class imbalance, very small objects, and low signal-to-noise ratio; under those conditions, pseudo-label-based teacher-student distillation may suffer from semantic noise, confidence bias, and teacher-student feature-space mismatch (Yu et al., 5 Dec 2025).

The broader purpose of CO-ISD is therefore twofold. First, it alleviates the computational overhead introduced by VFMs. Second, it replaces explicit teacher-student matching with an implicit transfer mechanism based on shared parameters and synchronized optimization. In the paper’s formulation, the central question is how to exploit VFM-derived global semantic priors during training, discard the expensive VFM-related path during inference, and avoid the instability associated with explicit distillation in an extremely imbalanced dense prediction problem.

2. Architectural context within FDEP

FDEP contains a frozen VFM representation branch, a main branch, and a lightweight branch. The VFM branch extracts semantic priors from the infrared image using frozen DINO-series features. In the main setting, the paper uses DINOv2 ViT-L/14 features from layers $i \in \{6,12,18,24\}$,

$$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$

These features are fused into the detector through the Semantic Alignment Modulation Fusion (SAMF) module. The lightweight branch, by contrast, preserves the original SIRST detector path without VFM feature fusion and is the branch retained for deployment (Yu et al., 5 Dec 2025).

Within SAMF, the detector top-level feature is

$F^{(t)} \in \mathbb{R}^{64 \times H \times W},$

and each selected VFM feature is projected and resized as

$\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$

with

$$\tilde P^{(i)} \in \mathbb{R}^{32 \times H \times W}.$$

The module then forms multiplicative and additive modulation maps,

$M_{\rm mul}^{(i)} = {\rm{Sigmoid}}({\rm{Conv}_{1 \times 1}(\tilde P^{(i)})),$

$M_{\rm add}^{(i)} = {\rm{ReLU}}({\rm{BN}}({\rm{Conv}_{1 \times 1}(\tilde P^{(i)}))),$

splits $F^{(t)}$ channel-wise into $[F_A^{(t)}, F_B^{(t)}]$, and applies dual-path modulation: $B_{\rm mul}^{(i)} = {\rm{Conv}_{3 \times 3}(F_A^{(t)} \odot M_{\rm mul}^{(i)}),$

$$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$0

Residual refinement is then performed via

$$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$1

and progressive fusion over $$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$2 yields the final semantically aligned feature

$$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$3

SAMF and CO-ISD are complementary: SAMF is the mechanism by which the main branch explicitly incorporates VFM priors, whereas CO-ISD is the mechanism by which the lightweight branch implicitly inherits those benefits and remains deployable without the VFM path (Yu et al., 5 Dec 2025).

3. Collaborative optimization and implicit self-distillation mechanism

CO-ISD operates on two co-optimized branches: a VFM-guided main branch and a lightweight branch. The paper states that the two branches “share the same encoder-decoder structure and parameter set,” and the best ablation result occurs when both encoder and decoder are shared. Thus, the intended constraint is that the detector encoder-decoder weights are common across both forward paths. The main branch receives semantically enriched features through SAMF, while the lightweight branch follows the original detector path; because both optimize the same underlying detector parameters, the main branch’s semantics-informed gradients shape the weights later used by the lightweight branch (Yu et al., 5 Dec 2025).

This is the sense in which the method is both collaborative and implicit. It is collaborative because both branches are trained simultaneously and contribute gradients in the same iteration. It is implicit because there is no explicit feature, logit, or soft-label matching term between the branches. The paper states that knowledge transfer occurs through “parameter sharing and synchronized backpropagation.” The main branch acts as the semantically stronger counterpart because it incorporates VFM priors, and the lightweight branch “passively inherits” representational capability through shared-parameter updates.

The branch losses are defined as

$$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$4

$$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$5

where $$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$6 is the edge-enhanced difficulty-mining loss, $$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$7 is the main-branch output, $$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$8 is the lightweight-branch output, and $$F_{\rm{DINO}}^{(i)} \in \mathbb{R}^{C_d \times H_i \times W_i}, \quad i \in \{6,12,18,24\}.$$9 is the ground-truth label. The CO-ISD objective is

$F^{(t)} \in \mathbb{R}^{64 \times H \times W},$0

with

$F^{(t)} \in \mathbb{R}^{64 \times H \times W},$1

Under shared encoder-decoder parameters, the implied update is a synchronized accumulation of the two gradient contributions onto the same parameter space. The paper explicitly contrasts this with explicit distillation and reports that no explicit distillation loss is used in the final method (Yu et al., 5 Dec 2025).

4. Empirical behavior and ablation evidence

The central empirical claim is that CO-ISD allows FDEP to benefit from VFM-enhanced training while retaining nearly the original inference cost. On SIRST3, the paper states that compared with “+ FDEP w/o CO-ISD,” the introduction of CO-ISD reduces inference time by 40.6% (from 0.160 to 0.095) to 74.4% (from 0.082 to 0.021), while maintaining comparable detection performance. In the MSDA-Net case, the original detector has runtime $F^{(t)} \in \mathbb{R}^{64 \times H \times W},$2 s and HSE $F^{(t)} \in \mathbb{R}^{64 \times H \times W},$3, “+ FDEP w/o CO-ISD” has runtime $F^{(t)} \in \mathbb{R}^{64 \times H \times W},$4 s and HSE $F^{(t)} \in \mathbb{R}^{64 \times H \times W},$5, and the final “+ FDEP Framework” with CO-ISD has runtime $F^{(t)} \in \mathbb{R}^{64 \times H \times W},$6 s and HSE $F^{(t)} \in \mathbb{R}^{64 \times H \times W},$7, thereby recovering the original runtime while preserving most of the gain (Yu et al., 5 Dec 2025).

A particularly direct ablation is the SIRST3 breakdown for MSDA-Net. Scheme2, defined as DINO + SAMF without CO-ISD, yields HSE $F^{(t)} \in \mathbb{R}^{64 \times H \times W},$8 at $F^{(t)} \in \mathbb{R}^{64 \times H \times W},$9 s. Scheme4, defined as DINO + SAMF + CO-ISD, yields HSE $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$0 at $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$1 s. The paper highlights this as a 63.0% runtime reduction,

$\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$2

with only a $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$3 HSE drop. The same section describes this as achieving “an optimal balance between performance and deployability.”

The most CO-ISD-specific ablation concerns parameter sharing and explicit distillation. Without explicit distillation loss, HSE on SIRST3 is $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$4 when sharing the encoder only, $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$5 when sharing the decoder only, and $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$6 when sharing encoder and decoder together. With explicit distillation loss, the corresponding HSE values are $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$7, $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$8, and $\tilde P^{(i)} = {\rm{Up}}({\rm{Conv}_{1 \times 1}(F_{\rm{DINO}}^{(i)})),$9. This shows two points stated by the paper: full encoder-decoder sharing is important, and adding explicit distillation slightly degrades the best shared-parameter configuration relative to pure CO-ISD (Yu et al., 5 Dec 2025).

5. Position within the distillation literature

CO-ISD belongs to a broader family of methods that couple multiple learning signals under shared optimization, but its mechanism is narrower and more deployment-oriented than many adjacent formulations. “Collaborative Teacher-Student Learning via Multiple Knowledge Transfer” introduces a hybrid objective with supervised loss, peer-to-peer online distillation, relation alignment, and self-distillation; it is close in spirit to CO-ISD because it combines collaborative optimization with self-guidance, but its self-distillation is explicit and depends on a pretraining stage and a frozen self-teacher (Sun et al., 2021).

“Distill on the Go: Online knowledge distillation in self-supervised learning” is another closely related point of comparison. It uses two jointly trained peer networks, no fixed external teacher, and temperature-scaled similarity-distribution matching during self-supervised learning. That places it near teacher-free, collaboratively optimized, implicitly distilling methods, although it is more precisely mutual or peer distillation than strict single-model self-distillation (Bhat et al., 2021).

A different connection comes from “Revisiting Self-Distillation,” which argues that self-distillation is best understood through loss-landscape geometry: the combined hard-label and soft-target objective leads to flatter minima and better generalization. This does not describe CO-ISD directly, but it suggests that auxiliary consistency mechanisms can matter not only as transfer channels but also as optimization regularizers (Pham et al., 2022).

More recent work extends the same design space in other domains. “A Kernel Perspective on Distillation-based Collaborative Learning” studies collaborative distillation via consensus predictions on public data and then proposes a neural algorithm with feature-kernel matching, emphasizing explicit collaborative distillation rather than self-distillation (Park et al., 2024). “EvoDriveVLA: Evolving Autonomous Driving Vision-Language-Action Model via Collaborative Perception-Planning Distillation” combines self-anchored visual distillation with oracle-guided trajectory distillation; its visual component is especially close to implicit self-distillation because the teacher is a frozen copy of the model’s own pre-finetuning visual encoder (Cao et al., 10 Mar 2026). “Self-Distilled Reinforcement Learning for Co-Evolving Agentic Recommender Systems” likewise combines collaborative optimization over interacting agents with self-distilled token-level credit assignment from historical trajectories, making it an unusually direct analogue of CO-ISD outside vision (Wang et al., 11 Apr 2026). Together, these papers place CO-ISD within a spectrum spanning explicit collaborative distillation, teacher-free mutual learning, self-anchored regularization, and trajectory- or representation-level self-generated supervision.

6. Interpretation, assumptions, and limitations

CO-ISD is not presented as a generic distillation principle independent of architectural context. Its efficacy depends on a semantically stronger main branch built from frozen VFM priors and SAMF, so the quality of the transferred signal depends on the usefulness of those priors for SIRST. The paper also shows that partial parameter sharing is insufficient: sharing only the encoder or only the decoder degrades HSE relative to full encoder-decoder sharing. This suggests that end-to-end gradient collaboration in a unified parameter space is not incidental but central to the method (Yu et al., 5 Dec 2025).

The method is also not lossless relative to the heavier VFM-assisted branch. Across several comparisons, the “+ FDEP Framework” variant with CO-ISD usually trails “+ FDEP w/o CO-ISD” slightly, even though it remains markedly more efficient at inference. The resulting tradeoff is therefore not elimination of the heavy branch’s performance gap, but preservation of most of its benefit at nearly original runtime. The paper further reports that explicit distillation is harmful in this setting, both in its dedicated ablation and in a supplementary discussion of pseudo-label-based collapse, which reinforces the paper’s claim that SIRST’s imbalance and semantic mismatch make explicit teacher-student transfer brittle (Yu et al., 5 Dec 2025).

Finally, CO-ISD solves inference-time overhead rather than training-time cost. Training still requires frozen VFM feature extraction, a VFM-guided main branch, and a lightweight branch. The framework is also scoped to encoder-decoder-based SIRST networks rather than arbitrary vision architectures. A plausible implication is that CO-ISD should be understood less as a universal self-distillation algorithm than as a task-specific parameter-sharing strategy for converting a semantically enriched training path into a deployment-efficient detector. In that sense, its defining contribution is architectural and optimization-theoretic at once: it replaces explicit branch matching with shared-parameter semantic transfer and thereby turns VFM-assisted training into lightweight inference (Yu et al., 5 Dec 2025).