Detector-Agnostic Fine-Tuning

Updated 23 June 2026

Detector-Agnostic Fine-Tuning is the practice of updating only the backbone of pre-trained models to ensure compatibility across diverse detection heads.
It leverages strategies like self-supervised contrastive loss and teacher-student representation matching to boost accuracy and data efficiency.
Empirical results reveal significant AP and mAP improvements with minimal inference overhead, highlighting its importance in cross-domain detection.

Detector-agnostic fine-tuning refers to a suite of methods and principles whereby model adaptation, calibration, or regularization are performed in such a way that the resulting (fine-tuned) model remains compatible with a wide range of detector architectures or detection heads, rather than being customized or restricted to a specific detector instance. These approaches are characterized by their decoupling of adaptation steps from the specific inference heads or domain-specific properties of individual detectors, enabling plug-and-play transfer, robust cross-detector alignment, and efficiency in resource usage. Detector-agnostic fine-tuning is a response to the proliferation of pre-trained foundation models and the demand for transferability and modularity in both deep learning and traditional signal-processing pipelines.

1. Formal Definitions and Canonical Properties

A detector-agnostic fine-tuning protocol is defined by its invariance to the choice of detection head, architecture, or specific pipeline instantiation. Formally, given a model $F(x)$ composed of a backbone $f_\phi$ and a detector-specific head $g_\psi$ , detector-agnostic fine-tuning modifies only a set of parameters within $f_\phi$ , leaving all $g_\psi$ (potentially of arbitrary structure) unchanged or compatible.

Key properties include:

Backbone-centered adaptation: Only the backbone encoding is trained or regularized; detector heads are reusable or initialized independently (Wan et al., 5 May 2026, Jang et al., 2024).
Plug-and-play compatibility: The adapted module can be inserted into multiple detection pipelines or detector geometries without retraining (Raikwar et al., 9 Sep 2025, Mokhtar et al., 28 Feb 2025).
Negligible inference overhead: Approaches such as DAT and DADT ensure no change to inference time or pipeline complexity (Wan et al., 5 May 2026, Jang et al., 2024).
Generalization across modalities: Demonstrated for 2D/3D vision, language processing, time-series, and generative physics simulation (Chen et al., 2023, Raikwar et al., 9 Sep 2025, Ye et al., 2019).

2. Representative Methodologies

A. Region-Level Self-Supervised Adaptation in Object Detection

Decoupled Adaptivity Training (DAT) provides a canonical workflow for detector-agnostic fine-tuning in open-vocabulary object detection:

Pseudo-label construction: The detector generates region crops with confident base-class or background assignments; these are used as labeled or negative examples for a frozen vision-LLM (VLM) backbone (Wan et al., 5 May 2026).
Backbone-only optimization: Fine-tuning is restricted to the last LayerNorm and projection matrices of the VLM visual encoder, with the detector and text encoder entirely fixed ( $\sim$ 0.8M parameters).
Self-supervised contrastive loss: A contrastive objective identifies true versus spurious region-class pairs, leveraging both labeled and negative (background or unknown) regions.
Global-local balance by weight-space interpolation: Pre-trained and fine-tuned backbone weights are linearly interpolated to preserve global semantics while enabling local region adaptation.

This enables significant AP gains across detectors and datasets, and is fully agnostic to the choice of base detector (Mask R-CNN, DINO, etc.) (Wan et al., 5 May 2026).

B. Teacher-Student Representation Matching for 3D Detection

Domain Adaptive Distill-Tuning (DADT) couples a frozen teacher backbone, simulating the pre-train sensor resolution, with a fine-tuned student exposed to target-domain data (Jang et al., 2024):

Object- and context-level alignments: L2 and contextual cross-attention losses match per-object and spatial–semantic features, stabilizing adaptation and preventing teacher drift.
Any detection head: Adaptation operates at the BEV feature level, so any 3D detection head (SECOND, PV-RCNN, etc.) is compatible.
Sensor-agnostic by pre-alignment: Simulate pre-train conditions in the teacher; the student is adapted for the real target distribution.

Substantial data-efficiency and domain-robustness gains are observed (e.g., strong mAP improvements with only $\sim$ 100 frames).

C. Cross-Detector Foundation Model Transfer in Particle Physics

Fine-tuning of machine-learned particle-flow architectures demonstrates detector-agnostic transfer, where the feature extractor is pre-trained on one detector geometry and successfully adapted to new geometries using orders-of-magnitude less data (Mokhtar et al., 28 Feb 2025). No geometry encoding is used at the input—geometry information manifests only via the distributions of standard kinematic features, making the fine-tuned backbone agnostic to the architecture of the detection head.

D. Calibration and Score-Derivative Approaches

Post-hoc calibration strategies introduce a scalar correction to detector outputs regardless of the original detector’s architecture, optimizing on a target-domain validation set while keeping the backbone fixed (Yang et al., 2 Feb 2026). This framework, grounded in Bayes decision theory, is detector-agnostic and shown to improve open-world image fake detection accuracy across a wide spectrum of models. Similarly, Fine-tuned Score Deviation (FSD) applies to any pretraining-data detector by comparing pre- and post-fine-tuning scores, using only a small auxiliary set of non-member data (Zhang et al., 2024).

3. Empirical Results and Cross-Domain Benchmarks

Detector-agnostic fine-tuning consistently yields:

Substantial reductions in adaptation data/time: For instance, calorimeter simulators can be adapted to new detector geometries with up to 25× less data and 20× less computation relative to training from scratch (Raikwar et al., 9 Sep 2025).
Improved detection accuracy and robustness: DAT improves novel-category AP on COCO from 50.3% to 70.1%, and DADT yields +4.4 to +20.9 AP improvements on Waymo with very limited data (Wan et al., 5 May 2026, Jang et al., 2024).
Compatibility across detection paradigms: Methods are applicable to anchor-based (YOLO, Faster R-CNN), anchor-free (FCOS), set-based transformer detectors (DETR/DINO), and even classic or hybrid anomaly detectors in time-series (Ye et al., 2019, Dunnett et al., 7 May 2026).
Negligible or zero inference penalty: All reported methods either incur no runtime cost or require only scalar post-processing at inference (Wan et al., 5 May 2026, Yang et al., 2 Feb 2026).

A table of notable detector-agnostic fine-tuning approaches is given below:

Method	Domain	Agnostic Target
DAT (Wan et al., 5 May 2026)	Open-vocab 2D OD	Any detector+VLM
DADT (Jang et al., 2024)	3D LiDAR OD	Any BEV backbone
CaloDiT-2 (Raikwar et al., 9 Sep 2025)	Calorimetry	Any calorimeter
MLPF transfer (Mokhtar et al., 28 Feb 2025)	Particle-flow	Any detector geometry
ATSDLN (Ye et al., 2019)	Time-series	EGADS-classifiers
Logit calib. (Yang et al., 2 Feb 2026)	AIGC Img. Det.	Any classifier

4. Principles for Designing and Applying Detector-Agnostic Fine-Tuning

Key principles and validated best practices include:

Parameter targeting: Restrict updates to backbone/intermediate layers, minimizing adaptation cost and maximizing transferability (Wan et al., 5 May 2026, Jang et al., 2024).
Multi-level representation matching: Employ both object-level and context-level alignment regularizers for stabilized adaptation (especially for 3D/point cloud data) (Jang et al., 2024).
Explicit modality-agnostic representations: Utilize universal representations (e.g., cylindrical meshes for calorimetry, shared feature extractors for time series) to decouple physical geometry from the learning architecture (Raikwar et al., 9 Sep 2025, Ye et al., 2019).
Self-supervised and calibration-based adaptation: Leverage negative examples or domain-validation sets without requiring manual annotations for unknown/novel classes or post-hoc correction (Yang et al., 2 Feb 2026, Jang et al., 2024).
Transfer learning and pre-trained initializations: Use pre-trained weights for the backbone, optionally freezing low-level layers, and adapt only higher layers or lightweight modules to achieve data-efficient generalization (Chen et al., 2023, Ye et al., 2019).

Pitfalls observed:

Overfitting the adapted module to scarce target data can erode generalization; representation regularization and teacher guidance are critical (Jang et al., 2024).
Excessively updating backbone parameters leads to catastrophic forgetting; weight interpolation or regularization is recommended (Wan et al., 5 May 2026).
Using the detector’s architecture-specific loss signal for fine-tuning without agnostic regularization typically fails to generalize across detection paradigms (Dunnett et al., 7 May 2026).

5. Applications and Impact Across Modalities

Detector-agnostic fine-tuning frameworks have enabled:

Rapid adaptation of simulation and analysis pipelines in particle physics and collider experiments, where new detector configs must be studied repeatedly (Raikwar et al., 9 Sep 2025, Mokhtar et al., 28 Feb 2025).
Open-vocabulary vision tasks, where new objects must be localized and recognized without closed-set label restrictions (Wan et al., 5 May 2026).
Robust fake, backdoor, and pretraining-data detection, through calibration steps or score deviation computations for images, text, and multimodal data (Yang et al., 2 Feb 2026, Zhang et al., 2024, Dunnett et al., 7 May 2026).
Modular anomaly detection in time series, where the detection method and its hyperparameters may be dynamically adapted per segment (Ye et al., 2019).

A plausible implication is that foundation models across modalities—vision, language, 3D sensing, and physics simulation—are trending towards architectures and adaptation recipes that explicitly support detector-agnostic fine-tuning, making them core to modern research and real-world deployment.

6. Outlook, Limitations, and Extensions

While detector-agnostic fine-tuning has demonstrated substantial data and compute efficiency, several limitations persist:

Cross-domain generalization: Most empirical studies focus on similar or related domains (e.g., similar calorimeters, high-energy $e^+e^-$ detectors); transfer to more dissimilar settings (hadronic collisions, new modalities) remains relatively unexplored (Mokhtar et al., 28 Feb 2025).
Explicit geometry and parameter encoding: Many approaches rely on implicit encoding of detector differences. Explicit architectural conditioning may further boost transferability (Raikwar et al., 9 Sep 2025).
catastrophic forgetting: Despite weight interpolation and representation alignment, maintaining source-domain performance under repeated adaptation cycles is an ongoing challenge.
Score-based and calibration methods: Scalar calibration may fail in multi-modal or heavily overlapped logit distributions; more complex recalibration strategies may be required (Yang et al., 2 Feb 2026).

Future directions are likely to focus on self-supervised pre-training for universal modalities, continual domain-regularized adaptation, explicit geometry/parameter representations, and scalable benchmarks measuring cross-detector robustness and usability.

References: