UniSpector: Towards Universal Open-set Defect Recognition via Spectral-Contrastive Visual Prompting

Abstract: Although industrial inspection systems should be capable of recognizing unprecedented defects, most existing approaches operate under a closed-set assumption, which prevents them from detecting novel anomalies. While visual prompting offers a scalable alternative for industrial inspection, existing methods often suffer from prompt embedding collapse due to high intra-class variance and subtle inter-class differences. To resolve this, we propose UniSpector, which shifts the focus from naive prompt-to-region matching to the principled design of a semantically structured and transferable prompt topology. UniSpector employs the Spatial-Spectral Prompt Encoder to extract orientation-invariant, fine-grained representations; these serve as a solid basis for the Contrastive Prompt Encoder to explicitly regularize the prompt space into a semantically organized angular manifold. Additionally, Prompt-guided Query Selection generates adaptive object queries aligned with the prompt. We introduce Inspect Anything, the first benchmark for visual-prompt-based open-set defect localization, where UniSpector significantly outperforms baselines by at least 19.7% and 15.8% in AP50b and AP50m, respectively. These results show that our method enable a scalable, retraining-free inspection paradigm for continuously evolving industrial environments, while offering critical insights into the design of generic visual prompting.

• The paper introduces a novel framework employing spatial–spectral encoding and angular contrastive regularization to robustly recognize open-set defects.
• It integrates a dual-path encoder and dynamic query selection to suppress intra-class variance and enhance inter-class separability.
• Results show significant performance gains on industrial benchmarks, with detection AP50 improving by 19.7% and segmentation by 15.8%.

Universal Open-set Defect Recognition With Spectral-Contrastive Visual Prompting

Introduction

Open-set defect recognition poses significant challenges in industrial visual inspection due to continuously emerging defect types and the presence of visually ambiguous or subtle abnormalities. Traditional closed-set defect detectors and anomaly detection methodologies are insufficient for scalable industrial deployment, either due to the need for exhaustive retraining or lacking the capacity to semantically separate and localize diverse anomalies. Existing visual prompting models, while circumventing some label constraints, suffer from embedding collapse in high intra-class variance and subtle inter-class scenarios. "UniSpector: Towards Universal Open-set Defect Recognition via Spectral-Contrastive Visual Prompting" (2604.02905) confronts these issues by introducing a novel paradigm for robust and transferable open-set recognition, leveraging a semantically structured prompt topology built on spatial–spectral representations and contrastive regularization.

Figure 1: Comparison of visual inspection paradigms: (a) Closed-set detectors fail for novel defects; (b) anomaly detectors lack semantic specificity; (c) visual prompting enables open-set alignment using exemplar prompts.

Problem Analysis in Industrial Context

Closed-set methods fail to accommodate novel or rare defects that perennially arise in industrial mass production due to the assumption of fixed class taxonomies. Even advanced visual grounding or text-prompted open-set detectors—such as GroundingDINO and YOLO-World—are often inadequate in practice because verbalizing subtle, technical defect instances is infeasible for both LLMs and human experts. Moreover, industrial anomaly classes typically exhibit high intra-class variance and low inter-class separability, as demonstrated on the InsA benchmark; this manifests as significant challenges for unsupervised anomaly detection or prompt-based recognition.

Figure 2: InsA samples show pronounced intra-class variance (top) and ambiguous visual boundaries between distinct classes (bottom), illustrating recognition complexity.

UniSpector: Methodology

UniSpector reframes prompt-based open-set recognition through an explicitly regularized semantic topology for visual prompts. The architecture consists of the following principal components:

1. Spatial–Spectral Prompt Encoder (SSPE): Extends beyond spatial-token feature extraction by fusing region-of-interest (RoI) spatial information with orientation-invariant spectral descriptors derived via the 2D Discrete Fourier Transform. This dual-path encodes both fine-grained texture and global structure, suppressing rotational and illumination-induced intra-class noise while maximizing inter-class discriminability.
2. Contrastive Prompt Encoder (CPE): Enforces an angular margin–based contrastive loss (inspired by ArcFace) on SSPE-learned prompt embeddings, yielding an angular manifold with compact intra-class clusters and pronounced angular inter-class separation. Structuring the embedding topology in this manner prevents semantic collapse and facilitates consistent semantical positioning of novel classes.
3. Prompt-guided Query Selection (PQS): Employs a differentiable Gumbel–Softmax top-K strategy to dynamically select prompt-aware queries directly from vision tokens, preserving end-to-end optimization and adaptability to the prompt's peculiarity over static or heuristic schemes.

Figure 3: Overview of UniSpector: (left) SSPE extracts spatial and spectral cues; (center) CPE regularizes prompt embeddings; (right) PQS dynamically samples prompt-adaptive queries.

The inference pipeline is architecturally decoupled: prompt embeddings are computed offline, and only prompt-guided prediction is performed at runtime, achieving efficient and flexible deployment.

Benchmarks and Evaluation

A new benchmark, Inspect Anything (InsA), aggregates seven diverse industrial defect datasets, offering both in-domain (with held-out unseen classes) and cross-domain splits. The evaluation protocol adopts an exemplar-driven scheme with set numbers of prompt images per class, providing robust aggregates over prompt/class partition seeds.

UniSpector achieves a 19.7% improvement in detection $\mathrm{AP}_{50}^b$ and 15.8% in segmentation $\mathrm{AP}_{50}^m$ compared to previous state-of-the-art visual prompting and grounding methods on InsA's open-set splits. In cross-domain scenarios, despite dataset-specific imaging and domain biases, UniSpector retains a significant performance lead. Baseline models—such as DINOv, T-Rex2, YOLOE, and classical text-prompted detectors—consistently lag, especially for subtle, heterogeneous, and visually ambiguous classes.

Embedding Topology and Transfer Generalization

The semantic structure and transfer capacity of the prompt embedding are critically assessed. UniSpector’s embeddings, by explicit contrastive regularization and spectral–spatial fusion, form well-separated, compact class clusters even for unseen types, enabling improved open-set generalization.

Figure 4: UniSpector (left) produces well-structured manifold clusters in 3D PCA prompt embedding space with clear class separation, contrasting with scattered DINOv (right).

Intra-class cosine similarity is shown to be consistently higher for UniSpector versus baselines Figure 5, validating the effectiveness of spectral and angular-topological regularization.

Figure 5: UniSpector maintains higher intra-class cosine similarity for both seen and unseen classes on RealIAD.

Performance is also robust to prompt count per class—stabilizing with as few as 1–3 prompts Figure 6—and features an adaptive prompt allocation strategy to ensure equity across class distributions Figure 7.

Figure 6: The effect of increasing prompt samples per class, showing diminishing returns and stable accuracy after few exemplars.

Figure 7: Adaptive prompt strategy prevents prompt-to-target imbalance in long-tailed class distributions.

Practical Insights and Robustness

UniSpector’s detection remains stable under prompt annotation noise (IoU perturbations) and moderate image degradation, indicating robustness suited for pragmatic, non-laboratory industrial settings.

Figure 8: Model accuracy remains robust under substantial prompt annotation perturbation (box IoU), indicating noise tolerance.

Qualitative results illustrate UniSpector’s capacity to accurately localize prompted defect types in challenging open-set scenarios—notably outperforming vanilla visual prompting (DINOv), especially for rotated or subtle cases.

Figure 9: UniSpector localizes prompted, unseen defects, succeeding where DINOv fails.

Ablation and Closed-set Competitiveness

Ablation studies confirm each module’s essential contribution, with the greatest gains arising from introducing CPE atop SSPE refinements. PQS further improves prompt-conditioned querying. In classical closed-set supervision, UniSpector matches or surpasses canonical approaches (e.g., YOLOv11, Mask2Former), verifiying that advances in open-set capability do not sacrifice closed-set accuracy.

Implications and Future Directions

UniSpector demonstrates that regularizing visual prompting with structured, semantically organized embedding topologies—bolstered by spectral cues—enables robust, retraining-free open-set defect recognition. This paradigm shift from naive prompt-to-region matching towards explicit prompt topology design addresses the bottleneck of industrial open-set adaptation. The architecture is inherently extensible to broader domains beyond defectomics, implying that spectral–contrastive learning with prompt-driven querying presents a generic, scalable alternative to costly retraining in evolving, specialized visual environments.

Open theoretical challenges remain: mitigating cross-domain imaging biases, designing domain-agnostic prompt representations, and integrating multimodal cues without reliance on language groundings. Practically, UniSpector's approach reduces annotation, curation, and retraining effort—paving the way for sustainable real-world deployment in rapidly changing manufacturing lines and other high-variance technical fields.

Conclusion

UniSpector provides a robust architectural and algorithmic foundation for universal, open-set defect recognition via the combination of spectral–spatial representations, angular manifold structuring, and dynamic query conditioning. This work shifts industrial visual inspection from brittle closed-set and anomaly-detection regimes to scalable, semantically coherent, and highly generalizable prompting frameworks, setting a strong precedent for subsequent research in prompt-based, retraining-free visual recognition.