PACO: Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery

Abstract: On-the-Fly Category Discovery (OCD) requires a model, trained on an offline support set, to recognize known classes while discovering new ones from an online streaming sequence. Existing methods focus heavily on offline training. They aim to learn discriminative representations on the support set so that novel classes can be separated at test time. However, their discovery mechanism at inference is typically reduced to a single threshold. We argue that this paradigm is fundamentally flawed as OCD is not a static classification problem, but a dynamic process. The model must continuously decide 1) whether a sample belongs to a known class, 2) matches an existing novel category, or 3) should initiate a new one. Moreover, prior methods treat the support set as fixed knowledge. They do not update their decision boundaries as new evidence arrives during inference. This leads to unstable and inconsistent category formation. Our experiments confirm these issues. With properly calibrated and adaptive thresholds, substantial improvements can be achieved, even without changing the representation. Motivated by this, we propose PACO, a support-set-calibrated, tree-structured online decision framework. The framework models inference as a sequence of hierarchical decisions, including known-class routing, birth-aware novel assignment, and attach-versus-create operations over a dynamic prototype memory. Furthermore, we simulate the proxy discovery process to initialize the thresholds during offline training to align with inference. Thresholds are continuously updated during inference using mature novel prototypes. Importantly, PACO requires no heavy training and no dataset-specific tuning. It can be directly integrated into existing OCD pipelines as an inference-time module. Extensive experiments show significant improvements over SOTA baselines across seven benchmarks.

• The paper presents PACO, a support-set calibrated pipeline that aligns proxy tasks with online statistical decision calibration for on-the-fly category discovery.
• The paper leverages dynamic prototype memory and adaptive thresholding to effectively mitigate class fragmentation and manage the evolving data stream.
• The paper demonstrates robust performance and computational efficiency across diverse datasets, confirming its practical applicability in continual learning scenarios.

Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery: An Expert Analysis

Introduction

The problem of On-the-Fly Category Discovery (OCD) addresses the simultaneous identification of known (base) and previously unobserved visual categories within a streaming setting, where online, single-pass decisions must be made as each sample arrives. Traditional models for open-set recognition and offline clustering are not well aligned with the operational constraints of OCD: immediate decision-making, non-stationary data distributions, and the necessity to minimize class fragmentation and contamination. "PACO: Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery" (2604.11484) proposes a support-set calibrated pipeline (PACO) that addresses the key pathologies of OCD through principled proxy task alignment, statistical decision calibration via a small initial labeled support set, and the introduction of dynamic, data-driven thresholds for category birth and attachment. The following essay provides an expert technical summary, exploring the PACO framework, its calibration strategies, dynamic thresholding mechanisms, empirical performance, and its broader implications for continual category discovery.

Support-Set Calibrated Decision Framework

PACO's pipeline is instantiated with a preliminary, fixed labeled support set, from which all decision thresholds and base-class geometries are derived prior to online streaming. The model uses a DINO-pretrained ViT-B/16 backbone and a fixed linear projection head. During offline initialization, statistical regularization is applied, exposing the model to a proxy "seen-versus-pseudo-unseen" task by partitioning classes into visible and hidden subsets, thus explicitly training the embedding space to separate base-class responses from pseudo-unseen ones.

The selection between support-mean prototypes and classifier-driven reference directions for base categories is resolved by auditing the base-class reference's reliability directly on the support set, using a simple metric-based criterion. This minimizes class fragmentation by adaptively selecting the most robust metric for subsequent streaming decisions.

Streaming Inference and Dynamic Prototype Memory

The core PACO inference pipeline maintains an evolving memory bank containing fixed base-class references and dynamically updated novel-class prototypes. As each query sample arrives, it is routed using calibrated thresholds:

• If the routing gate confidently matches to a base reference, the assignment is immediate.
• Otherwise, the system computes a calibrated birth statistic (rooted in a von Mises-Fisher model with explicit uniform background correction), determining whether a novel prototype must be created or the sample should be attached to an existing one.
• For candidate attachment to a novel prototype, PACO uses the maximum log-likelihood ratio, moderated by the cluster's current size and concentration and capped by a dynamically updated creation threshold.

Crucially, all threshold calibrations originate from support-set statistics rather than hand-tuned or heuristic parameters, reducing dataset-specific overfitting and retaining robustness to stream variation.

Adaptive Boundary Tightening and Online Calibration

A major PACO innovation is the online adaptation of the "birth" threshold, which governs novel category creation. The streamed memory bank allows estimation of a robust empirical bank threshold, which is mixed with the support-based prior by a coefficient that reflects the maturity and stability of the discovered novel prototypes as the stream progresses. Early in the stream, when novel structure is unreliable, boundary tightening is conservative; as the median size of novel clusters approaches that of base classes, the influence of stream-derived statistics is increased.

Figure 1: Evolution of the discovered category count and the mixing coefficient $\eta_t$ as the stream progresses on Stanford Cars, demonstrating early rapid expansion and late-stage threshold tightening.

This adaptivity mitigates excessive late-stream fragmentation and stabilizes prototype proliferation without sacrificing the prompt discovery of new classes at stream onset.

Empirical Analysis and Computational Efficiency

PACO demonstrates stability across multiple datasets of varying granularity (CIFAR-10/100, ImageNet-100, fine-grained CUB-200-2011, Stanford Cars, Oxford-IIIT Pet, Food-101). The introduction of support-set calibration reduces variance in accuracy across random seeds, especially for old classes in single-pass OCD, as shown by limited standard deviation in error analyses.

Figure 2: Strict--Hungarian All/Old/New accuracy as a function of the support ratio on CUB-200-2011, Stanford Cars, and Oxford-IIIT Pet, illustrating performance stability and the positive effect of increased support data.

The per-sample decision time for the PACO inference layer remains sub-millisecond over the course of streaming, regardless of prototype bank expansion, confirming suitability for real-time OCD deployments.

Figure 3: Per-sample latency of PACO's decision layer on Stanford Cars, showing consistently low inference time across stream progression.

Limitations and Theoretical Considerations

The performance of PACO is contingent on the quality and representativeness of the initial support set. In scenarios with extremely limited, imbalanced, or poorly separated support examples, initial calibration is less reliable, impacting downstream discovery. Additionally, the online, single-pass nature of OCD makes the system path-dependent; suboptimal early decisions can propagate, though this is mitigated by support-spread shrinkage and robust online boundary updates.

Practical and Theoretical Implications

From a practical standpoint, PACO decouples representation learning from complex online memory updates, allowing its support-set calibrated decision logic to be integrated with other OCD architectures with minimal modification. The plug-and-play nature of this pipeline (demonstrated in the transfer studies referenced in the appendix) advances the modularity and generalization of OCD systems.

Theoretically, PACO formalizes the relationship between support-calibrated proxy tasks and streaming threshold adjustment through explicit statistical regularization. Its von Mises-Fisher/statistical decision model grounds streaming inference in well-understood density modeling rather than ad hoc distance thresholds. As future data increases in non-stationarity and class granularity, such adaptive, support-calibrated frameworks are expected to underpin robust continual category discovery.

Conclusion

PACO establishes a principled, support-calibrated paradigm for OCD, unifying proxy-task guided representation, dynamic prototype memory, and adaptive thresholding within a statistically rigorous framework. Its experimental stability and computational efficiency suggest broad applicability across persistent, open-world visual streams, with clear modular pathways for extension and integration in future OCD research and deployment scenarios (2604.11484).