High-Speed Tracking with Kernelized Correlation Filters

• The paper introduces kernelized correlation filters that exploit the circulant structure of image patches to efficiently perform tracking using Fourier transforms.
• The methodology integrates ensemble strategies, multi-kernel learning, and deep feature fusion to balance accuracy and high-speed performance.
• Enhanced regularization and boundary-effect mitigation techniques ensure robust tracking even under occlusion, scale variation, and rapid motion.

High-speed tracking with kernelized correlation filters (KCF) is a computationally efficient object tracking paradigm that exploits the circulant structure of densely shifted image patches and the diagonalization property of the discrete Fourier transform (DFT). By formulating tracking as a regularized regression or classification problem and leveraging fast, closed-form solutions in the frequency domain, KCF and its variants enable frame rates reaching hundreds of frames per second on standard CPUs, while achieving strong accuracy on challenging benchmarks. Modern advances extend the basic KCF concept with ensembles, multi-kernel fusion, deep feature integration, boundary-effect mitigation, and sparsity-driven optimization, yielding robust pipelines that balance accuracy, adaptability, and speed for real-time applications (Henriques et al., 2014, Uzkent et al., 2018, Tang et al., 2018).

1. Mathematical Foundations of Kernelized Correlation Filters

Classic KCF methods formulate visual tracking as a ridge regression over all cyclic translations of the target appearance. Let $x$ be an image patch (possibly multi-channel, e.g., HOG features), $y$ a continuous label map (e.g., a centered Gaussian) and $h$ the filter to be learned. The objective function is

$$E(h) = \frac{1}{2} \| y - \sum_{c=1}^C h_c * x_c \|^2 + \frac{\lambda}{2} \sum_{c=1}^C \| h_c \|^2,$$

where $*$ denotes circular correlation. The key insight is that the set of all translations of $x$ forms a circulant matrix, which is diagonalized by the DFT. In the frequency domain, for each channel,

$$\hat{h} = \frac{\hat{x}^* \odot \hat{y}}{\hat{x}^* \odot \hat{x} + \lambda},$$

with $\odot$ denoting element-wise product.

To capture nonlinear relations, KCF employs a kernel extension, reformulating the problem in the dual. For a shift-invariant kernel (e.g., Gaussian),

$$\hat{\alpha} = \frac{\hat{y}}{\hat{k}^{xx} + \lambda},$$

where $\hat{k}^{xx}$ is the DFT of the first row of the Gram matrix. At detection, the response to patch $z$ is

$r(z) = F^{-1}( \hat{k}^{xz} \odot \hat{\alpha} ),$

and the tracking hypothesis is taken as the maximum of $r(z)$. Model updates use exponential smoothing in the frequency domain (Henriques et al., 2014, Uzkent et al., 2018).

2. Ensemble and Hybrid Architectures

Ensemble KCF (EnKCF) addresses challenges in translation and scale variation, implementing three separate KCFs: a large-area translation filter ($R_t^L$), a small-area translation filter ($R_t^S$), and a scale filter ($R_s$). Rather than running all filters each frame, EnKCF cycles through them in a fixed schedule over five frames—for example, $R_s$ every fifth frame, $R_t^L$ next, and $R_t^S$ as refinement—drastically lowering total computation without sacrificing accuracy. Transitions and drifts are mitigated by integrating a lightweight particle filter for state smoothing, modeling object location and velocity with Gaussian process noise and importance weighting based on KCF response maps. This approach achieves up to 416 fps on CPU with a precision of 53.0% (20-pixel threshold) and AUC of 40.2% on UAV123, outperforming other trackers at $\geq$300 fps by 5–20% in precision and 10–25% in AUC (Uzkent et al., 2018).

3. Multi-Kernel and Deep Feature Integration

Standard KCFs are limited by use of a single kernel. Tang et al. introduced multi-kernel learning (MKCF), combining $M$ base kernels as $k(x, x') = \sum_{m=1}^M d_m k_m(x, x')$, but direct optimization with joint kernel weights leads to high computational cost and kernel interference, limiting frame rates ($\sim$30 fps). The MKCFup reformulation introduces a tight upper bound to decouple the optimization for each kernel, allowing separate closed-form updates and preserving FFT-based speed. MKCFup achieves 150 fps, with precision@20px of 83.5% and AUC of 64.1% on OTB2013—over 10% AUC absolute gain relative to KCF and 8% over traditional MKCF—while retaining high-speed performance (Tang et al., 2018).

Deep feature integration is achieved by extracting hierarchical CNN feature maps—e.g., from different VGG layers; each is paired with an independent filter or KCF, with fusion either by averaging or by a learned lightweight convnet acting as a "fusion" head. KMC (Kernelized Multi-resolution Convnet) employs per-layer kernelized DCFs and stacks their response maps for regression via a small CNN. This approach yields real-time performance (30–50 fps on GPU) and outperforms classical KCF, especially in challenging scenarios, with typical AUC increases of 9–13 points (Wu et al., 2017, Wang et al., 2017, Wang et al., 2019).

4. Mitigation of Boundary Effects and Enhanced Regularization

Conventional correlation filter trackers introduce a boundary effect due to circular (cyclic) shifts, degrading discrimination when targets approach patch edges. nBEKCF (fast kernelized correlation filters without boundary effect) eliminates this artifact by densely sampling real (non-cyclic) image patches as training data, while retaining a cyclic structure as a basis expansion. This design separates the training and basis sets, so kernel correlation matrices are formed between all real data patches and cyclically shifted versions of the target template only.

To maintain computational efficiency in the absence of FFT-friendly circulant structure, nBEKCF introduces ACSII (Autocorrelation with Squared Integral Image) and CCIM (Cyclic Correlation with Integral Matrix), two spatial-domain algorithms for fast matrix construction. Empirically, nBEKCF with hand-crafted features achieves 50 fps (CPU) while improving AUC by 2–3 points on OTB-2013/OTB-2015 compared to BACF/SRDCF, and deep-feature variants reach top performance on VOT2018 and TrackingNet (Tang et al., 2018).

Several variants also employ mixed $\ell_2$–$\ell_1$ (Huber-type) regularization in the Fourier domain for additional robustness to outliers and corrupted channels. By carefully splitting the real and imaginary frequency components, the $\ell_1$ penalty yields a closed-form optimization per DFT bin, enabling frame rates exceeding 40 fps with consistent accuracy gains versus vanilla KCF (Guan et al., 2018).

5. Support Correlation Filters and SVM-CF Synergy

Support Correlation Filters (SCF) generalize the KCF approach by incorporating SVM objectives (squared hinge loss) over the circulant matrix of translated patches. The optimization alternates between error and model updates, utilizing the DFT for each, and converges rapidly. Kernelized SCF further extends this to nonlinear kernels, preserving the FFT-based computational complexity. With multi-channel features (HOG+CN), kernelized SCF achieves state-of-the-art mean DP (85.0%) and AUC (57.5%) at 35 fps (OTB50), outperforming both SVM-based trackers and classical KCF (Zuo et al., 2016).

6. Empirical Performance and Benchmarks

Comprehensive experiments across standard datasets (OTB100, UAV123, OTB2013/2015, NfS, VOT2018, GOT10k, TrackingNet) demonstrate the balance of accuracy and speed achieved by high-speed kernelized correlation filters. Typical results include:

Tracker	Dataset	Precision@20px	Success AUC	FPS
KCF (HOG)	OTB50	73.2%	50.7%	172
EnKCF	OTB100	70.1%	53.0%	340
EnKCF	UAV123	53.0%	40.2%	416
MKCFup (HOG+Color)	OTB2013	83.5%	64.1%	150
nBEKCF (HC)	OTB-2013	–	67.1%	50
nBEKCF (HC)	OTB-2015	–	64.3%	50
KSCF (HOG+CN)	OTB50	85.0%	57.5%	35

Comparisons indicate that ensemble and multi-kernel approaches (EnKCF, MKCFup) yield improvements of 5–20% in precision and AUC at moderate computational cost, while KCF and DCF baselines excel in efficiency. Deep-feature enhancements further improve robustness, albeit at reduced (but still real-time) throughput (Henriques et al., 2014, Uzkent et al., 2018, Tang et al., 2018, Tang et al., 2018, Zuo et al., 2016).

7. Implementation Considerations, Limitations, and Prospective Directions

All real-time KCF-based pipelines exploit highly optimized Fourier libraries (e.g., FFTW), minimal intermediate data storage, and rigorous application of circulant algebra. Hand-crafted features such as fHoG and Color-Naming maximize CPU speed; deep feature variants incorporating VGGNet or ResNet layers achieve higher accuracy, with CPU throughput of 30–50 fps and much higher with GPU acceleration (Uzkent et al., 2018, Tang et al., 2018, Wang et al., 2019).

Ensemble and particle-filtered methods require no offline training, adapting wholly online, and remain amenable to embedded and mobile deployment. However, limitations exist in handling full-target loss, severe occlusion, and 3D pose changes. Potential extensions include integrating long-term re-detection modules, adaptive filter scheduling, and further refining deep-feature-enabled KCF for GPU (Uzkent et al., 2018).

Continued research focuses on further closing the gap to state-of-the-art deep trackers in accuracy while maintaining or improving the real-time computational profile intrinsic to the kernelized correlation filter framework.