Best-Buddies Similarity (BBS) Overview

• BBS is a mutual nearest-neighbor similarity measure that identifies Best-Buddies Pairs to capture the structural agreement between point sets.
• It employs both hard and differentiable formulations to compute similarity scores that are robust to outliers, occlusions, and partial overlaps.
• BBS has been successfully applied in template matching, visual tracking, and 3D point cloud registration, showing superior accuracy over traditional methods.

Best-Buddies Similarity (BBS) is a parameter-free, mutual nearest-neighbor-based similarity measure between two point sets. BBS captures the structural agreement between sets by enumerating the pairs of points—termed Best-Buddies Pairs (BBPs)—where each point is simultaneously the nearest neighbor of the other with respect to a chosen metric. Originating in robust template matching, BBS has been systematically developed and extended for diverse applications such as image matching, visual tracking, and rigid 3D point cloud registration, including differentiable approximations for use within deep learning frameworks (Oron et al., 2016, Oron et al., 2016, Drory et al., 2020, Hezroni et al., 2021).

1. Formal Definition and Algorithmic Computation

Let $P = \{p_i\}_{i=1}^N$ and $Q = \{q_j\}_{j=1}^M$ be two point sets in $\mathbb{R}^d$, typically defined via feature extraction from image regions, deep neural embeddings, or raw 3D coordinates. The pairwise distance matrix $D$ is constructed with $D_{ij} = d(p_i, q_j)$ for a given metric $d(\cdot, \cdot)$. The nearest neighbor of $p_i$ in $Q$ is $\mathrm{NN}(p_i, Q) = \arg\min_{q \in Q} d(p_i, q)$. A pair $(p_i, q_j)$ forms a Best-Buddies Pair if and only if

$$\mathrm{NN}(p_i, Q) = q_j \quad \text{and} \quad \mathrm{NN}(q_j, P) = p_i.$$

The hard (indicator) version introduces $B_{ij} = 1$ if the above holds, $0$ otherwise. The BBS score is then

$$\mathrm{BBS}(P, Q) = \frac{1}{\min(N, M)} \sum_{i=1}^N \sum_{j=1}^M B_{ij},$$

yielding a similarity value in $[0,1]$ quantifying the proportion of high-fidelity mutual correspondences (Oron et al., 2016, Oron et al., 2016).

For differentiable optimization, the non-differentiable $\arg\min$ is replaced by the soft-argmin: $$\mathrm{SoftMin}_{i' \to j}(D_{i'j}) = \frac{\exp(-\alpha D_{ij})}{\sum_{i'} \exp(-\alpha D_{i'j})},$$ and analogously row-wise. The soft best-buddies matrix is then $$\tilde{B}_{ij} = \mathrm{SoftMin}_{i'\to j}(D_{i'j}) \cdot \mathrm{SoftMin}_{j'\to i}(D_{ij'})$$, where $\alpha$ is a tunable temperature (Drory et al., 2020, Hezroni et al., 2021).

The computational complexity for hard BBS is $O(NM)$ per evaluation, but random sampling (below) can ensure constant-time evaluation for large or unequal-sized sets (Oron et al., 2016).

2. Robustness Properties and Theoretical Analysis

BBS is fundamentally robust to outliers, occlusion, partial overlap, distractors, and complex deformations. Only pairs that are mutually the closest participate; spurious, one-sided correspondences do not increase the score. For missing data or occluded regions, points with no true partner will generally have large mutual distances, resulting in negligible BBP contributions. Notably, non-overlapping or outlier structures almost never contribute BBPs, thus not misleading the similarity measure (Oron et al., 2016, Drory et al., 2020, Hezroni et al., 2021).

Theoretical analysis reveals that, for large samples, the expected BBS between point sets drawn i.i.d. from densities $f_P$ and $f_Q$ converges to a function closely related to the $\chi^2$ distance: $$E[\mathrm{BBS}(P, Q)] = \frac{1}{\min(N_P, N_Q)} \sum_{i,j} E[\,bb(p_i, q_j)\,] \to \frac{1}{2} - \frac{1}{4} \chi^2(f_P, f_Q)$$ with

$$\chi^2(f_P, f_Q) = \int \frac{(f_P(p)-f_Q(p))^2}{f_P(p) + f_Q(p)}\, dp.$$

Thus, maximizing BBS is asymptotically equivalent to minimizing statistical discrepancy, but without requiring histogram binning (Oron et al., 2016).

3. Variants, Approximations, and Extensions

Handling Unequal Set Sizes

Original BBS exhibits upward bias when sets are of disparate cardinalities; as $|Q| \gg |P|$, every $p\in P$ eventually finds a "buddy," biasing BBS upwards. This is resolved via random sampling: uniformly sample $K \leq \min(N, M)$ points without replacement from both $P$ and $Q$ and compute BBS on these subsets. This approach removes size bias, ensures scale invariance, and guarantees $O(K^2)$ computational complexity (Oron et al., 2016).

Differentiable (Soft) Variants

For learning-based scenarios, smooth (soft) BBS is constructed, enabling integration into gradient-based optimization and deep learning frameworks. The soft formulation, combining row- and column-wise soft-argmin, provides a continuous analog to BBP assignment and is essential in DeepBBS and other registration losses (Drory et al., 2020, Hezroni et al., 2021).

Loss Functions for Registration

Four BBS-inspired losses have been proposed for 3D rigid registration:

• $L_{\mathrm{softBBS}}$: negative soft mutual-NN count.
• $L_{\mathrm{softBD}}$: soft buddy-weighted point-to-point distances.
• $\mathrm{BBR-N}$: soft buddy-weighted point-to-plane distances (using normals).
• $\mathrm{BBR-F}$: hard best-buddy filtering of point-to-plane distances.

The combination of BBP filtering and geometric distance minimization yields registration algorithms robust to noise, occlusion, sparsity, and distractors (Drory et al., 2020, Hezroni et al., 2021).

4. Practical Applications

BBS has been applied successfully in:

Template Matching

BBS outperforms classical measures (SSD, NCC, $\chi^2$ histogram, BDS) for template matching under unconstrained conditions, exhibiting high mean Average Precision (mAP) in the presence of background clutter, occlusions, and nonrigid deformations. BBPs localize on inliers, producing well-localized matching peaks (Oron et al., 2016).

Visual Tracking

Best-Buddies Tracking (BBT) employs (sampled) BBS within a particle filter framework for model-free online tracking. The tracker utilizes random-sampled BBS to handle scale changes and integrates bag-of-templates for appearance variation. Ensembles of trackers with different process noises yield robust performance under challenging conditions (Oron et al., 2016).

3D Point Cloud Registration

BBS and its soft variant, embedded in differentiable loss functions, drive rigid registration algorithms for 3D scans and LiDAR data—demonstrated in KITTI/Apollo-Southbay—offering robustness to sparsity, outliers, occlusion, and even distractor objects. DeepBBS incorporates SoftBBS in an end-to-end deep network, training feature embeddings specifically to optimize mutual-nearest neighbor matching for highly accurate pose recovery even for unseen object categories (Drory et al., 2020, Hezroni et al., 2021).

5. Empirical Results and Limitations

BBS-based methods consistently demonstrate the following empirical properties:

• Superior overlap accuracy and mAP in template matching benchmarks, with significant gains (5–30%) over traditional scores.
• In tracking, scale invariance and appearance adaptation via sampling and bag-of-points strategies.
• In registration, state-of-the-art alignment errors in both synthetic and challenging real-world datasets, exceeding symmetry-ICP and modern learning baselines, particularly under low point counts, partial overlap, or strong distractors (Oron et al., 2016, Oron et al., 2016, Drory et al., 2020, Hezroni et al., 2021).
• Robustness to unseen categories, with DeepBBS++ achieving two orders of magnitude reduction in rotation MAE versus prior art on held-out classes (Hezroni et al., 2021).

Limitations include failure in settings where few or no true BBPs are available, such as when template size is too small relative to the background, or under extremely strong distractors mimicking the target (Oron et al., 2016).

6. Implementation Considerations and Recommendations

Efficient computation is critical in practical settings. For image-based tasks, caching strategies exploit the incremental nature of the sliding window to reduce complexity substantially; patch-based subsampling further accelerates evaluation. In tracking and detection, random sampling ensures computational tractability and removes scale bias.

Best-practice recommendations include:

• Always normalizing by the minimum set size, or via random sampling for unequal sets.
• Using rich joint features (e.g., color+location, deep CNN embeddings) to define the metric $d(\cdot, \cdot)$, as BBS is parameter-free aside from feature choice.
• In detection or tracking, considering top-$k$ BBS modes and applying nonmaxima suppression to mitigate distractor peaks.
• In learning contexts, incorporating soft BBS losses for end-to-end trainability (Oron et al., 2016, Oron et al., 2016, Drory et al., 2020, Hezroni et al., 2021).

BBS is applicable beyond vision: wherever mutual nearest-neighbor relationships robustly delineate genuine correspondences, the BBS principle generalizes, offering a theoretically justified, outlier-tolerant, and parameter-minimal matching protocol.