---
title: Best-Buddies Similarity (BBS) Overview
url: https://www.emergentmind.com/topics/best-buddies-similarity-bbs
type: topic
---

# Best-Buddies Similarity (BBS) Overview

Best-Buddies Similarity (BBS) is a parameter-free, mutual nearest-neighbor-based similarity measure between two point sets. BBS captures the structural agreement between sets by enumerating the pairs of points—termed Best-Buddies Pairs (BBPs)—where each point is simultaneously the nearest neighbor of the other with respect to a chosen metric. Originating in robust template matching, BBS has been systematically developed and extended for diverse applications such as image matching, visual tracking, and rigid 3D point cloud registration, including differentiable approximations for use within deep learning frameworks [1609.01571][1611.00148][2010.01912][2110.03016].

## 1. Formal Definition and Algorithmic Computation

Let $P = \{p_i\}_{i=1}^N$ and $Q = \{q_j\}_{j=1}^M$ be two point sets in $\mathbb{R}^d$, typically defined via feature extraction from image regions, deep neural embeddings, or raw 3D coordinates. The pairwise distance matrix $D$ is constructed with $D_{ij} = d(p_i, q_j)$ for a given metric $d(\cdot, \cdot)$. The nearest neighbor of $p_i$ in $Q$ is $\mathrm{NN}(p_i, Q) = \arg\min_{q \in Q} d(p_i, q)$. A pair $(p_i, q_j)$ forms a Best-Buddies Pair if and only if 
\[
\mathrm{NN}(p_i, Q) = q_j \quad \text{and} \quad \mathrm{NN}(q_j, P) = p_i.
\]
The hard (indicator) version introduces $B_{ij} = 1$ if the above holds, $0$ otherwise. The BBS score is then
\[
\mathrm{BBS}(P, Q) = \frac{1}{\min(N, M)} \sum_{i=1}^N \sum_{j=1}^M B_{ij},
\]
yielding a similarity value in $[0,1]$ quantifying the proportion of high-fidelity mutual correspondences [1609.01571][1611.00148]. 

For differentiable optimization, the non-differentiable $\arg\min$ is replaced by the soft-argmin:
\[
\mathrm{SoftMin}_{i' \to j}(D_{i'j}) = \frac{\exp(-\alpha D_{ij})}{\sum_{i'} \exp(-\alpha D_{i'j})},
\]
and analogously row-wise. The soft best-buddies matrix is then $\tilde{B}_{ij} = \mathrm{SoftMin}_{i'\to j}(D_{i'j}) \cdot \mathrm{SoftMin}_{j'\to i}(D_{ij'})$, where $\alpha$ is a tunable temperature [2010.01912][2110.03016].

The computational complexity for hard BBS is $O(NM)$ per evaluation, but random sampling (below) can ensure constant-time evaluation for large or unequal-sized sets [1611.00148].

## 2. Robustness Properties and Theoretical Analysis

BBS is fundamentally robust to outliers, occlusion, partial overlap, distractors, and complex deformations. Only pairs that are mutually the closest participate; spurious, one-sided correspondences do not increase the score. For missing data or occluded regions, points with no true partner will generally have large mutual distances, resulting in negligible BBP contributions. Notably, non-overlapping or outlier structures almost never contribute BBPs, thus not misleading the similarity measure [1609.01571][2010.01912][2110.03016].

Theoretical analysis reveals that, for large samples, the expected BBS between point sets drawn i.i.d. from densities $f_P$ and $f_Q$ converges to a function closely related to the $\chi^2$ distance:
\[
E[\mathrm{BBS}(P, Q)] = \frac{1}{\min(N_P, N_Q)} \sum_{i,j} E[\,bb(p_i, q_j)\,] \to \frac{1}{2} - \frac{1}{4} \chi^2(f_P, f_Q)
\]
with
\[
\chi^2(f_P, f_Q) = \int \frac{(f_P(p)-f_Q(p))^2}{f_P(p) + f_Q(p)}\, dp.
\]
Thus, maximizing BBS is asymptotically equivalent to minimizing statistical discrepancy, but without requiring histogram binning [1609.01571].

## 3. Variants, Approximations, and Extensions

### Handling Unequal Set Sizes

Original BBS exhibits upward bias when sets are of disparate cardinalities; as $|Q| \gg |P|$, every $p\in P$ eventually finds a "buddy," biasing BBS upwards. This is resolved via random sampling: uniformly sample $K \leq \min(N, M)$ points without replacement from both $P$ and $Q$ and compute BBS on these subsets. This approach removes size bias, ensures scale invariance, and guarantees $O(K^2)$ computational complexity [1611.00148].

### Differentiable (Soft) Variants

For learning-based scenarios, smooth (soft) BBS is constructed, enabling integration into gradient-based optimization and deep learning frameworks. The soft formulation, combining row- and column-wise soft-argmin, provides a continuous analog to BBP assignment and is essential in DeepBBS and other registration losses [2010.01912][2110.03016].

### Loss Functions for Registration

Four BBS-inspired losses have been proposed for 3D rigid registration:
- $L_{\mathrm{softBBS}}$: negative soft mutual-NN count.
- $L_{\mathrm{softBD}}$: soft buddy-weighted point-to-point distances.
- $\mathrm{BBR-N}$: soft buddy-weighted point-to-plane distances (using normals).
- $\mathrm{BBR-F}$: hard best-buddy filtering of point-to-plane distances.

The combination of BBP filtering and geometric distance minimization yields registration algorithms robust to noise, occlusion, sparsity, and distractors [2010.01912][2110.03016].

## 4. Practical Applications

BBS has been applied successfully in:

### Template Matching

BBS outperforms classical measures (SSD, NCC, $\chi^2$ histogram, BDS) for template matching under unconstrained conditions, exhibiting high mean Average Precision (mAP) in the presence of background clutter, occlusions, and nonrigid deformations. BBPs localize on inliers, producing well-localized matching peaks [1609.01571].

### Visual Tracking

Best-Buddies Tracking (BBT) employs (sampled) BBS within a particle filter framework for model-free online tracking. The tracker utilizes random-sampled BBS to handle scale changes and integrates bag-of-templates for appearance variation. Ensembles of trackers with different process noises yield robust performance under challenging conditions [1611.00148].

### 3D Point Cloud Registration

BBS and its soft variant, embedded in differentiable loss functions, drive rigid registration algorithms for 3D scans and LiDAR data—demonstrated in KITTI/Apollo-Southbay—offering robustness to sparsity, outliers, occlusion, and even distractor objects. DeepBBS incorporates SoftBBS in an end-to-end deep network, training feature embeddings specifically to optimize mutual-nearest neighbor matching for highly accurate pose recovery even for unseen object categories [2010.01912][2110.03016].

## 5. Empirical Results and Limitations

BBS-based methods consistently demonstrate the following empirical properties:
- Superior overlap accuracy and mAP in template matching benchmarks, with significant gains (5–30%) over traditional scores.
- In tracking, scale invariance and appearance adaptation via sampling and bag-of-points strategies.
- In registration, state-of-the-art alignment errors in both synthetic and challenging real-world datasets, exceeding symmetry-ICP and modern learning baselines, particularly under low point counts, partial overlap, or strong distractors [1609.01571][1611.00148][2010.01912][2110.03016].
- Robustness to unseen categories, with DeepBBS++ achieving two orders of magnitude reduction in rotation MAE versus prior art on held-out classes [2110.03016].

Limitations include failure in settings where few or no true BBPs are available, such as when template size is too small relative to the background, or under extremely strong distractors mimicking the target [1609.01571].

## 6. Implementation Considerations and Recommendations

Efficient computation is critical in practical settings. For image-based tasks, caching strategies exploit the incremental nature of the sliding window to reduce complexity substantially; patch-based subsampling further accelerates evaluation. In tracking and detection, random sampling ensures computational tractability and removes scale bias.

Best-practice recommendations include:
- Always normalizing by the minimum set size, or via random sampling for unequal sets.
- Using rich joint features (e.g., color+location, deep CNN embeddings) to define the metric $d(\cdot, \cdot)$, as BBS is parameter-free aside from feature choice.
- In detection or tracking, considering top-$k$ BBS modes and applying nonmaxima suppression to mitigate distractor peaks.
- In learning contexts, incorporating soft BBS losses for end-to-end trainability [1609.01571][1611.00148][2010.01912][2110.03016].

BBS is applicable beyond vision: wherever mutual nearest-neighbor relationships robustly delineate genuine correspondences, the BBS principle generalizes, offering a theoretically justified, outlier-tolerant, and parameter-minimal matching protocol.

Source: https://www.emergentmind.com/topics/best-buddies-similarity-bbs