Mean Local Group Avg Precision (mLGAP)

• Mean Local Group Average Precision (mLGAP) is a metric that evaluates hashing-based retrieval systems by jointly measuring ranking precision and binary code dispersion.
• It employs a penalty function to reduce the impact of hash code collisions, thereby ensuring a more uniform utilization of the binary space.
• Empirical evaluations on datasets like CIFAR-10 and CIFAR-100 show that mLGAP provides a more balanced assessment compared to traditional mAP metrics.

Mean Local Group Average Precision (mLGAP) is a performance metric introduced to address the key deficiencies of existing evaluation frameworks for hashing-based retrieval systems, particularly the inability of Mean Average Precision (mAP) to account for the utilization of hash codes. mLGAP quantifies retrieval quality while explicitly encouraging effective and uniform dispersion of binary hash codes in Hamming space, mitigating the collision-induced issues inherent in mAP-driven training regimes (Ding et al., 2018).

1. Motivation and Limitations of mAP in Hashing-based Retrieval

Hashing-based retrieval systems map data points $u \in \mathbb{U}$ to compact binary vectors $b = f(u) \in \{-1, 1\}^k$, facilitating efficient similarity search via Hamming distance. mAP, the prevailing evaluation metric, exclusively measures the ranking accuracy of retrieval outputs. However, optimizing solely for mAP frequently results in severe hash code collisions, in which samples from the same class collapse into identical or very similar binary codes. This process reduces the effective “vocabulary” of available binary codes, degrades fine-grained retrieval (rendering within-ball ranking impossible), and leads to overall poorer utilization of the hash space.

A key proposition presented by Ding et al. asserts that achieving perfect global mAP (mAP = 1) forces the fraction of used codes to no more than two-thirds of all $2^k$ possible codes. Consequently, an mAP-oriented objective inherently promotes poor code dispersion, undermining the practical quality of retrieval outcomes (Ding et al., 2018).

2. Mathematical Formulation of mLGAP

mLGAP evaluates retrieval quality by combining ranking precision within local Hamming balls and a penalty reflecting the spread (or dispersion) of codes among those retrieved items. Given:

• $B = \{b_1, \dots, b_N\}$: Query codes,
• $X = \{x_1, \dots, x_M\}$: Database codes,
• $d_H(\cdot, \cdot)$: Hamming distance.

For each query code $b_j$ and radius $r \geq 0$, the “local group” is:

$S_r(b_j) = \{x_i \in X \mid d_H(b_j, x_i) \leq r\}$

A penalty function $\varphi(S_r(b_j)) \in [0,1]$ quantifies the uniformity of code usage within $S_r(b_j)$: 1 denotes perfect uniformity (no collisions), while 0 indicates total collision. A canonical choice is:

$$\varphi(S) = \frac{\sum_{c \in \mathrm{codes}(S)} \#\{x \in S : x = c\}}{(\max_{c \in \mathrm{codes}(S)} \#\{x \in S : x = c\}) \cdot |\mathrm{codes}(S)|}$$

where $\mathrm{codes}(S)$ is the set of unique codes in $S$.

The Local Group Average Precision (LGAP) for query $b_j$ at radius $r$ is:

$$\mathrm{LGAP}(b_j)@r = \frac{1}{r+1}\sum_{k=0}^r \left( \frac{\#\{\text{relevant in }S_k(b_j)\}}{|S_k(b_j)|} \cdot \varphi(S_k(b_j)) \right)$$

The Mean Local Group Average Precision (mLGAP) over the set of queries is:

$$\mathrm{mLGAP}(B) = \frac{1}{N} \sum_{j=1}^N \mathrm{LGAP}(b_j)$$

3. Computation Procedure for mLGAP

The computation of mLGAP operates as follows:

for each query b_j in B: for k = 0 to r_max: S_k = { x in X : d_H(b_j, x) <= k } if len(S_k) > 0: precision[k] = #relevant_in(S_k) / len(S_k) phi[k] = penalty(S_k) # As defined above else: precision[k] = 0 phi[k] = 0 LGAP[j] = (1/(r_max+1)) * sum(precision[k] * phi[k] for k in range(r_max+1)) mLGAP = (1/len(B)) * sum(LGAP[j] for j in range(len(B)))

Here, “#relevant_in(S_k)” refers to the number of codes in $S_k$ sharing the true label with the query $b_j$.

4. Balancing Retrieval Accuracy and Code Utilization

mLGAP explicitly rewards systems that jointly optimize both ranking precision and binary space utilization. The penalty factor $\varphi(S_k)$ directly lowers the contribution of local precision when hash code collisions are prevalent, counteracting the mAP tendency to favor highly clustered codes. Even with high local precision, a low $\varphi$ enforces a substantial reduction in LGAP. The theoretical analysis (including an orthodrome-based argument) demonstrates that mAP optimization alone collapses the code space, using ≤ $2/3$ of the codes, whereas mLGAP incentivizes dispersal of true positives throughout the hash space (Ding et al., 2018).

5. Experimental Characterization

Empirical results on CIFAR-10 and CIFAR-100 validate the distinctiveness of mLGAP relative to mAP. For instance, Deep Supervised Hashing (DSH) achieves high mAP (0.60 at 12 bits on CIFAR-10) but much lower mLGAP (≈0.26) due to concentrated collisions. Introducing a “dispersion buffer” term to the DSH loss, which encourages same-class codes to reside within an annulus $[r_1, r_2]$ instead of a point, increases mLGAP (to ≈0.28 at 12 bits) but results in lower mAP (≈0.30). The same pattern holds for CIFAR-100: under dispersion-focused training, uniformity is markedly improved as captured by mLGAP, while mAP is no longer a sufficient indicator of retrieval performance (Ding et al., 2018).

Method/Setting	mAP (CIFAR-10, 12b)	mLGAP (CIFAR-10, 12b)
DSH (original loss)	0.60	~0.26
DSH (+ dispersion buffer)	0.30	~0.28

6. Practical Recommendations for mLGAP Adoption

• Select the Hamming radius $r$ to match targeted retrieval ranges, typically $r = 2$ or $3$.
• Efficiently implement code histograms within each Hamming ball to compute $\varphi$.
• Integrate mLGAP monitoring during model selection and loss design to avoid degenerate solutions dominated by collisions.
• When a system exhibits high mAP but low $\varphi$, introducing a code dispersion term (e.g., enforcing minimal pairwise Hamming distance within a buffer zone) is advised.
• Report both mLGAP and mAP in comparative studies to expose the true extent of code space utilization and retrieval quality, as mLGAP surfaces performance characteristics that mAP alone can obscure (Ding et al., 2018).

7. Significance and Context

mLGAP responds to a long-standing need for robust evaluation of hashing-based retrieval systems, reconciling the twin aims of ranking accuracy and effective binary space usage. Its introduction provides researchers and practitioners with a metric that reflects not only whether a system retrieves correct results but also whether it does so by leveraging the capacity of the hash space. This dual focus is particularly significant as database sizes and code lengths increase, making code collisions a central bottleneck in large-scale retrieval scenarios. mLGAP is thus positioned as an essential complement to mAP for rigorous assessment and advancement of hashing-based retrieval methods (Ding et al., 2018).