End-to-end training of object class detectors for mean average precision

Abstract: We present a method for training CNN-based object class detectors directly using mean average precision (mAP) as the training loss, in a truly end-to-end fashion that includes non-maximum suppression (NMS) at training time. This contrasts with the traditional approach of training a CNN for a window classification loss, then applying NMS only at test time, when mAP is used as the evaluation metric in place of classification accuracy. However, mAP following NMS forms a piecewise-constant structured loss over thousands of windows, with gradients that do not convey useful information for gradient descent. Hence, we define new, general gradient-like quantities for piecewise constant functions, which have wide applicability. We describe how to calculate these efficiently for mAP following NMS, enabling to train a detector based on Fast R-CNN directly for mAP. This model achieves equivalent performance to the standard Fast R-CNN on the PASCAL VOC 2007 and 2012 datasets, while being conceptually more appealing as the very same model and loss are used at both training and test time.

• The paper proposes training CNN object detectors end-to-end by using mean average precision (mAP) directly as the training loss, deviating from traditional window classification loss.
• It addresses the non-differentiability of mAP by introducing novel gradient-like measures (SDE and MEE) suitable for optimizing piecewise-constant functions.
• The method integrates Non-Maximum Suppression (NMS) into the training process, aligning the training objective with the test-time evaluation metric for improved consistency and potential performance gains.

End-to-End Training of Object Class Detectors Using Mean Average Precision

The paper "End-to-end training of object class detectors for mean average precision" authored by Paul Henderson and Vittorio Ferrari presents a method designed to directly train convolutional neural network (CNN)-based object detectors using mean average precision (mAP) as the training loss. Traditionally, CNNs have been trained using window classification loss followed by non-maximum suppression (NMS) applied only at inference time. This work diverges from that paradigm by integrating NMS into the training process, promising a unified and consistent approach to both training and evaluation.

Key Contributions

A significant challenge in utilizing mAP as a training loss lies in its nature as a piecewise-constant function, rendering conventional gradient descent inapplicable due to the undefined or zero gradient at most points. This paper addresses this by introducing gradient-like constructs suitable for piecewise-constant functions and demonstrating efficient gradient calculations for mAP post-NMS.

1. Novel Gradient-Like Measures: The authors propose general, gradient-like definitions for piecewise-constant functions. Two primary methods are introduced: a symmetric difference estimator (SDE) and a mean envelope estimator (MEE). Both approaches seek to estimate meaningful gradients in regions where standard derivatives provide no informative signal, which are crucial for optimizing non-differentiable metrics.
2. Integration with NMS: Their method effectively incorporates NMS within the training loop, thereby allowing the network to be genuinely trained for the task it is intended to perform at test time, i.e., localizing and classifying high-confidence detections post-NMS.
3. Implementation and Performance: The framework, tested on the PASCAL VOC 2007 and 2012 datasets, uses Fast R-CNN as a baseline. The authors demonstrate that training directly for mAP achieves performance comparable to traditional approaches. Moreover, leveraging larger training datasets shows even greater improvements, suggesting a performance ceiling that traditional methods may not exploit as thoroughly.

Implications and Observations

From a practical perspective, the ability to optimize directly for the metric used at inference time can potentially simplify system architectures by eliminating discrepancies between training and test losses. This move towards end-to-end learning offers more natural and logically coherent training processes, aligning network optimization with ultimate task objectives.

Theoretically, this work opens avenues for applying similar gradient-like definitions to other structured, non-convex objectives that have historically resisted traditional optimization techniques. Moreover, this could stimulate developments in directly training for other complex metrics beyond object detection, such as those found in ranking problems.

In terms of network architecture adaptability, their approach highlights a framework that can be integrated into existing neural paradigms with minimal alterations to optimization strategies. This approach aligns well with contemporary trends in machine learning, which favor streamlined, comprehensive systems over composite, hand-tuned pipelines.

Future Directions

The authors hint at the broader applicability of their findings beyond the object detection domain. In information retrieval tasks and similar fields where metrics such as discounted cumulative gain are pivotal, these pseudo-gradient strategies could be directly employed or adapted. Future research might entail exploring the dynamics of these gradient-like quantities in more varied settings and across different neural architectures.

Additionally, while the current study focuses on the mAP metric, further experimentation with volatility and biases of other metrics, when used as training losses, could yield insightful data on how to maintain stability and robustness throughout training.

In summary, this paper enriches the toolkit for training modern object detection systems by aligning the training objectives with test-time evaluation metrics and offering a conceptual advancement in the application of gradient descent to piecewise-constant functions. As deep learning systems continue to evolve, these developments will likely play a critical role in numerous applications within artificial intelligence and beyond.