Tightness-aware Evaluation Protocol for Scene Text Detection (1904.00813v1)

Abstract: Evaluation protocols play key role in the developmental progress of text detection methods. There are strict requirements to ensure that the evaluation methods are fair, objective and reasonable. However, existing metrics exhibit some obvious drawbacks: 1) They are not goal-oriented; 2) they cannot recognize the tightness of detection methods; 3) existing one-to-many and many-to-one solutions involve inherent loopholes and deficiencies. Therefore, this paper proposes a novel evaluation protocol called Tightness-aware Intersect-over-Union (TIoU) metric that could quantify completeness of ground truth, compactness of detection, and tightness of matching degree. Specifically, instead of merely using the IoU value, two common detection behaviors are properly considered; meanwhile, directly using the score of TIoU to recognize the tightness. In addition, we further propose a straightforward method to address the annotation granularity issue, which can fairly evaluate word and text-line detections simultaneously. By adopting the detection results from published methods and general object detection frameworks, comprehensive experiments on ICDAR 2013 and ICDAR 2015 datasets are conducted to compare recent metrics and the proposed TIoU metric. The comparison demonstrated some promising new prospects, e.g., determining the methods and frameworks for which the detection is tighter and more beneficial to recognize. Our method is extremely simple; however, the novelty is none other than the proposed metric can utilize simplest but reasonable improvements to lead to many interesting and insightful prospects and solving most the issues of the previous metrics. The code is publicly available at https://github.com/Yuliang-Liu/TIoU-metric .

• The paper introduces a novel Tightness-aware Intersect-over-Union (TIoU) metric that penalizes both incomplete detection and extraneous content in scene text regions.
• It assesses detection performance using a continuous evaluation scale that better captures the tightness and completeness of text detections.
• Experimental results on ICDAR benchmarks show that TIoU correlates more closely with recognition performance than traditional IoU metrics.

Tightness-aware Evaluation Protocol for Scene Text Detection

In the paper "Tightness-aware Evaluation Protocol for Scene Text Detection," the authors propose a novel evaluation metric termed Tightness-aware Intersect-over-Union (TIoU) to improve the evaluation of scene text detection methods. This metric aims to account for deficiencies in existing evaluation approaches, specifically regarding their inability to effectively quantify the tightness and completeness of detection in scene text detection tasks.

Motivation and Challenges

Scene text detection is distinct from generic object detection tasks as it requires bounding boxes to tightly encompass text to facilitate recognition. Current mainstream metrics, such as IoU, do not address the peculiarities of text detection, such as handling overlapping text instances and distinguishing between one-to-many or many-to-one detection scenarios. The authors identify several shortcomings of existing metrics, such as their reliance on threshold values that fail to distinguish detection quality differences like tightness and completeness, and the failure to effectively manage complex matching scenarios.

Proposed Metric: TIoU

To remedy these limitations, the TIoU metric introduces penalties for detections that inadequately cover ground truth (GT) text regions or include extraneous outlier regions. The metric is designed to provide a continuous scale of evaluation, allowing more granular assessment of detection tightness, completeness, and matching accuracy:

• Completeness: TIoU enforces penalties when detection fails to recall full text regions, addressing the issue of partial detections.
• Compactness: The metric penalizes detections containing extraneous background or text, thereby promoting tighter results.
• Tightness-awareness: TIoU allows discriminating between detections that barely meet the IoU threshold and those that more accurately represent the GT.

Evaluation and Results

The paper employs a comprehensive experimental evaluation on ICDAR 2013 and ICDAR 2015 datasets using a range of published detection methods and general object detection frameworks. The results illustrate that TIoU aligns more closely with end-to-end recognition performance compared to traditional IoU metrics, exposing discrepancies in current methods' ability to produce usable text regions for recognition purposes. TIoU also reveals a significant reduction in perceived performance when traditionally evaluated methods are assessed under this new metric, highlighting hidden inefficiencies and potential areas for improvement.

Implications and Future Directions

The introduction of the TIoU metric offers promising improvements in evaluating scene text detection by providing a more nuanced understanding of detection performance. The results suggest that further work in crafting methodologies that account for both detection and recognition tasks are warranted. Additionally, TIoU could be leveraged to guide model training, emphasizing tighter detection boundaries to enhance recognition outcomes.

Future endeavors could explore integrating TIoU in training regimes or applying the metric to incrementally update training samples in semi-supervised learning contexts, potentially leading to models that more closely align with recognition-driven objectives.