TedEval: A Fair Evaluation Metric for Scene Text Detectors (1907.01227v1)

Abstract: Despite the recent success of scene text detection methods, common evaluation metrics fail to provide a fair and reliable comparison among detectors. They have obvious drawbacks in reflecting the inherent characteristic of text detection tasks, unable to address issues such as granularity, multiline, and character incompleteness. In this paper, we propose a novel evaluation protocol called TedEval (Text detector Evaluation), which evaluates text detections by an instance-level matching and a character-level scoring. Based on a firm standard rewarding behaviors that result in successful recognition, TedEval can act as a reliable standard for comparing and quantizing the detection quality throughout all difficulty levels. In this regard, we believe that TedEval can play a key role in developing state-of-the-art scene text detectors. The code is publicly available at https://github.com/clovaai/TedEval.

• The paper introduces TedEval, a fair evaluation metric that uses instance-level matching and character-level scoring to enhance scene text detection assessment.
• It addresses limitations of metrics like IoU and DetEval by effectively capturing detection granularity and completeness.
• Empirical results on ICDAR datasets show TedEval’s sensitivity in improving H-mean scores for state-of-the-art detectors.

An Analysis of TedEval: A Fair Evaluation Metric for Scene Text Detectors

The paper "TedEval: A Fair Evaluation Metric for Scene Text Detectors" addresses a pivotal challenge in the field of scene text detection—namely, the development of a robust and comprehensive evaluation metric for effective and equitable assessment of text detection algorithms. Despite advancements in scene text detection algorithms, established evaluation metrics are often deficient in addressing the intrinsic characteristics of text detection tasks. This paper presents TedEval, a novel evaluation protocol aimed at remedying these deficiencies.

Key Concepts and Approach

The paper critiques commonly-used metrics such as IoU (Intersection over Union) and DetEval, highlighting their shortcomings in adequately reflecting granularity and completeness—the two primary challenges in scene text detection evaluation. The IoU metric originated from object detection frameworks, failing to accommodate scene text's inherent multi-character structure accurately. DetEval, while more tailored for text, is criticized for leniency by permitting incomplete detections that may compromise the recognition phase.

TedEval's design introduces an instance-level matching protocol paired with a character-level scoring mechanism. Specifically, this approach allows for one-to-one, one-to-many, and many-to-one matching to better align with ground truth annotations. An innovative aspect of TedEval is its use of pseudo character centers derived from word-level bounding boxes and associated word lengths, enabling a sophisticated character-level scoring method without explicit character annotations. This scoring method provides a nuanced account of text detection accuracy by penalizing instances of missing or overlapping characters, a step beyond traditional binary evaluation scores.

Empirical Validation and Results

TedEval has been empirically validated using widely recognized scene text detection datasets, ICDAR 2013 and ICDAR 2015. The results underscore TedEval's efficacy in addressing granularity and completeness. Comparisons with existing metrics revealed variances in H-mean score changes, highlighting metrics' discordance when applied to state-of-the-art detectors. Such variations indicate TedEval's ability to more accurately capture text detection quality across different complexities and tasks.

The paper features a compelling quantitative analysis, presenting R (Recall), P (Precision), and H (Harmonic mean) scores. For instance, the EAST detector exhibits a notable improvement in H-mean scores when assessed with TedEval as opposed to DetEval, demonstrating TedEval's sensitivity to detection granularity and completeness. Additionally, statistical factors such as granularity and completeness are analyzed for a range of detectors, offering insight into detection challenges and metric performance.

Implications and Future Directions

TedEval is poised to be a vital tool in advancing the methodological rigor in scene text detection evaluations. The authors postulate that it will serve as a reliable standard for evaluating and advancing state-of-the-art text detection methodologies. Future prospects involve extending TedEval to accommodate polygon annotations and enhancing its applicability across tasks by mitigating dependencies on bounding box vertex order. This aligns with broader objectives in artificial intelligence, where precision in model evaluation directly impacts the trajectory of algorithm refinement and practical deployment.

In summary, TedEval presents a meaningful progression in the evaluative landscape for scene text detection, offering measurable improvements over established metrics. Its methodical approach to granularity and completeness sets a precedent for future endeavors aiming to refine text detector assessments, paving the way for more precise and equitable evaluations within the field.