When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition (2207.11463v1)

Abstract: Recently, most handwritten mathematical expression recognition (HMER) methods adopt the encoder-decoder networks, which directly predict the markup sequences from formula images with the attention mechanism. However, such methods may fail to accurately read formulas with complicated structure or generate long markup sequences, as the attention results are often inaccurate due to the large variance of writing styles or spatial layouts. To alleviate this problem, we propose an unconventional network for HMER named Counting-Aware Network (CAN), which jointly optimizes two tasks: HMER and symbol counting. Specifically, we design a weakly-supervised counting module that can predict the number of each symbol class without the symbol-level position annotations, and then plug it into a typical attention-based encoder-decoder model for HMER. Experiments on the benchmark datasets for HMER validate that both joint optimization and counting results are beneficial for correcting the prediction errors of encoder-decoder models, and CAN consistently outperforms the state-of-the-art methods. In particular, compared with an encoder-decoder model for HMER, the extra time cost caused by the proposed counting module is marginal. The source code is available at https://github.com/LBH1024/CAN.

• The paper introduces a novel network that integrates symbol counting with traditional HMER methods to boost prediction accuracy.
• It employs a weakly-supervised Multi-Scale Counting Module and a Counting-Combined Attentional Decoder within an encoder-decoder framework.
• The approach achieves state-of-the-art results on datasets like CROHME and promises advancements in education technology and digital document processing.

Overview of "When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition"

The paper "When Counting Meets HMER" introduces a novel approach to Handwritten Mathematical Expression Recognition (HMER) by integrating a Counting-Aware Network (CAN) with traditional encoder-decoder architectures. The innovation lies in addressing the challenges faced by existing attention-based systems, particularly inaccuracies arising from complex structures and lengthy markup sequences due to variance in writing style and spatial layout.

Methodology

The authors propose the CAN, which combines HMER with symbol counting tasks to enhance prediction accuracy. The CAN incorporates a weakly-supervised Multi-Scale Counting Module (MSCM) capable of estimating symbol counts without requiring position annotations. The MSCM is seamlessly integrated into conventional encoder-decoder frameworks, jointly optimizing both HMER and symbol counting. Notably, the added computational cost remains marginal.

Components of CAN

1. Multi-Scale Counting Module (MSCM): This module predicts symbol counts using multi-scale feature extraction and a channel attention mechanism. It operates without full supervision, relying only on existing markup sequence annotations.
2. Counting-Combined Attentional Decoder (CCAD): The decoder enhances the recognition process using a combination of local context vectors, hidden states, and counting vectors as global information.

Results

The CAN demonstrates superior performance over state-of-the-art methods across several benchmark datasets, including the CROHME 2014, 2016, and 2019 datasets. For instance, CAN-DWAP achieved an expression recognition rate (ExpRate) of 57.00% on CROHME 2014, outperforming contemporary methods. The improvements are further validated on the HME100K dataset, which introduces more complex real-world scenarios.

Implications

The integration of counting enhances the spatial awareness and symbol prediction accuracy of traditional HMER systems. This approach is not only theoretically enriching by exploring the synergy between counting and recognition tasks but also practically significant, with potential applications in education technology, automated grading systems, and digital document processing.

Future Directions

While CAN significantly improves HMER performance, it encounters challenges with diverse writing styles and intricate symbolic structures. Future research could explore enhanced structural grammar modeling to further increase recognition robustness. Additionally, adapting the approach to a broader variety of handwriting datasets could render the method universally applicable.

Conclusion

The proposed Counting-Aware Network marks an important contribution to the HMER landscape, reflecting a shift towards multi-task learning frameworks that integrate auxiliary tasks like counting to bolster primary task performance. This paper lays the groundwork for future advancements in both methodology and application domains in AI-driven handwritten text recognition.