A Technical Report: Entity Extraction using Both Character-based and Token-based Similarity

Abstract: Entity extraction is fundamental to many text mining tasks such as organisation name recognition. A popular approach to entity extraction is based on matching sub-string candidates in a document against a dictionary of entities. To handle spelling errors and name variations of entities, usually the matching is approximate and edit or Jaccard distance is used to measure dissimilarity between sub-string candidates and the entities. For approximate entity extraction from free text, existing work considers solely character-based or solely token-based similarity and hence cannot simultaneously deal with minor variations at token level and typos. In this paper, we address this problem by considering both character-based similarity and token-based similarity (i.e. two-level similarity). Measuring one-level (e.g. character-based) similarity is computationally expensive, and measuring two-level similarity is dramatically more expensive. By exploiting the properties of the two-level similarity and the weights of tokens, we develop novel techniques to significantly reduce the number of sub-string candidates that require computation of two-level similarity against the dictionary of entities. A comprehensive experimental study on real world datasets show that our algorithm can efficiently extract entities from documents and produce a high F1 score in the range of [0.91, 0.97].