Question Answering Over Temporal Knowledge Graphs (2106.01515v1)

Abstract: Temporal Knowledge Graphs (Temporal KGs) extend regular Knowledge Graphs by providing temporal scopes (start and end times) on each edge in the KG. While Question Answering over KG (KGQA) has received some attention from the research community, QA over Temporal KGs (Temporal KGQA) is a relatively unexplored area. Lack of broad coverage datasets has been another factor limiting progress in this area. We address this challenge by presenting CRONQUESTIONS, the largest known Temporal KGQA dataset, clearly stratified into buckets of structural complexity. CRONQUESTIONS expands the only known previous dataset by a factor of 340x. We find that various state-of-the-art KGQA methods fall far short of the desired performance on this new dataset. In response, we also propose CRONKGQA, a transformer-based solution that exploits recent advances in Temporal KG embeddings, and achieves performance superior to all baselines, with an increase of 120% in accuracy over the next best performing method. Through extensive experiments, we give detailed insights into the workings of CRONKGQA, as well as situations where significant further improvements appear possible. In addition to the dataset, we have released our code as well.

Question Answering Over Temporal Knowledge Graphs

The paper explores a niche yet crucial area of NLP and machine learning: Question Answering over Temporal Knowledge Graphs (Temporal KGQA). While Knowledge Graph Question Answering (KGQA) itself has seen extensive research, the temporal dimension in knowledge graphs (KRs) introduces unique challenges that remain less explored. This paper provides insights into these challenges and proposes a novel dataset and methodology to address them.

Temporal Knowledge Graphs and Their Significance

Knowledge Graphs are structured data representations storing entities and their relationships. Temporal Knowledge Graphs add a temporal component to these relationships by assigning time intervals to each edge in the graph. This temporal aspect is critical in domains where the validity of facts changes over time—for instance, historical datasets, financial data, and event timelines.

Introduction of the CronQuestions Dataset

Recognizing the gap in available resources for Temporal KGQA, the authors present CronQuestions, a significant contribution to the field. This dataset is the largest of its kind, consisting of a temporal knowledge graph derived from WikiData and a comprehensive set of natural language questions classified by their structural complexity. With a total of approximately 410k questions, CronQuestions provides an ample training ground for AI models, dwarfing previous datasets by a factor of 340x in size.

Methodological Innovations: CronKGQA

To benchmark Temporal KGQA performance, the paper critiques current state-of-the-art KGQA methodologies and notes their inadequacy in handling temporal queries. In response, the authors develop CronKGQA, a transformer-based solution that integrates Temporal KG embeddings. The embedding techniques experimented with include ComplEx, TComplEx, and TimePlex, each providing distinct ways to encode temporal data alongside entities and relations.

CronKGQA enhances the traditional model by incorporating two separated question embeddings specifically targeting entity prediction and time prediction, structured around the TComplEx scoring function. This dual-embedding approach significantly outperforms baseline models in both hits@1 and hits@10 metrics. Specifically, CronKGQA achieves a 120% improvement in accuracy over existing methods.

Key Findings and Results

The paper's empirical analysis reveals the following:

1. LLMs like BERT and T5, when used solely, fail to meet the task's demands, emphasizing the necessity for Temporal KG embeddings.
2. The integration of temporal embeddings significantly boosts performance, demonstrating the necessity of temporal context in KGQA tasks.
3. Question complexity and type impact model performance, with simple reasoning questions yielding near-perfect accuracy, while complex queries involving temporal reasoning indicate room for methodological improvement.

Implications for Future Research

The introduction of CronQuestions and the performance improvements showcased by CronKGQA establish a new baseline for Temporal KGQA. Future AI developments in this field will likely build on these findings, possibly exploring more sophisticated temporal reasoning mechanisms, hybrid models leveraging both rule-based and learning-based methods, and expanding the dataset to incorporate additional sources of temporal data.

In conclusion, the paper not only fills a critical resource gap in the domain of Temporal KGQA but also sets the stage for further explorations into temporal reasoning in AI, with potential applications across various industries requiring temporal knowledge processing.