Autoregressive Entity Retrieval (2010.00904v3)

Abstract: Entities are at the center of how we represent and aggregate knowledge. For instance, Encyclopedias such as Wikipedia are structured by entities (e.g., one per Wikipedia article). The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering. Current approaches can be understood as classifiers among atomic labels, one for each entity. Their weight vectors are dense entity representations produced by encoding entity meta information such as their descriptions. This approach has several shortcomings: (i) context and entity affinity is mainly captured through a vector dot product, potentially missing fine-grained interactions; (ii) a large memory footprint is needed to store dense representations when considering large entity sets; (iii) an appropriately hard set of negative data has to be subsampled at training time. In this work, we propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion. This mitigates the aforementioned technical issues since: (i) the autoregressive formulation directly captures relations between context and entity name, effectively cross encoding both; (ii) the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with vocabulary size, not entity count; (iii) the softmax loss is computed without subsampling negative data. We experiment with more than 20 datasets on entity disambiguation, end-to-end entity linking and document retrieval tasks, achieving new state-of-the-art or very competitive results while using a tiny fraction of the memory footprint of competing systems. Finally, we demonstrate that new entities can be added by simply specifying their names. Code and pre-trained models at https://github.com/facebookresearch/GENRE.

Autoregressive Entity Retrieval: Summary and Insights

The paper "Autoregressive Entity Retrieval" introduces GENRE, a novel approach to entity retrieval that generates entity names in an autoregressive manner. This method addresses key challenges associated with traditional retrieval methods which use classification approaches with dense vector representations.

Key Contributions

• Generative Approach: Unlike conventional methods that match inputs to entities through bi-encoder dot products, GENRE treats entity retrieval as a sequence-to-sequence problem and generates entity names token-by-token. This allows for more nuanced interactions between entity names and context.
• Efficiency Improvements:
  • Memory Footprint: GENRE reduces memory requirements since the architecture's parameters scale with vocabulary size rather than the number of entities.
  • Training Loss: The exact softmax loss is computed efficiently without needing negative data subsampling, offering an advantage in training complexity.
• Compatibility and Flexibility: New entities can be added by simply specifying their names, making the system versatile in dynamic knowledge environments.

Experimental Evaluation

GENRE's performance is evaluated across over 20 datasets covering tasks like entity disambiguation, end-to-end entity linking, and document retrieval:

• Entity Disambiguation: The model achieves competitive results across in-domain and out-of-domain scenarios with average improvements over existing methods.
• Entity Linking: In end-to-end linking, GENRE shows superiority in in-domain performance and robust out-of-domain adaptability, despite challenges with coreference datasets.
• Page-level Document Retrieval: Utilized on the KILT benchmark, GENRE establishes new state-of-the-art results across various tasks with significantly reduced memory requirements compared to other models.

Technical Insights

• Constrained Beam Search: This decoding strategy ensures generated outputs are valid entity names, improving retrieval accuracy without exhaustively scoring the entire candidate set.
• Structured Name Space Utilization: The autoregressive generation of structured and compositional entity names leverages the regularity and predictability in their interaction with context, providing a robust retrieval mechanism even with cold-start scenarios.

Implications and Future Directions

GENRE's introduction of a generative retrieval paradigm marks a significant shift in how entity linking and retrieval can be approached. By focusing on the structural properties of entity identifiers, this method not only reduces resource burdens but also enhances retrieval effectiveness in knowledge-dynamic contexts.

Future research could explore further applications of similar autoregressive and sequence generation models to other domains that involve large-scale entity management and retrieval tasks. Additionally, expanding the approach to handle enhanced language constructs such as multi-language identifiers or context-dependent naming conventions might push the boundary of its applicability and accuracy.

Overall, GENRE sets a foundation for more intelligent and efficient knowledge system architectures that could seamlessly integrate with evolving information landscapes.