Learning Lineage Constraints for Data Science Operations

Abstract: Data science workflows often integrate functionalities from a diverse set of libraries and frameworks. Tasks such as debugging require data lineage that crosses library boundaries. The problem is that the way that "lineage" is represented is often intimately tied to particular data models and data manipulation paradigms. Inspired by the use of intermediate representations (IRs) in cross-library performance optimizations, this vision paper proposes a similar architecture for lineage - how do we specify logical lineage across libraries in a common parameterized way? In practice, cross-library workflows will contain both known operations and unknown operations, so a key design of XProv to link both materialized lineage graphs of data transformations and the aforementioned abstracted logical patterns. We further discuss early ideas on how to infer logical patterns when only the materialized graphs are available.