Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems (1903.12394v3)

Abstract: Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for informed machine learning which illustrates its building blocks and distinguishes it from conventional machine learning. We introduce a taxonomy that serves as a classification framework for informed machine learning approaches. It considers the source of knowledge, its representation, and its integration into the machine learning pipeline. Based on this taxonomy, we survey related research and describe how different knowledge representations such as algebraic equations, logic rules, or simulation results can be used in learning systems. This evaluation of numerous papers on the basis of our taxonomy uncovers key methods in the field of informed machine learning.

• The paper introduces a comprehensive taxonomy that classifies methods by knowledge source, representation, and integration.
• It details techniques to incorporate algebraic equations, differential equations, simulations, and logic rules into ML pipelines.
• It highlights future directions for adaptive learning systems that balance domain-specific knowledge with data-driven insights.

Integrating Prior Knowledge into Machine Learning: A Taxonomic Perspective

The paper "Informed Machine Learning -- A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems" offers a comprehensive overview of integrating prior knowledge into machine learning. This concept, termed informed machine learning, addresses the limitations of purely data-driven models, especially in scenarios with insufficient training data or the necessity for models to adhere to specific physical constraints.

Taxonomy Development

The paper introduces a well-structured taxonomy that categorizes informed machine learning approaches along three primary dimensions: knowledge source, knowledge representation, and knowledge integration. This classification framework is based on an extensive literature survey, providing a systematic overview of the approaches in the field.

Knowledge Source

Categorized sources include:

• Scientific Knowledge: Predominantly formalized rules from domains like physics or biology.
• World Knowledge: Common everyday knowledge often observed implicitly or through linguistic rules.
• Expert Knowledge: Intuitive insights often used in specific domains such as engineering or medicine.

Knowledge Representation

The paper identifies several representation methods:

• Algebraic Equations: Used to represent relationships between variables, these are frequently integrated into learning algorithms as constraints.
• Differential Equations: Common in dynamic systems modeling, these are often integrated into neural networks through gradient-based methods.
• Simulation Results: These are usually employed to enhance or generate synthetic training data.
• Spatial Invariances: Integration into models to reflect transformations like rotation or translation, typical in image classification tasks.
• Logic Rules and Knowledge Graphs: Used for structuring models or influencing their learning paths, leveraging techniques such as neural-symbolic systems.

Knowledge Integration

Knowledge is woven into machine learning systems through various facets of the pipeline:

• Training Data Augmentation: Utilizing simulations to generate synthetic data.
• Hypothesis Set Structuring: Informing the architecture of models or defining hyperparameters.
• Learning Algorithm Adjustment: Incorporation through constraints or regularizers in loss functions.
• Final Hypothesis Validation: Ensuring that predictions adhere to known theoretical constraints.

Methodological Insights

The authors emphasize integrating structured knowledge such as algebraic and differential equations within learning algorithms to act as constraints or augmentations to traditional loss functions. The integration of simulations into training data showcases the growing importance of hybrid models leveraging both data-driven and knowledge-based insights.

Furthermore, the role of knowledge graphs and probabilistic relations in neural architectures suggests potential pathways for combining human-like reasoning with algorithmic precision, heralding advances in relational understanding and semantic integration.

Implications and Future Directions

The theoretical benefits of informed machine learning are evident, potentially reducing the reliance on massive training datasets and advancing model interpretability and performance. Practically, these integrative approaches promise improvements across fields with well-established knowledge bases but limited data availability.

The paper suggests several challenges, including the accurate weighting of knowledge versus data-driven insights and the integration of complex domain-specific knowledge into flexible learning frameworks. Future research may focus on adaptive learning algorithms that dynamically balance these sources of information, along with the development of robust frameworks for the seamless integration of symbolic and connectionist methods.

In conclusion, informed machine learning presents a pragmatic expansion of contemporary machine learning paradigms, bridging empirical data techniques with structured domain-specific knowledge to enhance model efficacy and reliability. As indicated by the paper, the methodical expansion of this field promises significant advancements in the application and theoretical foundations of AI systems.