Towards a Systems Theory of Algorithms (2401.14029v2)

Abstract: Traditionally, numerical algorithms are seen as isolated pieces of code confined to an {\em in silico} existence. However, this perspective is not appropriate for many modern computational approaches in control, learning, or optimization, wherein {\em in vivo} algorithms interact with their environment. Examples of such {\em open algorithms} include various real-time optimization-based control strategies, reinforcement learning, decision-making architectures, online optimization, and many more. Further, even {\em closed} algorithms in learning or optimization are increasingly abstracted in block diagrams with interacting dynamic modules and pipelines. In this opinion paper, we state our vision on a to-be-cultivated {\em systems theory of algorithms} and argue in favor of viewing algorithms as open dynamical systems interacting with other algorithms, physical systems, humans, or databases. Remarkably, the manifold tools developed under the umbrella of systems theory are well suited for addressing a range of challenges in the algorithmic domain. We survey various instances where the principles of algorithmic systems theory are being developed and outline pertinent modeling, analysis, and design challenges.

• The paper introduces a paradigm shift by modeling algorithms as open, interactive dynamical systems rather than isolated code blocks.
• It employs systems theory to analyze stability, error propagation, and feedback interactions in real-time, offloading traditional limitations.
• The framework offers practical insights for optimizing control, distributed learning, and data pipelines in complex computational environments.

Towards a Systems Theory of Algorithms

The paper "Towards a Systems Theory of Algorithms" offers an expert perspective on the evolving nature of algorithms in various computational domains. Rather than viewing algorithms as isolated code blocks, the authors propose a paradigm where algorithms are considered as open dynamical systems. This approach aligns with the increasing interactivity and complexity of modern computational systems, particularly in areas such as control, optimization, and learning.

The core argument presented is that the traditional viewpoint—where algorithms are closed, autonomous, and isolated—fits inadequately with contemporary needs. Instead, the authors advocate for a model where algorithms are open systems, interacting with their environment, which includes other algorithms, physical systems, humans, or databases. This shift is depicted through a systems-theoretic lens, offering insights and tools that can be applied to paper algorithmic interconnections and complexity.

Algorithmic Perspectives

The authors delineate between the traditional and proposed viewpoints of algorithms:

• Traditional Viewpoint:
  • Closed and autonomous execution.
  • Operates offline, often with batch data.
  • Considered perfect in execution until predetermined termination.
  • Viewed as monolithic and isolated from external interactions.
• Proposed Viewpoint:
  • Open with real-time data interaction and outputs.
  • Executed online in a causal manner.
  • Subject to execution imperfections and early termination.
  • Contains internal structures that facilitate external interfacing.

The paper suggests that adopting a systems theory perspective enables a more holistic understanding of the roles and interactions of algorithms within broader systems. This approach allows for addressing issues of stability, uncertainty propagation, and further algorithmic challenges.

Practical and Theoretical Implications

The authors highlight several areas where this systems-theoretic framework is not only applicable but advantageous:

• Optimization and Control: The paper discusses the application of systems theory in real-time algorithms, such as those used in model predictive control or distributed optimization. By modeling these algorithms as dynamical systems, it is possible to analyze their stability and performance rigorously.
• Machine Learning and Data Pipelines: In learning systems, algorithms are often components within larger architectures. Viewing these as interconnected systems can help in understanding error propagation and transient performance.
• Feedback Interconnections: Algorithms frequently operate within feedback loops, interacting with dynamic environments. Systems theory provides a robust framework for studying these feedback interactions, offering tools like small-gain theorems and dissipativity analysis.

Challenges and Future Directions

The paper identifies several grand challenges that can be approached from a systems-theoretic perspective, including:

• Modeling algorithmic interactions with complex environments, particularly when algorithms influence their own data sources (performative prediction).
• Architecting and analyzing pipelines and control stacks, exploring modularity and error propagation.
• Developing methodologies for dealing with uncertainty and convergence in feedback-laden algorithmic systems.

Conclusion

The authors propose a shift in perspective towards viewing algorithms as dynamic, open systems. This paradigm has potential implications across control, optimization, machine learning, and beyond. While systems theory offers valuable tools and insights, the authors caution against simplistic applications of these concepts and suggest integrating methods from computer science and other fields for a comprehensive approach. The paper aims to stimulate research and practical applications within this emerging paradigm, positioning the control and systems community to engage with these contemporary algorithmic challenges.