Adjoint and Its roles in Sciences, Engineering, and Mathematics: A Tutorial

Abstract: This paper synergizes the roles of adjoint in various disciplines of mathematics, sciences, and engineering. Though the materials developed and presented are not new -- as each or some could be found in (or inferred from) publications in different fields -- we believe this is the first effort to systematically unify these materials on the same mathematical footing starting from the basic definitions. We aim to provide a unified perspective and understanding of adjoint applications. As a result, this work could give broader views and better insights into the application of adjoint beyond a single community. By rigorously establishing general results and then developing materials specific to each application, we bring forth details on how abstract concepts/definitions can be translated into particular applications and the connections among them. This paper is written as an interdisciplinary tutorial on adjoint with discussions and with many examples from different fields including linear algebra (e.g. eigendecomposition and the singular value decomposition), ordinary differential equations (asymptotic stability of an epidemic model), partial differential equations (well-posedness of elliptic, hyperbolic, and Friedrichs' types), neural networks (backpropagation of feed-forward deep neural networks), least squares and inverse problems (with Tikhonov regularization), and PDE-constrained optimization. The exposition covers results and applications in both finite-dimensional and infinite-dimensional Hilbert spaces.