On the Role of Tikhonov Regularizations in Standard Optimization Problems

Abstract: Tikhonov regularization is a common technique used when solving poorly behaved optimization problems. Often, and with good reason, this technique is applied by practitioners in an ad hoc fashion. In this note, we systematically illustrate the role of Tikhonov regularizations in two simple, yet instructive examples. In one example, we use regular perturbation theory to predict the impact Tikhonov regularizations have on condition numbers of symmetric, positive semi-definite matrices. We then use a numerical example to confirm our result. In another example, we construct an exactly solvable optimal control problem that exhibits a boundary layer phenomenon. Since optimal control problems are rarely exactly solvable, this brings clarity to how vital Tikhonov regularizations are for the class of problems this example represents. We solve the problem numerically using MATLAB's built-in optimization software and compare results with a Galerkin-type/genetic algorithm. We find the second method generally outperforms MATLAB's optimization routines and we provide the MATLAB code used to generate the numerics. Finally, to demonstrate the utility of Tikhonov regularization from the second example in a more technologically realistic application, we show that regularization guides the design of gradient-index optical devices toward those which are significantly less expensive to fabricate.