Preliminary Announcement

Instructor: Professor Andrej V. Cherkaev,  Department of Mathematics Office: JWB 225  Email: cherk@math.utah.edu Tel : +1 801 - 581 6822  Fax: +1 801 - 581 4148	Search for the Perfection:  An image from  Bridgeman Art Library

The desire for optimality (perfection) is inherent for humans. The search for extremes inspires mountaineers, scientists, mathematicians, and the rest of the human race. A beautiful and practical mathematical theory of optimization (i.e. search-for-optimum strategies) is developed since the sixties when computers become available. Every new generation of computers allows for attacking new types of problems and calls for new methods. The goal of the theory is the creation of reliable methods to catch the extremum of a function by an intelligent arrangement of its evaluations (measurements). This theory is vitally important for modern engineering and planning that incorporate optimization at every step of the complicated decision making process.

This course discusses classical direct search-for-optimum methods, such as Golden Mean, Conjugate Gradients, Modified Newton Method, methods for constrained optimization, including Linear and Quadratic Programming, and others. We will also briefly review genetic algorithms that mimic evolution and stochastic algorithms that account for uncertainties of mathematical models. The course work includes several homework assignments that ask to implement the studied methods and a final project, that will also be orally presented in the class.

The textbook Practical optimization by Philip Gill, Walter Murray, and Margaret H. Wright  is interesting and readable (a British colleague of my recommended it as "a bloody good book"); the authors are among the top experts in the field. The book discusses pros and cons of various methods, often in the context of specific applications. The review part of the course will also use the instructor's notes.

Prerequisite: Calculus, ODE, elementary programming.

To learn more about the course, please visit instructor's website
www.math.utah.edu/~cherk; then follow the online instructions.

Contents

 

Preliminary Plan  Some Frivolous Remarks  Introduction

Intro: Optimization and modeling  Intro: Assumptions of the theory Classification of optimization problems Links

Preliminary Plan

1. Introduction (1 week) Modelling and Optimization.

Types of problems and methods, examples. Course projects.
2. Extremum of a function of one variable (2 weeks)

Fibbonacci and Golden Mean method, Secant method, Newton method.
HW 1
3. Unconstrained optimization (4 weeks)

Gradient search, Newton method and its modifications, Conjugate Gradient method, flip-flop type methods. Example: Mean Square approximation and its modifications.
HW 2
4. Constrained optimization (4 weeks)

Lagrangians and Augmented Lagrangian Methods. Linear Programming. Quadratic programming. Nonlinear programming.
HW 3
5. Review of various methods (2 week)

Stochastic search. Genetic algorithms. Discrete methods. Game approach.
HW 4
6. Projects presentations (1 week)

 
The final grade is composed as follows: 60% - the homework, 40% - the project.

---

Go to Contents
Go to the top

Some Frivolous Remarks on Optimization

• The inherent human desire to optimize is cerebrated in the famous Dante quotation:

All that is superfluous displeases God and Nature
All that displeases God and Nature is evil.
In engineering, optimal projects are considered beautiful and rational, and the far-from-optimal ones are called ugly and meaningless. Obviously, every engineer tries to create the best project and he/she relies on optimization methods to achieve the goal.
• The essence of an optimization problem: Catching a black cat in a dark room in minimum time.

(A constrained optimization problem corresponds to a room full of furniture.) A dim light is needed: Hence optimization methods explore assumptions about the character of response of the goal function to varying parameters and suggest the best way to change them. The variety of a priori assumptions corresponds to the variety of optimization methods. This variety explains why there is no silver bullet in optimization theory.
• Optimization theory is developed by ingenious and creative people, who regularly appeal to vivid common-sense association, formulating them in a general mathematical form. For instance, the theory steers numerical searches through canyons and passes (saddles), towards the peaks; it fights the curse of dimensionality, models evolution, gambling, etc. The optimizing algorithms are mathematical models of intellectual and intuitive decision-making.
• The general principle by Maupertuis proclaims:    If there occur some changes in nature, the amount of action necessary for this change must be as small as possible. This principle rightly proclaims that the nature always finds the best way to reach a goal (see calculus of variations webpage for the discussion). The trust in this principle leads to an interesting inverse optimization problem: find the essence of optimality of a natural "project."


 
 

Introduction to Optimization

Everyone who studied calculus knows that an extremum of a smooth function is reached at a stationary point where its gradient vanishes. Some may also remember the Weierstrass theorem which proclaims that the minimum and the maximum of a function in a closed finite domain do exist. Does this mean that the problem is solved?

A small thing remains: To actually find that maximum. This problem is the subject of the optimization theory that deals with algorithms for search of the extremum. More precisely, we are looking for an algorithm to approach a proximity of this maximum and we are allowed to evaluate the function (to measure it) in a finite number of points. Below, some links to mathematical societies and group in optimization are placed that testify how popular the optimization theory is today: Many hundreds of groups are intensively working on it around the globe.

---

Go to Contents
Go to the top

Optimization and Modeling

The modeling of the optimizing process is conducted along with the optimization. Inaccuracy of the model is emphasized in optimization problem, since optimization usually brings the control parameters to the edge, where a model may fail to accurately describe the prototype. For example, when a linearized model is optimized, the optimum often corresponds to infinite value of the linearized control. (Click here for an example) On the other hand, the roughness of the model should not be viewed as a negative factor, since the simplicity of a model is as important as the accuracy. Recall the joke about the most accurate geographical map: It is done in the 1:1 scale.

Unlike the models of a physical phenomena, an optimization models critically depend on designer's will.  Firstly, different aspects of the process are emphasized or neglected depending on the optimization goal. Secondly, it is not easy to set the goal and the specific constains for optimization.

Naturally, one wants to produce more goods, with lowest cost and highest quality. To optimize the production, one either may constrain by some level the cost and the quality and maximize the quantity, or constrain the quantity and quality and minimize the cost, or constrain the quantity and the cost and maximize the quality. There is no way to avoid the difficult choice of the values of constraints.  The mathematical tricks go not farther than: "Better be healthy and wealthy than poor and ill". True, still not too exciting.

It seems that the maximization of the profit solves the problem by applying an universal criterion. Still, the short-term and long-term profits require very different strategies; and it is necessary to assign the level of insurance, to account for possible market variations, etc.

Sometimes, the solution of an optimization problem shows unexpected features: for example, an optimal trajectory zigzags infinitely often. Such behavior points to an unexpected, but optimal behavior of the solution. It should not be rejected as a mathematical extravaganza, but thought through! (Click here for some discussion.)

Go to Contents
Go to the top

---

Assumptions of the Theory

A smoothness or sometimes the convexity of the function is a priori assumed. Without some assumptions, no rational algorithms can be suggested. For example, it is impossible to suggest a nontrivial algorithm for choosing the oldest person from the alphabetical telephone directory. The search methods approximate -- directly or indirectly -- the behavior of the function in the neighborhood of measurements. Various methods assume different types of the approximation.

My maximum is higher than your maximum! The optimized function may have more than one local maximum. Generally, there are no ways to predict the behavior of the function everywhere in the permitted domain. Most of the methods pilot the search to a local maximum without a guarantee that this maximum is also a global one.

Several classical optimization problems are of special interest. Those are: maximum of an one-dimensional unimodal function, the mean square approximation, linear and quadratic programming. Those problems and their modifications serve as testing grounds for optimization algorithms and they used for algorithms evaluation

• Numerous optimization methods are designed for various number of independent controls (dimensionality). These ranges corresponds to one variable, several (two to ten), dozens, hundreds, or tens of thousands.
• An important special class -- discrete optimization -- corresponds to a large number of controls that take only integer or Boolean values.
• Another special case -- calculus of variations -- corresponds to infinitely many controls.
• A significant parameter is the cost of evaluation of the function: It ranges from milliseconds of the computer time to lengthy and costly measurements in natural experiments. Accordingly, the realistic goals and the methods are different.
• The sharpness of the maximum is of a special interest. It is very difficult to find a maximum of a flat function especially in the presence of numerical errors. For instance, a point of the maximum elevation in a plain is much harder to determine than to locate a peak in the mountains. A sharp maximum also requires some special methods since the gradient is discontinuous; it does not exist in the optimal point.
• The presence of constraints and their character is very important for the choice of the strategy. The constrained problem typically possess the sharp maximum in a corner of the permitted domain while the smooth unconstrained function reaches its maximum in a stationary point.
•  Stochastic optimization deals  with noise in measurements or uncertainty of some factors in the model. The optimization process should account for possibly nonaccurate data (click to see an example). Here, the goal may be to maximize the expected profit.
• Another approach plays the game: Design versus uncertainty, considering for the worst possible scenario. Find here an example of such game. ,
• Structural Optimization is an interesting and tricky class of optimization problem. It is characterized by a large number of variables which represent the shape of the design and and stiffness of the available materials.