The Extrema of Multivariable Functions

Finding the extrema of a function of one variable means differentiating and setting the derivative to zero; the multivariable case is somewhat more involved: a stationary point is only a candidate, and you still have to decide whether it is a local maximum, a local minimum or a saddle point.

The necessary condition for an extremum is that all the partial derivatives vanish simultaneously — that is, that the gradient is zero, and this connects directly to the “gradient direction”. To classify the point you need the second-order derivatives: a positive definite Hessian means a local minimum, a negative definite one means a local maximum, and an indefinite one means a saddle point. This chapter gives the test in its precise form, and stresses that when the Hessian test fails you still need another method (such as an analysis along a specific direction).

The notion of a “saddle point” comes up again and again here — it is just as important in the loss functions of machine learning, and it is worth building an accurate intuition for it here.