Directional Derivatives and the Gradient

Once differentiability is in place, the rate of change of a function in an arbitrary direction has a unified definition. The directional derivative is the projection of the total differential onto a unit direction, while the gradient organises these infinitely many directional derivatives into a single vector.

The most important conclusion of this chapter is: the gradient direction is the direction in which the directional derivative attains its maximum, and its magnitude is that maximum. In other words, the gradient points in the direction in which the function “rises fastest”. The applications section explains why optimisation algorithms go in the direction opposite to the gradient, and why the gradient is always perpendicular to a contour line.

Understanding the directional derivative and the gradient separately is worthwhile: the former is a scalar and depends on the choice of direction, whereas the latter is a unique vector that does not.