L1 Course Introduction

Source: Annotated lecture PDF. Page references use PDF page numbers.

Overview

• Gradients describe local change; directional derivatives explain steepest ascent and descent.
• Convexity and Taylor approximations provide the language for optimisation.
• Matrices represent linear transformations relative to a chosen basis.

Gradients and directional derivatives

For a differentiable scalar function $f:{\mathbb{R}}^n\to \mathbb{R}$,

$$\nabla f(x)=\left[\begin{array}{c}\partial f/\partial x_1\\ ⋮\\ \partial f/\partial x_n\end{array}\right],D_{u}f(x)=\nabla f(x{)}^{T}u.$$

Each gradient component measures change along one coordinate axis. For a unit vector $u$, the directional derivative is the rate of change per unit distance in that direction. Without normalisation, the vector’s length also scales the rate.

Short derivation: why the dot product?

Define $g(h)=f(x+hu)$. The chain rule gives

$$g^{\prime }(0)=\sum _i\frac{\partial f(x)}{\partial x_i}{u}_i=\nabla f(x{)}^{T}u.$$

The annotated chain-rule calculation is on p. 9. The useful idea is to turn movement through several coordinates into a function of one step parameter, $h$.

For $\nabla f(x)\ne 0$ and $\Vert u{\Vert }_2=1$,

$$-\Vert \nabla f(x){\Vert }_2\le \nabla f(x{)}^{T}u\le \Vert \nabla f(x){\Vert }_2.$$

Thus the unit steepest ascent/descent directions are $\pm \nabla f(x)/\Vert \nabla f(x){\Vert }_2$. The comparison assumes equal-length directions.

Geometry: gradient perpendicular to a level curve

Key takeaway: along a level curve, $f$ stays constant, so its derivative along a tangent is zero. Consequently, a nonzero gradient is perpendicular to that tangent. The diagram and handwritten explanation preserve this geometry.

Worked example: direction versus distance

From p. 10, let $f(x,y)=350-2x^2-1.5y^2$. Then

$$\nabla f(1,1)=(-4,-3{)}^T,D_{(-5,-2)}f(1,1)=(-4)(-5)+(-3)(-2)=26.$$

This is the rate with respect to the step parameter. Per unit distance along the same direction it is $26/\sqrt{29}$. The greatest possible unit-direction increase is $5$, attained along $(-4,-3{)}^T/5$.

Convexity

A set $S$ is convex if every segment between two of its points stays in $S$:

$$tx+(1-t)y\in S(x,y\in S,0\le t\le 1).$$

A function on a convex domain is convex if

$$f(tx+(1-t)y)\le tf(x)+(1-t)f(y).$$

Its graph lies below the chord joining two graph points. See the diagrams on p. 13. This becomes useful in Lecture 2: convexity makes local minima global.

Taylor approximations and the Hessian

For a small displacement $s$ from $w$, first-order and second-order approximations are

$$f(w+s)\approx f(w)+\nabla f(w{)}^{T}s,$$ $$f(w+s)\approx f(w)+\nabla f(w{)}^{T}s+\frac{1}{2}{s}^T{\nabla }^{2}f(w)s.$$

The Hessian has entries $[{\nabla }^{2}f(w){]}_{ij}={\partial }^{2}f/\partial w_i\partial w_j$. The gradient supplies slope; the Hessian supplies curvature. $C^1$ means continuous first derivatives; $C^2$ means continuous second derivatives. These are local approximations, not exact descriptions of an arbitrary function far from $w$.

Source: Taylor approximations, p. 14 and Hessian, p. 15.

Bases, transformations and similarity

A basis $B=(b_1,\dots ,b_n)$ lets us express each vector uniquely as $v={\sum }_i{x}_i{b}_i$. Its coordinate column $[v{]}_B=(x_1,\dots ,x_n{)}^T$ depends on the basis and its ordering.

A transformation is linear when $T(ax+by)=aT(x)+bT(y)$. A matrix describes it once bases are chosen; matrix–vector multiplication applies it.

If $P$ contains the new basis vectors in old coordinates, then

$$x_{\mathrm{old}}=P{x}_{\mathrm{new}},A_{\mathrm{new}}=P^{-1}{A}_{\mathrm{old}}P.$$

The similarity formula represents the same transformation in different coordinates: convert to old coordinates, apply the transformation, then convert back. $P$ must be invertible. See p. 22.

Geometry: moving a point versus changing coordinates

Key takeaway: $\mathrm{diag}(2,3)(1,1{)}^T=(2,3{)}^T$ can describe moving a point with fixed axes. The same numerical coordinate change can instead arise by keeping the point fixed and shrinking the coordinate units to $1/2$ and $1/3$. Keep the geometric object separate from its coordinates; p. 26 develops the basis interpretation.

Important results

• Directional derivative: $D_{u}f=\nabla f^{T}u$; use unit vectors when comparing slopes per unit distance.
• Negative gradient gives Euclidean steepest descent when the gradient is nonzero.
• Taylor’s linear term predicts change; its quadratic term captures curvature.
• A matrix representation depends on the basis, while the underlying transformation need not change.