Projections, visually — onto a line and onto a plane

bgiven vector (target y)

aspans a line (feature column x)

pprojection: the part of b in span(a) (fit ŷ)

eerror: the part of b not in span(a) (residual)

Drag the tips of a and b

Projecting onto a line

All multiples ca form a line, the span of a. The projection p is the point on that line closest to b.

At the closest point, the error e = b − p meets the line at a right angle:

a · (b − ca) = 0 ⇒ c = a·ba·a, p = ca

Try this

Drag b onto the dashed line: e shrinks to zero and p = b.
Make b perpendicular to a: the projection is 0.
Stretch or flip a without changing its line: p stays put and only c changes. The projection depends on the span, not on a itself.

Least squares: with one feature and no intercept, a is the feature column x, b is the target y, and c = x·y / x·x is the fitted slope β̂.

Snap to grid Show span of a Values on vectors Show legend

bgiven vector (target y)

a₁ a₂span a plane (feature columns of X)

pprojection: the part of b in span(a₁, a₂) (fit ŷ)

eerror: the part of b not in span(a₁, a₂) (residual)

Drag to rotate · drag a tip to move a vector · scroll to zoom

Projecting onto a plane

a₁ and a₂ span a plane. Put them side by side as the columns of A = [a₁ a₂]. Every point in the plane is then Ac = c₁a₁ + c₂a₂.

The closest point p = Ac leaves an error e = b − p perpendicular to the whole plane, so to both a₁ and a₂:

A^T(b − Ac) = 0 ⇒ A^TA c = A^Tb

These are the normal equations. Solving them gives c = (A^TA)⁻¹A^Tb.

Try this

Click Edge-on to look along the plane: the right angle between e and the plane is plain to see.
Drag the tip of b into the plane: e vanishes.
Click a₂ ← a₁ + a₂. The plane is the same, so p doesn't move, but c₁ and c₂ change.

Least squares: A is the design matrix X (one column per feature), b is y, c is β̂, p is the fit ŷ, and e is the residual. With n samples the same picture lives in ℝⁿ; here n = 3, so we can look at it.

Show c₁a₁ + c₂a₂ Values on vectors Show legend Auto-rotate