Logistic regression, visually

Draw a line on the log-odds scale, squash it into a probability, and train it by walking downhill.

Estimating class probabilities (a classifier) with a regression model

Problem (direct approach): if we estimate P̂(Y = 1 | x) = w · x, the estimates aren't bounded. If some sample x′ gets an 80% chance of class 1, P̂(Y = 1 | x′) = w · x′ = 0.8, then doubling it gives a 160% chance: P̂(Y = 1 | 2x′) = w · 2x′ = 1.6. The linearity is the problem!

Approach: instead of using regression to estimate the probability directly, use it to estimate the log odds z = ln P(Y = 1 | x)P(Y = 0 | x), which can be any number. With one feature, that gives the model

z = w₀ + w₁x ⇒ p̂ = P̂(Y = 1 | x) = σ(z) = 11 + e^−z

Predict 1 when p̂ > 0.5, which is when z > 0. The decision boundary is the x where the line crosses 0: x = −w₀ / w₁.

A sample's loss is −ln of the probability the model gave its true class; the loss of w is the mean over the samples (the cross-entropy, in nats).

Training: walk downhill on the loss

Goal: choose w₀ and w₁ with the smallest mean cross-entropy loss. Unlike MSE, there is no formula for the best w.

Approach: the same as for MSE. Start anywhere, step against the gradient, and repeat.

loss(w) = −1n ∑_i [ y_i ln p̂_i + (1 − y_i) ln(1 − p̂_i) ]

∇loss(w) = 1n ∑_i (p̂_i − y_i) 1x_i w ← w − η ∇loss(w)

This has the same form as the MSE gradient −2n∑ r_i1x_i, with the error now p̂_i − y_i: the red segment from each sample to the sigmoid. The loss is in nats (ln). In bits (log₂, as on the problem sets) every loss and gradient is 1/ln 2 ≈ 1.44 times larger: the same surface, and the same walk with η times ln 2.

Try this

Hover a sample in the data strip and follow it: its x gives a score z on the line (left), and the sigmoid turns that into p̂ (right).
In the log-odds panel, drag the right square handle up. What happens to the sigmoid? Now drag it below the left one.
Answer
A larger slope w₁ makes the sigmoid steeper. A negative w₁ flips it: p̂ now falls as x grows.
Read the right edge of the log-odds panel: what probability is z = 2? z = 4? Find the same points on the sigmoid.
Answer
0.88 and 0.98. Each +1 of log odds multiplies the odds p̂ / (1 − p̂) by e ≈ 2.7.
Fit w by hand: make the loss as small as you can. Then check Show best fit.

From w = 0 (every p̂ = 0.5), press Gradient step a few times. Which samples pull hardest on w?
Answer
The badly predicted ones, with the longest red segments p̂_i − y_i. A sample already predicted close to its label barely counts.
Press Run, then hover the trail on the surface to see the sigmoid at each step.
Switch the right panel to Log odds and step. The line moves as in regression, but what is it fitting?
Answer
Not the samples themselves: their log odds would be ±∞. The loss only cares about σ(z) at each sample.
Raise η far to the right and press Run. What does the walk do?
Answer
It overshoots the valley and bounces around it without settling: the loss chart turns to noise. Each gradient is an average of (p̂_i − y_i) [1, x_i], which is bounded, so unlike MSE the walk never blows up.
Slide Separation up until the badge says the classes are separable, then press Run. What happens to w, the loss and the sigmoid?
Answer
w keeps growing along a ray: the boundary settles in the gap while the sigmoid sharpens toward a step. The loss falls toward 0 but never gets there, so the run never converges. Regularization (lesson 13) fixes this by charging for large w.
With separable data, click a sample deep inside its class to flip it, then Run again.
Answer
The classes overlap again, so the loss has a minimum and the walk converges.
Drag a sample far into the other class's territory. Watch the surface's bottom move.

Data (x_i, y_i)

Log odds

In either panel, drag a square handle to tilt the line, or empty space to shift it. The probability panel plots against z (the sigmoid itself) or x. Drag a sample sideways; click it to flip its class.

Logistic regression, visually

Estimating class probabilities (a classifier) with a regression model

Training: walk downhill on the loss

Try this

Data (x_i, y_i)

Log odds

Probability

Loss surface

Loss by step

Estimating class probabilities (a classifier) with a regression model

Training: walk downhill on the loss

Try this

Data (xi, yi)

Log odds

Probability

Loss surface

Loss by step

Data (x_i, y_i)