Binary metrics, visually

Many binary classification metrics are ratios of counts from one 2 × 2 table.

The confusion matrix

Goal: describe how well a binary classifier (a coin flip, a medical test) does, with a number that answers the question you care about.

Approach: count the samples in each of the four cells: actual class (row) by prediction (column). Every metric divides some of the cells by a row, a column or the whole table.

Try this

Edit the counts in the table to change the prior: move samples from the Actual 0 row to the Actual 1 row. Which metrics move, and which stay put?
In the Mosaic view, drag the line between the rows. Which metrics move, and which stay put?
On Mammogram, you can raise either TPR or TNR by one percentage point. Which would you rather raise to improve accuracy?
Answer
TNR, by far. Accuracy = prior × TPR + (1 − prior) × TNR, so each row counts in proportion to its size. One more point of TPR helps 1% of the 10 women with cancer (0.1 more right; accuracy +0.01 points); one more point of TNR helps 1% of the 990 without (9.9 more right; +0.99 points), 99 times as much. Accuracy barely notices the rare class, even though a missed cancer is the costlier mistake.

Rows to scale

The ROC curve

Goal: rate a classifier with one number that does not depend on picking a threshold.

Approach: a classifier gives each sample a score and estimates 1 when score ≥ t. Each threshold t gives one confusion matrix, so one point (FPR, TPR). Sweeping t from high to low traces the ROC (receiver operating characteristic) curve; the area under it (AUC) scores every threshold at once.

Try this

Press Sweep, or drag the threshold from right to left. Why does the curve only ever move up or right?
Answer
Moving the threshold left only turns predictions of 0 into 1, never the reverse. A Actual 1 sample now predicted 1 is one more catch, so TPR rises: the curve steps up. A Actual 0 sample now predicted 1 is one more false alarm, so FPR rises: the curve steps right. Neither rate can ever fall.
Set the separation to 0. Where does the curve go?
Answer
Near the diagonal, with AUC near 0.5: the scores carry no information about the class, so at any threshold the classifier flags the same share of each class (TPR ≈ FPR). That is guessing.
Press Estimate backwards: 1 is now predicted where the score is low. What happens to the curve and the AUC?
Answer
The curve mirrors through the center, below the diagonal: each point (FPR, TPR) becomes (1 − FPR, 1 − TPR), and AUC becomes 1 − AUC. A classifier that is worse than guessing is still useful: estimate the opposite.
Move the prior from 50% to 5%. Does the ROC curve change shape?
Answer
Hardly: TPR and FPR each use one true class, so the prior does not enter (the wiggles are sampling noise).

Drag the threshold on the scores, or drag along the curve. The arrow keys move it one sample. Rest the pointer on a metric at right for what it means in plain English.