Loss Functions

Topic — Every fitted model is the answer to "which parameters minimise this quantity", and the quantity is something you choose. This page argues that the choice is not a detail of the reporting: it changes the fitted parameters, it changes which points can influence them at all, and — for one extremely natural-looking choice — it makes the problem unsolvable by any gradient method.

---

In plain words

A model has no idea what "good" means until you tell it. The loss function is how you tell it: a formula that turns one prediction into a penalty. Training is then nothing more than hunting for the parameters that make the total penalty as small as possible.

Everything on this page follows from three consequences of that.

1. Change the penalty and you get a different model. Not better or worse — different. The same thirty points, fitted three ways, gave three different slopes.
2. Some penalties ignore how big a mistake is. Absolute error only asks which side of the line a point landed on, never how far. That makes it immune to a wild outlier — and equally deaf to a large value that was real signal.
3. Some penalties cannot be used at all. "Count the mistakes" sounds like the obvious thing to minimise. It can't be done, and the reason is worth having a picture for.

That third one is the least intuitive, so here is the picture. Training works like a hiker in thick fog who can only feel the slope of the ground under their boots. On a hillside that is enough — they always know which way is down. But put them on a staircase: every spot they can feel is perfectly flat, so they have no reason to move in any direction, even though there are steps all around them. Counting mistakes builds a staircase. Squared error and log loss build hillsides.

The fix, used everywhere in machine learning, is to minimise a smooth stand-in (a surrogate loss) and report the count you actually cared about afterwards. That is the single most useful idea here.

tip

How to read this page

Every claim below is backed by a printed number, often to six or seven digits. On a first pass, read the bold sentences and the section headings and let the digits go past — they are there so you can verify the claim later, not so you can memorise them. The Summary at the bottom collects them if you want them.

Words used on this page

Term	What it means here
Residual r	yᵢ − ŷᵢ — actual minus predicted, for one example
Gradient	The derivative of the cost with respect to each parameter
Leverage	How far a point's x sits from the average x. Points at the far ends can tilt the line; points in the middle barely can
Subgradient	What you use where a loss has a kink instead of a smooth slope, e.g. |r| at r = 0
Surrogate loss	A smooth loss you minimise because the thing you care about (accuracy, F1) has no usable slope
Calibrated	Of the cases predicted at 0.7, about 70% actually happen
Margin m	y·f(x) — positive when the prediction is on the correct side, larger when further from the boundary

---

The loss is the specification, not a detail

The words get used loosely, so it is worth being precise once.

Term	Applies to	Definition
Loss	One example	The penalty for one prediction, e.g. (ŷᵢ − yᵢ)²
Cost / objective	The whole dataset	The aggregate that is minimised, e.g. (1/n)Σ(ŷᵢ − yᵢ)²
Metric	The whole dataset	What you report. Need not be differentiable, and need not match the loss

Training minimises the cost. Reporting quotes the metric. They are frequently different quantities, and the reason is the whole of this page: many good metrics cannot be minimised, so you minimise a differentiable stand-in and report the metric afterwards.

The error measures computed earlier were all metrics — summaries of residuals after a fit. Squared error was doing double duty there: it was also the thing being minimised. That coincidence is the exception.

---

Three losses, three different lines

Thirty points on y = 2x + 5 with light noise. Fit three times, changing only what is minimised: squared error (LinearRegression), absolute error (QuantileRegressor(quantile=0.5)), and Huber (HuberRegressor). Then corrupt exactly one target to 60 — first a point in the middle of the x range, then the point at the end.

Output

1. ONE DATASET, THREE LOSSES   (truth: slope 2.0, intercept 5.0)
   loss      clean slope     int    centre-out     int    edge-out     int
   squared        1.9490    5.16        1.9780    6.51      2.6066    3.04
   absolute       1.9752    4.94        1.9432    5.25      1.9752    4.94
   Huber          1.9526    5.14        1.9531    5.16      1.9581    5.12

The clean fits already differ — 1.9490, 1.9752, 1.9526 — on identical data. Three losses, three answers, none of them wrong. That is the first point and it is easy to skip past: there is no such thing as "the" line of best fit until you say what "best" means.

Now the corruption. Read the centre-out columns first, because they are the ones that look wrong.

The centre outlier barely moved any slope. Squared error went 1.9490 → 1.9780; absolute error actually moved further, 1.9752 → 1.9432. If you were testing robustness, this comparison would tell you squared error is more robust than absolute error, which is false.

The reason has nothing to do with the loss. A point at x ≈ x̄ has almost no leverage — it contributes nearly zero to Sxy and Sxx, so it can hardly tilt the line at all. What it can do is lift the line, and it does: the squared-error intercept moved 5.16 → 6.51.

The edge outlier is where the losses separate. Same corruption, now on the highest-leverage point: squared error's slope goes 1.9490 → 2.6066, a 34% error. Absolute error goes 1.9752 → 1.9752. Huber, 1.9526 → 1.9581.

danger

Testing outlier robustness with a low-leverage point measures nothing

Both corruptions moved y by the same amount. One produced a +0.0290 slope shift and the other +0.6576, from the same loss. The variable being tested was position, not the loss function.

This is the same failure mode as the 1/√n measurement done with one fit per sample size — a plausible experiment whose design cannot detect the effect it is looking for. It was the first version of this demo, and it produced a confident, backwards conclusion.

---

Absolute error does not care how wrong an outlier is

1.9752 → 1.9752 is worth more than a passing note. It is not "small"; it is exact, and there is a clean reason.

The plain version first. Squared error asks each point "how far off am I?" and pulls on the line in proportion to the answer — so a point that is ten times further away pulls ten times harder. Absolute error asks only "am I above the line or below it?" Every point above the line pulls up by the same fixed amount, every point below pulls down by the same fixed amount, and that is the entire input. Move a point from just above the line to miles above the line and the answer to the only question being asked has not changed, so nothing about the fit changes either.

In symbols, the gradient of absolute-error cost is:

$\frac{\partial}{\partial\theta} \sum \lvert y_i - \theta^T x_i \rvert = -\sum \text{sign}(r_i) \cdot x_i$

Every residual enters through sign(rᵢ) and nothing else. The magnitude of a residual is absent from the gradient entirely. So moving a point further from the line changes the fit only if it changes which side of the line that point is on:

Output

   absolute loss depends only on sign(residual), never on its size:
   sign(r[15]) clean -1 -> corrupted +1   FLIPPED   (row 15)
   sign(r[29]) clean +1 -> corrupted +1   unchanged   (row 29)

   push the high-leverage point further and further away:
      y[29]   absolute slope   squared slope
         60         1.975169          2.6066
        200         1.975169          5.2260
      10000         1.975169        188.5808

Row 15's residual was negative under the clean fit and became positive when set to 60 — the subgradient changed, so the fit moved. Row 29's residual was already positive and stayed positive — the subgradient is identical, so the fit does not move at all.

The consequence is the strongest number on this page. Pushing that point to 200 and then 10000: absolute error returns 1.975169 all three times, to every printed digit. Squared error returns 2.6066, 5.2260, 188.5808. One loss is indifferent to how wrong the outlier is; the other is approximately proportional to it.

note

This is also absolute error's weakness, stated honestly

Ignoring magnitude is exactly what makes the median fit robust, and exactly what makes it insensitive — it discards real information when the large residual is genuine signal rather than corruption. A point at y = 10000 might be the most informative row in the dataset. Absolute error will not notice.

Robustness is not a free upgrade; it is a decision to stop listening to extremes.

Where the difference comes from

The per-residual arithmetic explains all of it:

Output

2. WHAT ONE RESIDUAL COSTS UNDER EACH LOSS
       r   squared  absolute  Huber d=1
     0.0      0.00      0.00       0.00
     0.5      0.25      0.50       0.12
     1.0      1.00      1.00       0.50
     2.0      4.00      2.00       1.50
     5.0     25.00      5.00       4.50
    10.0    100.00     10.00       9.50
     sum    130.25     18.50      16.12
   the single r = 10 is 76.8% of the squared total, 54.1% of the absolute total

Under squared error the single r = 10 accounts for 76.8% of the total. The optimiser is therefore mostly working on that one point, and the fit it produces is mostly a compromise with it. Under absolute error the same residual is 54.1% of the total. Same data, and the objective has a different opinion about what matters.

Note the crossover at r = 1: below it, squared error penalises less than absolute error (0.25 against 0.50), above it, much more. Squared error is lenient about small misses and severe about large ones, which is a real modelling statement, not a technicality.

Huber is the deliberate hybrid: quadratic within δ of zero, linear beyond it. At r = 0.5 it reads 0.12 (quadratic, ½r²); at r = 10 it reads 9.50 (linear, |r| − ½). That gives it squared error's smooth gradient near the optimum and absolute error's bounded influence far from it, which is why its slope moved only 1.9526 → 1.9581.

Loss	Formula	Optimises for	Gradient	Outlier influence
Squared (L2)	$r^2$	The mean	$2r$ — proportional to error	Unbounded
Absolute (L1)	$\lvert r \rvert$	The median	$\text{sign}(r)$ — constant	Bounded, and magnitude-blind
Huber	$\frac{1}{2}r^2$ or $\delta(\lvert r \rvert - \delta/2)$	Between the two	$r$ then $\delta \cdot \text{sign}(r)$	Bounded beyond $\delta$

tip

Squared error fits the mean; absolute error fits the median

This is the cleanest way to remember the difference. Minimising $\sum(y - c)^2$ over a constant $c$ gives $c = \bar{y}$; minimising $\sum \lvert y - c \rvert$ gives $c = \text{median}(y)$. The regression versions inherit exactly those properties, including the median's indifference to how extreme the extremes are.

The corollary matters for reporting: if you fit with absolute error, do not be surprised when the residuals do not sum to zero. That is a property of least squares, not of fitting in general.

---

The Loss Functions Sandbox

Experiment with the interactive tool below. Modify the residual $r$ and the Huber threshold $\delta$ to watch how the penalty cost of a single residual builds under L2 (Squared), L1 (Absolute), and Huber losses, visualizing their exact curves and crossover thresholds.

Loss Curves Sandbox

Single Residual (r): 2.0

Huber Threshold (δ): 1.5

-- L2 (Squared)-- L1 (Absolute)● Huber (Hybrid)

L2 (Squared) Cost: 4.00 (r²)

L1 (Absolute) Cost: 2.00 (|r|)

Huber Cost (δ = 1.5): 1.88

Linear robust region (δ(|r| - δ/2))

---

The loss decides whether the problem is solvable at all

Everything above concerned which answer you get. This section is about whether you get one.

This is the staircase from the opening, measured. The claim to test: a loss made of counts has no slope to follow, so a gradient method cannot move at all. The way to test it is to nudge one weight by a hair and ask whether the loss changed. If nothing changed, the slope is zero and the optimiser is standing on a flat step.

For classification, the obvious thing to minimise is the number of mistakes. It is what you care about, it needs no parameters, and it is the metric you will report. Take a two-feature binary problem, sweep one weight across 2000 values, and compute the numerical derivative of misclassification count at each:

Output

3. SOLVABILITY   (gradient surveyed over 2000 values of w1, w2 held at 0.3)
   0/1 loss   1998 of 2000 gradients are exactly 0.0   max |gradient| = 833.3333
   log loss      0 of 2000 gradients are exactly 0.0   max |gradient| = 0.6903
   200 gradient descent steps from the same start (12/60 correct):
      minimising 0/1 loss -> 12/60      minimising log loss -> 60/60

1998 of 2000 gradients are exactly 0.0. Not small — zero, bit-for-bit. Misclassification count is a step function: nudging a weight slightly changes no prediction at all, so the count is locally constant everywhere except at the 2 sampled points where some example crosses the boundary. There, the derivative is 833.3333 — one example flipping (1/60) divided by the step size (2 × 10⁻⁵).

So the derivative of 0/1 loss is either exactly zero or an artefact of the step size. It is never informative. Compare log loss: 0 of 2000 gradients are zero, and the largest is 0.6903 — a bounded, meaningful number at every point.

Running gradient descent on each confirms the consequence. From an identical, deliberately bad start at 12/60 correct:

• minimising 0/1 loss: 12/60 after 200 steps. It did not move. It cannot.
• minimising log loss: 60/60. Every row correct.

danger

You cannot optimise accuracy directly, and this is why

Accuracy, precision, recall and F1 are all built from counts, so all of them are piecewise constant in the parameters. A gradient method has nothing to follow. This is not a limitation of gradient descent — it is the reason surrogate losses exist.

The pattern is universal in supervised learning: minimise a smooth surrogate (log loss, hinge), report the discrete metric (accuracy, F1). Anyone who says "just optimise the metric you care about" is describing something a first-order method cannot do.

Two properties are being asked of a loss here, and they are separate:

Requirement	Why	0/1 loss
Differentiable (almost everywhere)	A gradient must exist to follow	Fails — flat with jumps
Informative gradient	The gradient must point somewhere useful	Fails — zero where it exists
Convex (desirable, not required)	Guarantees one minimum	Not convex either

Absolute error is interesting against this table: it is not differentiable at r = 0, yet it works fine, because the non-differentiable set has measure zero and a subgradient exists. Non-differentiability at isolated points is tolerable. Being flat everywhere is not.

---

The Hiker's Staircase vs. Hillside Sandbox

Experiment with the interactive complexity sandbox below. Move the weight $w_1$ (representing the hiker searching for the minimum) to watch how the 0/1 loss builds flat steps (zero gradient almost everywhere) while Cross-Entropy / Log Loss builds a smooth, convex slope.

Staircase vs. Hillside Simulator

Nudge the weight $w_1$ to watch how a hiker on the 0/1 staircase is completely stranded, while log loss offers a smooth slope.

Model Weight (w₁): -0.50

● 0/1 Loss (Staircase)● Log Loss (Hillside)

0/1 Loss: 0.60 (Staircase)

Gradient: 0.0000 (Flat step!)

Hiker is lost in fog; has no downhill direction.

Log Loss Cost: 0.6566

Gradient (Slope): -0.7311 (Confident!)

Smooth active derivative points exactly down the hill.

---

Loss by task

With that constraint in place, the standard choices follow.

Task	Loss	Formula	Minimises
Regression	Squared error / MSE	$(y - \hat{y})^2$	Distance to the mean
Regression, outliers present	Absolute error, Huber	$\lvert y - \hat{y} \rvert$	Distance to the median
Binary classification	Cross-entropy / log loss	$-[y \cdot \ln p + (1-y) \cdot \ln(1-p)]$	Divergence from the true distribution
Multi-class	Categorical cross-entropy	$-\sum y_k \cdot \ln p_k$	Same, over $k$ classes
Classification with margins (SVM)	Hinge	$\max(0, 1 - m)$	Margin violations

Log loss is unbounded, and that is the design

Output

4. LOSS BY TASK
   log loss = −ln(p) on the true class:
            p    −ln(p)
     9.90e-01      0.01
     9.00e-01      0.11
     5.00e-01      0.69
     1.00e-01      2.30
     1.00e-02      4.61
     1.00e-07     16.12

$p = 0.5$ — no opinion — costs $0.69$, which is $\ln 2$. A confident correct answer at $p = 0.99$ costs $0.01$. A confident wrong answer at $p = 10^{-7}$ costs 16.12, and there is no upper limit: as $p \to 0$ the loss goes to infinity.

That asymmetry is the point. Log loss does not score whether you were right; it scores how much probability you assigned to what actually happened. Being confidently wrong is punished without bound, which is what forces a model to produce calibrated probabilities rather than confident guesses.

warning

log_loss on a probability of exactly zero

Because $-\ln(0) = \infty$, any implementation must clip. scikit-learn's log_loss clips predictions away from 0 and 1 internally, so it returns a large finite number rather than inf. If you write the loss yourself, clip explicitly — a single $p = 0.0$ on a positive example will otherwise turn your entire cost into inf or nan and every gradient with it.

Hinge loss cares about the margin, then stops caring

Hinge loss is a fussy examiner who stops marking once you are safely past. Get a point onto the right side of the boundary with room to spare and it scores zero — no complaint, no pull on the fit at all. Only the points near the boundary, or on the wrong side of it, still cost anything. That is why a support vector machine ends up defined by a handful of points: the rest have been marked zero and have nothing left to say.

Formally, hinge is stated in terms of the margin $m = y \cdot f(x)$, positive when the prediction is on the correct side and larger when it is further from the boundary:

Output

   loss against the margin m (positive = correct side, |m| = confidence):
        m   hinge  sq hinge  (sig(m)−1)^2   0/1
      2.0    0.00      0.00        0.0142     0
      1.0    0.00      0.00        0.0723     0
      0.5    0.50      0.25        0.1425     0
      0.0    1.00      1.00        0.2500     1
     -1.0    2.00      4.00        0.5344     1
     -2.0    3.00      9.00        0.7758     1

Read the hinge column. At $m = 2$ and $m = 1$ it is exactly 0.00 — a correctly classified point comfortably past the margin contributes nothing, so it exerts no pull on the fit whatsoever. Below $m = 1$ the loss rises linearly. That flat region is the whole idea of a support vector machine: only the points near the boundary are support vectors, and the rest could be deleted without changing the model.

Compare the columns:

• 0/1 is a step: it drops from 1 to 0 between $m = 0$ and $m = 0.5$ and is flat either side — the flatness this page has already shown to be fatal.
• hinge is 0/1's tightest convex upper bound with a useful gradient: constant slope −1 while violated, 0 when satisfied.
• squared_hinge grows quadratically ($9.00$ at $m = -2$ against hinge's $3.00$), so it is smooth at the kink but far more outlier-sensitive.
• $(\sigma(m) - 1)^2$ — squared error applied to a probability — never reaches zero and never gets steep. At $m = -2$, badly wrong, it is only 0.7758, less than 3× its value at $m = 0$. That flatness is the subject of the next section.

Does the choice actually change the classifier?

Same 60 rows, same seed, same estimator, five losses:

Output

   the same linear classifier trained on the same 60 rows, five losses:
   loss              max_iter=1000  max_iter=200000   stopped on tol?
   log_loss                  60/60            60/60   yes / yes
   hinge                     59/60            59/60   yes / yes
   squared_hinge             58/60            58/60    no / no
   modified_huber            59/60            59/60   yes / yes
   squared_error             57/60            57/60   yes / yes
   hinge has no probabilities: AttributeError: This 'SGDClassifier' has no attribute 'predict_proba'

The spread is real but small: 57/60 to 60/60, three rows out of sixty. Crucially it survives raising max_iter from 1000 to 200000 and tightening tol — every accuracy is identical in both columns, so this is a property of the losses rather than of how long the optimiser ran.

squared_hinge is the one that never hits its tolerance, in either column. It still lands at 58/60 both times, so the non-convergence did not cost it anything here — but a non-convergence warning sitting next to an accuracy comparison is a reason to check, not to publish.

note

An earlier version of this measurement produced a false finding

The first run of this comparison reported all five losses clustered at chance level and concluded that the choice of loss makes no difference to accuracy. The labels had accidentally been generated from a separate random draw rather than from the feature columns, so they carried no information about $X$ at all and every classifier was guessing.

Two lessons. A result showing "no effect" deserves the same scrutiny as one showing a large effect — more, because it is easier to believe. And any classification demo should print an accuracy that is clearly above chance before any comparison between methods is read.

The last line of that output is the difference that dwarfs three rows of accuracy:

AttributeError: This 'SGDClassifier' has no attribute 'predict_proba'

Hinge loss produces a decision value, not a probability, so predict_proba does not exist on the fitted estimator — not a low-quality probability, no probability. log_loss and modified_huber give you one; hinge and squared_hinge do not. If anything downstream needs a probability — a threshold, a cost-weighted decision, a calibration plot — that choice is made when the loss is chosen, and it is irreversible without refitting.

---

Cross-entropy versus squared error on a sigmoid

Squared error on a classifier's probability output is a natural-looking mistake: the output is a number, the target is 0 or 1, so square the difference. The table above showed it stays flat when badly wrong.

The plain statement of what goes wrong: squared error's alarm goes quiet exactly when the fire is worst. A classifier ends in a squashing function (the sigmoid) that maps any score onto a probability between 0 and 1. Squashing means that far out at either end, a large change in the score barely moves the probability. Squared error measures the mistake in probability, so when the model is confidently, catastrophically wrong it sees a tiny distance and asks for a tiny correction — precisely when it should be shouting. Cross-entropy is built to cancel the squashing out, so its correction stays proportional to the actual mistake.

Here is that in arithmetic, differentiating with respect to the pre-sigmoid score $z$.

With $p = \sigma(z)$ and true label $y = 1$:

$\text{cross-entropy: } \frac{\partial L}{\partial z} = p - y \qquad \qquad \text{the sigmoid's derivative cancels}$

$\text{squared error: } \frac{\partial L}{\partial z} = 2(p - y) \cdot \sigma'(z) \qquad \qquad \text{and } \sigma'(z) = p(1 - p)$

$\sigma'(z)$ is the problem. It peaks at $0.25$ when $p = 0.5$ and collapses towards zero at both extremes:

Output

5. GRADIENT AT THE OUTPUT, y = 1   (cross-entropy vs squared error)
       z  p = sig(z)     CE dL/dz     SE dL/dz     ratio
     -10     0.00005   -1.000e+00   -9.079e-05    11014x
      -4     0.01799   -9.820e-01   -3.469e-02       28x
      -1     0.26894   -7.311e-01   -2.875e-01        3x
       0     0.50000   -5.000e-01   -2.500e-01        2x
       1     0.73106   -2.689e-01   -1.058e-01        3x
       4     0.98201   -1.799e-02   -6.354e-04       28x
      10     0.99995   -4.540e-05   -4.122e-09    11014x

Read the $z = -10$ row. The true label is $1$ and the model predicts $p = 0.00005$ — as wrong as it is possible to be. Cross-entropy's gradient is $-1.000$ — maximal, demanding a large correction. Squared error's is $-9.079e-05$, effectively nothing. The ratio is 11014×.

The mechanism is that squared error multiplies the error by $\sigma'(z)$, and $\sigma'$ is smallest exactly where the model is most confident. A confidently wrong prediction is therefore where squared error learns slowest. Cross-entropy's $-\ln p$ was constructed so that the $\sigma'$ term cancels, leaving $p - y$: the gradient is the error itself.

---

The Sigmoid Vanishing Gradient Explorer

Experiment with the interactive slider below. Move the pre-sigmoid logit score $z$ (when true target is $y = 1$) to watch how OLS squared error's gradient vanishes completely on a sigmoid (learning rate drops $11,014\times$!), while Cross-Entropy stays strong.

Sigmoid Saturation & Vanishing Gradient Explorer

Nudge $z$ to see how squared error completely goes blind on wrong, confident predictions, while Cross-Entropy remains strong.

Pre-sigmoid Logit (z): -4.0

Sigmoid Activation Curve p = σ(z)

Probability Output (p): 0.01799 (Target: 1.0)

CE dL/dz: -0.98201

Squared Error dL/dz: -0.03469

Learning Ratio: 28x slower

Vanishing Gradient Trap! Because the sigmoid saturates, its derivative σ'(z) drops exponentially. Under squared error, the learning pull collapses. At z = -4.0, Cross-Entropy learns 28x faster!

tip

Cross-entropy is not "better", it is the loss whose gradient survives the sigmoid

The cancellation is the design, not a happy accident. $-\ln p$ is chosen because its derivative is $-1/p$ and $\sigma' = p(1-p)$, so the product is $-(1-p)$, giving $p - y$ overall. Any smooth loss paired with a saturating output needs this check.

The same reasoning explains the 28× at $z = \pm 4$. The problem is not confined to extreme values — $p = 0.018$ is not an unusual prediction, and squared error is already learning 28 times more slowly there.

The $z = +10$ row is the mirror image: the model is confidently right, and both gradients are tiny — correctly, since nothing needs changing. The ratio is again 11014×, but of two negligible numbers. Confident correctness is where a small gradient is the right answer.

---

2. Implementation Lab

tip

Lab Exercise: Loss Functions

An interactive, fully-functional Google Colab / Jupyter Notebook is available to experiment with this concept live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Click "Run all" or execute cells individually using Shift + Enter.

Things to try inside the script:

• Experiment 1 (Outliers vs. Losses): Run section 1, edit target values from 60 to 5000, and observe how absolute loss is completely bulletproof, while OLS is pulled into orbit.
• Experiment 2 (Staircase search): In section 3, replace the 0/1 gradient descent block with random search of 1,000 runs, demonstrating why flat-gradient staircases force stochastic random trials instead of gradients.
• Experiment 3 (Sigmoid saturation): Redo the derivative of section 5 using tanh instead of sigmoid, checking if the vanishing gradient trap persists.

---

3. Common Mistakes

1. One Canonical Best Line

• The Misconception: "There is only one mathematically 'correct' best-fit line for any dataset."
• Wrong Thinking: Believing OLS is the only legitimate model.
• The Right Principle: "Best" is purely a statement about your loss. Clean data fitted with squared error, absolute error, and Huber loss gave three different slopes: 1.9490, 1.9752, and 1.9526. None of them is "wrong"—each satisfies a different mathematical specification.

2. Low-Leverage Outlier robustness

• The Misconception: "I can test how robust a loss function is by adding a wild outlier anywhere in my dataset."
• Wrong Thinking: Adding a giant outlier at the mean of $x$ (center of data) and drawing conclusions from how the slope changes.
• The Right Principle: A point near the mean of $x$ has zero leverage over the slope of a regression line (it only lifts the intercept). To test outlier robustness, the corruption must be placed at the extreme edge of $x$ where leverage is high.

3. Direct Accuracy Optimization

• The Misconception: "I should optimize accuracy, F1, or mistake counts directly."
• Wrong Thinking: Trying to pass classification accuracy or 0/1 mistake counts to gradient descent.
• The Right Principle: Count-based metrics are flat everywhere except at discrete steps, yielding exactly zero gradient ($1998$ of $2000$ sampled locations was exactly $0.0$). You cannot use gradient-descent on staircase landscapes; you must minimize a smooth surrogate loss (like cross-entropy or hinge) instead, and report accuracy afterwards.

4. Squared Error on Sigmoid

• The Misconception: "Squaring probability differences is a natural way to fit a binary classifier."
• Wrong Thinking: Believing squared error is optimal for sigmoid probability outputs.
• The Right Principle: Under squared error, the gradient is multiplied by the sigmoid derivative $\sigma'(z)$. Because the sigmoid saturates and flattens out on wrong confident predictions, $\sigma'$ collapses to zero—making squared error learn $11,014\times$ slower than Cross-Entropy when it is most wrong.

---

4. Summary

Core Theory

• Squared Loss (L2): Minimizes $\sum(y-\hat{y})^2$. Fits the mathematical mean. Highly outlier-sensitive.
• Absolute Loss (L1): Minimizes $\sum \lvert y-\hat{y} \rvert$. Fits the median. Completely blind to residual size, looking only at residual sign.
• Huber Loss: Quadratic within $\delta$, linear beyond it, offering L2 smoothness at the optimum and L1 robustness far from it.
• Surrogate Losses: Smooth differentiable functions (hinge, log loss) minimized in place of non-differentiable count-based metrics (accuracy, F1).

Engineering Realities

• Step size artifact: G gradients of 0/1 loss are either exactly $0$ or high-value noise (e.g. $833.33$) caused by step-size parameters.
• Hinge loss support: Hinge loss is exactly $0.0$ for margins $m \ge 1$, which is why support vector machines are defined only by a few points.
• Clipping Log Loss: Log loss on $p=0.0$ yields infinity. Production libraries clip probabilities to prevent NaN/Infinity gradients.
• Sigmoid trap: Pairing a saturating sigmoid with squared error collapses learning rates, whereas cross-entropy derivative cancels sigmoid saturation out completely.

info

Key Takeaways

• Absolute limitation: Because L1 absolute loss looks strictly at residual sign, pushing a high-leverage point from $60$ to $10,000$ yields exactly the same slope ($1.975169$).
• Probability constraint: SGDClassifier trained under Hinge loss yields raw decision margins, raising AttributeError if predict_proba is called.
• Robustness trade-off: Robustness is not a free upgrade—it is a conscious choice to discard the magnitude of extreme measurements, which could otherwise carry valuable signal.

Next in this section: Gradient Descent — the method that follows one of these gradients downhill, the two different plots that share its name, and why the step size shrinks near the minimum without anyone programming it.

---

5. Active Recall

Attempt each question first, then click to reveal detailed answers, mathematical derivations, and sample explanations.

Interactive Checkpoint Quiz

Interactive Checkpoint: The Absolute Leverage

If we corrupt a high-leverage point on absolute L1 loss, why does the slope stay perfectly constant ($1.975169$) even if we push the target value from 60 to 10,000?

Review Flashcards

1. [THEORY] Why is a low-leverage outlier (at $x \approx \bar{x}$) completely useless for testing loss robustness?

• Leverage Effect: Outliers sitting near the mean coordinate $\bar{x}$ have $(x_i - \bar{x}) \approx 0$, meaning they carry zero leverage and cannot pivot or tilt the slope of a regression line.
• The Lift: Instead, low-leverage outliers merely shift the vertical intercept $b_0$ upward, which affects both OLS and L1 similarly.
• The Correct Test: Outliers must be placed at the extreme ends of $x$ (high leverage) where they act as heavy anchors to test whether a loss is truly robust.

2. [THEORY] Show mathematically why minimizing L2 loss fits the mean while L1 fits the median.

• L2 Fits Mean: Let $L = \sum (y_i - c)^2$. Take the derivative with respect to $c$ and set to zero: $\frac{\partial L}{\partial c} = -2 \sum (y_i - c) = 0 \implies \sum y_i - n \cdot c = 0 \implies c = \frac{\sum y_i}{n} = \bar{y} \text{ (mean)}$
• L1 Fits Median: Let $L = \sum \lvert y_i - c \rvert$. Take the derivative with respect to $c$ and set to zero (where defined): $\frac{\partial L}{\partial c} = -\sum \text{sign}(y_i - c) = 0 \implies \sum_{y_i > c} 1 = \sum_{y_i < c} 1$
• This condition states that the number of data points above $c$ must equal the number of points below $c$, which is the exact definition of the median.

3. [ANALYZE] Why can't we use standard gradient-descent algorithms to optimize classification accuracy directly?

• The Staircase Landscape: Classification accuracy is based on mistake counts (0/1 loss). Counts are discrete step functions—if you nudge a model weight slightly, predictions stay the same until a threshold is crossed, making the loss landscape perfectly flat with sudden jumps.
• Zero Gradient: Because of this staircase shape, the gradient is exactly zero almost everywhere ($1998$ of $2000$ sampled points). A gradient-descent hiker has no slopes to follow and stays completely lost.
• The Surrogate Solution: We must minimize a smooth surrogate loss (like cross-entropy or hinge) that has continuous slopes everywhere, and report accuracy afterwards.

4. [ANALYZE] Explain the "Sigmoid Trap" and why OLS fails to learn on confident wrong probability predictions.

• The Sigmoid Derivative: Standard OLS squared error loss is $L = (p - y)^2$ where $p = \sigma(z)$. Differentiating with respect to the pre-sigmoid logit $z$ gives: $\frac{\partial L}{\partial z} = 2(p - y) \cdot \sigma'(z)$
• Vanishing Gradient: When a model is confidently wrong (e.g., $y=1$ but predicting $p \approx 0.00005$ far on the left curve of the sigmoid), the derivative $\sigma'(z) = p(1-p)$ collapses exponentially to $0.00$.
• The Collapse: This vanishing derivative multiplies the entire OLS gradient, making the learning pull $11,014\times$ weaker than Cross-Entropy exactly when the model is most wrong.

5. [THEORY] How does Hinge loss create "Support Vectors" geometrically?

• Margin Threshold: Hinge loss is defined as $\max(0, 1 - m)$ where $m = y \cdot f(x)$ is the margin score.
• Zero-Pull Regions: If a point is on the correct side of the boundary with a margin $m \ge 1$, its loss is exactly $0.00$ and its gradient is $0.00$.
• The Support Vectors: Because correctly classified points comfortable past the margin have no pull, they do not influence the optimization. The final decision boundary is defined solely by the remaining points (the ones violating or sitting on the margin boundary), which are the support vectors.