📄️ Loss Functions
The loss function is how you tell a model what "good" means, and changing it changes the answer. Three losses gave three different slopes on identical data; absolute error ignored an outlier at 60, 200 and 10000 alike; counting mistakes turned out to be unminimisable, with 1998 of 2000 gradients exactly zero; and squared error on a sigmoid was 11014x weaker than cross-entropy exactly where the model was most wrong.
📄️ Gradient Descent
Walking downhill in fog, made precise — why two different plots both get called the gradient descent curve, why the algorithm slows near the bottom with nothing in the loop programming a brake, why flipping one minus sign sends the cost to 6.33e+07, and why the neat round bowl in every textbook is the one case that never happens in practice.
📄️ The Learning Rate
The learning rate is your stride length, and there is an exact cutoff between working and exploding that you can compute instead of guess — verified to 3.3e-16. Also why an alpha 0.4% too large looks merely slow for a thousand iterations, why overshooting is the fastest regime rather than a fault, and why the textbook optimal alpha measured 2.20x slower than a slightly smaller one.
📄️ Batch, Stochastic and Mini-Batch
One decision left — how many rows the model looks at before each step. "Stochastic is faster" turns out to need a unit attached, because batch beat single-row descent on passes, on seconds and on final cost, single-row descent could not converge at all at a constant step size, and yet mini-batch 32 reached a loose target after reading 4% of one pass where batch read 50 full passes and still missed.