Minimising a loss — gradient descent, the learning rate, and how much data each step sees.
View all tagsOne decision left — how many rows the model looks at before each step. "Stochastic is faster" turns out to need a unit attached, because batch beat single-row descent on passes, on seconds and on final cost, single-row descent could not converge at all at a constant step size, and yet mini-batch 32 reached a loose target after reading 4% of one pass where batch read 50 full passes and still missed.
Walking downhill in fog, made precise — why two different plots both get called the gradient descent curve, why the algorithm slows near the bottom with nothing in the loop programming a brake, why flipping one minus sign sends the cost to 6.33e+07, and why the neat round bowl in every textbook is the one case that never happens in practice.
The loss function is how you tell a model what "good" means, and changing it changes the answer. Three losses gave three different slopes on identical data; absolute error ignored an outlier at 60, 200 and 10000 alike; counting mistakes turned out to be unminimisable, with 1998 of 2000 gradients exactly zero; and squared error on a sigmoid was 11014x weaker than cross-entropy exactly where the model was most wrong.
The learning rate is your stride length, and there is an exact cutoff between working and exploding that you can compute instead of guess — verified to 3.3e-16. Also why an alpha 0.4% too large looks merely slow for a thousand iterations, why overshooting is the fastest regime rather than a fault, and why the textbook optimal alpha measured 2.20x slower than a slightly smaller one.