The Sigmoid and Fitting

Topic — How do we fit an S-shaped curve to discrete binary data? In linear regression, parameters are chosen by minimizing Mean Squared Error through a closed-form formula. In logistic regression, squared error creates non-convex surfaces with treacherous local minima. This chapter derives Binary Cross-Entropy (Log Loss) from the principle of Maximum Likelihood Estimation (MLE), steps through hand-worked parameter fitting, and tracks gradient descent convergence.

---

1. The Core Concept

Logistic regression combines a linear combination of input features with the non-linear sigmoid activation function:

$z = w^T x + b = w_1 x_1 + w_2 x_2 + \dots + w_d x_d + b$

$\hat{y} = P(y = 1 \mid x) = \sigma(z) = \frac{1}{1 + e^{-z}}$

Because the output $\hat{y}$ represents the probability of a Bernoulli random variable, we cannot optimize parameters $w$ and $b$ using Ordinary Least Squares. Instead, we use Maximum Likelihood Estimation (MLE) to find the parameters that make the observed training data most probable.

tip

If you only take one thing from this page

Log Loss (Binary Cross-Entropy) is not an arbitrary choice: it is the direct negative log-likelihood of a Bernoulli distribution. It is strictly convex with respect to linear weights, guaranteeing that gradient descent always converges to the global minimum without getting trapped in local valleys.

---

2. Interactive Loss Penalty Sandbox: Log Loss vs. MSE

Compare how Binary Cross-Entropy and Mean Squared Error respond to prediction certainty:

Loss Function Penalty Explorer

Select the true label and adjust predicted probability ŷ to inspect error divergence:

True Label (y):

Predicted Probability ŷ: 50.0% (0.500)

Binary Cross-Entropy (Log Loss)

0.6931

Penalty: Asymptotic to infinity

Mean Squared Error (MSE)

0.2500

Penalty: Strictly bounded by 1.0

Intuition: Near correct predictions, both losses are low. But when predictions are confident and wrong, Log Loss grows exponentially, while MSE plateaus and causes vanishing gradients.

---

3. Deriving Log Loss from Maximum Likelihood Estimation

Why do we use $-\sum [y \ln(\hat{y}) + (1-y) \ln(1-\hat{y})]$? It is not an arbitrary heuristic.

1. The Bernoulli Likelihood

• For a single sample $(x_i, y_i)$, the probability of observing outcome $y_i \in \{0, 1\}$ is given by the Bernoulli PMF: $P(Y = y_i \mid x_i) = \hat{y}_i^{y_i} (1 - \hat{y}_i)^{1 - y_i}$
  • If $y_i = 1$, this simplifies to $\hat{y}_i^1 (1 - \hat{y}_i)^0 = \hat{y}_i$.
  • If $y_i = 0$, this simplifies to $\hat{y}_i^0 (1 - \hat{y}_i)^1 = 1 - \hat{y}_i$.
• Assuming independent and identically distributed (i.i.d.) observations, the joint likelihood across all $n$ samples is the product: $L(w, b) = \prod_{i=1}^{n} \hat{y}_i^{y_i} (1 - \hat{y}_i)^{1 - y_i}$

2. The Log-Likelihood

• Multiplying thousands of fractional probabilities causes severe floating-point underflow.
• Taking the natural logarithm converts the product into a numerically stable summation: $\ln L(w, b) = \ln \left( \prod_{i=1}^{n} \hat{y}_i^{y_i} (1 - \hat{y}_i)^{1 - y_i} \right)$ $\ln L(w, b) = \sum_{i=1}^{n} \left[ y_i \ln(\hat{y}_i) + (1 - y_i) \ln(1 - \hat{y}_i) \right]$

3. Negative Log-Likelihood (Loss)

• In optimization, algorithms are conventionally formulated to minimize cost rather than maximize likelihood.
• Multiplying by $-1$ and averaging over $n$ samples produces the standard Binary Cross-Entropy Loss function: $J(w, b) = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \ln(\hat{y}_i) + (1 - y_i) \ln(1 - \hat{y}_i) \right]$
• Maximizing likelihood is mathematically identical to minimizing log loss.

---

4. Interactive Parameter Fitting Sandbox

Experiment with fitting a logistic regression model manually on 5 students ($x=[2, 4, 6, 8, 10]$, $y=[0, 0, 1, 1, 1]$):

Interactive 5-Student Parameter Fitting

Adjust slope m and intercept b to minimize mean log loss across the 5 students:

Slope (m): 0.35

Intercept (b): -1.40

Current Mean Log Loss:

0.3671

Optimal Benchmark: ~0.3648
(at m = 0.40, b = -1.60)

Hours (x)	Actual (y)	z = mx + b	P(Pass) = σ(z)	Sample Loss
2	0	-0.70	33.2%	0.4032
4	0	0.00	50.0%	0.6931
6	1	0.70	66.8%	0.4032
8	1	1.40	80.2%	0.2204
10	1	2.10	89.1%	0.1155

---

5. Optimization Mechanics: Gradient Descent

Unlike Ordinary Least Squares linear regression, setting $\nabla J(w, b) = 0$ yields a transcendental equation that cannot be solved analytically in closed form. We must use iterative optimization (Gradient Descent).

The Elegance of the Gradient

Using the chain rule, the derivative of the sigmoid is:

$\frac{d\sigma(z)}{dz} = \sigma(z)(1 - \sigma(z)) = \hat{y}(1 - \hat{y})$

When taking the partial derivative of Log Loss $J(w, b)$ with respect to weight $w_j$, the non-linear terms cancel out cleanly:

$\frac{\partial J}{\partial w_j} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i) x_{ij}$

$\frac{\partial J}{\partial b} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)$

info

The Universal Gradient Pattern

Notice that the gradient for Logistic Regression is identical in algebraic form to the gradient of Linear Regression:

$\text{Error} \times \text{Input} = (\hat{y}_i - y_i) x_{ij}$

The only difference is that for Logistic Regression, $\hat{y}_i = \sigma(w^T x_i + b)$ is bounded between $0$ and $1$.

---

6. Implementation Lab

tip

Lab Exercise: Log Loss Calculations & Gradient Descent Convergence

An interactive, fully functional Google Colab / Jupyter Notebook is available to experiment with manual parameter grid searches, compare Log Loss against MSE, and track SGD convergence live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Run cells sequentially to benchmark candidate parameter pairs and train an SGD classifier.

Key Experiments to Run:

• Experiment 1 (Grid Search over Parameters): Verify how candidate parameters $m=0.40, b=-1.60$ achieve minimum log loss ($1.8241$) on the 5-student toy dataset.
• Experiment 2 (Extreme Error Divergence): Compare a prediction of $\hat{y}=0.001$ on an actual positive: observe MSE plateauing at $0.998$ while Log Loss spikes to $6.908$.
• Experiment 3 (Tracking SGD Convergence): Train SGDClassifier(loss='log_loss') on 50 students across 50 epochs, watching loss smoothly descend from $1.45$ to $0.41$.

---

7. Common Mistakes

1. Believing Closed-Form Solutions Exist

• The Misconception: "We can compute optimal logistic regression weights directly using a matrix formula like $(X^T X)^{-1} X^T y$."
• Wrong Thinking: Searching for an analytical closed-form solution.
• The Right Principle: The non-linear sigmoid inside the log-likelihood makes the derivative non-linear in parameters. Numerical optimization (Gradient Descent, Newton-CG, or L-BFGS) is mathematically required.

2. Using MSE with Sigmoidal Activations

• The Misconception: "Minimizing Mean Squared Error $\sum (y - \sigma(w^T x + b))^2$ should work just as well as log loss."
• Wrong Thinking: Combining MSE with sigmoid activations for classification.
• The Right Principle: MSE combined with a sigmoid creates a non-convex cost function full of saddle points and local minima. Furthermore, confident wrong predictions produce tiny gradients (vanishing gradient problem), stopping learning entirely.

3. Neglecting Numerical Stability in Logarithms

• The Misconception: "Computing $-\ln(\hat{y})$ in Python directly is fine for all model outputs."
• Wrong Thinking: Passing raw floating-point 0.0 or 1.0 to np.log().
• The Right Principle: If $\hat{y} = 0.0$, $\ln(0)$ evaluates to $-\infty$ or raises a ZeroDivisionError / RuntimeWarning. Production implementations must always clamp probabilities using an epsilon ($\hat{y} \in [\epsilon, 1 - \epsilon]$ with $\epsilon = 10^{-15}$).

4. Forgetting to Scale Features Before Optimization

• The Misconception: "Logistic regression outputs probabilities, so raw feature scales do not matter."
• Wrong Thinking: Training gradient descent on unscaled features (e.g. Age in $[20, 60]$ vs. Income in $[20000, 150000]$).
• The Right Principle: Unscaled features create distorted, elliptical loss contours, causing gradient descent to oscillate wildly and converge slowly. Always scale features via StandardScaler inside a Pipeline.

5. Omitting Regularization on Separable Data

• The Misconception: "If classes are perfectly linearly separable, logistic regression achieves 100% accuracy and zero loss."
• Wrong Thinking: Fitting unregularized logistic regression to perfectly separable data.
• The Right Principle: On perfectly separable data, unregularized maximum likelihood drives weights to $\pm \infty$ to force $\sigma(z) \to 1.0$ and $0.0$, causing numerical overflow. Always apply L2 regularization (C parameter in scikit-learn).

---

8. Summary

Theoretical Foundations

• Model Hypothesis: $\hat{y} = \sigma(w^T x + b) = \frac{1}{1 + e^{-(w^T x + b)}}$.
• Cost Function: Binary Cross-Entropy $J(w, b) = -\frac{1}{n} \sum [y \ln \hat{y} + (1-y) \ln(1-\hat{y})]$.
• Derivation: Derived directly from the log-likelihood of a Bernoulli distribution.
• Convexity: Log loss is strictly convex with respect to linear weights, guaranteeing a single global minimum.

Optimization Realities

• No Closed Form: Requires numerical optimization (Gradient Descent, L-BFGS, Newton-Raphson).
• Gradient Equation: $\frac{\partial J}{\partial w_j} = \frac{1}{n} \sum (\hat{y}_i - y_i) x_{ij}$.
• Loss Gradient Advantage: Steep gradients for confident mistakes prevent vanishing gradient stalls.
• Feature Scaling: Standardize features to ensure isotropic circular contours for fast gradient descent.

info

Key Takeaways

• Log Loss is MLE: Minimizing Binary Cross-Entropy is mathematically equivalent to maximizing the likelihood of observing the training data under a Bernoulli assumption.
• Avoid MSE on Classification: MSE creates non-convex surfaces and vanishes when predictions are confidently incorrect.
• Gradient Descent is Universal: The gradient update formula $(\hat{y} - y)x$ is shared between Linear and Logistic regression, providing seamless mathematical continuity.

Next Section: Multiclass and Practical Use - extending binary logistic regression to multiclass problems via One-vs-Rest (OvR) and Multinomial Softmax.

---

9. Active Recall

Test your understanding by answering first, then clicking to reveal the underlying principles.

Interactive Checkpoint Quiz

Interactive Checkpoint: The Vanishing Gradient in MSE

If we train a logistic classifier ŷ = σ(z) using Mean Squared Error loss L = (y - ŷ)², what happens to the gradient when the model makes a confident wrong prediction (e.g. true y = 1 but predicted ŷ ≈ 0.001)?

Review Flashcards

1. [THEORY] Why does minimizing Binary Cross-Entropy guarantee finding the global optimum?

• Convexity: The negative log-likelihood function $J(w, b)$ of logistic regression is strictly convex with respect to the weights $w$ and bias $b$.
• Its Hessian matrix $\nabla^2 J = \frac{1}{n} X^T D X$ (where $D$ is a diagonal matrix with entries $\hat{y}_i(1 - \hat{y}_i) > 0$) is positive semi-definite everywhere.
• Because there are no local minima or saddle points, gradient descent is mathematically guaranteed to converge to the unique global minimum.

2. [MATH] Why does the sigmoid derivative cancel out when differentiating Binary Cross-Entropy?

• Using the chain rule: $\frac{\partial J_i}{\partial z} = \frac{\partial J_i}{\partial \hat{y}_i} \cdot \frac{\partial \hat{y}_i}{\partial z}$
• $\frac{\partial J_i}{\partial \hat{y}_i} = -\frac{y_i}{\hat{y}_i} + \frac{1 - y_i}{1 - \hat{y}_i} = \frac{\hat{y}_i - y_i}{\hat{y}_i(1 - \hat{y}_i)}$.
• $\frac{\partial \hat{y}_i}{\partial z} = \sigma'(z) = \hat{y}_i(1 - \hat{y}_i)$.
• Multiplying them together cancels the denominator $\hat{y}_i(1 - \hat{y}_i)$ completely, leaving: $\frac{\partial J_i}{\partial z} = \hat{y}_i - y_i$
• This cancellation eliminates the vanishing gradient problem.

3. [PRACTICE] What happens to logistic regression weights if the training classes are perfectly linearly separable and no regularization is applied?

• Weight Explosion: If classes are perfectly separable, the model can achieve a log loss of zero only if $\hat{y}_i = 1$ for all positives and $\hat{y}_i = 0$ for all negatives.
• Because $\sigma(z) \to 1$ only as $z \to +\infty$ and $\sigma(z) \to 0$ only as $z \to -\infty$, gradient descent continues increasing the magnitude of weights $\|w\| \to \infty$.
• The Remedy: Apply L2 weight regularization (e.g. C=1.0), which adds $\frac{1}{2C} \|w\|^2$ penalty to the cost function, preventing weights from growing unbounded.

4. [ANALYZE] If a model predicts probability ŷ = 0.99 for an actual positive sample, what is the log loss? What if ŷ = 0.50?

• For $\hat{y} = 0.99$: Loss $= -\ln(0.99) \approx 0.010$. The penalty is virtually zero because the model is confident and correct.
• For $\hat{y} = 0.50$: Loss $= -\ln(0.50) = \ln(2) \approx 0.693$. A loss of $\approx 0.693$ represents complete uncertainty (random coin flip guessing).
• Baseline Benchmark: Any trained binary classification model that exhibits a mean log loss $> 0.693$ is performing worse than a trivial coin toss.

5. [PRACTICE] Why are second-order optimization algorithms like L-BFGS or Newton-CG often used instead of basic SGD for small-to-medium logistic regression?

• Curvature Information: Second-order methods utilize the Hessian matrix (curvature of the loss surface).
• Because logistic regression is strictly convex, Newton-Raphson steps jump directly toward the optimum without requiring manual learning rate tuning ($\alpha$).
• For datasets with $N \le 100,000$, L-BFGS converges in $10-20$ iterations, whereas basic SGD may require hundreds of epochs.