Why Logistic Regression

Topic — Linear regression is designed to predict unbounded continuous quantities like housing prices or temperature. When applied to classification problems (pass/fail, default/repay, spam/ham), linear regression fails catastrophically: it produces nonsensical negative or $>100\%$ probabilities and allows distant outliers to warp the decision boundary. This chapter explores these structural failure modes and introduces the sigmoid activation function as the proper mathematical gateway to probabilistic classification.

---

1. The Core Concept

In supervised binary classification, target labels are discrete binary states:

$y \in \{0, 1\}$

Rather than outputting hard discrete labels directly, modern machine learning models estimate the conditional posterior probability:

$P(y = 1 \mid x)$

A valid probability must strictly adhere to the Kolmogorov axioms, specifically bounded within the unit interval:

$0 \le P(y = 1 \mid x) \le 1$

While Ordinary Least Squares (OLS) linear regression fits a hyper-plane $y = w^T x + b$, its outputs are completely unconstrained ($y \in (-\infty, +\infty)$). Wrapping this linear score inside a monotonic, non-linear sigmoid squashing function guarantees valid probabilities.

tip

If you only take one thing from this page

Despite its historical name, Logistic Regression is a classification algorithm, not a regression algorithm. It models the log-odds of a binary event as a linear function of features, mapping real-valued inputs into valid $[0, 1]$ probabilities via the sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$.

---

2. Interactive Sigmoid & Probability Sandbox

Use the interactive slider below to vary the linear score $z = w^T x + b$ and observe how the sigmoid function maps real numbers into probabilities, odds, and discrete classifications:

Interactive Sigmoid & Odds Sandbox

Adjust the linear logit score $z$ to inspect the transformation into probability $\sigma(z)$:

Linear Logit Score (z): 0.00

Decision Threshold: 0.50

Probability P(y=1)

50.0%

σ(z) = 0.5000

Odds Ratio [p / (1-p)]

1.00:1

e^z ratio

Binary Decision

Class 1

Rule: p ≥ 0.50

---

3. The Two Fatal Flaws of Linear Regression

Why can we not simply fit Ordinary Least Squares $y = mx + b$ and apply a cutoff at $0.5$?

Flaw 1: Nonsensical Probabilities

• The Problem: Linear equations are unbounded rays: as $x \to \pm \infty$, $y \to \pm \infty$.
• Real-world Disaster: If predicting student pass/fail based on study hours ($y = 0.16x - 0.58$):
  • Student studying 1 hour: predicted score is $-0.42$ (Negative probability!).
  • Student studying 15 hours: predicted score is $1.82$ (Probability of 182%!).
• Statistical Violation: Violates Kolmogorov probability axioms. Downstream decision systems (expected value, risk budgeting) fail when passed negative probabilities.

Flaw 2: Severe Outlier Sensitivity

• The Problem: OLS minimizes squared residuals $\sum (y_i - \hat{y}_i)^2$.
• The Distortion: A student who studied 50 hours and passed easily has $y_i = 1$. The linear model predicts $\hat{y} = 7.6$.
• The Paradox: Because $7.6 \gg 1.0$, OLS treats this successful student as an enormous error with squared loss $(1 - 7.6)^2 = 43.56$.
• The Collapse: To minimize this artificial penalty, OLS rotates its slope downwards, pulling down predictions for borderline students and misclassifying true positives.

---

4. Interactive Outlier Shock Comparator

Observe how adding an extreme data point tilts a linear regression line while leaving Logistic Regression's decision boundary stable:

Outlier Vulnerability Comparator

Compare how an extreme observation (50 study hours) warps model parameters:

Linear Regression (OLS)

• Slope: 0.164 (Sharp)
• 6-Hour Student Prediction: 0.40 / Pass boundary
• Impact: Standard linear fit across range.

Logistic Regression (Sigmoid)

• Boundary: Stable at x ≈ 3.8 hours
• 6-Hour Student Prediction: 0.82 (PASS )
• Impact: Sigmoid plateau asymptotes cleanly at 1.0; extreme points create zero gradient distortion!

---

5. Mathematical Anatomy of the Sigmoid Curve

The standard logistic sigmoid function $\sigma(z)$ maps any real number $z \in (-\infty, +\infty)$ into the open unit interval $(0, 1)$:

$\sigma(z) = \frac{1}{1 + e^{-z}} = \frac{e^z}{1 + e^z}$

1. Numerator: 1

• Acts as the upper asymptotic ceiling.
• As $z \to +\infty$, $e^{-z} \to 0$, forcing $\sigma(z) \to \frac{1}{1 + 0} = 1.0$.

2. Denominator Term: e^(-z)

• The exponential engine driving smooth non-linear saturation.
• When $z = 0$, $e^{-0} = 1.0$, producing the exact symmetry center: $\sigma(0) = \frac{1}{1 + 1} = 0.5$.
• As $z \to -\infty$, $e^{-z} \to +\infty$, driving $\sigma(z) \to 0.0$.

3. Fundamental Symmetry

• The Identity: $\sigma(-z) = 1 - \sigma(z)$.
• Proof: $1 - \sigma(z) = 1 - \frac{1}{1 + e^{-z}} = \frac{e^{-z}}{1 + e^{-z}} = \frac{1}{e^z + 1} = \sigma(-z)$
• Engineering Value: The probability of failure ($1 - p$) equals the sigmoid of the negated linear score.

Key Sigmoid Benchmarks (Mental Math)

Input ($z$)	Expression ($e^{-z}$)	Probability ($\sigma(z)$)	Practical Interpretation
$-5.0$	$148.41$	$0.7\%$	Near certainty of Negative Class
$-2.0$	$7.39$	$11.9\%$	Low confidence Negative
$0.0$	$1.00$	$50.0\%$	Maximum uncertainty (Decision Boundary)
$+2.0$	$0.135$	$88.1\%$	Strong confidence Positive
$+5.0$	$0.0067$	$99.3\%$	Near certainty of Positive Class

---

6. Implementation Lab

tip

Lab Exercise: Linear Classification Failures & Sigmoid Mechanics

An interactive, fully functional Google Colab / Jupyter Notebook is available to demonstrate linear regression probability failures, outlier line tilts, and sigmoid symmetry live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Run cells sequentially to reproduce linear probability violations and sigmoid stability.

Key Experiments to Run:

• Experiment 1 (Negative & >100% Errors): Fit OLS linear regression on 10 student records to observe predicted values of $-0.255$ and $+1.218$.
• Experiment 2 (Outlier Shock Test): Inject a single student at $(50, \text{Fail})$ to watch the linear slope collapse by $31\%$ while Logistic Regression maintains accurate classifications.
• Experiment 3 (Sigmoid Symmetry Verification): Empirically test $\sigma(-z) == 1 - \sigma(z)$ across floating-point ranges using scipy.special.expit.

---

7. Common Mistakes

1. Confusing Classification with Regression

• The Misconception: "Logistic Regression has 'Regression' in its name, so it is used to predict continuous numeric values."
• Wrong Thinking: Using LogisticRegression to forecast prices or temperature.
• The Right Principle: The name is an artifact of statistics (modeling the continuous log-odds parameter). Logistic Regression is fundamentally a classification algorithm designed for discrete categorical targets.

2. Using OLS for Probability Estimation

• The Misconception: "We can fit linear regression to 0/1 labels and simply clamp predictions into [0, 1] using min/max."
• Wrong Thinking: Patching linear models with artificial clipping.
• The Right Principle: Clamping ignores the non-linear relationship of probabilities near 0 and 1, distorts variance, and remains fragile to outlier-induced boundary shifts. Use Logistic Regression natively.

3. Blindly Fixing Threshold at 0.5

• The Misconception: "A model should always predict positive when $P \ge 0.5$."
• Wrong Thinking: Treating the 0.5 mathematical midpoint as a universal business rule.
• The Right Principle: Thresholds must reflect downstream asymmetric costs. In fraud detection or cancer diagnostics, false negatives are catastrophic; lower the threshold to 0.1 or 0.2 to maximize recall.

4. Confusing Sigmoid with Tanh

• The Misconception: "Sigmoid and Hyperbolic Tangent (Tanh) are interchangeable activation functions."
• Wrong Thinking: Using Tanh to model probabilities.
• The Right Principle: Sigmoid maps to $(0, 1)$, matching probability axioms. Tanh maps to $(-1, 1)$, zero-centered for neural network hidden states, but invalid for direct binary probabilities.

5. Fitting Linear Regression Outliers

• The Misconception: "Squared error loss ($MSE$) is suitable for training binary classification models."
• Wrong Thinking: Minimizing $(y - \hat{y})^2$ on binary targets.
• The Right Principle: Squared error penalizes correct but confident predictions ($y=1, \hat{y}=3 \implies \text{loss}=4$). Logistic regression uses Binary Cross-Entropy (Log Loss), which creates zero penalty for confident correct classifications.

---

8. Summary

Linear Regression Failures

• Unbounded Output: Linear predictions escape $[0, 1]$, producing impossible negative and $>100\%$ probabilities.
• Outlier Sensitivity: OLS penalizes confident correct samples ($y=1, \hat{y} \gg 1$), tilting the decision boundary.
• Loss Function Mismatch: Mean Squared Error creates non-convex optimization surfaces when paired with non-linear activations.

The Sigmoid Solution

• Strict Bounding: $\sigma(z) = \frac{1}{1 + e^{-z}}$ mathematically guarantees outputs in open unit interval $(0, 1)$.
• Symmetry: Perfect anti-symmetry centered at $(0, 0.5)$ via $\sigma(-z) = 1 - \sigma(z)$.
• Robust Asymptotes: Extreme values plateau smoothly, rendering parameter updates robust to outliers.

info

Key Takeaways

• Probabilities require sigmoid: Linear models cannot enforce the unit interval constraint $[0, 1]$.
• Sigmoid symmetry simplifies math: $\sigma(-z) = 1 - \sigma(z)$ ensures that modeling $P(y=1)$ automatically defines $P(y=0)$.
• Asymptotes protect against outliers: The plateau of the S-curve prevents extreme inputs from distorting classifications near the decision threshold.

Next Section: The Sigmoid and Fitting - deriving the log-loss function, maximum likelihood estimation, and gradient descent optimization.

---

9. Active Recall

Test your understanding by answering first, then clicking to reveal the underlying principles.

Interactive Checkpoint Quiz

Interactive Checkpoint: OLS vs. Sigmoid Loss

Why does fitting a linear regression model with Mean Squared Error (MSE) fail when an extremely obvious positive sample (e.g. $x=100, y=1$) is present in the training set?

Review Flashcards

1. [THEORY] Why is the sigmoid function preferred over a simple step function (Heaviside) for classification?

• Differentiability: A step function has a derivative of zero everywhere except at $z=0$, where it is undefined. Gradient-based optimization (gradient descent) cannot compute parameter updates.
• Continuous Probabilities: The sigmoid function is smooth, continuously differentiable everywhere ($\sigma'(z) = \sigma(z)(1 - \sigma(z))$), providing informative gradients that indicate not just whether a sample is classified correctly, but how confident the prediction is.

2. [MATH] Prove the sigmoid symmetry property: σ(-z) = 1 - σ(z).

• Start with the right-hand side: $1 - \sigma(z) = 1 - \frac{1}{1 + e^{-z}} = \frac{(1 + e^{-z}) - 1}{1 + e^{-z}} = \frac{e^{-z}}{1 + e^{-z}}$
• Multiply numerator and denominator by $e^z$: $\frac{e^{-z} \cdot e^z}{(1 + e^{-z}) \cdot e^z} = \frac{1}{e^z + 1} = \frac{1}{1 + e^z} = \sigma(-z)$
• Thus, $\sigma(-z) = 1 - \sigma(z)$ holds identically for all $z \in \mathbb{R}$.

3. [PRACTICE] A logistic regression model outputs a linear logit score of z = 2.197. What is the predicted probability and odds?

• Odds: $\text{Odds} = e^z = e^{2.197} \approx 9.0$ (i.e. 9 to 1 odds).
• Probability: $P = \frac{\text{Odds}}{1 + \text{Odds}} = \frac{9.0}{1 + 9.0} = \frac{9.0}{10.0} = 0.90$ ($90\%$).
• Direct Sigmoid: $\sigma(2.197) = \frac{1}{1 + e^{-2.197}} = \frac{1}{1 + 0.111} \approx 0.90$.

4. [ANALYZE] Why is setting the classification threshold to 0.5 dangerous in an automated credit card fraud detection system?

• Asymmetric Costs: A false positive (flagging a legit $50 transaction) causes minor customer annoyance. A false negative (missing a$5,000 fraudulent charge) causes direct financial loss.
• Severe Imbalance: Fraud represents $\approx 0.1\%$ of transactions. A default 0.5 threshold will fail to flag suspicious transactions that have a 20% or 30% probability of fraud.
• Solution: Lower the threshold to 0.05 or 0.10 to prioritize recall over precision.

5. [THEORY] What geometric shape is the decision boundary of Logistic Regression in feature space?

• A Linear Hyperplane: The decision boundary occurs where $P(y=1) = 0.5$, which corresponds to $z = 0$.
• Since $z = w_1 x_1 + w_2 x_2 + \dots + w_d x_d + b = 0$, the decision boundary is a straight line in 2D, a flat plane in 3D, and a hyperplane in higher dimensions.
• Despite the non-linear sigmoid activation, Logistic Regression is fundamentally a linear classifier.