Overfitting, Underfitting, and Regularization

Topic — A machine learning model fits training data with 100% precision, yet fails catastrophically on unseen customer transactions: overfitting. Another model is so mathematically rigid that it misses real signals entirely: underfitting. Regularization introduces a mathematical constraint into the loss function to penalize excessive complexity. This guide explores the bias-variance tradeoff, contrasts L1 (Lasso), L2 (Ridge), and Elastic Net penalties, and details geometric constraint boundaries.

---

1. The Core Concept

The fundamental objective of supervised learning is not resubstitution memorization, but out-of-sample generalization:

$\mathbb{E}_{(x, y) \sim \mathcal{D}}\left[ \mathcal{L}(y, \hat{f}(x)) \right]$

Every machine learning model balances two competing sources of error captured by the Bias-Variance Decomposition:

$\text{Total Expected Error} = \text{Bias}[\hat{f}(x)]^2 + \text{Var}[\hat{f}(x)] + \sigma^2$

1. Bias (Underfitting): Error resulting from oversimplified mathematical assumptions. A linear model fitting quadratic data cannot capture true curvature.
2. Variance (Overfitting): Error resulting from hypersensitivity to random fluctuations in the training set. A high-degree polynomial bends to pass through every noise point.
3. Irreducible Error ($\sigma^2$): Intrinsic noise in the data collection process that no algorithm can eliminate.

tip

If you only take one thing from this page

Overfitting means your model has low bias but high variance (memorizing noise). Underfitting means your model has high bias but low variance (too rigid). Regularization adds a penalty term $\lambda \cdot \Omega(w)$ to shrink weights toward zero, trading a tiny increase in training bias for a massive reduction in test variance.

---

2. Interactive Bias-Variance Tradeoff Simulator

Vary the polynomial degree $d$ below to observe how model capacity shapes training error versus test generalization error:

Interactive Bias-Variance Tradeoff Simulator

Adjust polynomial complexity (Degree d) to watch the U-shaped test error curve emerge:

Model Complexity (Polynomial Degree d):Degree 3

Training Error (MSE)0.1200

Monotonically drops toward 0.0

Test Generalization Error (MSE)0.1400

Classic U-shaped curve

Optimal Generalization (The Sweet Spot)

Captures true non-linear structure while ignoring point-wise measurement noise. Test error reaches its global minimum at d = 3.

---

3. Regularization Mechanics: L1 vs. L2 vs. Elastic Net

Regularization controls model complexity by adding a mathematical penalty to the objective function:

$\text{Cost}(\mathbf{w}) = \text{Loss}(\mathbf{y}, \hat{\mathbf{y}}) + \lambda \cdot \Omega(\mathbf{w})$

Regularization Penalty and Gradient Explorer

Adjust coefficient magnitude $w$ and penalty strength $\lambda$ to compare cost penalties:

Weight Coefficient (w):1.20

Penalty Strength (λ / alpha):1.0

L1 (Lasso Penalty)

1.200

λ · |w|

L2 (Ridge Penalty)

0.720

0.5 · λ · w²

Elastic Net (50/50 Mix)

0.960

0.5·L1 + 0.5·L2

---

4. Comparing Regularization Architectures

1. L2 Regularization (Ridge)

• Penalty Term: $\Omega(w) = \frac{1}{2} \sum_{j=1}^{d} w_j^2 = \frac{1}{2} \|\mathbf{w}\|_2^2$.
• Gradient Contribution: $\frac{\partial \Omega}{\partial w_j} = w_j$.
• Shrinkage Behavior: Weight update subtracts a fraction proportional to $w_j$: $w_j \leftarrow w_j (1 - \alpha \lambda) - \alpha \frac{\partial \text{Loss}}{\partial w_j}$
• Sparsity: Never exactly zero. Smoothly shrinks coefficients toward zero, preserving all features.
• Geometry: Spherical / circular constraint surface without sharp corners.
• Best For: Multicollinearity (correlated features), preserving all predictive signals.

2. L1 Regularization (Lasso)

• Penalty Term: $\Omega(w) = \sum_{j=1}^{d} |w_j| = \|\mathbf{w}\|_1$.
• Gradient Contribution: $\frac{\partial \Omega}{\partial w_j} = \text{sign}(w_j)$ (constant magnitude $\pm 1$).
• Shrinkage Behavior: Constant step reduction regardless of weight magnitude: $w_j \leftarrow w_j - \alpha \lambda \cdot \text{sign}(w_j)$
• Sparsity: Drives uninformative weights to exact zero ($w_j = 0$). Performs automatic feature selection.
• Geometry: Diamond / $L_1$ polytope with sharp corners on the coordinate axes.
• Best For: Sparse feature selection when many features are pure noise.

3. Elastic Net

• Penalty Term: Convex linear combination of L1 and L2 penalties: $\Omega(w) = \rho \|\mathbf{w}\|_1 + \frac{1 - \rho}{2} \|\mathbf{w}\|_2^2$
• The Grouping Effect: When multiple features are strongly correlated, Lasso arbitrarily picks one and drops the others. Elastic Net groups them, shrinking them together while maintaining sparsity.
• Best For: High-dimensional data ($p \gg n$) with correlated predictor clusters.

---

5. Geometric Intuition: Why Lasso Yields Zeros

Why does $L_1$ regularization produce exact zeros (sparsity), while $L_2$ regularization never does?

In constrained optimization (Lagrange form), regularized regression solves:

$\min_{\mathbf{w}} \text{Loss}(\mathbf{w}) \quad \text{subject to} \quad \Omega(\mathbf{w}) \le C$

The optimal solution occurs at the point of tangency where the expanding unconstrained Loss Contours (ellipses) first touch the Regularization Constraint Region.

Interactive Geometry of Sparsity Simulator

Rotate the unconstrained OLS optimum angle (θ) to observe how the expanding loss ellipses interact with the constraint boundary:

Unconstrained OLS Angle (θ):32°

Constraint Budget (C):1.0

Optimal Constrained Solution (w*)

w₁ = 1.00, w₂ = 0.00

EXACT SPARSITY: One weight is zero (Corner Solution)!

Why Lasso sets weights to zero: The diamond constraint has sharp vertices sitting directly on the coordinate axes ($(C, 0)$ and $(0, C)$). Because the corners protrude furthest, the expanding loss ellipse almost always makes initial contact at a sharp corner across a wide range of angles ($15°$ to $75°$). Touching a corner forces the orthogonal weight to be identically zero.

---

6. Implementation Lab

tip

Lab Exercise: Bias-Variance Curves & Regularization Shrinkage

An interactive, fully functional Google Colab / Jupyter Notebook is available to evaluate polynomial degree overfitting, contrast Lasso vs. Ridge coefficient sparsity, and tune $\lambda$ live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Run cells sequentially to generate synthetic non-linear polynomial data and inspect weight vectors.

Key Experiments to Run:

• Experiment 1 (Polynomial Overfitting Sweep): Track train vs. test MSE from Degree 1 to 12 to observe training error plunge to $0.008$ while test error explodes to $1.893$.
• Experiment 2 (L1 Sparsity Demonstration): Fit OLS, Ridge, Lasso, and Elastic Net on 5 features (where 3 are noise) to observe Lasso drive noisy coefficients to exact zero ($0.000$).
• Experiment 3 (Tuning Alpha): Sweep $\alpha \in [0.001, 100.0]$ in Ridge regression to trace the U-shaped transition from overfitting to underfitting.

---

7. Common Mistakes

1. Regularizing the Intercept (Bias)

• The Misconception: "The regularization penalty should apply to all parameters including the intercept $b$."
• Wrong Thinking: Adding $\lambda b^2$ to the cost function.
• The Right Principle: The intercept represents the global base rate of the target. Penalizing $b$ shifts predictions away from the target mean. Scikit-learn models (fit_intercept=True) never regularize the bias term.

2. Regularizing Without Feature Standardization

• The Misconception: "Regularization works identically regardless of whether features are scaled."
• Wrong Thinking: Applying L1 or L2 penalties to unstandardized features.
• The Right Principle: The penalty $\sum w_j^2$ treats all weights identically. Features with large scales naturally have tiny weights, escaping penalization, while features with tiny scales are penalized unfairly. Always standardize features with StandardScaler inside a Pipeline.

3. Confusing Alpha and C Parameters

• The Misconception: "Setting C=10 in Logistic Regression means strong regularization."
• Wrong Thinking: Inverting the meaning of $C$ and $\alpha$.
• The Right Principle: In linear models (Ridge, Lasso), alpha ($\lambda$) represents penalty strength (large $\alpha$ = strong regularization). In LogisticRegression and SVC, C is the inverse penalty ($C = \frac{1}{\lambda}$). Large $C$ means weak regularization!

4. Believing L2 Performs Feature Selection

• The Misconception: "Ridge regression sets unimportant features to zero if alpha is large enough."
• Wrong Thinking: Expecting L2 to produce sparse models.
• The Right Principle: Because the L2 gradient $\lambda w$ diminishes as $w \to 0$, L2 coefficients asymptotically approach zero but never reach it. Only L1 (Lasso) produces exact zeros.

5. Assuming Zero Training Error Implies Perfection

• The Misconception: "Our model achieved 100% training accuracy, so the algorithm is optimal."
• Wrong Thinking: Evaluating model quality using resubstitution accuracy.
• The Right Principle: Zero training error is the cardinal diagnostic signature of severe overfitting. True model quality is determined strictly by out-of-sample cross-validation error.

---

8. Summary

Error Regimes

• Underfitting (High Bias): Model too simple. Both training and test errors are high.
• Overfitting (High Variance): Model too complex. Low training error, exploding test error.
• The Sweet Spot: Optimal complexity minimizing total out-of-sample expected loss.
• Regularization Role: Trades a small amount of training bias to massively reduce test variance.

Penalty Selection

• L2 (Ridge): $\lambda \|\mathbf{w}\|_2^2$. Proportional shrinkage. Retains all features. Tames multicollinearity.
• L1 (Lasso): $\lambda \|\mathbf{w}\|_1$. Constant shrinkage. Drives weights to exact zero. Sparse feature selection.
• Elastic Net: Combines L1 and L2. Handles correlated feature clusters.
• Standardization Mandate: Always scale features before regularized fitting.

info

Key Takeaways

• Regularization controls variance: Without regularization, complex models memorize sample noise rather than population signal.
• Corners create sparsity: The diamond geometry of L1 forces loss contours to touch on coordinate axes, performing automated feature selection.
• Alpha vs C inversion: Never forget that in scikit-learn, alpha increases regularization, while C decreases regularization.

Next Section: Regularization Strength and Solvers - tuning $\lambda$ via cross-validation, path algorithms (LassoCV, RidgeCV), and numerical solvers.

---

9. Active Recall

Test your understanding by answering first, then clicking to reveal the underlying principles.

Interactive Checkpoint Quiz

Interactive Checkpoint: Diagnosing Generalization Regimes

A machine learning practitioner trains a 20-layer neural network on tabular customer churn data. The model achieves 99.8% training accuracy, but cross-validation accuracy on held-out folds is only 64.2%. What is the primary diagnosis, and what intervention is appropriate?

Review Flashcards

1. [THEORY] Why does L1 regularization produce sparse models (exact zeros) while L2 regularization does not?

• The Gradient Perspective:
  • L2 penalty is $\frac{1}{2} \lambda w^2$. Its gradient is $\lambda w$. As $w \to 0$, the gradient diminishes proportionally to zero, shrinking weights slower and slower without reaching zero.
  • L1 penalty is $\lambda |w|$. Its subgradient is a constant $\pm \lambda$ regardless of magnitude. It continues pushing weights toward zero with full force until snapping to exact zero.
• The Geometric Perspective: The $L_1$ constraint region is a diamond with sharp corners on the coordinate axes. The unconstrained loss ellipses almost always touch the diamond at a corner where coordinates equal zero.

2. [MATH] In the Bias-Variance decomposition, what represents the theoretical lower bound on model error?

• Irreducible Error ($\sigma^2$): $\mathbb{E}[(y - \hat{f}(x))^2] = \text{Bias}^2 + \text{Variance} + \sigma^2$
• Even an omniscient Bayes-optimal model with zero bias and infinite training data (zero variance) will still suffer an expected error of $\sigma^2$.
• It represents unmeasured confounding variables, physical sensor limits, and intrinsic randomness.

3. [ANALYZE] When should you choose Elastic Net over standard Lasso (L1)?

• Correlated Feature Groups: When features are strongly collinear (e.g. 10 gene markers that activate together), Lasso arbitrarily selects one and zeroes out the remaining 9. Elastic Net groups them and preserves their collective signal.
• High Dimensions ($p > n$): In datasets where features exceed sample count, Lasso can select at most $n$ non-zero features before saturating. Elastic Net can select all $p$ features if necessary.

4. [PRACTICE] Why is feature scaling strictly mandatory before applying Ridge or Lasso regression?

• Scale Asymmetry: The penalty term $\sum w_j^2$ or $\sum |w_j|$ does not know what units features are measured in.
• If Feature 1 is Salary in rupees ($10^6$) and Feature 2 is Age in years ($10^1$), the OLS weight for Salary will naturally be tiny ($10^{-6}$), while the weight for Age will be large ($10^0$).
• Regularization will aggressively crush the Age weight while completely ignoring Salary! Standardizing with StandardScaler places all coefficients on a level playing field.

5. [THEORY] Why is the bias term (intercept b) excluded from the regularization penalty in scikit-learn?

• Centering the Output: The intercept $b$ captures the unconditional mean of the target variable $\mathbb{E}[y]$ when all standardized features are zero.
• Penalizing $b$ toward zero would force the prediction line down toward the origin, destroying accuracy even if the slope is perfectly learned.
• Regularization is designed to constrain feature interactions (slopes), not the global vertical baseline.