Bias-Variance Tradeoff

Overview

The bias-variance tradeoff is one of the most important concepts in machine learning. It explains why models fail and how to balance model complexity.

Core Insight:
  Total Error = Bias² + Variance + Irreducible Error

You CANNOT minimize both bias and variance simultaneously.
Balancing them is the art of machine learning.

---

Part 1: Intuitive Understanding

What is Bias?

Bias = Error from model being too SIMPLE
       = Systematic error across different datasets
       = Model's assumptions are too strong

Analogy: 
  Archer always shoots LEFT of target
  (biased aim, not random)

Examples of Biased Models:

Problem: Predicting house price from size alone
  ✗ Ignores location (major factor)
  ✗ Ignores age, condition, amenities
  ✓ Simple but systematically wrong

Solution: Add more features or use non-linear model

What is Variance?

Variance = Error from model being too COMPLEX
         = Model's sensitivity to small data changes
         = Different datasets → wildly different predictions

Analogy:
  Archer's shots are scattered all over
  (inconsistent, high spread)

Examples of High Variance Models:

Problem: Polynomial regression degree 10 on 5 data points
  ✗ Fits training data perfectly
  ✗ But tiny data change → huge prediction change
  ✓ Complex but unstable

Solution: Reduce complexity, get more data, use regularization

The Tradeoff Visualized

Model Complexity →

Underfitting          Good Fit           Overfitting
(High Bias)      (Balance)              (High Variance)

    Model too simple      Model just right    Model too complex
    ╱╲                    ╱─╲                 ╱──╲
   ╱  ╲                  ╱    ╲               ╱     ╲
  ╱    ╲                ╱      ╲             ╱________╲
  
Train: ░░░░░░░░░░░░    Train: ░░░░░░░░░░░░  Train: ░
Test:  ░░░░░░░░░░░░    Test:  ░░░░░░░░░░░░  Test:  ░░░░░░░░░░░░░
       (High error)           (Low error)         (High test error!)

---

Part 2: Formal Definitions

Bias Formally

Bias(f̂) = E[f̂(x)] - f(x)

where:
  f̂(x)      = predicted value from our model
  f(x)       = true value
  E[...]     = expected value (average over all datasets)

Interpretation:
  • 0 bias = model predicts correctly on average
  • High bias = model systematically wrong

Variance Formally

Var(f̂) = E[(f̂(x) - E[f̂(x)])²]

where:
  f̂(x)       = predicted value from our model
  E[f̂(x)]    = average prediction across all datasets
  
Interpretation:
  • Low variance = predictions consistent across datasets
  • High variance = predictions vary widely

Bias-Variance Decomposition

Expected Test Error = Bias² + Variance + σ²

where:
  Bias²        = (systematic error)²
  Variance     = average squared deviation from mean prediction
  σ²           = irreducible error (noise in data)

σ² is FIXED (cannot reduce it, it's data noise)
You control Bias² and Variance!

---

Part 3: The Tradeoff Curve

Theoretical Relationship

Total Error = Bias² + Variance + Noise

As model complexity increases:

  Bias decreases        (fits more patterns)
  BUT Variance increases (becomes unstable)

                Total Error
                    ╱╲
              Variance ╱  ╲
               /              ╲ Bias²
              ╱                  ╲
         ╱________________________╲
        Simple            Complex
        Model            Model
        
    Optimal complexity is where sum is minimized

Practical Error Curve

Error
  │     Overfitting Region
  │     (High Variance)
  │        ╱╲
  │       ╱  ╲
  │      ╱    ╲
  │     ╱      ╲
  │    ╱        ╲ Optimal Zone
  │   ╱          ╲
  │  ╱            ╲
  │ ╱              ╲ Underfitting
  │╱________________╲ (High Bias)
  └─────────────────────→ Model Complexity

Key Points:
  • Too simple: high bias, low variance
  • Too complex: low bias, high variance
  • Sweet spot: balanced error

---

Part 4: Detecting Bias vs Variance

Train vs Test Error Analysis

Metric              Underfitting        Good Fit          Overfitting
                    (High Bias)                           (High Variance)
────────────────────────────────────────────────────────────────────
Train Error         ✗ HIGH              ✓ LOW             ✓ VERY LOW
Test Error          ✗ HIGH              ✓ LOW             ✗ VERY HIGH
Gap (Test-Train)    ✓ SMALL             ✓ SMALL           ✗ LARGE
────────────────────────────────────────────────────────────────────
Problem             Model too           Model just        Model too
                    simple              right             complex
Solution            Increase            Keep as is        Reduce
                    complexity                           complexity

Diagnosis Chart

         Train Error
            ↓
       Low      High
       │          │
T  H   │ GOOD │ HIGH BIAS
e  I   ├──────┼──────────
s  G   │      │
t  H   │ HIGH │ BOTH
       │VARIANCE│
Low    │      │
       └──────┴──────────→ Training Error

Visual Diagnosis

Training Data vs Test Data Plot:

1. High Bias (Underfitting):
   Train Error = 0.30
   Test Error = 0.32
   (Both high, gap small)
   
   Solution: More complex model

2. Good Fit:
   Train Error = 0.08
   Test Error = 0.09
   (Both low, gap small)
   
   Solution: Use this model

3. High Variance (Overfitting):
   Train Error = 0.01
   Test Error = 0.25
   (Gap HUGE!)
   
   Solution: Simpler model, more data, regularization

---

Part 5: Bias-Variance in Action

Worked Example: Polynomial Regression

Setup

True relationship: y = 2 + 3x + ε
                   (linear with noise)

Data: 50 points from this relationship

Degree 1 (Linear Model)

Model: ŷ = b₀ + b₁x

Result:
  Training Error:  0.95
  Test Error:      0.98
  
  Bias:   HIGH (linear assumptions fit data well)
  Variance: LOW (won't change much with different data)
  
Diagnosis: GOOD FIT

Degree 3 (Cubic Polynomial)

Model: ŷ = b₀ + b₁x + b₂x² + b₃x³

Result:
  Training Error:  0.92
  Test Error:      0.96
  
  Bias:   LOWER (captures more curvature)
  Variance: LOW (still stable)
  
Diagnosis: STILL GOOD (slight improvement)

Degree 10 (Very Complex)

Model: ŷ = b₀ + b₁x + ... + b₁₀x¹⁰

Result:
  Training Error:  0.001    ← Almost perfect!
  Test Error:      0.85     ← Terrible on new data!
  
  Bias:   LOW (fits everything)
  Variance: VERY HIGH (wiggly curve, unstable)
  
Diagnosis: OVERFITTING (high variance)

Summary

Complexity      Bias    Variance   Test Error   Problem
─────────────────────────────────────────────────────
Linear (1)      HIGH    LOW        0.98         ✓ OK
Cubic (3)       LOWER   LOW        0.96         ✓ GOOD
Degree 10       LOW     VERY HIGH  0.85         ✗ OVERFITTING

Lesson: Don't always use most complex model!

---

Part 6: Mathematical Example

Concrete Bias-Variance Decomposition

True function: f(x) = x
Model 1: f̂₁(x) = 0 (always predicts 0)
Model 2: f̂₂(x) = x + random noise from dataset

At x = 1:

Model 1 (Constant 0)

True value f(1) = 1

Predictions on different datasets:
  Dataset 1: f̂ = 0
  Dataset 2: f̂ = 0
  Dataset 3: f̂ = 0
  
E[f̂] = 0 (average of all predictions)

Bias = E[f̂] - f = 0 - 1 = -1
Bias² = 1

Variance = E[(f̂ - E[f̂])²]
         = E[(0 - 0)²]
         = 0

Total Error = Bias² + Variance
            = 1 + 0
            = 1

Interpretation:
  • HIGH BIAS (always wrong by 1)
  • NO VARIANCE (always same prediction)
  • UNDERFITTING

Model 2 (Fits data + noise)

True value f(1) = 1

Predictions on different datasets:
  Dataset 1: f̂ = 0.8 (because noise=-0.2)
  Dataset 2: f̂ = 1.3 (because noise=+0.3)
  Dataset 3: f̂ = 1.0 (because noise=0)
  
E[f̂] = 1.03 ≈ 1 (average, close to true)

Bias = E[f̂] - f = 1.03 - 1 = 0.03
Bias² ≈ 0 (close to zero!)

Variance = E[(f̂ - E[f̂])²]
         = E[(f̂ - 1.03)²]
         = average squared deviation
         ≈ 0.1 (varies by ±0.2 or 0.3)

Total Error = Bias² + Variance
            = 0 + 0.1
            = 0.1

Interpretation:
  • LOW BIAS (unbiased on average)
  • SOME VARIANCE (predictions vary)
  • BETTER overall

---

Part 7: Bias-Variance vs Overfitting/Underfitting

Relationship

Underfitting = High Bias + Low Variance
Overfitting = Low Bias + High Variance
Good Fit = Low Bias + Low Variance

                     Variance
                    Low    High
        ┌─────────────────────┐
Low     │ GOOD FIT │ OVER    │
Bias    ├──────────┤ FITTING │
        │          │         │
High    │ UNDER    │ BOTH    │
        │ FITTING  │ PROBLEMS│
        └─────────────────────┘

Key Insight

Overfitting ≠ Variance
  High variance model might underfit
  
Underfitting ≠ Bias
  High bias model might overfit (rare but possible)

But typically:
  • High bias → underfitting
  • High variance → overfitting

---

Part 8: Solutions

For High Bias (Underfitting)

Problem: Model too simple, systematic errors

Solutions (ranked by effectiveness):

1. INCREASE MODEL COMPLEXITY
   • Add more features
   • Use polynomial instead of linear
   • Use more flexible algorithm
   
2. DECREASE REGULARIZATION
   • Reduce λ (if using Ridge/Lasso)
   • Allow larger coefficients
   
3. TRAIN LONGER
   • More iterations (if using gradient descent)
   • Might not help if model fundamentally limited

4. ENGINEER BETTER FEATURES
   • Create interaction terms (x₁ × x₂)
   • Transform features (log, sqrt)
   • Domain knowledge
   
Examples:
  House price with size only:
    Add: age, location, condition, bedrooms
  
  Linear regression on non-linear data:
    Add: x², x³ terms

For High Variance (Overfitting)

Problem: Model too complex, unstable predictions

Solutions (ranked by effectiveness):

1. GET MORE DATA
   • Simpler models needed with less data
   • 2x data often reduces variance significantly
   • Most effective but often impractical
   
2. REDUCE MODEL COMPLEXITY
   • Remove features (especially noisy ones)
   • Use simpler algorithm
   • Lower polynomial degree
   
3. INCREASE REGULARIZATION
   • Increase λ (penalize large coefficients)
   • Ridge: shrink all coefficients
   • Lasso: eliminate some coefficients
   
4. USE CROSS-VALIDATION
   • Detect overfitting early
   • Choose regularization parameter
   • Better error estimates
   
Examples:
  Polynomial degree 10 on small dataset:
    Reduce to degree 2-3
  
  100 features on 50 observations:
    Use feature selection or regularization

---

Part 9: Bias-Variance in Different Models

Linear Regression

By default: LOW BIAS, LOW VARIANCE
  • Makes strong linearity assumption
  • Stable coefficients
  
When problems occur:
  • If true relationship non-linear → HIGH BIAS
  • If too many features relative to samples → HIGH VARIANCE

Decision Trees

By default: LOW BIAS, HIGH VARIANCE
  • Can fit any pattern (very flexible)
  • Small data change → completely different tree
  
Solutions:
  • Pruning (remove branches) → reduce variance
  • Ensemble (bag many trees) → reduce variance

K-Nearest Neighbors

k=1: LOW BIAS, VERY HIGH VARIANCE
  • Fits each point exactly
  • Extremely sensitive to noise
  
k=n: VERY HIGH BIAS, LOW VARIANCE
  • Predicts mean for everything
  • Very stable but wrong
  
Optimal k: Somewhere in middle

Neural Networks

Deep network: LOW BIAS, HIGH VARIANCE
  • Can learn very complex patterns
  • But memorizes noise/training quirks
  
Shallow network: HIGH BIAS, LOW VARIANCE
  • Cannot learn complex patterns
  • But stable predictions
  
Solutions:
  • Dropout (disable random neurons)
  • Batch normalization
  • L1/L2 regularization
  • Early stopping

---

Part 10: Bias-Variance in Cross-Validation

How CV Estimates Bias-Variance

K-fold CV with k=5:

Fold 1: Train on [2,3,4,5], Test on [1]
Fold 2: Train on [1,3,4,5], Test on [2]
Fold 3: Train on [1,2,4,5], Test on [3]
Fold 4: Train on [1,2,3,5], Test on [4]
Fold 5: Train on [1,2,3,4], Test on [5]

Results:
  • Average CV error ≈ true test error
  • High variance CV (std dev) → unstable model
  • Low variance CV (std dev) → stable model

Example:
  CV scores: [0.10, 0.11, 0.09, 0.10, 0.11]
  Mean: 0.10, Std: 0.01 → LOW VARIANCE model
  
  CV scores: [0.05, 0.25, 0.15, 0.30, 0.08]
  Mean: 0.16, Std: 0.11 → HIGH VARIANCE model (unstable!)

---

Part 11: Learning Curves

What are Learning Curves?

Plots showing how error changes with training set size

Helps diagnose bias vs variance issues

High Bias (Underfitting)

Error
  │ Training Error      ─ ─ ─ ─ ─ ─
  │                    ╱
  │ ─ ─ ─ ─ ─ ─ ─ ─ ─
  │ Test Error       ╱
  │                ╱
  │              ╱
  │            ╱
  └─────────────────→ Training Set Size
  
Characteristics:
  • Both curves high and flat
  • Getting more data won't help much
  • Need to change model, not get more data

High Variance (Overfitting)

Error
  │ Test Error      ─ ─ ─ ╲
  │                        ╲
  │                         ╲
  │                          ╲
  │                           ╲ converges slowly
  │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
  │ Training Error ╱
  │              ╱
  │            ╱
  │          ╱
  └─────────────────→ Training Set Size
  
Characteristics:
  • Large gap between train and test
  • Gap narrows as more data added
  • Getting more data WILL help!

Good Fit

Error
  │ Test Error    ─ ╲
  │                 ╲
  │ Training Error  ╲ ─ ─ ─
  │ and Test Error  ╱
  │    converge    ╱
  │              ╱
  │            ╱
  │          ╱ (small gap)
  └─────────────────→ Training Set Size

---

Part 12: Exam Questions & Solutions

Q1: Diagnosis from Error Metrics

Q: Your model has:

• Training error: 0.05
• Test error: 0.30

What's the problem and how would you fix it?

A:

Problem: HIGH VARIANCE (OVERFITTING)

Evidence:
  • Training error very low (0.05)
  • Test error very high (0.30)
  • Large gap (0.25) indicates overfitting

Solutions (prioritized):
  1. Get more data (if possible) - most effective
  2. Reduce model complexity
  3. Add regularization (Ridge/Lasso)
  4. Use feature selection (remove noisy features)
  5. Use cross-validation to choose hyperparameters

---

Q2: Learning Curve Interpretation

Q: Your learning curve shows:

• Training error: 0.15 (at large dataset)
• Test error: 0.18 (at large dataset)
• Still converging (not flat)

What do you conclude?

A:

Conclusion: LIKELY HIGH BIAS

Evidence:
  • Both errors high and similar
  • Still room to improve
  • Not due to variance (train ≈ test)

Action: Increase model complexity
  • Add more features
  • Use more flexible model
  • Increase polynomial degree
  
Note: Getting more data won't help much
      (already have enough data to show problem)

---

Q3: Model Complexity Choice

Q: You're choosing between: Model A: Linear regression (R²=0.72, Test RMSE=0.45) Model B: Polynomial degree 5 (R²=0.89, Test RMSE=0.52)

Which to use and why?

A:

Choose: Model A (Linear)

Reasoning:
  • Model A: Lower test RMSE (0.45 < 0.52)
  • Model B: Higher training R² but worse test performance
  • Model B likely overfitting
  
Bias-Variance perspective:
  Model A: Balanced bias-variance
  Model B: Low bias but high variance
  
When test error increases while training improves:
  → Clear sign of overfitting
  → Simpler model is better

---

Q4: Ridge vs Lasso for Variance Reduction

Q: You have high variance (overfitting) with multiple regression. Should you use Ridge or Lasso?

A:

Either Ridge or Lasso works, but:

Ridge Regression (L2):
  ✓ Shrinks all coefficients
  ✓ Reduces variance
  ✓ Keeps all features
  ✓ Use when all features might be relevant

Lasso Regression (L1):
  ✓ Shrinks and eliminates some coefficients
  ✓ Reduces variance
  ✓ Performs feature selection
  ✓ Use when want to identify key features

Recommendation:
  If you suspect some features are noise → Lasso
  If all features should be relevant → Ridge
  If unsure → Elastic Net (combines both)

---

Q5: Bias-Variance Tradeoff Decision

Q: Given scenario:

• You can either: (A) Add 5 new features, or (B) Get 2x more data

You suspect high variance. Which helps more and why?

A:

Choose: (B) Get 2x more data

Why:
  (A) Add 5 new features:
      ✗ Increases model complexity
      ✗ Worsens variance
      ✗ Wrong for high variance problem
  
  (B) Get 2x more data:
      ✓ Reduces variance directly
      ✓ Model becomes more stable
      ✓ Learning curves show this helps most
      ✓ Correct solution for high variance

High variance solution hierarchy:
  1. More data (most effective)
  2. Reduce complexity
  3. Regularization
  4. Feature selection
  
NOT: Add more features (worsens variance)

---

Part 13: Key Concepts Summary

Bias-Variance Decomposition

Expected Test Error = Bias² + Variance + Irreducible Noise

Bias²: How far from true value on average
Variance: How much predictions vary across datasets
Noise: Random error in data (cannot reduce)

The Fundamental Tradeoff

As model complexity increases:
  Bias decreases (fits more patterns)
  Variance increases (becomes unstable)

Goal: Find complexity that minimizes total error

Diagnosis Table

Error Pattern	Bias	Variance	Solution
Train ≈ Test (both high)	HIGH	LOW	Increase complexity
Train << Test	LOW	HIGH	Decrease complexity
Train ≈ Test (both low)	LOW	LOW	Use this model

When to Worry

High Bias Indicators:
  ✗ Training error high
  ✗ Model systematic error
  → Need more complex model

High Variance Indicators:
  ✗ Large train-test gap
  ✗ Model unstable
  ✓ Learning curve shows test error barely improving
  → Need simpler model or more data

---

Part 14: Practice Problems

Problem 1: Error Analysis

Scenario: Predicting student GPA

Model: Linear regression on 10 features

Results:
  Training error: 0.08
  Validation error: 0.09
  Test error: 0.10

Questions:
a) Does this indicate bias or variance problem?
b) Would you add more features?
c) Would you get more data?
d) How would you improve the model?

Problem 2: Learning Curve Interpretation

Scenario: Image classification model

At n=1000:
  Train error: 0.02
  Test error: 0.15

At n=10000:
  Train error: 0.03
  Test error: 0.12

At n=100000:
  Train error: 0.04
  Test error: 0.10

Questions:
a) What bias-variance issue is occurring?
b) Does getting more data help? By how much?
c) What's the best next step?

Problem 3: Model Selection

Three models available:

Model 1: Decision Tree (max_depth=3)
  CV Score: 0.78 ± 0.02

Model 2: Decision Tree (max_depth=15)
  CV Score: 0.85 ± 0.08

Model 3: Decision Tree (max_depth=30)
  CV Score: 0.84 ± 0.15

Questions:
a) Which model shows best average performance?
b) Which model is most stable?
c) Which would you choose for production? Why?

---

Part 15: Real-World Applications

Example 1: Email Spam Detection

Simple model (logistic regression):
  Train accuracy: 95%
  Test accuracy: 94%
  → Low bias, low variance ✓

Complex model (neural network):
  Train accuracy: 99%
  Test accuracy: 88%
  → Low bias, high variance ✗

Solution: Use simple model
         Or regularize complex model

Example 2: Stock Price Prediction

Linear model: 
  Consistently wrong (high bias)
  → Trends are non-linear

Polynomial model:
  Fits training data perfectly but
  Horrible on test data (high variance)
  → Overfitting to noise

Ensemble of medium-complexity models:
  Balanced bias and variance ✓
  → Better generalization

Example 3: Medical Diagnosis

Problem: Predicting disease from symptoms

Simple model:
  Misses some cases (false negatives)
  High bias

Complex model:
  Many false positives
  High variance
  
Solution: Adjust decision threshold
         Balance sensitivity vs specificity
         Use regularization

---

Tips for CT Exam

✅ DO

• Always compare train vs test error - immediate diagnosis
• Mention specific solutions - not just "reduce bias"
• Reference learning curves - shows understanding of theory
• Use the bias-variance decomposition formula - shows mathematical knowledge
• Connect to model complexity - key relationship

❌ DON'T

• Don't confuse bias and variance
• Don't say "get more features" for variance problems
• Don't ignore the irreducible error term
• Don't use only training error for diagnosis
• Don't recommend solutions without explaining why

---

Summary Table

Concept	Definition	Problem	Solution
Bias	Systematic error	Underfitting	More complex model
Variance	Instability	Overfitting	Simpler model / More data
Tradeoff	Cannot minimize both	Must balance	Cross-validation
High Bias	Train Error ≈ Test Error (HIGH)	Simple model	Complexity ↑
High Variance	Train Error << Test Error	Complex model	Complexity ↓

---

Next Topics

→ Performance Metrics
→ Gradient Descent
→ Regularization