Multiple Linear Regression

Topic — The previous page fitted one feature and found something unresolved: age had a real effect of $-800$ that its marginal correlation of $-0.0392$ completely hid, and only became visible once size's contribution was removed. Doing that removal automatically, for every feature at once, is what multiple regression is. This page works out how, and then asks which measure can be trusted to tell you whether an added feature helped.

---

The model, and the meaning of one coefficient

Each feature gets its own coefficient:

$y = b_0 + b_1 x_1 + b_2 x_2 + \dots + b_k x_k$

With one feature this is a line, with two a plane, with more a hyperplane — impossible to picture and no different algebraically.

The definition of $b_i$ is the part that carries all the weight:

Holding every other feature in the model constant, one additional unit of $x_i$ is associated with a $b_i$ increase in $y$.

Both qualifications are load-bearing, and the first one is not a caveat but a description of what the arithmetic actually does. "Holding every other feature constant" means $b_i$ is fitted on the variation in $x_i$ that the other features cannot explain — exactly the residual-then-refit operation the previous page did by hand. "Is associated with" is not "causes"; regression measures association, and nothing about fitting a hyperplane establishes direction.

---

Solving for two coefficients by hand

The two-feature case has a closed form small enough to compute on paper, and doing so shows where the "holding constant" behaviour comes from mechanically.

Working in deviations from the means — $d_{1i} = x_{1i} - \bar{x}_1$, and so on — the least-squares coefficients satisfy two simultaneous equations, the normal equations:

$S_{11} \cdot b_1 + S_{12} \cdot b_2 = S_{1y}$

$S_{12} \cdot b_1 + S_{22} \cdot b_2 = S_{2y}$

$\text{where } S_{11} = \sum d_{1i}^2 \qquad S_{22} = \sum d_{2i}^2 \qquad S_{12} = \sum d_{1i}d_{2i}$

$S_{1y} = \sum d_{1i}d_{yi} \qquad S_{2y} = \sum d_{2i}d_{yi}$

$S_{11}$ and $S_{22}$ measure how much each feature varies. $S_{12}$ measures how much they vary together — and it is the only term that couples the two equations. Set $S_{12} = 0$ and they decouple completely: $b_1 = S_{1y}/S_{11}$ and $b_2 = S_{2y}/S_{22}$, each identical to its simple-regression value. Everything interesting about multiple regression lives in $S_{12}$.

---

The Collinearity & Determinant Solver

Experiment with the interactive solver below. Modify the coupling term $S_{12}$ (covariance between features) and the individual variances $S_{11}, S_{22}$ to watch how the simultaneous normal equations couple and decouple, how the determinant shrinks, and how the coefficients $b_1, b_2$ behave.

Interactive Normal Equations Solver

Feature 1 Var (S₁₁): 5.0

Feature 2 Var (S₂₂): 5.0

Coupling term (S₁₂): 2.00

The Coupled Normal Equations:

5.0·b₁ + 2.00·b₂ = 8

2.00·b₁ + 5.0·b₂ = 9

Determinant: det = S₁₁S₂₂ − S₁₂²

det = 21.0000

b₁ (Slope 1) = 1.0476

b₂ (Slope 2) = 1.3810

b₀ (Intercept) = 3.4286

Fitted: ŷ = 3.43 + 1.05x₁ + 1.38x₂

tip

That determinant is the whole story of collinearity

det = S₁₁S₂₂ − S₁₂² sits in the denominator of both coefficients. $S_{12}$ is the coupling term, so as the two features become more alike, $S_{12}^2$ grows toward $S_{11}S_{22}$ and the determinant heads to zero — dividing by something arbitrarily small. This is the algebraic source of every instability later on this page, and at exactly $S_{12}^2 = S_{11}S_{22}$ the determinant is zero and there is no unique solution at all, which is the next page's subject.

The same structure generalises: with $k$ features you get $k$ simultaneous equations, which is unwieldy by hand and elegant in matrix form — hence the next page.

---

What "holding all else constant" does to a coefficient

A coefficient's value depends on what else is in the model. Two cases on the house-price data, where the truth is $\text{price} = 3000 \cdot \text{size} - 800 \cdot \text{age} + 50000$:

Output

2. WHAT 'HOLDING ALL ELSE CONSTANT' DOES TO A COEFFICIENT
   size's DIRECT effect is 3000 in both models.
   but rooms = 2 + 0.05·size, so in the second model size ALSO acts
   through rooms: 3000 + 5000×0.05 = 3250 is its total marginal effect.

   pair              corr  size alone  size joint    shift
   size, age       0.0614     3043.86     3062.84   +18.98
   size, rooms     0.8244     3218.22     2921.04  -297.18

With a near-orthogonal partner, the coefficient barely moves. size and age correlate at $0.0614$, and size's coefficient shifts by $+18.98$ — under 1% — between the solo and joint fits. $S_{12} \approx 0$, so the equations nearly decouple and each feature is estimated as though the other were not there.

With a correlated partner, it moves substantially and for a specific reason. rooms is built as $2 + 0.05 \cdot \text{size}$, correlating with size at $0.8244$. Now size influences price through two channels: directly at $3000$, and through rooms at $5000 \times 0.05 = 250$. Its total marginal effect is therefore $3250$, and its direct effect is $3000$.

Total versus Direct Effect Sandbox

Toggle the model partner below to watch how size's coefficient transitions from measuring the total effect (when fitted alone) to measuring only the direct effect (when fitted jointly with rooms).

Total versus Direct Effect Sandbox

Toggle the model to see how holding `rooms` constant isolates the direct physical effect of home size.

Model A: Size fitted alone

Fitted Coefficient: ~3218.22

Because `rooms` is absent, `size` gets credit for both its direct value (3000) and the value of the extra rooms (5000 × 0.05 = 250) that larger size implies. It measures the total marginal effect.

danger

A coefficient is not a property of a feature

It is a property of a feature in a particular model. Add or drop a correlated column and it changes — $3218.22$ to $2921.04$ here. Reporting "size is worth 3000 per unit" without stating what else was in the model is not a rounding-level imprecision; it is an incomplete statement.

---

Marginal correlation is a poor screen

The obvious way to choose features is to correlate them with the target. On this data that ranking is:

• $\text{corr(size, price)} = +0.9849$
• $\text{corr(age, price)} = -0.0392$
• $\text{corr(door\_colour, price)} = +0.0419$

age, with a real effect of $-800$ per year, ranks below door_colour, which is pure noise. Any threshold that keeps door_colour keeps age, and any threshold that drops age drops nothing else of value — the screen simply carries no information about these two.

Now fit them in combination:

Features	$k$	Train $R^2$	CV $R^2$	Adjusted $R^2$
size	1	0.9700	0.9661	0.9698
size, age	2	0.9800	0.9773	0.9797
size, age, door_colour	3	0.9800	0.9766	0.9795
size, age + 20 noise	22	0.9838	0.9732	0.9802

Three readings, in increasing order of importance:

age genuinely helps. CV $R^2$ rises $0.9661 \to 0.9773$ when it joins size, despite that marginal correlation of $-0.0392$. The joint fit finds the effect the screen missed: once size is accounted for, age correlates with what remains at $-0.5758$.

door_colour is harmless in small numbers. Train $R^2$ unchanged at $0.9800$, CV $R^2$ down a hair to $0.9766$. One irrelevant column costs almost nothing.

Twenty noise columns raised train $R^2$ to $0.9838$ while CV $R^2$ fell to $0.9732$. Both numbers moved, in opposite directions, from the same change.

danger

Training $R^2$ never decreases when you add a feature

Not "rarely" — never, as a mathematical fact. The model can always set a new coefficient to zero and do exactly as well as before, so the best achievable fit is at least as good as it was. Twenty columns of noise bought $+0.0038$ of training $R^2$ and cost $-0.0041$ of measured out-of-sample performance.

So training $R^2$ cannot be used to decide whether a feature belongs.

The standard answer to that problem is Adjusted $R^2$, which penalises the model for the number of features it uses. Look at its column above: $0.9797$ for size, age, and 0.9802 for the same model plus twenty columns of pure noise. It ranks the noise model higher.

---

Does Adjusted R² actually catch the noise?

That single comparison could be luck, so here is the sweep: noise columns from 0 to 100 on $n = 120$ rows, every row averaged over 30 independent regenerations of both the data and the noise.

Adjusted R² vs. CV R² Noise Sweeper

Adjust the slider below to add between 0 and 100 columns of pure Gaussian noise to the baseline size, age model. Watch how Training $R^2$, Adjusted $R^2$, and Cross-Validated $R^2$ behave in real-time.

Interactive Complexity Sweeper

Add pure noise columns to watch Adjusted R² fail to penalize overfitting while CV R² catches it.

Useless Noise Columns: 0 (k = 2 total features)

● Train R²● Adjusted R²● CV R²

Train R²: 0.9782 (Inflated by overfitting)

Adjusted R²: 0.9779 (Tricked! Baseline was 0.9779)

CV R²: 0.9751 (Collapsing!)

Why the penalty fails

Adjusted $R^2$ is:

$\text{adj } R^2 = 1 - (1 - R^2) \times \frac{n - 1}{n - k - 1}$

and the mechanism is a race between two quantities that happen to grow at similar rates. As $k$ rises, the penalty factor $\frac{n-1}{n-k-1}$ grows — at $k = 102, n = 120$ it is $119/17 = 7.0$, a sevenfold inflation of the unexplained variance. But $(1 - R^2)$ is shrinking at almost exactly the rate that offsets it, because spurious columns inflate train $R^2$: from $0.9782$ to $0.9971$, so $(1 - R^2)$ goes $0.0218 \to 0.0029$, a factor of 7.5.

Multiply the two and you get roughly what you started with. The penalty is real, correctly implemented, and almost exactly cancelled by the overfitting it is meant to expose.

danger

Adjusted R² is not an overfitting guard

At $n = 120$ it failed to flag 100 columns of pure noise. It is a mild correction for the fact that $R^2$ cannot decrease — useful for comparing two models that differ by a feature or two, and no substitute for measuring out-of-sample performance.

The number that worked here is cross-validated $R^2$, which fell monotonically and by $0.3847$. When train $R^2$ and CV $R^2$ diverge, the training number is the one lying — and Adjusted $R^2$ is close enough to a training number to lie with it.

This is also why the honest form of the earlier claim matters: Adjusted $R^2$ and cross-validation are often presented as two interchangeable remedies for feature-count inflation. On this data they are not remotely interchangeable. Only one of them worked.

---

When coefficients stop meaning anything

The determinant warning from the hand-worked fit, measured. Two features, each with a true coefficient of exactly $3.00$, fitted 40 times at increasing correlation between them:

$\text{corr}(x_1, x_2)$	Mean $coef_1$	Mean $coef_2$	Sum	SD of $coef_1$
0.000	2.9927	3.0017	5.9944	0.0646
0.500	2.9918	3.0019	5.9937	0.0718
0.900	2.9893	3.0038	5.9931	0.1426
0.990	2.9810	3.0118	5.9928	0.4500
0.999	2.9554	3.0373	5.9928	1.4312

Read the columns carefully, because the result is subtle:

• The means stay correct. $2.9554$ and $3.0373$ at the extreme — still centred near $3.00$. Collinearity does not bias the coefficients.
• The sum stays exact. $5.9928$ in the last three rows, against a true total of $6.00$, and it barely moves across the whole table.
• The spread explodes. Standard deviation of the first coefficient goes $0.0646 \to 1.4312$ — 22× larger — and most of that happens between $0.99$ and $0.999$.

The interpretation is precise and worth stating carefully: when two features carry nearly the same information, the data determines their combined effect perfectly well and says almost nothing about how to divide it between them. Any individual fit picks a split more or less arbitrarily, and the next sample picks a different one.

This is multicollinearity. Its consequences:

Affected	How
Predictions	Barely — the sum is what drives them, and the sum is stable
Individual coefficients	Severely — unstable, and can even flip sign
Interpretation	Fatally — "size adds 3000" is not a supportable claim
Standard errors	Inflated, so significance tests fail to reject

What ridge does about it

Ridge regression adds a penalty on the size of the coefficients, which resolves the ambiguity by preferring the split that keeps them small. At $\text{corr} = 0.999$:

Model	Mean $coef_1$	Mean $coef_2$	SD of $coef_1$	Sum	Test $R^2$
LinearRegression	3.2657	2.7314	1.8311	5.9971	0.9710
Ridge(alpha=1.0)	3.0206	2.9563	0.2378	5.9769	0.9713

The stability gain is dramatic. Coefficient spread drops $1.8311 \to 0.2378$, an 87% reduction, and the split becomes even: the gap between the two means narrows from $0.5342$ to $0.0644$.

The prediction gain is nil. Test $R^2$ goes $0.9710 \to 0.9713$ — an improvement of 0.0003, which is nothing.

That pairing is the lesson, and it confirms the thesis of the table above from the opposite direction: since collinearity was never damaging the predictions, fixing collinearity cannot improve them. What ridge buys is coefficients you can report and reproduce. Whether that is worth a small bias depends entirely on whether anyone is going to read the coefficients.

note

Regularisation gets its own treatment later

Ridge appears here only as the direct answer to collinearity. The penalty term, choosing its strength, and the different behaviour of lasso — which drives coefficients to exactly zero rather than merely shrinking them — belong with generalisation and model control.

---

2. Implementation Lab

tip

Lab Exercise: Multiple Linear Regression

An interactive, fully-functional Google Colab / Jupyter Notebook is available to experiment with this concept live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Click "Run all" or execute cells individually using Shift + Enter.

Things to try inside the script:

• Experiment 1 (Determinant Crash): In section 1, change x2 to exactly 2 * x1. Watch det = 0 and OLS crash with a singular matrix division-by-zero, while Ridge handles it safely.
• Experiment 2 (Spurious k/n ratios): In section 4, raise $n$ from 120 to 2000 while keeping the noise counts. Find the ratio where Adjusted $R^2$ finally starts falling.
• Experiment 3 (Ridge sweep): In section 6, sweep alpha over [0.01, 0.1, 1, 10, 100] to find the perfect trade-off between coefficient stability and test $R^2$ score.

---

3. Common Mistakes

1. Absolute Coefficients

• The Misconception: "A coefficient is an absolute property of a feature."
• Wrong Thinking: Believing size is always worth 3000 per unit, regardless of what other columns (like rooms) are included in the fit.
• The Right Principle: A coefficient is a property of a feature in a particular model context. Adding or dropping correlated columns immediately shifts other weights (e.g., $3218.22 \to 2921.04$ once rooms joined) as credit is shared.

2. Solo Correlation Screening

• The Misconception: "I should drop features that have near-zero correlation with the target."
• Wrong Thinking: Removing age because its solo correlation with house price is $-0.0392$.
• The Right Principle: Dominant features can completely mask secondary signals. Once dominant features (like size) are controlled for, hidden correlations become extremely clear (e.g., age partial correlation was $-0.5758$, significantly boosting CV $R^2$).

3. Trusting Adjusted R²

• The Misconception: "Adjusted R² is a robust guard against overfitting."
• Wrong Thinking: Adding 100 noise columns because Adjusted $R^2$ read $0.9794$ (higher than the clean 2-feature baseline of $0.9779$).
• The Right Principle: Adjusted $R^2$ is not an overfitting guard. Spurious columns inflate training $R^2$ so rapidly that the inflation nearly cancels out the feature-count penalty. Always trust out-of-sample Cross-Validation scoring.

4. Collinearity biases fits

• The Misconception: "Multicollinearity introduces a bias into our regression coefficients."
• Wrong Thinking: Believing collinearity makes the averages of fitted coefficients drift away from the true values.
• The Right Principle: Collinearity does not bias coefficients—their means stay exactly correct. Instead, it makes coefficient variance explode, creating massive instability where individual weights swing wildly.

---

4. Summary

Core Theory

• Normal Equations: Simultaneous equations satisfied by OLS weights. Coupled by the covariance term $S_{12}$.
• Decoupling Rule: If $S_{12} = 0$, the equations decouple, and multiple regression collapses into independent simple regressions.
• Collinearity Root: The determinant $S_{11}S_{22} - S_{12}^2$ sits in the denominator of OLS coefficients. As correlation climbs, determinant heads to zero, blowing up coefficient sizes.
• Total vs. Direct Effect: Solo fits capture total effects across all channels; joint fits isolate only the direct effect of each feature.

Engineering Realities

• R² Never Decreases: Training $R^2$ is mathematically guaranteed to increase or stay constant as columns are added, making it useless for complexity evaluation.
• Adjusted R² Flatness: Adjusted $R^2$ fails completely under heavy noise columns because overfitting and degrees-of-freedom penalties cancel out.
• Generalization general: Out-of-sample Cross-Validated $R^2$ is the only metric that monotonically captures generalization collapse.
• Ridge regularisation: Ridge adds an L2 size penalty to coefficients, stabilizing their spread (variance) under extreme collinearity without biasing the means.

info

Key Takeaways

• Sum Invariance: Under multicollinearity, while individual coefficients are unstable and change wildly, their combined sum remains perfectly stable and predictions remain robust.
• Interpretation vs. Prediction: Collinearity completely destroys model interpretation ("how much size adds"), but has almost no effect on prediction performance.
• Ridge trade-off: Ridge regression does not improve accuracy (test $R^2$ stays flat); it buys reportable, stable, and reproducible coefficients.

Next in this section: The Matrix Formulation — the same solution for any number of features in one algebraic step, and what happens when correlation reaches exactly 1.0.

---

5. Active Recall

Attempt each question first, then click to reveal detailed answers, mathematical derivations, and sample explanations.

Interactive Checkpoint Quiz

Interactive Checkpoint: The Collinearity Conundrum

Under severe multicollinearity (e.g. $r = 0.999$), what happens to our model's predictions and its individual coefficients?

Review Flashcards

1. [THEORY] What does $S_{12} = 0$ mean, and what happens to the simultaneous normal equations?

• The Covariance Rule: $S_{12} = \sum d_{1i} d_{2i}$ measures the covariance (shared variation) between features $x_1$ and $x_2$.
• If $S_{12} = 0$, the features are completely orthogonal (uncorrelated).
• Substituting $S_{12} = 0$ into the simultaneous normal equations: $S_{11} b_1 + 0 = S_{1y} \implies b_1 = \frac{S_{1y}}{S_{11}}$ $0 + S_{22} b_2 = S_{2y} \implies b_2 = \frac{S_{2y}}{S_{22}}$
• The equations decouple completely. Multiple linear regression collapses into two completely independent simple linear regressions, with both slopes equal to their solo values.

2. [ANALYZE] How does the determinant $S_{11}S_{22} - S_{12}^2$ head to zero, and why does this explode coefficient variance?

• The Math: Solve the normal equations using Cramer's rule. The determinant sits in the denominator of both coefficients $b_1$ and $b_2$: $b_1 = \frac{S_{1y}S_{22} - S_{12}S_{2y}}{S_{11}S_{22} - S_{12}^2}$
• As features become more collinear, their covariance $S_{12}^2$ increases, heading towards the maximum limit $S_{11}S_{22}$ (where $r = 1.0$).
• This forces the denominator (determinant) closer and closer to $0$. Dividing by an arbitrarily tiny number makes the calculated coefficient value extremely sensitive to minor fluctuations in target measurements ($S_{1y}, S_{2y}$), causing coefficient variance to explode.

3. [THEORY] Why is it mathematically impossible for training $R^2$ to decrease when you add a feature?

• The Optimization Rule: OLS operates by finding coefficients $b_j$ that minimize the Sum of Squared Errors (SSE): $\text{Minimize } \sum (y_i - b_0 - b_1 x_{1i} - \dots - b_k x_{ki})^2$
• If you add a new feature $x_{k+1}$, the optimizer has a new parameter $b_{k+1}$ to tune.
• In the worst-case scenario where the new feature is completely useless noise, the optimizer can simply assign $b_{k+1} = 0$, achieving the exact same SSE (and thus the exact same $R^2$) as before.
• Because the search space of the optimization has expanded, the minimum SSE can only stay the same or decrease, meaning training $R^2$ can never decrease.

4. [ANALYZE] Why does Adjusted $R^2$ completely fail as an overfitting guard against multiple noise columns?

• The Formula: $\text{adj } R^2 = 1 - (1 - R^2) \times \frac{n-1}{n-k-1}$.
• Adding noise columns triggers a literal race between two opposing terms:
  1. As $k$ increases, the penalty ratio $\frac{n-1}{n-k-1}$ grows (at $k=102, n=120$, it inflates unexplained variance by 7.0×).
  2. However, because of overfitting, the training unexplained variance $(1 - R^2)$ shrinks from $0.0218 \to 0.0029$ (a 7.5× reduction).
• The Result: The two factors nearly cancel each other out, making Adjusted $R^2$ remain completely flat and completely blind to the fact that the model is fitting pure noise.

5. [THEORY] How does Ridge regularisation resolve coefficient instability without improving prediction accuracy?

• Coefficient Stability: Ridge regression adds an L2-norm penalty ($\alpha \sum b_j^2$) to the OLS cost function. When features are collinear, infinite splits of the coefficients yield the same prediction. OLS picks a split randomly, but Ridge enforces stability by picking the split that keeps both coefficients small and evenly distributed, cutting standard deviation by over 85%.
• No Prediction Gain: Because OLS was already finding a stable combined sum, its predictions were already robust. Restricting the coefficients via Ridge does not change this sum—it merely changes the individual allocation, leaving prediction score virtually unchanged ($\approx +0.0003$ gain).