Simple Linear Regression

Topic — The first actual algorithm, and the one worth understanding completely, because almost everything later is a variation on it. This page fits a line to a single feature and works the arithmetic all the way through by hand — two coefficients, five error measures, no estimator doing anything you cannot check on paper.

---

Regression versus classification

The distinction from Types of Learning, restated because this is where it starts to matter:

	Regression	Classification
Predicts	A continuous number	A category from a fixed set
Target type	Numeric	Categorical
Does "close" count?	Yes — ₹52 lakh for a ₹50 lakh house is good	No — "cat" for a dog is simply wrong
Error measured as	A distance	A count of mistakes
Output range	Unbounded	One of $k$ labels

The load-bearing difference is the third row. Because closeness counts, regression can be scored by how far off it was — which is what makes squared error the natural loss, and what makes the whole algebraic solution below possible.

---

The line, and where its two numbers come from

One feature, one target, a straight line:

$y = b_0 + b_1 \cdot x$

Symbol	Name	Meaning
$b_1$	slope / coefficient	How much $y$ changes per one-unit increase in $x$
$b_0$	intercept	The value of $y$ when $x = 0$

Fitting means choosing $b_0$ and $b_1$ to minimize the sum of squared vertical distances between each point and the line. That phrase is usually where explanations stop. It shouldn't — the two numbers come out of a short calculation you can do on paper, and doing it once removes most of the mystery from everything that follows.

The formulas

Minimizing $\sum (y_i - b_0 - b_1 x_i)^2$ — differentiate with respect to each coefficient, set both derivatives to zero, solve — gives exactly this:

$b_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}}$

$b_0 = \bar{y} - b_1 \cdot \bar{x}$

Two things are worth noticing before computing anything.

$b_1$ is a ratio of sums about the means. The numerator asks how $x$ and $y$ deviate together; the denominator asks how much $x$ deviates at all. If $x$ barely varies, the denominator is tiny and the slope is unstable — which is the same defect that makes a constant column fail a pre-flight audit.

$b_0$ is not fitted independently. Once the slope is known, the intercept is forced: the line must pass through $(\bar{x}, \bar{y})$. Every least-squares line goes through the mean of the data. That is not a convention, it falls out of the algebra.

---

The Simple Regression Workbench

Experiment with the interactive tool below. Modify the $y$ values of the five data points to see how the deviation table, the regression coefficients ($b_1, b_0$), and the residual squares update in real-time.

Interactive Regression Workbench

Blue lines cross at the mean point (3.0, 4.0). The green regression line is mathematically forced to pass through this point. Red shaded squares show the physical magnitude of residual².

Adjust Point Targets (y):

Point 1: x = 1, y = 2.0

Point 2: x = 2, y = 4.0

Point 3: x = 3, y = 5.0

Point 4: x = 4, y = 4.0

Point 5: x = 5, y = 5.0

Slope (b₁): 0.6000

Intercept (b₀): 2.2000

Model Equation: ŷ = 2.20 + 0.60x

R² Score: 0.6000

x	y	(x - x̄)	(y - ȳ)	(x - x̄)(y - ȳ)	(x - x̄)²	ŷ	r	r²
1	2.0	-2.0	-2.0	4.00	4.00	2.80	-0.80	0.64
2	4.0	-1.0	0.0	0.00	1.00	3.40	0.60	0.36
3	5.0	0.0	1.0	0.00	0.00	4.00	1.00	1.00
4	4.0	1.0	0.0	0.00	1.00	4.60	-0.60	0.36
5	5.0	2.0	1.0	2.00	4.00	5.20	-0.20	0.04
Sums →				Sxy = 6.00	Sxx = 10.00	—	—	SSE = 2.40

tip

Read the third row of the deviation table

At $x = 3$, $(x - \bar{x}) = 0$, so that point contributes nothing to either sum — its $y$ value cannot influence the slope at all! Use the buttons above to increment or decrement the $y$ value of Point 3 (where $x = 3$). Observe that the Slope ($b_1$) remains completely unchanged — only the Intercept ($b_0$) moves.

A point sitting at the mean of $x$ has no leverage over the line's tilt, only over its height. That is the beginning of the idea of leverage, and it is why outliers far out along $x$ are so much more dangerous than outliers in the middle.

---

How wrong is it? The five error measures

The fit produces predictions, the predictions produce residuals, and every standard error measure is a different summary of those same residuals. Nothing new is needed — they are all computed from the numbers already calculated in the workbench table.

Two aggregates carry everything else:

• SSE (Sum of Squared Errors): $\sum (y_i - \hat{y}_i)^2 = 2.4$ — the quantity the fit minimized.
• SST (Total Sum of Squares): $\sum (y_i - \bar{y})^2 = 6.0$ — the error you would make predicting the mean every time, which is the baseline any model has to beat.

Measure	Formula	Value	Reads as
MAE	$\sum \lvert r_i \rvert / n$	0.6400	Average miss, in the units of $y$
MSE	$SSE / n$	0.4800	Average squared miss — units are $y^2$
RMSE	$\sqrt{MSE}$	0.6928	Back in the units of $y$, but outlier-sensitive
$R^2$	$1 - SSE/SST$	0.6000	Fraction of variance beaten off the mean baseline
Adjusted $R^2$	$1 - \frac{(1-R^2)(n-1)}{n-k-1}$	0.4667	$R^2$, penalized for using $k$ features

All four of the first group match scikit-learn's metrics exactly. Adjusted $R^2$ has no scikit-learn function — you must compute it yourself.

Three load-bearing insights from these numbers:

• $R^2 = 0.6000$ represents a comparison, not an accuracy. It means our model has cleared away 60% of the baseline variance (SST) that a constant mean-guesser ($\bar{y}$) would suffer. It is unbounded below: a negative $R^2$ is common on test sets and means your model is worse than predicting the mean.
• The gap between MAE ($0.6400$) and RMSE ($0.6928$) measures outlier severity. Because RMSE squares errors, large outliers pull it upward, widening the gap. A wide gap signals that a few massive misses are dominating your model's performance.
• Adjusted $R^2$ falls severely ($0.6000 \to 0.4667$) due to small sample size. With $n = 5$ and $k = 1$, the degree-of-freedom penalty $\frac{n-1}{n-k-1} = 1.33$ is extremely punitive. On realistic datasets with large $n$, this gap shrinks unless you flood the model with useless noise columns.

note

Choosing between them is a separate question

This page computes the five so the arithmetic is never mysterious. Which one to report — and the traps in each, including a genuine sign inversion in scikit-learn's cross-validation scoring — belongs with regression performance metrics, along with residual diagnostics. Compute first, choose later.

---

r, b₁ and R² are three views of one fact

For a one-feature regression with an intercept, the correlation coefficient and the fit are not separate pieces of information. Two exact identities:

$b_1 = r \times \frac{s_y}{s_x} \qquad \qquad R^2 = r^2$

On the five default points above, $r = 0.774597$, and both hold to every digit printed:

• r × (s_y/s_x) = 0.600000 (equal to $b_1$)
• r² = 0.600000 (equal to $R^2$)

The $ddof$ question — population or sample standard deviation — is the one people expect to matter here, and it does not: the identity is exact either way because the denominator cancels out of the algebra. One less thing to get wrong.

The first identity says a correlation becomes a slope once you rescale it by the spread of the two variables. The second is narrower than it looks:

danger

$R^2 = r^2$ holds only for simple regression

With one feature and an intercept, $R^2$ is the squared correlation between $x$ and $y$. With two or more features it is not — there is no single $r$ to square, and $R^2$ becomes the squared correlation between $y$ and the model's predictions $\hat{y}$. Carrying the $R^2 = r^2$ intuition into multiple regression is a common and quiet error.

---

How much data does it take? The 1/√n Law

The recovered coefficients are never exactly right, because noise in real-world data is real. The useful question is how fast the estimation error shrinks as we collect more rows. Theory is specific: the standard deviation of the estimated slope is:

$SD(\hat{b}_1) = \frac{\sigma}{\sqrt{n \cdot Var(x)}}$

Thus, the error falls as $1/\sqrt{n}$, and $RMSE \times \sqrt{n}$ should be a constant. With a true noise $\sigma = 2$ and $x \sim Uniform(0, 10)$, the variance of $x$ is $100 / 12 = 8.333$, making that constant exactly $2 / \sqrt{8.333} = 0.6928$.

[ADVANCED DEEP-DIVE] Run 500 Real-Time Monte Carlo Simulations

Adjust the sliders below to change $n$ (sample size) and $\sigma$ (noise standard deviation). The simulator will instantly run 500 Monte Carlo fits in your browser to calculate the empirical standard error of the slope and compare it directly to mathematical theory.

Interactive 1/√n Error Simulator

Run 500 regressions instantly. See how the spread of our slope estimate shrinks as $n$ increases.

Sample Size ($n$): 200

Noise Level (σ): 2.0

Empirical Error SD

0.0000

Standard error calculated from 500 fits

Theoretical Error SD

0.0000

Calculated as σ / √(n * Var(x))

Difference: NaN%. Note how doubling the noise (σ) doubles the error, but halving the error requires four times the sample size (n).

warning

Testing this with one fit per sample size gives a false answer

$|\hat{b}_1 - 2.5|$ is a random quantity, and $1/\sqrt{n}$ describes how its spread scales, not how any single draw behaves. Draw once per $n$ and the numbers bounce around enough to look like no law at all — they did, on our first attempt at this measurement. Averaging over many fits per $n$ is not statistical decoration here; it is the difference between the right conclusion and a false one.

---

What "linear" actually restricts

The most common misreading of linear regression is that it can only fit straight lines. It cannot fit curves in its coefficients — a completely different restriction.

The data below is a parabola, $y = 2x^2 - 3x + 5$ plus noise:

Output

5. 'LINEAR' MEANS LINEAR IN THE COEFFICIENTS
   given x            R² = 0.4559   coefficients [-2.8500]
   given x, x²        R² = 0.9826   coefficients [-3.0394  1.9711]
   given x, x², x³    R² = 0.9826   coefficients [-3.1011  1.9705  0.0113]

Given only $x$, the model manages $R^2 = 0.4559$ — it is trying to fit a curve with a line. Given $x$ and $x^2$ as two columns, it reaches 0.9826, and its coefficients [-3.0394, 1.9711] recover the true $-3$ and $2$.

Nothing about the estimator changed. $x^2$ is just another column of numbers, and the model is still computing a weighted sum:

$y = b_0 + b_1 \cdot x + b_2 \cdot (x^2)$

which is linear in $b_0, b_1, b_2$ even though it draws a parabola. Adding $x^3$ gained nothing — $R^2$ is identical at $0.9826$, and the new coefficient came out near zero ($0.0113$).

tip

"Linear" is a statement about the coefficients, not the shape

A model is linear if the prediction is a weighted sum of its inputs — whatever those inputs are.

This is why feature engineering is so powerful here: $x^2, \log(x), x_1 \times x_2$, and $\sin(2\pi h / 24)$ are all just columns, and each extends what a linear model can express without changing the model at all. It is also why adding interaction terms can raise $R^2$ dramatically.

What linear regression genuinely cannot do is discover those columns for itself.

---

The assumptions

Linear regression is a model of the world, and it assumes four things.

Assumption	Means	Violated when
Linearity	The relationship really is a weighted sum	Curvature is left in the residuals
Independence	Rows do not influence each other	Time series; repeated measures per subject
Homoscedasticity	Error spread is constant across the range	Errors fan out as $y$ grows
Normality of residuals	Errors are roughly normally distributed	Needed for confidence intervals, not for fitting

Two are worth stressing. Independence is the same requirement that made a random train/test split invalid on grouped data — one violation, two consequences. And normality of residuals is often over-emphasized: the coefficients are unbiased without it. It matters for inference, not for prediction.

Checking them is residual work — plotting residuals against fitted values and looking for pattern, spread, or curvature — which belongs with regression performance metrics.

---

Correlation with the target, and why it misleads

The obvious way to decide whether a feature is worth having is to correlate it with the target. Fit each of three features on its own against house price, where the truth is:

$\text{price} = 3000 \cdot \text{size} - 800 \cdot \text{age} + 50000$

and door_colour is pure noise:

Output

6. CORRELATION WITH THE TARGET   (true: price = 3000·size − 800·age + 50000)
   feature            corr   fitted b1  R² alone
   size            +0.9849     3043.86    0.9700
   age             -0.0392     -332.50    0.0015
   door_colour     +0.0419     5160.69    0.0018

size behaves as expected: correlation $+0.9849$ and $R^2 = 0.9700$. The other two rows are the interesting ones.

• age has a real effect of $-800$ per year and a marginal correlation of $-0.0392$ (essentially zero). Its solo $R^2$ is 0.0015. On this evidence you would drop it!
• door_colour is meaningless and gets a fitted coefficient of $5160.69$ and solo $R^2$ of 0.0018 — which is actually higher than age's!

Why the useful feature is invisible

Not a mystery. Price variance is dominated by other terms:

• size spread: $150 \text{ units} \times 3000 = 450,000$
• age spread: $50 \text{ units} \times 800 = 40,000$
• noise: $\sigma = 20,000$

age's effect is real but roughly a tenth of size's. Correlating age against raw price asks it to show up against all of that giant size variation at once.

Hidden Feature Reveal Visualizer

Click the button below to toggle between correlating raw price vs. age and correlating size-adjusted price vs. age to see the effect instantly.

Interactive Hidden Feature Reveal

Toggle the size adjustment to watch the correlation with House Age pop out of the noise.

Current Correlation: -0.0392

Messy Raw Price: The house sizes vary from 50 to 200 sq. yards. Because size has such a colossal impact, it completely drowns out the smaller age effect, leaving a correlation of practically zero (r = -0.0392).

tip

This is what "holding all else constant" buys you

A coefficient in a multiple regression is measured on exactly this basis — the variation left over once the other features have been accounted for. That is why a multiple regression can find an effect that no marginal correlation shows, and it is the single strongest argument for fitting features jointly rather than screening them one at a time.

---

2. Implementation Lab

tip

Lab Exercise: Simple Linear Regression

An interactive, fully-functional Google Colab / Jupyter Notebook is available to experiment with this concept live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Click "Run all" or execute cells individually using Shift + Enter.

Things to try inside the script:

• Experiment 1: Modify the target value of Point 3 and watch the slope stay perfectly constant.
• Experiment 2: Adjust the simulation noise level $\sigma$ in section 4 to see how standard errors change.
• Experiment 3: Inspect section 6 and watch how multiple regression automatically recovers the correct parameters of hidden features.

---

3. Common Mistakes

1. Linear = Straight Lines Only

• The Misconception: "I thought linear regression could only fit straight lines in 2D space."
• Wrong Thinking: Assuming you cannot fit curved data trends like parabolas, exponentials, or sinusoids.
• The Right Principle: "Linear" refers to being linear in the coefficients ($b_0, b_1, b_2$), not the feature shape. By feeding $x^2$ or $\log(x)$ as new columns, the model fits complex curves perfectly while remaining a simple linear estimator.

2. Independent Intercept

• The Misconception: "I thought the intercept $b_0$ is optimized completely independently of the slope $b_1$."
• Wrong Thinking: Believing the optimizer picks both parameters freely.
• The Right Principle: Once the slope $b_1$ is computed, the intercept is locked as $b_0 = \bar{y} - b_1\bar{x}$. Every least-squares regression line must pass through the mean coordinates $(\bar{x}, \bar{y})$.

3. Marginal Feature Screening

• The Misconception: "If a feature has almost zero correlation with the target, it is useless and should be dropped."
• Wrong Thinking: Screening out features (like age in our housing dataset) based on their solo correlation coefficient with $y$.
• The Right Principle: High-variance features can completely drown out weaker but important predictors. Always fit features jointly; multiple regression automatically measures each coefficient after holding the others constant.

4. Adjusted R² function

• The Misconception: "I'll just import adjusted_r2_score from sklearn.metrics."
• Wrong Thinking: Spending hours looking for a non-existent scikit-learn function.
• The Right Principle: Scikit-learn has no Adjusted $R^2$ function. You must compute it yourself using $1 - \frac{(1-R^2)(n-1)}{n-k-1}$.

---

4. Summary

Core Theory

• Least Squares Fit: Minimizes the sum of squared vertical residuals (SSE).
• The Force Rule: The line is mathematically forced to pass through the mean coordinate point $(\bar{x}, \bar{y})$.
• $R^2$ Meaning: Compares the model's performance against predicting the target mean ($\bar{y}$) every single time.
• $R^2 = r^2$: Holds true for simple (one-feature) regression only.

Engineering Realities

• Leverage Trap: Points at $x = \bar{x}$ have zero influence on slope. Outliers far from the mean have severe leverage.
• The $1/\sqrt{n}$ Law: Shrinking the estimation error by half requires four times the training data.
• Masked Features: A vital feature's correlation can be masked entirely by a high-variance feature unless evaluated jointly.
• No Sklearn Adjusted $R^2$: Must be written manually to avoid importing non-existent metric functions.

info

Key Takeaways

• Convexity: Squared error is chosen because it is convex and differentiable, leading to a closed-form analytical solution.
• Outliers: Squaring the residuals makes the line highly sensitive to outlier values, especially those at high-leverage positions.
• Linear Limitations: Linear models cannot discover feature interactions or transformations (like $x^2$); they must be engineered manually.

Next in this section: Multiple Linear Regression — more than one feature, the normal equations solved by hand, and what happens to a coefficient when two features carry the same information.

---

5. Active Recall

Attempt each question first, then click to reveal detailed answers, mathematical derivations, and sample explanations.

Interactive Checkpoint Quiz

Interactive Checkpoint: The R² Trap

A data scientist fits a multiple regression model with 10 features and observes R² = 0.81. They claim the correlation coefficient r between their predictions and the true labels is exactly 0.90. Is this claim valid?

Review Flashcards

1. [THEORY] Why is the intercept $b_0$ not fitted independently of $b_1$?

• The Derivation: The least-squares loss is $L = \sum (y_i - b_0 - b_1 x_i)^2$.
• Taking the partial derivative with respect to $b_0$ gives: $\frac{\partial L}{\partial b_0} = -2 \sum (y_i - b_0 - b_1 x_i) = 0$
• Dividing by $-2$ and expanding the sum: $\sum y_i - n \cdot b_0 - b_1 \sum x_i = 0$
• Dividing through by $n$: $\bar{y} - b_0 - b_1 \bar{x} = 0 \implies b_0 = \bar{y} - b_1 \bar{x}$
• The Reality: This mathematical proof guarantees that the fitted regression line is always forced to pass through the exact mean point $(\bar{x}, \bar{y})$.

2. [ANALYZE] How do outliers far from the mean of $x$ affect the slope compared to outliers near the mean of $x$?

• Leverage Principle: The denominator of $b_1$ is $S_{xx} = \sum (x_i - \bar{x})^2$.
• The influence (or leverage) of any point on the slope is proportional to its squared distance from the mean of $x$, $(x_i - \bar{x})^2$.
• If a point sits exactly at $x_i = \bar{x}$, its $(x_i - \bar{x}) = 0$, meaning it has zero leverage over the slope $b_1$ (it only affects the vertical position of the line, i.e., $b_0$).
• An outlier far out on the $x$-axis has enormous $(x_i - \bar{x})^2$, acting as a heavy anchor that can single-handedly pivot the entire slope of the line.

3. [THEORY] What baseline does $R^2$ compare the model against, and how can $R^2$ be negative?

• The Baseline: $R^2$ compares the model's SSE against the Sum of Squares Total (SST), where $SST = \sum (y_i - \bar{y})^2$. SST represents the error of a "constant baseline model" that always predicts the mean $\bar{y}$.
• Negative $R^2$: Since $R^2 = 1 - \frac{SSE}{SST}$, it becomes negative when $SSE > SST$.
• This occurs when the model's predictions make larger errors than the simple constant mean predictor. This is common when testing a model on unseen out-of-sample data, proving the model is worse than useless.

4. [ANALYZE] If a feature is highly predictive of the target, why might its correlation with the target look near-zero?

• The Masking Effect: Correlation measures marginal (one-on-one) relationship. If there is a dominant feature that accounts for 90% of the target's variance, a secondary but highly useful feature's signal is completely buried under that massive leftover variance.
• The Solution: Subtracting the dominant feature's contribution ("partialing out" the effect) reveals the true, strong correlation of the secondary feature. Multiple regression does this automatically by fitting all coefficients jointly.

5. [THEORY] What does "linear" in linear regression actually restrict? Can it fit a curve?

• The Restriction: "Linear" restricts the model to be linear in its coefficients ($b_0, b_1, b_2$), not linear in its input shapes.
• Fitting Curves: It can fit a curve (like a parabola $y = b_0 + b_1 x + b_2 x^2$) because $x^2$ is treated by the estimator as just another column of numeric values. The formula remains a simple weighted sum of inputs.
• The Limitation: A linear model cannot discover these transformations itself; they must be engineered as features manually.