Feature Scaling and Transformation

Topic - Everything is finally numeric. Now the columns are on absurdly different scales - and some models are crippled by that while others are provably indifferent to it. This page is about knowing which situation you're in, then choosing a scaler that survives your data.

---

The problem

One dataset, four of its columns:

Output

                      min        max       std
mean area        143.5000  2501.0000  351.9141
worst area       185.2000  4254.0000  569.3570
mean radius        6.9810    28.1100    3.5240
mean smoothness    0.0526     0.1634    0.0141
   largest std 569.36 (worst area), smallest 0.002646 (fractal dimension error)
   ratio: 215,171x

The most variable column has 215,171× the spread of the least variable one. Both describe the same tumours; they're simply measured in different units.

Now consider what a distance calculation does with that. Euclidean distance between two samples sums squared differences across all columns. A 100-unit difference in worst area contributes 100² = 10,000. A difference spanning the entire range of mean smoothness - about 0.11 - contributes 0.0121. The area column outvotes the smoothness column by roughly a million to one.

The model isn't weighing evidence. It's reading the units.

🎛️ Live Scaler Simulator

See how an outlier (1000) crushes regular values in MinMaxScaler but gets safely isolated in RobustScaler:

Inject Extreme Outlier (1000)

Raw 1

10

Scaled Value:

-1.342

Raw 2

20

Scaled Value:

-0.447

Raw 3

30

Scaled Value:

0.447

Raw 4

40

Scaled Value:

1.342

Standardizes features to mean 0 and unit variance. Unbounded range.

---

Standardisation (Z-score normalisation)

Explore the mathematical foundations of standard scaling and normalisation:

📊 Standardisation (Z-Score)

Subtract the mean, divide by the standard deviation:

$z = \frac{x - \mu}{\sigma}$

The result has mean 0 and standard deviation 1. Values are not bounded - an extreme input stays extreme, just expressed in standard deviations.

# Scikit-learn equivalent
StandardScaler()

📏 Min-Max Normalisation

Subtract the minimum, divide by the range:

$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$

The result is bounded to $[0, 1]$ by construction, with the smallest value at exactly $0$ and the largest at exactly $1$.

# Scikit-learn equivalent
MinMaxScaler()

Metric / Feature	Standardisation	Min-Max
Output Bounds	Unbounded (mean 0, std 1)	strictly bounded $[0, 1]$
Formula anchors	population mean, std	column minimum, maximum
Outlier sensitivity	Moderate	Severe
Preserves distribution shape?	Yes	Yes
Default Choice	Usually this	When bounds are required (e.g., image pixels)

Neither changes the shape of a distribution - both are linear rescalings. A skewed column stays exactly as skewed after either. Fixing shape is what transformations, further down, are for.

---

Which models actually need it

This is the question worth answering precisely, and it's measurable. Same data, same folds, only scaling differs:

Output

   model                     unscaled   scaled     gain
   KNeighbors(5)               0.9279   0.9649  +0.0369
   SVC(rbf)                    0.9122   0.9736  +0.0615
   LogisticRegression          0.9526   0.9807  +0.0281
   RandomForest                0.9631   0.9631  +0.0000

SVC gained 6.2 points. RandomForest gained exactly zero - identical to four decimal places.

That zero isn't luck, it's structural:

Model family	Needs scaling?	Why
kNN, K-Means, SVM	Yes - critically	They compute distances; units become weights
Logistic / linear regression	Yes, when regularised	The penalty shrinks all coefficients equally, so scale decides who gets penalised
Neural networks	Yes	Gradient descent converges badly on mismatched scales
PCA	Yes	It maximises variance, so the largest-variance column dominates
Decision trees, Random Forest, Boosting	No	They split on thresholds, and thresholds are scale-free

Why trees are immune

A tree asks "is area ≤ 750?" Rescale that column and the question simply becomes "is area_scaled ≤ 0.31?" - the same split, partitioning the same rows.

Any monotonic transformation preserves the ordering of values, and a tree only ever uses ordering. So scaling, log transforms, and square roots are all invisible to it. This is a genuinely useful property: it means you can skip scaling entirely for tree ensembles, and it explains why boosted trees are so forgiving of raw, messy features.

tip

Unsure? Scale anyway

Scaling costs nothing for models that don't need it - the RandomForest row proves that, at +0.0000. It's substantial for models that do. Inside a Pipeline it's one extra line.

The asymmetry is decisive: forgetting to scale kNN cost 3.7 points here; scaling a forest unnecessarily cost nothing at all.

---

One outlier, four scalers

Both formulas above are computed from the data, so a single extreme value poisons them. Here are 200 values drawn around 50, plus a single 5000:

Output

   scaler            uses                range of the 200     span
   StandardScaler    mean / std        -0.1460 .. -0.0043   0.1417
   MinMaxScaler      min / max           0.0000 .. 0.0099   0.0099
   RobustScaler      median / IQR       -1.7187 .. 1.5329   3.2516
   MaxAbsScaler      max absolute        0.0049 .. 0.0148   0.0099

Read the span column - how much room the 200 real values got:

MinMaxScaler compressed them into 0.0099 - under 1% of its output range. The outlier claimed 1.0 for itself and everything real is squashed against zero, effectively indistinguishable. This is the worst case because min and max are each set by exactly one data point.

StandardScaler gave them 0.1417 - better, but still badly compressed. The mean was dragged upward and the std inflated by the single outlier, so genuine variation shrank to a seventh of what it should be.

RobustScaler gave them 3.2516 - the real structure survives intact. It centres on the median and divides by the interquartile range, and neither statistic notices one extreme value.

Scaler	Centre	Scale	Robust?
StandardScaler	mean	std	No
MinMaxScaler	min	max - min	No - worst
RobustScaler	median	IQR	Yes
MaxAbsScaler	none	max abs	No - but preserves sparsity

warning

MinMaxScaler and outliers do not mix

If a column might contain an extreme value, min-max is the wrong choice - one point defines your entire range. Use RobustScaler, or handle the outliers first - see Outliers and Feature Engineering.

MinMaxScaler earns its place when bounded output is a requirement - image pixel values, or a neural network layer expecting [0, 1] - not as a default.

Normalizer is not a scaler - it works on rows

A genuinely common confusion, worth seeing directly:

Output

   input rows      [1.0, 2.0, 3.0] and [100.0, 200.0, 300.0]
   StandardScaler  [[-1.0, -1.0, -1.0], [1.0, 1.0, 1.0]]
   Normalizer      [[0.267, 0.535, 0.802], [0.267, 0.535, 0.802]]

StandardScaler works down columns - each feature is rescaled independently. Normalizer works across rows - each sample is scaled to unit length.

Notice what happened: the two input rows differ by a factor of 100, and Normalizer made them identical. It discarded magnitude entirely, keeping only direction. That's the right thing for text frequency vectors, where document length shouldn't matter - and completely wrong if magnitude carries information.

Everything else on this page scales columns. Normalizer scales rows. It is not a substitute for any of them.

---

Transformations: changing the shape

Scaling moves and stretches a distribution. It never changes its shape. When a column is heavily skewed, shape is the problem - and a linear model asked to fit a variable whose values span four orders of magnitude will be dominated by the tail.

Skew measures asymmetry, where 0 is symmetric:

Output

   column            raw    log1p     sqrt  Yeo-Johnson  Quantile
   MedInc           1.65     0.23     0.69        -0.00      0.18
   AveRooms        20.70     1.39     4.85        -0.17     -0.00
   AveOccup        97.63     3.88    43.34        -0.11      0.03
   Population       4.94    -1.04     1.22         0.11      0.03

Three things to read from that table.

Log transforms are powerful. AveOccup went from skew 97.63 to 3.88 - a column so right-tailed as to be unusable for a linear model became merely awkward. AveRooms went from 20.70 to 1.39.

Square root is a weaker version of the same idea. AveOccup only reached 43.34. Both compress large values more than small ones; log just compresses far harder.

Log is not automatically correct - it can overshoot. Look at Population: skew +4.94 became −1.04. The transform overcorrected, converting a right tail into a left one. Applying log reflexively to anything skewed is how that happens.

log versus log1p

Output

    np.log1p([0.0, 1.0, 10.0]) = [0.0, 0.693, 2.398]
    np.log([0.0, 1.0, 10.0])   = [-inf, 0.0, 2.303]   <- -inf

log(0) is −inf, and one −inf propagates through every downstream computation. log1p(x) computes log(1 + x), so zero maps to zero. If a column can contain zeros, use log1p.

Negative values break logarithms entirely - no shifting trick makes that principled.

Let the data choose the transform

PowerTransformer fits the transformation strength to the column rather than assuming it:

PowerTransformer(method="yeo-johnson")   # default; accepts negatives and zeros
PowerTransformer(method="box-cox")       # strictly positive input only

It got every column in that table to a skew between −0.17 and +0.11 - including Population, which log overcorrected, and including columns with negative values that log cannot touch:

Output

PowerTransformer(yeo-johnson) on [-5.0, 0.0, 5.0] -> [-1.225, -0.0, 1.225]

QuantileTransformer(output_distribution="normal") is the blunter instrument - it maps values to their ranks and then onto a normal distribution, forcing near-perfect symmetry (−0.00, 0.03) regardless of input shape. It's effective and it discards the actual spacing between values, keeping only their order.

Transform	Handles 0	Handles negatives	Strength
np.log	No	No	Fixed, strong
np.log1p	Yes	No	Fixed, strong
np.sqrt	Yes	No	Fixed, mild
PowerTransformer (Yeo-Johnson)	Yes	Yes	Fitted per column
PowerTransformer (Box-Cox)	No	No	Fitted per column
QuantileTransformer	Yes	Yes	Forced - rank-based

note

Transforming the target changes what you are optimising

Everything above concerns features. Log-transforming the target is a different decision: minimising squared error on log(y) minimises relative error on y, not absolute error.

For a house-price model that means treating a £20k miss on a £100k house as equivalent to a £200k miss on a £1M house. Sometimes exactly right, sometimes badly wrong - but it is a modelling choice, not a preprocessing step. And remember predictions come back in log space and need np.expm1 to be interpretable.

---

Sparse data

The previous page established that one-hot output should stay sparse - 1,000 categories cost 0.8 MB sparse against 400 MB dense. Centering breaks that:

Output

  sparse one-hot: (5000, 50), stored non-zeros = 5,000 (2.0% of cells)
  StandardScaler() on sparse -> ValueError
    Cannot center sparse matrices: pass `with_mean=False` instead.
  StandardScaler(with_mean=False): OK, still sparse = True, nnz = 5,000

Subtracting the mean would turn every one of those zeros into a non-zero, converting a 2%-full matrix into a 100%-full one. scikit-learn refuses rather than silently exhausting your memory - a good error.

For sparse input use StandardScaler(with_mean=False) or MaxAbsScaler, which never centres and so preserves sparsity by design.

---

How much does scaling before the split really leak?

The rule is to fit on training data only. It's worth knowing the size of the effect, measured over 30 different splits:

Output

   StandardScaler  honest 0.9643   leaked 0.9645   inflation +0.0002
   MinMaxScaler    honest 0.9663   leaked 0.9657   inflation -0.0006

+0.0002 and −0.0006. Both are noise. MinMaxScaler actually scored marginally worse when leaked.

That's an honest result and it deserves stating plainly: scaling leakage is real in principle and usually negligible in magnitude. It is nothing like the target-encoding leak on the previous page, which manufactured 21 accuracy points out of pure noise.

The reason is what gets leaked. Target encoding leaks the target itself, per row. Scaling leaks one mean and one standard deviation aggregated over hundreds of rows - a vanishing amount of information about any individual row.

tip

Keep the discipline anyway - for reasons other than fear

Fit on train only, but not because the numbers above are alarming. Because:

• On a small dataset, or with MinMaxScaler where one test outlier can define the range, the effect is larger and less predictable
• A Pipeline gives you correctness for free, and you want one regardless once you reach the train/test split
• At predict time there is no test set to compute statistics from, so training-set statistics are what you must ship

Be accurate about which leaks matter. Overstating this one makes it harder to take the genuinely dangerous ones seriously.

---

Implementation Lab

tip

Lab Exercise: Scaling & Transformation

An interactive, fully-functional Google Colab / Jupyter Notebook is available to experiment with these concepts live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Click "Run all" or execute cells individually using Shift + Enter.

Things to try inside the script:

• Experiment 1: Evaluate the accuracy gain on SVM / kNN models when scaling is turned on vs. off.
• Experiment 2: Observe how a single outlier compresses MinMaxScaler's non-outlier data to under 1% of the range.
• Experiment 3: Inspect the Yeo-Johnson exponent values calculated for California housing metrics to understand the automatic distribution fitting.

Comparative Implementation (Production vs. Scratch)

Compare production-ready pipelines with a manual, leakage-prone implementation:

📦 Production (Scikit-Learn Pipeline)

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

# Safe and isolated scaling fitted ONLY on training folds
model = make_pipeline(
    StandardScaler(),
    KNeighborsClassifier()
)

🧮 Manual Scaling (Dangerous Leak)

# DANGEROUS: Fit-transforming over entire dataset before splitting
# leaks global mean and standard deviation into the validation sets.
X_scaled = StandardScaler().fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

---

Common Mistakes

Explore some of the most critical engineering pitfalls in feature scaling and transformation and how to avoid them:

🚨 Pitfall 1: Unnecessary Tree Scaling

• The Misconception: "Scaling is a mandatory step that always improves RandomForest / XGBoost performance."
• ❌ Wrong Thinking: Decision tree nodes split on thresholds, which are completely scale-invariant. Scaling a tree ensemble yields exactly +0.0000 accuracy gain.
• ✅ The Right Principle: Focus scaling efforts strictly on distance-based models (kNN, SVM), linear estimators, neural networks, or PCA. You can skip scaling for pure tree ensembles.

🚨 Pitfall 2: MinMaxScaler Outlier Poisoning

• The Misconception: "MinMaxScaler is always the best choice because bounded [0, 1] scales look tidier."
• ❌ Wrong Thinking: If your column contains extreme outliers, MinMaxScaler will compress 99% of your legitimate data points into a narrow 1% band, rendering them indistinguishable.
• ✅ The Right Principle: Use RobustScaler (median / IQR based) when outliers are present, or handle outliers first before using standard scaling.

🚨 Pitfall 3: Row vs Column Scaling

• The Misconception: "Normalizer is a drop-in substitute for StandardScaler."
• ❌ Wrong Thinking: Normalizer scales rows (samples) to unit length, completely discarding magnitude. StandardScaler scales columns (features) independently.
• ✅ The Right Principle: Use column-based scalers (StandardScaler, RobustScaler) for feature-scale alignment. Reserve Normalizer for row-wise frequency representations like TF-IDF text features.

🚨 Pitfall 4: Reflexive Log Transforms

• The Misconception: "I should apply plain np.log to any right-skewed feature."
• ❌ Wrong Thinking: Plain np.log will fail with an -inf on zero values and raise errors on negative numbers. Reflexive log scaling can also overcorrect, flipping right-skews into negative left-skews.
• ✅ The Right Principle: Use np.log1p to handle zeros safely. Even better, use PowerTransformer(method="yeo-johnson") to let scikit-learn automatically find the optimal skew-reducing exponent for both positive and negative values.

🚨 Pitfall 5: Centering Sparse Matrices

• The Misconception: "I can run standard StandardScaler on high-cardinality one-hot outputs."
• ❌ Wrong Thinking: StandardScaler subtracts the mean, which immediately converts all zero entries into non-zero values, causing sparse representations to densify and blow up memory (e.g. from 0.8 MB to 400 MB), resulting in a ValueError.
• ✅ The Right Principle: Always set with_mean=False in StandardScaler, or use MaxAbsScaler() to scale sparse matrices without destroying their memory-saving properties.

---

Summary

🤖 Scaler Selection

• Mean 0, Std 1: Use StandardScaler() for linear and distance models under normal distributions.
• Bounded Ranges: Use MinMaxScaler() to force bounds within $[0, 1]$ (e.g., neural networks or pixels).
• Outlier Resilience: Use RobustScaler() (median and IQR-based) to safely insulate regular data from extreme spikes.
• Sparse Matrix Preservation: Use StandardScaler(with_mean=False) or MaxAbsScaler() to prevent memory blowups.

🔄 Distribution Transformations

• Strict Positive Skew: Use np.log1p(x) or np.sqrt(x) to compress long right tails containing zeros.
• Flexible Auto-Fitting: Use PowerTransformer(method="yeo-johnson") to automatically fit the optimal exponent across both positive and negative values.
• Perfect Symmetry: Use QuantileTransformer(output_distribution="normal") to force uniform ranks, at the cost of losing inter-value spacings.

info

📌 Key Takeaways

• 🎯 Distance models collapse unscaled: Algorithms like SVM (+6.2 accuracy points) and kNN (+3.7) are critically crippled by raw unit spreads, whereas tree ensembles (+0.0000) are structurally immune.
• ⚠️ Outliers crush MinMaxScaler: A single outlier squashes legitimate data down to less than 1% of the scaled range; RobustScaler prevents this via median centering.
• 🔄 Log1p handles zeros safely: Normal np.log creates -inf on zero columns; use np.log1p or Yeo-Johnson power transforms instead.

Next in this section: Outliers and Feature Engineering

See also: Encoding Categorical Data for the sparsity this must preserve · Missing Values for why imputation comes first · The Toolkit and the Pipeline for fit versus transform

---

Active Recall Flashcards

Attempt each question first, then click to reveal detailed answers, mathematical derivations, and sample explanations.

❓ 1. [THEORY] Why do distance-based models (SVM, kNN) require feature scaling while decision trees are completely invariant?

• Distance Models: Compute straight-line (Euclidean) spatial metrics. Features on wildly larger scales outvote smaller-variance columns (e.g. Area vs Smoothness) by millions-to-one, effectively ignoring evidence.
• Decision Trees: Split nodes strictly on single-feature thresholds (e.g. $x_j \le C$). Since any monotonic scaling transformation preserves the absolute sorting order of elements, the exact same partitions are achieved before and after scaling, resulting in exactly $+0.0000$ accuracy gain.

❓ 2. [THEORY] What are standardisation and min-max equations, and what are their respective output bounds?

• Standardisation (Z-Score): $z = \frac{x - \mu}{\sigma}$ with unbounded output range (centered at mean 0 with variance 1).
• Min-Max Scaling: $x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$ with strict output range $[0, 1]$.

❓ 3. [ANALYZE] Why is MinMaxScaler uniquely vulnerable to a single outlier, and how does RobustScaler solve this?

• MinMaxScaler Vulnerability: Uses the single absolute minimum ($x_{\min}$) and maximum ($x_{\max}$) data points. A single large outlier inflates $x_{\max}$ to an extreme value, compressing 99% of legitimate variation into a tiny fraction of the scale (e.g. $[0, 0.01]$).
• RobustScaler Resolution: Centers using the median and scales by the Interquartile Range (IQR = Q3 - Q1). Since both statistics are based on rank percentiles, they are completely unaffected by extreme individual spikes, keeping the scaled distribution's core wide and stable.

❓ 4. [THEORY] What is the difference between StandardScaler and Normalizer in scikit-learn?

• StandardScaler: Operates column-wise (down features), standardizing each variable's scale across all samples.
• Normalizer: Operates row-wise (across features per sample), scaling each individual row to unit Euclidean ($L2$) norm. It completely discards magnitude to retain only vector direction (useful for text frequencies).

❓ 5. [THEORY] Why does np.log fail on zero columns, and how does np.log1p resolve this?

• np.log Failure: Mathematical $\log(0) = -\infty$, which introduces non-numeric types and breaks downstream matrix calculations.
• np.log1p Solution: Computes $\log(1 + x)$. Since $\log(1) = 0$, zero values map cleanly to zero, preserving original sparse zeroes while compressing right-tailed distributions.

❓ 6. [THEORY] What does PowerTransformer do that fixed log/sqrt transforms cannot?

• Fixed transforms (like log/sqrt) apply a constant compression function that may overshoot (e.g., turning Population skew from $+4.94$ to $-1.04$).
• PowerTransformer fits a parametrizable exponent (using Box-Cox or Yeo-Johnson) to find the exact mathematical transformation that maximizes normality. In addition, its Yeo-Johnson formulation natively supports negative values, which logarithms cannot handle.

❓ 7. [ANALYZE] What happens to a sparse matrix when centered, and why does StandardScaler raise a ValueError?

• Sparse Centering: Subtracting the mean requires subtracting a non-zero value from every single sparse zero element. This converts an efficient sparse matrix (e.g., 2% non-zero) into a 100% dense matrix.
• Scikit-Learn Safeguard: It raises a ValueError to prevent your machine from running out of RAM and crashing. The fix is to pass with_mean=False to StandardScaler.

❓ 8. [ANALYZE] Why is scaling parameter leakage (e.g., +0.0002) historically negligible compared to target-encoding leakage (+21 points)?

• Scaling Leakage: Leaks aggregate parameters (a single column mean and standard deviation) computed over hundreds of rows, which conveys almost zero private information about any individual row's target label.
• Target-Encoding Leakage: Leaks the exact row target $y$ itself in a slightly blurred format directly into the feature $X$, creating a direct circular dependency on the answer.

❓ 9. [ANALYZE] Given that scaling leakage is negligible, why must we still enforce fitting scalers strictly on training data?

It remains an essential production guardrail because:

1. At predict time, you receive data one row at a time (making it mathematically impossible to calculate standard deviations on inference splits).
2. For small datasets or with MinMaxScaler, single-point test outliers can wildly and unpredictably distort the validation scores.
3. Encapsulating the scaler fit in a pipeline ensures identical, reproducible transformation boundaries across testing, validation, and live production endpoints.

❓ 10. [THEORY] Why must you always perform imputation before scaling?

Because missing values (NaN or None) break the mathematical calculations for mean, standard deviation, and range limits. Attempting to fit a scaler on a column containing missing values will result in a crash or propagate NaN values across your entire matrix.