Feature Scaling and Transformation

Topic - One clinical feature is measured in square millimeters (spread of $569$ units); another is measured in cellular smoothness (spread of $0.014$ units). When distance, gradient, and regularization algorithms consume unscaled metrics, the feature with the largest arbitrary unit outvotes the smaller feature by a million to one.

This chapter establishes when feature scaling is mandatory, why decision trees are mathematically indifferent, and how non-linear transformations reshape skewed distributions.

---

1. The Problem: Physical Units Outvote Evidence

Consider four features drawn from the Breast Cancer Wisconsin diagnostic dataset:

Feature Variance Disparity

Feature Column          Min         Max         Std Deviation
worst area             185.20      4254.00      569.3570
mean area              143.50      2501.00      351.9141
mean radius              6.98        28.11        3.5240
mean smoothness          0.05         0.16        0.0141
fractal dimension        0.001        0.02        0.0026

The most variable column (worst area) has 215,171x the variance of the least variable column (fractal dimension error). Both describe the same physical tumors; they are simply measured in different physical units.

The Euclidean Distance Collapse

In Euclidean metric space, the straight-line distance between two patient vectors $u$ and $v$ is calculated as:

$d(u, v) = \sqrt{\sum_{j=1}^D (u_j - v_j)^2}$

• A difference of $100\text{ mm}^2$ in worst area contributes $100^2 = \mathbf{10,000}$ to the sum.
• A difference spanning the entire physiological range of mean smoothness ($\sim 0.11$) contributes $0.11^2 = \mathbf{0.0121}$.

The area feature outvotes the smoothness feature by nearly one million to one. The model is not weighing predictive evidence — it is reading the physical measuring tape.

---

Interactive Euclidean Distance Distortion Sandbox

Adjust the observed differences between two patient samples in Worst Area and Mean Smoothness:

Worst Area Difference: 150 mm²

Mean Smoothness Difference: 0.050

Unscaled Raw Distance

150.0

Area determines 100.000% of metric

Standardized Z-Score Distance

3.58

Area contribution balanced to 0.5%

Key Takeaway: Without scaling, unscaled Area contributes 100.000% of the spatial calculation, rendering cellular smoothness completely invisible to kNN, SVM, and clustering algorithms. Standardizing scales restores balanced, unit-free geometry.

---

2. Standardisation (Z-Score) vs. Min-Max Normalisation

Linear scaling shifts and divides coordinates using sample statistics. The two primary formulations are:

1. Standardisation (Z-Score)

Subtract the sample mean ($\mu$), divide by the standard deviation ($\sigma$):

$z = \frac{x - \mu}{\sigma}$

• Resulting Bounds: Unbounded (mean is exactly $0$, standard deviation is exactly $1$).
• Outlier Behavior: Moderate. Outliers remain outliers, but are expressed in terms of standard deviation offsets.
• Scikit-Learn Implementation: StandardScaler()

2. Min-Max Normalisation

Subtract the minimum ($x_{\min}$), divide by the feature range ($x_{\max} - x_{\min}$):

$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$

• Resulting Bounds: Strictly bounded to $[0, 1]$ (smallest value maps to $0$, largest maps to $1$).
• Outlier Behavior: Extreme. A single outlier anchors $x_{\max}$, compressing all legitimate observations into a narrow band near zero.
• Scikit-Learn Implementation: MinMaxScaler()

The Essential Truth: Linear Scalers Never Change Shape

Both Standardisation and Min-Max scaling are linear transformations ($f(x) = ax + b$). They translate and scale the axis, but never change the shape of a distribution. A heavily right-skewed feature remains identically right-skewed after scaling. Changing distribution shapes requires non-linear power transformations.

---

3. Which Models Actually Need It (And Why Trees Are Immune)

The performance impact of feature scaling is not theoretical — it is directly measurable across model families on the exact same data partitions:

Empirical Accuracy Gain from Feature Scaling

Model Family                Unscaled Accuracy   Scaled Accuracy     Realized Gain
K-Nearest Neighbors (k=5)         0.9279            0.9649          +0.0369 (+3.7%)
Support Vector Machine (RBF)      0.9122            0.9736          +0.0615 (+6.2%)
Logistic Regression (L2)          0.9526            0.9807          +0.0281 (+2.8%)
Random Forest Classifier          0.9631            0.9631          +0.0000 (Zero Gain)

Support Vector Machines gained 6.2 points, while Random Forest gained exactly zero. That zero is not coincidence; it is an architectural property of tree-based partitioning.

---

Interactive Model Sensitivity Matrix

Select a model family to inspect its scaling requirement, empirical gain, and architectural reason:

Support Vector Machine (RBF Kernel)

Scaling Requirement: CRITICAL

Empirical Benchmark Accuracy Gain: +6.15%

Architectural Reason: SVM constructs maximum-margin hyperplanes using distance metrics ($K(x, x') = \exp(-\gamma ||x - x'||^2)$). When scales differ by 1000x, the kernel distance is dominated entirely by the high-variance feature, collapsing classification boundaries.

Why Trees are Scale-Invariant

A decision tree asks: "Is $\text{Area} \le 750$?" If you rescale that feature via standardisation or min-max scaling, the question becomes: "Is $\text{Area}_{\text{scaled}} \le 0.31$?"

The split partitions the exact same subset of samples. Any monotonic transformation preserves the sorting order of values, and decision trees split purely on rank orderings. Trees, Random Forests, and Gradient Boosted Trees (XGBoost, LightGBM) are 100% immune to feature scale.

tip

When Unsure: Scale Anyway

Scaling costs nothing for models that do not need it — Random Forest scored +0.0000 accuracy gain either way. But omitting scaling severely damages models that do (+6.2% loss on SVM). Inside a Pipeline, scaling is a single line that guarantees protection.

---

4. One Outlier, Four Scalers

Because scaling statistics are computed from data, a single extreme outlier corrupts parameter estimation.

Consider a sample of 200 normal observations centered around $50$, plus a single extreme outlier at $5000$:

Output Range of 200 Legitimate Values

Scaler              Formula Anchors       Output Span of the 200 Real Values
MinMaxScaler        Min / Max             0.0000 .. 0.0099 (Span: 0.0099)
StandardScaler      Mean / Std Deviation -0.1460 .. -0.0043 (Span: 0.1417)
RobustScaler        Median / IQR         -1.7187 .. 1.5329 (Span: 3.2516)
MaxAbsScaler        Max Absolute Value    0.0049 .. 0.0148 (Span: 0.0099)

• MinMaxScaler squashes legitimate data into 0.9% of the output space: The single outlier claims $1.0$ for itself, forcing the remaining 200 legitimate points into an indistinguishable clump near zero.
• StandardScaler suffers moderate compression (Span: 0.1417): The outlier inflates the sample mean and standard deviation, shrinking the legitimate distribution.
• RobustScaler preserves the entire structure (Span: 3.2516): Because it centers on the median and divides by the Interquartile Range (IQR), the single outlier has zero influence on the scaling parameters.

---

Interactive Scaler vs. Outlier Playground

Inject an extreme outlier (1000) to observe how MinMaxScaler crushes regular observations compared to RobustScaler:

Inject Extreme Outlier (1000)

Point 1

10

Scaled:

-1.342

Point 2

20

Scaled:

-0.447

Point 3

30

Scaled:

0.447

Point 4

40

Scaled:

1.342

Normalizer is Not a Scaler: It Scales Rows, Not Columns

A common architectural confusion is treating Normalizer as a scaler:

• StandardScaler / MinMaxScaler / RobustScaler: Scale down columns independently (across all samples).
• Normalizer: Scales across rows (rescaling each individual sample vector to unit Euclidean length $||x||_2 = 1$).

Row Normalization Example

Input Rows:     [1.0, 2.0, 3.0] and [100.0, 200.0, 300.0]  (differ by 100x)
Normalizer:     [0.267, 0.535, 0.802] and [0.267, 0.535, 0.802]  (IDENTICAL)

Normalizer discards magnitude completely, keeping only vector direction. It is designed for text frequency vectors (TF-IDF) where document length should not matter, but will ruin standard tabular features where absolute magnitude carries signal.

---

5. Transformations: Changing the Distribution Shape

While scaling shifts and stretches distributions linearly, non-linear transformations reshape distributions to compress long right tails and reduce skewness:

Skewness Reduction across California Housing Metrics

Column Name       Raw Skew    np.log1p    np.sqrt    Yeo-Johnson    QuantileTransformer
MedInc             +1.65       +0.23      +0.69         -0.00             +0.18
AveRooms          +20.70       +1.39      +4.85         -0.17             -0.00
AveOccup          +97.63       +3.88     +43.34         -0.11             +0.03
Population         +4.94       -1.04      +1.22         +0.11             +0.03

The Log-Overshoot Trap

Log transforms compress large numbers aggressively. For extremely right-tailed columns (AveOccup), log transforms reduce skew from $97.63 \to 3.88$.

However, applying log reflexively can overshoot: on Population, raw skew was $+4.94$, but log1p overcorrected to $-1.04$, flipping a right tail into a negative left tail!

---

Interactive Non-Linear Skewness Transformer Sandbox

Observe how different non-linear transforms reshape empirical features from the California housing dataset:

Select Feature Column

Select Transformation

Feature: AveOccup   |   Applied Transform: log1p

Resulting Skew: +3.88

Extreme skewness. sqrt is insufficient (43.34); Yeo-Johnson achieves near-perfect normality (-0.11).

log vs. log1p & Automated Power Transformations

• np.log(0) is $-\infty$: A single zero value creates a floating-point exception that destroys all downstream computations.
• np.log1p(x): Computes $\log(1 + x)$. Zero maps cleanly to $0$ ($\log(1) = 0$), preserving sparsity while safely compressing positive skewness.
• PowerTransformer(method="yeo-johnson"): Fits an optimal power parameter ($\lambda$) per column via maximum likelihood estimation. It natively handles zero and negative numbers, avoiding log-overshoot.

---

6. Sparse Data Preservation

High-cardinality one-hot encoding outputs sparse matrices (scipy.sparse.csr_matrix) to save memory (e.g. 0.8 MB sparse vs. 400 MB dense).

Applying standard centering breaks sparsity:

# RAISES ValueError: Cannot center sparse matrices: pass `with_mean=False` instead.
StandardScaler().fit_transform(X_sparse)

Subtracting the column mean from a sparse matrix converts every stored zero into a non-zero value, converting a 2%-full matrix into a 100%-dense allocation that exhausts RAM.

Solution: Use StandardScaler(with_mean=False) or MaxAbsScaler(), which scales by dividing by the maximum absolute coordinate without subtracting the mean, preserving zero entries by design.

---

7. How Much Does Scaling Before the Split Really Leak?

To quantify the true impact of pre-split scaling leakage, consider a 30-split Monte Carlo trial comparing honest vs. leaked scaling:

Empirical Scaling Leakage Measurement

Scaler              Honest CV Score     Leaked Pre-Split Score      Inflation
StandardScaler          0.9643                  0.9645              +0.0002
MinMaxScaler            0.9663                  0.9657              -0.0006

The leakage inflation is +0.0002 and -0.0006 — pure statistical noise.

Why Scaling Leakage is Small

• Target Encoding Leak: Leaks each individual row's target $y$ directly into $X$, generating +21 points of fake accuracy.
• Scaling Leak: Leaks only a single aggregate mean ($\mu$) and standard deviation ($\sigma$) computed across hundreds of rows, revealing near-zero private information about any individual sample.

tip

Maintain Pipeline Discipline Regardless:

While scaling leakage is small on large datasets, you must still fit scalers strictly on training folds:

1. On small datasets (N < 100), single test outliers can distort training boundaries significantly.
2. In production, live requests arrive one sample at a time — it is physically impossible to compute standard deviations on single-sample inference requests.
3. Placing scalers inside a Pipeline enforces reproducible boundaries with zero extra code.

---

8. Implementation Lab

tip

Implementation Lab: Feature Scaling and Distribution Transformation

Trace accuracy gains across distance, linear, and tree-based models, observe how outliers compress MinMaxScaler ranges, evaluate Yeo-Johnson power transformations on skewed features, and inspect sparse scaling with with_mean=False.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Click "Runtime" > "Run all" or execute cells individually using Shift + Enter.
3. Follow the guided experiments: test the 215,171x variance disparity, compare RobustScaler against MinMaxScaler, and measure log1p overshoot.

Comparative Implementation: Production Pipeline vs. Scratch

1. Production Leak-Free Pipeline

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.svm import SVC

# Scaler parameters fit strictly on training folds inside cross-validation
model = make_pipeline(
    RobustScaler(),
    SVC(kernel="rbf")
)
model.fit(X_train, y_train)

2. Manual Scaling from Scratch (NumPy)

import numpy as np

# Vectorized Z-Score Normalisation
mean = np.mean(X_train, axis=0)
std = np.std(X_train, axis=0)

# Protect against divide-by-zero on constant columns
std[std == 0.0] = 1.0

X_train_scaled = (X_train - mean) / std
X_test_scaled  = (X_test - mean) / std  # Apply frozen training statistics!

---

9. Common Mistakes

1. Unnecessary Tree Scaling

• The Misconception: "Scaling is mandatory for all machine learning models, including Random Forest and XGBoost."
• Wrong Thinking: Believing that decision tree splits are affected by coordinate scales.
• The Right Principle: Decision trees split on single-feature rank thresholds ($x_j \le C$). Monotonic scaling preserves sorting orders identically, yielding exactly +0.0000 accuracy gain.

2. MinMaxScaler Outlier Poisoning

• The Misconception: "MinMaxScaler is superior because bounded [0, 1] intervals look cleaner."
• Wrong Thinking: Applying MinMaxScaler to columns with extreme values.
• The Right Principle: A single outlier anchors the maximum, compressing 99% of regular data into a tiny 1% band. Use RobustScaler (median and IQR) when outliers are present.

3. Row vs. Column Scaling

• The Misconception: "Normalizer is a drop-in alternative to StandardScaler."
• Wrong Thinking: Using Normalizer to scale feature columns.
• The Right Principle: Normalizer scales across rows (individual samples) to unit length, discarding magnitude. Use StandardScaler to scale down columns.

4. Reflexive Log Transforms

• The Misconception: "Apply np.log to any feature with a right-skewed tail."
• Wrong Thinking: Using np.log on columns containing zeros ($-\infty$) or moderate skew (causing log-overshoot).
• The Right Principle: Use np.log1p to handle zeros safely, or PowerTransformer(method="yeo-johnson") to let scikit-learn optimize the power parameter automatically.

5. Centering Sparse Matrices

• The Misconception: "Apply StandardScaler() to high-cardinality one-hot outputs."
• Wrong Thinking: Subtracting the column mean turns all zero entries into non-zeros, causing a 500x memory explosion.
• The Right Principle: Set with_mean=False in StandardScaler, or use MaxAbsScaler().

---

10. Summary

Scaler Selection

• StandardScaler: Unbounded mean 0, variance 1. Default for linear models, SVMs, and neural networks.
• MinMaxScaler: Bounded $[0, 1]$. Required for image pixel arrays or neural network inputs. Vulnerable to outliers.
• RobustScaler: Median and IQR-based. Robust against extreme spikes.
• MaxAbsScaler: Divides by maximum absolute value without centering. Preserves sparse matrix memory.

Distribution Transformations

• np.log1p: Safely compresses long right tails while mapping zero cleanly to zero.
• PowerTransformer: Optimizes exponent ($\lambda$) via Yeo-Johnson; supports negative numbers and prevents log-overshoot.
• QuantileTransformer: Forces rank uniform/normal output; destroys linear spacing.
• Tree Invariance: Monotonic scaling yields $+0.0000$ gain on decision trees.

info

Key Takeaways

• Distance Models Collapse Unscaled: Algorithms relying on Euclidean metrics (SVM +6.2%, kNN +3.7%) are blinded by unscaled units, while tree ensembles (+0.0000%) are structurally immune.
• Outliers Crush MinMaxScaler: A single outlier squashes legitimate data into less than 1% of the output space; RobustScaler prevents this via median centering.
• Log1p Handles Zeros Safely: np.log produces $-\infty$ on zero columns; use np.log1p or Yeo-Johnson transformations.
• Scaling Leakage is Small but Real: Aggregate scaling leaks only +0.0002 points, but training-set isolation remains mandatory for production single-row serving.

Next in this section: Outliers and Feature Engineering - detecting anomalies and engineering domain predictors.

See also: Encoding Categorical Data · Missing Values · The Toolkit and the Pipeline

---

11. Active Recall Flashcards

Attempt each conceptual diagnostic question first, then click to reveal the underlying derivation and explanation.

1. [THEORY] Why do distance-based models (SVM, kNN) require feature scaling while decision trees are completely invariant?

• Distance Models: Compute spatial distances using metric tensors (e.g. Euclidean distance $||u - v||_2$). Features on large scales (Area std = 569) outvote smaller-variance features (Smoothness std = 0.014) by over a million-to-one, ignoring informative predictive signals.
• Decision Trees: Evaluate splits on individual feature thresholds ($x_j \le C$). Because any monotonic transformation preserves rank ordering (u_j < v_j \iff f(u_j) < f(v_j)), the exact same partition of samples is produced before and after scaling, yielding $+0.0000$ accuracy gain.

2. [THEORY] State the mathematical formulations and output ranges for Standardisation (Z-Score) and Min-Max Normalisation.

• Standardisation (Z-Score): $z = \frac{x - \mu}{\sigma}$. Output range is unbounded, centered at mean $0$ with standard deviation $1$.
• Min-Max Normalisation: $x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$. Output range is strictly bounded to $[0, 1]$.

3. [ANALYZE] Why is MinMaxScaler uniquely vulnerable to a single outlier, and how does RobustScaler solve this?

• MinMaxScaler Vulnerability: Depends on the single absolute minimum ($x_{\min}$) and maximum ($x_{\max}$). A single large outlier anchors $x_{\max}$, compressing all legitimate non-outlier data into less than 1% of the output scale ($[0, 0.01]$).
• RobustScaler Resolution: Centers using the median and scales by the Interquartile Range (IQR = Q3 - Q1). Because percentiles depend on sample counts rather than extreme magnitudes, individual outliers have zero effect on scaling parameters.

4. [THEORY] What is the difference between StandardScaler and Normalizer in scikit-learn?

• StandardScaler: Operates column-wise (down features), standardizing each variable's variance independently across all samples.
• Normalizer: Operates row-wise (across features per sample), scaling each individual sample vector to unit Euclidean ($L_2$) length. It discards magnitude completely to preserve only vector direction (ideal for text frequency angles).

5. [THEORY] Why does np.log fail on zero columns, and how does np.log1p resolve this?

• np.log Failure: $\log(0) = -\infty$, injecting non-numeric floating-point values that destroy gradient calculations.
• np.log1p Solution: Computes $\log(1 + x)$. Since $\log(1 + 0) = 0$, zero values map cleanly to zero, safely compressing positive skewness while preserving sparsity.

6. [THEORY] What does PowerTransformer (Yeo-Johnson) do that fixed log and square root transforms cannot?

• Fixed transforms (log, sqrt) apply a rigid compression curve that can overshoot (e.g. flipping Population skew from $+4.94$ to $-1.04$).
• PowerTransformer optimizes an exponent parameter ($\lambda$) per column via maximum likelihood estimation to find the exact power that maximizes normality. Furthermore, its Yeo-Johnson formulation natively supports negative numbers, which logarithms cannot process.

7. [ANALYZE] What happens to a sparse matrix when centered, and why does StandardScaler raise a ValueError?

• Sparse Centering: Subtracting the column mean requires subtracting a non-zero value from every zero cell, converting a 2%-full sparse matrix into a 100%-full dense matrix.
• Scikit-Learn Safeguard: Raises a ValueError to prevent out-of-memory container crashes. The fix is to set with_mean=False in StandardScaler or use MaxAbsScaler().

8. [ANALYZE] Why is pre-split scaling leakage (+0.0002) statistically negligible compared to target-encoding leakage (+21 points)?

• Scaling Leakage: Leaks aggregate parameters (a single column mean and standard deviation) computed over hundreds of samples, revealing near-zero private information about any individual row's target label.
• Target-Encoding Leakage: Directly leaks the exact row target $y$ itself in a slightly blurred format directly into feature $X$, creating a circular dependency on the answer.

9. [ANALYZE] Given that scaling leakage is negligible, why must we still enforce fitting scalers strictly on training data?

1. Single-Row Production Serving: Live API endpoints receive data one sample at a time; calculating test-set standard deviations is physically impossible.
2. Small Datasets (N < 100): Test outliers on small validation sets can shift scaling parameters significantly.
3. Pipeline Reproducibility: Wrapping scalers inside a Pipeline guarantees identical, bug-free transformation boundaries across training and testing.

10. [THEORY] Why must missing value imputation always precede feature scaling?

Missing values (NaN) break mathematical calculations for sample mean, variance, and min/max bounds. Fitting a scaler on un-imputed data causes crashes or propagates NaN values across output matrices.