Outliers and Feature Engineering

Topic - Removing outliers sounds like standard data hygiene until you examine what an outlier actually is: a data entry defect to delete, a genuine extreme to preserve, or a novelty to study? This chapter examines statistical detection rules, the masking cliff under Z-scores, why Huber regression only cures $y$-outliers, and the feature engineering patterns that supply relationships linear models cannot construct on their own.

Treating all anomalous points identically destroys valuable predictive signal. Knowing when an anomaly is an error versus when it is the entire signal is the mark of mature data engineering.

---

1. The Three Types of "Outlier"

Before calculating a single threshold, identify which generative process produced the anomaly:

Outlier Category	Underlying Reality	Practical Example	Recommended Action
Data Defects (Errors)	Recording mistake, sensor glitch, or corrupt ETL parser	A patient weight recorded as 70,000 grams instead of 70 kg.	Delete or impute. These points are pure noise.
Genuine Extremes	Rare, legitimate events from the true underlying distribution	A luxury penthouse selling for $50M in a city housing dataset.	Keep or winsorise/transform. Removing them blinds models to high-value reality.
Novelties / Anomalies	Generated by a different causal process entirely	A credit card charge originating from an unauthorized offshore server.	Isolate and study. In fraud detection, novelties are the primary objective.

---

2. The IQR Rule & Reading Boxplots

The most universal non-parametric outlier detection heuristic is John Tukey's 1.5x IQR Rule:

$\text{IQR} = Q_3 - Q_1$

$\text{Lower Bound} = Q_1 - 1.5 \times \text{IQR}, \qquad \text{Upper Bound} = Q_3 + 1.5 \times \text{IQR}$

Boxplot Anatomical Boundaries

     Lower Fence      Q1 (25th)     Median (50th)   Q3 (75th)      Upper Fence
  ---|--------------[===========|===============]--------------|---  o  o (Outliers)
     Q1 - 1.5*IQR                   IQR                            Q3 + 1.5*IQR

Where 1.5 Comes From and What It Costs on Skewed Data

• Under a Normal Distribution: The distance $Q_1 - 1.5 \times \text{IQR}$ corresponds mathematically to $-2.698\sigma$, and $Q_3 + 1.5 \times \text{IQR}$ corresponds to $+2.698\sigma$. The interval captures 99.30% of all observations, flagging roughly 0.70% as potential outliers.
• Under a Skewed Distribution: On heavily right-skewed data (such as income or website visits), the natural distribution tail extends far to the right. Applying the 1.5x IQR rule directly on skewed raw data will flag up to 11% of completely legitimate, clean records as false outliers!

tip

Always Reduce Skew Before Applying the IQR Rule

Never run Q1 - 1.5*IQR directly on heavily skewed features. Apply a variance-stabilizing transform (np.log1p or PowerTransformer(method="yeo-johnson")) first to symmetrize the distribution, then calculate IQR fences.

---

3. Z-Scores, and Why Outliers Hide Each Other (The Masking Cliff)

The classic parametric outlier detector computes the sample Z-score:

$z = \frac{x - \mu}{\sigma}$

An observation is conventionally flagged if $|z| > 3$.

The Masking Trap

Standard Z-scores suffer from a fatal flaw called Masking: the very outliers you are trying to detect are included in the calculation of the sample mean ($\mu$) and standard deviation ($\sigma$).

When multiple outliers contaminate a dataset:

1. They pull the sample mean $\mu$ toward themselves.
2. They inflate the standard deviation $\sigma$ dramatically.

Because $\sigma$ appears in the denominator, the Z-scores of all outliers collapse simultaneously, dropping below the $|z| > 3$ threshold — falling off a masking cliff where the detector becomes completely blind.

---

Interactive Masking Cliff Visualizer: Z-Score vs. Modified Z-Score (MAD)

Inject extreme outliers (value 100) into clean baseline data [12, 14, 15, 15, 16, 18, 19, 20] to observe the masking cliff:

Number of Injected Outliers (100s): 4

Sample Mean (μ)

44.1

Sample Std Dev (σ)

39.6

Standard Z-Score

1.41

MASKED (Z <= 3.0)

Modified Z-Score (MAD)

13.74

Always Flagged (> 3.5)

The Masking Cliff Occurred: Because 4 outliers inflated the sample standard deviation to 39.6, the Z-score of the 100s plummeted to 1.41. They completely escape the standard |z| > 3 detector!

The Solution: Modified Z-Score with MAD

Boris Iglewicz and David Hoaglin (1993) introduced the Modified Z-Score, replacing sample mean and standard deviation with the median and Median Absolute Deviation (MAD):

$\text{MAD} = \text{median}\left(|x_i - \text{median}(X)|\right)$

$\text{Modified } Z_i = \frac{0.6745 \times (x_i - \text{median}(X))}{\text{MAD}}$

Because median and MAD have a 50% breakdown point (up to half the dataset can be corrupted before the statistics break down), the Modified Z-Score is completely immune to masking. Any observation with $|\text{Modified } Z_i| > 3.5$ is flagged cleanly.

---

4. The Outlier No Single Column Can See: Multivariate Anomalies

Univariate checks scan columns one at a time. However, data points can be completely normal along every single individual feature while being an extreme anomaly in combination.

The 155 cm, 85 kg Anomaly

Consider a biometric dataset containing Height and Weight ($r = 0.92$):

• Height = 155 cm: A normal height for an adult ($Z \approx -1.2$).
• Weight = 85 kg: A normal weight for an adult ($Z \approx +1.4$).

Both features pass univariate checks cleanly. However, an individual who is simultaneously 155 cm tall and weighs 85 kg represents an extreme anomaly that shatters the correlation structure:

---

Interactive Multivariate Covariance Anomaly Simulator

Adjust Height and Weight to observe how an observation can pass univariate checks while breaking covariance bounds:

Height: 155 cm

Weight: 85 kg

Height Z-Score

-1.50

Normal (|Z| <= 3)

Weight Z-Score

1.25

Normal (|Z| <= 3)

Mahalanobis Distance (DM)

6.15 σ

ANOMALY DETECTED

Multivariate Blindness Confirmed: Both Height (-1.50) and Weight (1.25) pass univariate checks with flying colors (|Z| < 2.0). Yet Mahalanobis distance (6.15 σ) flags this point as an extreme anomaly because it completely breaks the height-weight covariance structure!

Why Isolation Forest Struggles vs. Elliptic Envelope

• IsolationForest: Isolates anomalies using randomized axis-parallel cuts (horizontal and vertical lines). Because the 155 cm, 85 kg point falls well within the individual coordinate ranges of both axes, axis-parallel lines require many splits to isolate it.
• EllipticEnvelope: Fits a robust covariance matrix ($\Sigma$) to compute the Mahalanobis Distance: $D_M(x) = \sqrt{(x - \mu)^T \Sigma^{-1} (x - \mu)}$ It captures the orientation of the covariance ellipse, instantly isolating points that violate the correlation slope.

---

5. Outlier Treatment: What Actually Works

To evaluate outlier treatments objectively, consider a benchmark on 280 training rows with 20 corrupted rows (a 10x recording error), evaluated on a clean test set:

Empirical Treatment Benchmark

Treatment Applied                       Test R² Score       Recovery Status
Clean Data (Theoretical Ideal)              0.9902          100% Recovery
Do Nothing (Raw Training Data)              0.1310          Catastrophic Failure
Remove Flagged Rows (IQR Rule)              0.9867          Near-Total Recovery (99.6%)
Cap at Bounds (Winsorisation)               0.8083          Partial Recovery
Robust Model (HuberRegressor on X-outliers) 0.1232          Failed Completely

Why Robust Models Fix y-Outliers, Not X-Outliers

HuberRegressor is designed to resist outliers by down-weighting observations with large residuals ($|y - \hat{y}| > \delta$). However, its effectiveness depends entirely on where the outlier sits:

Huber vs. OLS across Corruption Types

Corruption Location         OLS Test R²     HuberRegressor Test R²  Outcome
Outliers in Target (y)        -2.3392               0.9901          Transformative Recovery
Outliers in Features (X)       0.1310               0.1232          Zero Improvement

• Target ($y$) Outliers: The point has normal features $X$, but a wild target $y$. The vertical residual is massive. HuberRegressor down-weights the residual cleanly, delivering $R^2 = 0.9901$.
• Feature ($X$) Outliers (High Leverage): The point sits far out along the $X$-axis. Like a long physical lever arm, it drags the regression line directly to itself! Because the line is pulled directly to the point, its residual remains tiny. Huber never down-weights it!

---

Interactive Leverage vs. Residual Simulator

Toggle between Target Outliers and High-Leverage Feature Outliers to see why robust loss functions behave differently:

OLS Test R²

-2.3392

HuberRegressor Test R²

0.9901

Huber Effectiveness

Total Recovery

Target Noise Handled: The observation has normal feature values but an extreme target value. The vertical residual is huge, allowing Huber loss to down-weight it and restore near-perfect R² (0.9901).

---

Interactive Checkpoint: Robust Regression

If your training set has outliers in its feature columns (X-outliers), why doesn't fitting a HuberRegressor help prevent model distortion?

Decision Tree for Outlier Treatment

danger

Never Clean the Test Set

Removing outliers from training data is a valid decision about what patterns your model should learn. Removing outliers from the test set is fraudulent: it fabricates an evaluation report on a synthetic world that does not exist. Real production data will contain extreme outliers; if your model crashes or predicts nonsensical numbers, that is a discovery you must make during validation.

---

6. Feature Engineering: Supplying What Models Cannot Form

Feature engineering adds information that is implicit in the data but structured in a form algorithms cannot learn on their own.

Just as with scaling and encoding, linear models require extensive feature engineering, while deep tree ensembles generate interactions for themselves.

Cyclical Features: Hour 23 and Hour 0 Are Neighbors

Suppose an outcome depends on night-time hours ($22, 23, 0, 1, 2$). This sequence wraps past midnight. On a raw integer scale ($0..23$), hour 23 and hour 0 sit at opposite physical extremes:

Accuracy on Night-Time Events

Feature Representation              Logistic Regression     Random Forest
Raw Hour (0..23)                    0.6933 (BASELINE)       0.8337
Trigonometric (sin / cos) of Hour   0.8203                  0.8337
One-Hot Encoded Hour (24 columns)   0.8337                  0.8337
Majority Class Baseline: 0.6933

Logistic regression on raw hour scored 0.6933 — the majority class baseline. It learned literally nothing because a single linear coefficient $\beta \cdot \text{hour}$ can only express monotonic trends ("later hours are more likely").

Mapping hours onto a circle using sine and cosine wraps midnight seamlessly:

$\text{hour\_sin} = \sin\left(\frac{2\pi \times \text{hour}}{24}\right), \qquad \text{hour\_cos} = \cos\left(\frac{2\pi \times \text{hour}}{24}\right)$

---

Interactive Cyclical Feature Clock Visualizer

Select an hour of the day to inspect how sine and cosine coordinates preserve the continuous wrap-around past midnight:

Current Hour: 23:00

hour_sin

-0.259

hour_cos

0.966

Linear Distance to Hour 0

23 units

Cyclical Distance to Hour 0

0.261 units

Midnight Wrap-Around Preserved: On a linear scale, Hour 23 is 23 units away from Hour 0. But in circular trigonometric coordinates, Hour 23 and Hour 0 are immediate neighbors (distance = 0.261), allowing linear models to boost accuracy from 0.69 to 0.82!

High-Yield Feature Engineering Families

Feature Family	Concrete Example	Why It Unlocks Accuracy
Ratios	debt / income, price per m²	Linear models cannot divide; ratios are often the true underlying causal driver.
Interactions	length * width (Area)	Linear models compute weighted sums ($\sum w_j x_j$); they cannot multiply two features.
Cyclical	$\sin / \cos$ of hour, month	Preserves continuous boundaries across midnight or calendar year ends.
Datetime Extractions	is_weekend, hour_of_day	Raw timestamps are unusable continuous integers; business logic lives in cycles.
Aggregates	Past customer average spend	Collapses one-to-many transactional history into a single sample vector.
Domain Formulas	$\text{BMI} = \text{weight} / \text{height}^2$	Encodes specialized physical or financial laws no algorithm can infer from sparse data.

---

7. Implementation Lab

tip

Implementation Lab: Outliers and Advanced Feature Engineering

Observe the standard Z-score masking cliff live, compare univariate scans against bivariate Mahalanobis distance via EllipticEnvelope, evaluate the failure of HuberRegressor on high-leverage points, and construct cyclical sine/cosine encoders.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Click "Runtime" > "Run all" or execute cells individually using Shift + Enter.
3. Follow the guided experiments: test the masking cliff by adding identical outliers, evaluate PolynomialFeatures(interaction_only=True), and measure Cyclical hour gains.

Comparative Implementation: Feature Interactions & Cyclical Encoding

1. Production Preprocessing Pipeline

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import FunctionTransformer, RobustScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

def encode_cyclical_hour(X):
    # Maps hour column (0-23) onto 2D unit circle
    hour = X[:, 0]
    sin = np.sin(2 * np.pi * hour / 24)[:, None]
    cos = np.cos(2 * np.pi * hour / 24)[:, None]
    return np.hstack([sin, cos])

preprocessor = ColumnTransformer(
    transformers=[
        ("cyclical", FunctionTransformer(encode_cyclical_hour), ["hour"]),
        ("robust_num", RobustScaler(), ["salary", "debt_ratio"])
    ]
)

model = Pipeline([
    ("prep", preprocessor),
    ("clf", LogisticRegression())
])

2. Modified Z-Score with MAD from Scratch

import numpy as np

def modified_z_score(x):
    """
    Computes Iglewicz & Hoaglin modified z-score using median and MAD.
    Immune to the masking cliff.
    """
    med = np.median(x)
    mad = np.median(np.abs(x - med))
    if mad == 0:
        return np.zeros_like(x)
    
    mod_z = 0.6745 * (x - med) / mad
    return mod_z

# Outlier mask: values with |mod_z| > 3.5
outliers = np.abs(modified_z_score(data)) > 3.5

---

8. Common Mistakes

1. Blind Outlier Deletion

• The Misconception: "Every point flagged by a statistical outlier detection rule is corrupt and should be deleted immediately."
• Wrong Thinking: Deleting flagged rows purges genuine extreme records and critical fraud signals, eliminating the most valuable rows in the dataset.
• The Right Principle: Only delete confirmed data entry defects. Genuine extremes should be preserved (or winsorised/transformed for linear models).

2. Raw Skew Detection

• The Misconception: "Applying the IQR rule directly on skewed features isolates data errors."
• Wrong Thinking: On heavily skewed distributions, the 1.5x IQR rule flags up to 11% of clean, legitimate observations as false outliers.
• The Right Principle: Always apply skew-reducing transformations (log1p or Yeo-Johnson) before running outlier detection rules.

3. Standard Z-Score Masking

• The Misconception: "The standard $|z| > 3$ rule is the most robust way to find univariate outliers."
• Wrong Thinking: Multiple outliers inflate the sample standard deviation, driving their own Z-scores below 3.0 and falling off the masking cliff.
• The Right Principle: Use Modified Z-scores with Median Absolute Deviation (MAD), which has a 50% breakdown point and resists masking.

4. Univariate Blindness

• The Misconception: "Running outlier detection column-by-column catches every anomaly."
• Wrong Thinking: Single-column rules cannot see correlation-breaking anomalies (e.g. 155 cm height, 85 kg weight).
• The Right Principle: Use multivariate tools like EllipticEnvelope (Mahalanobis distance) to detect violations of joint covariance structure.

5. Huber Regressor on X-outliers

• The Misconception: "If features contain extreme outliers, I can just use a HuberRegressor robust loss."
• Wrong Thinking: Huber regressor only resists target ($y$) outliers by down-weighting large residuals. Feature ($X$) outliers exert high leverage, dragging the fitted line to themselves so their residuals remain tiny.
• The Right Principle: Huber is only useful for $y$-corruption. For $X$-corruption, you must explicitly detect and remove, winsorise, or transform the outliers.

---

9. Summary

Outlier Detection & Action

• Univariate Robustness: Replace standard Z-scores with Modified Z-scores (median/MAD-based) to structurally prevent masking.
• Correlation Breaks: Deploy EllipticEnvelope (Mahalanobis distance) to flag impossible multivariate combinations.
• Isolation Forest limits: Use for extremes in individual coordinates; use covariance-aware models for correlated variables.
• X vs y Outliers: Huber robust regression only fixes y outliers. Feature outliers must be capped, transformed, or removed.

Feature Engineering

• Cyclical Wrapping: Convert time parameters into sin and cos coordinates to bridge midnight boundaries.
• Multiplicative interactions: Supply length * width explicitly to linear models so they can fit scale-sensitive area metrics.
• High-Yield Ratios: Supply debt / income directly, as linear models cannot divide.
• Data Leakage: Never calculate aggregates or means over test rows; isolate calculations to training splits only.

info

Key Takeaways

• Outliers Are Not All Errors: Recording errors must be deleted or imputed. Genuine extremes and novelties contain critical signal and must be preserved.
• The Masking Cliff: Standard Z-scores suffer a masking cliff as outliers inflate sample variance. Median and MAD remain structurally immune.
• Linear Models Need Help: Linear models score baseline performance on cyclical features like hour, whereas tree-based models decode the boundaries automatically.
• Never Clean the Test Set: Removing outliers from test partitions fabricates evaluation reports on non-existent worlds.

Next in this section: Train / Test Split - honest evaluation, stratification, and avoiding split-time data leakage.

See also: Feature Scaling and Transformation · Encoding Categorical Data · Missing Values

---

10. Active Recall Flashcards

Attempt each conceptual diagnostic question first, then click to reveal the underlying derivation and explanation.

1. [THEORY] What are the three distinct categories of outliers, and how should their treatment differ?

1. Errors (Data Defects): Mistyped or wrong data (e.g. weight in grams instead of kg). Treatment: Delete or impute.
2. Genuine Extremes: Real, rare data points from the same distribution (e.g. a mansion in housing data). Treatment: Keep (or transform/winsorise for linear models).
3. Novelties / Anomalies: Data points from a different generative process (e.g. fraud). Treatment: Keep and study, as they contain the primary signal.

2. [THEORY] Why is the standard IQR 1.5x rule dangerous to apply directly on skewed features?

• The 1.5x IQR threshold is calibrated strictly for symmetric normal distributions (flagging about 0.70% of points).
• On heavily right-skewed clean distributions, the tail is naturally thick, causing the rule to flag up to 11% of perfectly ordinary, legitimate data points as outliers. You must transform features to reduce skew before applying outlier detection.

3. [THEORY] What is masking in outlier detection, and what causes the abrupt masking "cliff" under Z-scores?

• Masking: Occurs when multiple outliers inflate the very statistics (mean and standard deviation) used to measure them, allowing them to hide in plain sight.
• The Cliff: Because multiple outliers share nearly identical extreme values, they inflate standard deviation concurrently. Once standard deviation crosses a critical threshold, the Z-scores of all outliers drop below $|z| = 3$ simultaneously, causing them to vanish from the detector all at once.

4. [THEORY] How does modified Z-score using MAD prevent masking?

Instead of mean and standard deviation, modified Z-score is computed as:

$\text{Modified } z = \frac{0.6745 \times (x - \text{median})}{\text{MAD}}$

Because median and Median Absolute Deviation (MAD) are based on ranked middle values, they resist changes even if up to 50% of the sample is corrupted, preventing outliers from inflating the scale and escaping detection.

5. [ANALYZE] How can a data record be completely unflagged by univariate checks yet be the most extreme outlier in the dataset?

• A multivariate outlier violates the relationship (covariance structure) between features rather than feature limits.
• For example, a height of 155 cm is a normal short height, and a weight of 85 kg is a normal heavy weight. Neither column is univariately flagged. However, a person who is both 155 cm and 85 kg represents an impossible joint distribution. This covariance break can only be detected via multivariate tools like Mahalanobis distance.

6. [ANALYZE] Why did IsolationForest fail to flag the 155 cm, 85 kg outlier while EllipticEnvelope succeeded?

• IsolationForest: Isolates points using random axis-parallel cuts (perpendicular lines along coordinate axes). Since the outlier's height and weight are both within standard coordinates, it cannot be easily fenced off with axis-parallel lines.
• EllipticEnvelope: Fits a full covariance matrix to measure Mahalanobis distance. It directly captures the correlation slope (0.92) and flags the point because it falls far off the principal diagonal.

7. [ANALYZE] Why does HuberRegressor robust loss completely fail on X-outliers (features) while succeeding on y-outliers (targets)?

• Huber regressor is a residual-based robust model: it down-weights points with large residuals (errors).
• Outliers in features ($X$) represent high-leverage points. They sit far along the feature axis and have so much lever-arm power that they drag the fitted OLS line directly to themselves. Because the line is pulled directly to the $X$-outlier, its residual stays tiny. Since its residual is small, Huber never down-weights it.

8. [THEORY] Why is raw hour (0-23) completely useless for a Logistic Regression model when modeling cyclical night events?

• Linear models fit a single monotonic coefficient ($\beta$) per feature: it can only express "higher hours lead to higher probability."
• Night-time spans midnight, wrapping from hour 22, 23 to 0, 1, 2. A single monotonic coefficient cannot capture a relationship where both the highest hours (23) and lowest hours (0) have positive coefficients, resulting in zero performance.

9. [PROG] How do you transform a cyclical hour column into standard coordinates, and why are two columns required?

We map hours onto a circle of circumference $2\pi$ using sine and cosine waves:

$\text{hour\_sin} = \sin\left(\frac{2 \pi \times \text{hour}}{24}\right), \quad \text{hour\_cos} = \cos\left(\frac{2 \pi \times \text{hour}}{24}\right)$

• Two columns are mandatory because a single cyclical coordinate is ambiguous (e.g. sine has identical values at both sunrise and sunset; cosine is required to disambiguate the direction). Together, they cleanly map hours as coordinates on a circle.

10. [ANALYZE] Why are ratio features (e.g. debt / income) highly effective additions for linear models?

Linear models can only compute weighted sums ($\beta_1 x_1 + \beta_2 x_2$). They are mathematically incapable of performing division or multiplication on features. Providing a ratio like debt / income explicitly supplies a critical scale-free feature that would be impossible for the model to construct on its own.