Missing Values

Topic - SimpleImputer(strategy="mean") takes two lines and always executes without raising an error or warning, returning a fully populated matrix. This chapter examines what those two lines actually do to your data distributions, when mean substitution is statistically legitimate, and why why a value is missing matters far more than which function you use to fill it.

Filling missing entries is not just data formatting — it is an act of statistical estimation. Imputing improperly introduces silent biases and artificially narrows variance.

---

1. Rubin's Missingness Taxonomy: MCAR vs. MAR vs. MNAR

In mathematical statistics, missing data mechanisms are structured into three distinct regimes defined by Donald Rubin (1976). The mechanism determines whether missing information can be recovered at all:

Mechanism	Formal Definition	Real-World Example	Can Imputation Recover It?
MCAR (Missing Completely at Random)	Missingness is strictly independent of both observed and unobserved variables: $P(M \mid Y_{obs}, Y_{mis}) = P(M)$.	A laboratory test tube drops and shatters randomly on the floor.	Yes. Surviving rows form an unbiased representative sample.
MAR (Missing at Random)	Missingness depends on other observed features, but not the missing value itself: $P(M \mid Y_{obs}, Y_{mis}) = P(M \mid Y_{obs})$.	Younger survey respondents are less likely to report salary, but Age is recorded in the table.	Yes. Multivariate algorithms (such as KNN or MICE) can reconstruct it using observed columns.
MNAR (Missing Not at Random)	Missingness depends directly on the unobserved missing value itself: $P(M \mid Y_{obs}, Y_{mis}) \neq P(M \mid Y_{obs})$.	High-income earners refuse to disclose their salaries on tax forms due to wealth privacy.	No. The missing data is systematically biased; no statistical imputer can recover the true mean.

---

Interactive Checkpoint: Classifying Data Gaps

Scenario: A clinical sensor stops recording blood pressure metrics whenever a patient's temperature spikes above 104°F (and temperature is recorded separately in the same table). What is the missingness mechanism of the blood pressure column?

The California District Benchmark: MCAR vs. MNAR

To quantify how mechanism dictates accuracy, consider a controlled experiment on 2,000 California districts where the true median income is known ($\mu = 3.851$, $\sigma = 1.936$). Masking 30% of values under MCAR vs. MNAR reveals the operational reality:

---

Interactive Missingness Mechanism & Bias Simulator

Compare how removing 30% of records under MCAR vs. MNAR affects baseline statistics and mean imputation outcomes:

True Ground Truth Mean

3.851

Observed Sample Mean

3.815

Mean-Imputed Mean

3.815

Systematic Bias

-0.9%

Key Takeaway: Under MCAR, surviving data is unbiased (3.815 vs 3.851). Imputing the mean preserves the central tendency accurately.

danger

No Statistical Imputer Can Fix MNAR

Under MNAR, advanced imputers (like KNN or Iterative MICE) cannot recover the truth because every observed training neighbor belongs to the visible portion of the distribution. When missingness is MNAR, valid engineering solutions are:

1. Preserve the signal with add_indicator=True (let the model learn from the absence of data).
2. Collect supplementary data targeting the missing demographic.
3. Explicitly state the boundary limitation of the resulting model.

---

2. What Mean Imputation Costs: The 19% Variance Collapse

Even under valid MCAR conditions where mean imputation is statistically legitimate, replacing gaps with the column mean introduces severe mathematical distortion:

State	Sample Mean	Standard Deviation	Variance Loss
Complete Truth	3.851	1.936	0.0%
MCAR (30% missing, before imputing)	3.815	1.885	-2.6%
MCAR + Mean Imputation	3.815	1.573	-18.8% collapse

Why Variance Collapses

This collapse is pure arithmetic: substituting 600 missing values with the exact constant 3.815 means those 600 records contribute exactly zero squared deviation to the variance sum:

$\sigma^2 = \frac{1}{N} \sum_{i=1}^N (x_i - \mu)^2$

The Three Inescapable Side Effects

1. Confidence Intervals Shrink Artificially: Because $\sigma$ is understated by 19%, standard errors ($SE = \sigma / \sqrt{N}$) are too small, leading to false statistical significance ($p$-hacking by accident).
2. Correlations are Diluted: A block of identical constants cannot co-vary with any other feature. Correlation coefficients between the imputed column and target variables fall monotonically.
3. The Distribution Develops an Artificial Spike: The continuous density distribution develops a massive, non-existent spike (Dirac delta peak) right at the mean.

---

3. Listwise Deletion & The Compounding Formula $(1 - p)^k$

The simplest alternative to imputation is dropping incomplete rows: listwise deletion (df.dropna()).

While deletion invents no synthetic values, it is mathematically punishing on wide datasets. If a dataset has $k$ features, each missing a fraction $p$ independently, the fraction of surviving complete rows is:

$\text{Share of Complete Rows Kept} = (1 - p)^k$

---

Interactive Listwise Deletion Compound Loss Calculator

Adjust the number of features and per-column missingness rate to observe how listwise deletion compounds exponentially across columns:

Number of Columns ($k$): 8

Missing Rate per Column ($p$): 10%

Surviving Complete Rows

43.0%

Discarded Rows

57.0%

The Compounding Formula: (1 - 0.10)^8 = 43.0%. Notice that with only 8 columns at 10% missingness each, calling df.dropna() quietly discards 57% of your entire sample!

Deletion vs. Imputation Trade-off Matrix

Metric	Listwise Deletion (df.dropna())	Statistical Imputation (SimpleImputer)
Fabricates Data?	No — preserves raw authenticity	Yes — substitutes point estimates
Preserves Sample Size?	No — loses $1 - (1-p)^k$ of rows	Yes — retains 100% of samples
Variance Preservation	High — sample spread remains natural	Poor — shrinks variance by substituting constants
Safe Under MNAR?	No — selectively removes non-reporting rows	No — imputes biased conditional values
Recommended When	Tiny missing fraction (<2\%), large $N$	Missingness spread across many columns

---

4. Choosing a Strategy & The Categorical Trap

SimpleImputer offers four substitution strategies:

Strategy	Replacement Value	Ideal Target Domain
strategy="mean"	Column mean ($\mu$)	Symmetric, normal numerical features with zero outliers
strategy="median"	Column median	Skewed numerical features or tables with extreme outliers
strategy="most_frequent"	Statistical mode	Categorical features or discrete ordinal integers
strategy="constant"	fill_value	Text fields (e.g. "Unknown") or sentinel indicators

warning

The Categorical Mean Trap: Producing Impossible Numbers

Applying strategy="mean" to binary or categorical columns stored as integers (0 and 1) produces silent nonsense:

cat = pd.DataFrame({"employed": [1, 1, 0, 1, 1, np.nan, 1, 0]})
imputed = SimpleImputer(strategy="mean").fit_transform(cat)
print("Imputed value:", imputed[5, 0])

Output

Imputed value: 0.714286  <- IMPOSSIBLE VALUE

Nobody is 71.4% employed. The column represents a binary state, not a quantity. A scikit-learn model will consume 0.714286 without error, distorting decision boundaries. Always use strategy="most_frequent" on categorical columns.

---

Interactive Strategy Selector: Outliers & Skewness

Sample column containing typical values [2.5, 3.0, 2.8, 3.2, 2.9], one missing entry, and an extreme outlier [80.0]:

Evaluated Fill Value: 2.95

Robust to Outliers

The median centers at 2.95, completely ignoring the extreme outlier and substituting a realistic value matching typical neighborhood occupancies.

Preserving Information with add_indicator=True

If data is missing under MAR or MNAR, the fact that a value was missing is itself an informative feature.

By default, imputation destroys this signal. Passing add_indicator=True to SimpleImputer instructs scikit-learn to append binary indicator columns:

imp = SimpleImputer(strategy="median", add_indicator=True)
X_imputed = imp.fit_transform(X)

Output Matrix

 age  salary  age_missing  salary_missing
25.0 50000.0          0.0             0.0
30.0 72500.0          0.0             1.0  <- Salary was missing
35.0 70000.0          0.0             0.0
32.5 80000.0          1.0             0.0  <- Age was missing

Downstream algorithms can now learn patterns like "clients with missing salary have a 3x higher default risk", preserving predictive utility even if the imputed point estimate is imperfect.

---

5. When Smarter Imputers Pay Off: KNN vs. Iterative (MICE)

While SimpleImputer treats each column in total isolation, multivariate imputers utilize correlations across features:

• KNNImputer: Finds the $k$ most similar rows based on Euclidean distance across observed columns, and calculates a weighted average.
• IterativeImputer (MICE): Models each feature as a regression function of all other features in round-robin fashion until convergence.

The Correlation Rule: Sophistication Requires Signal

In empirical trials masking 20% of values, recovery error (where 0.0 = perfect, 1.0 = guessing the mean) reveals an essential rule:

Feature Predictability	SimpleImputer(mean)	KNNImputer(k=5)	IterativeImputer	Outcome
High Correlation ($r = 0.83$, Bedrooms vs. Rooms)	1.475	0.725	0.727	KNN cuts error in half (51% improvement)
Zero Correlation ($r = 0.03$, House Age vs. Occupancy)	0.984	1.056	0.984	KNN is WORSE than guessing the mean!

tip

Always Inspect Correlation Before Reaching for Complex Imputers:

df.corr()[column_with_gaps].abs().sort_values(ascending=False)

If the highest inter-feature correlation is weak (r < 0.3), KNNImputer averages neighbors across irrelevant dimensional space, injecting noise. Multivariate imputers pay off in direct proportion to feature correlation, and not otherwise.

---

Interactive Smart Imputer Decision Advisor

Tune the observed inter-feature correlation and distribution shape to determine the optimal imputation strategy:

Max Inter-Feature Correlation ($r$): 0.80

Column Distribution Shape

Recommended Architecture

IterativeImputer (MICE) or KNNImputer

Strong linear correlation and normal distribution make multivariate regression highly effective, cutting recovery error by up to 50% compared to mean substitution.

---

6. The fit / transform Contract & Preventing Data Leakage

Imputation is the foundational demonstration of scikit-learn's cardinal rule: fit learns, transform applies.

# 1. Fit computes statistics strictly on the training partition
imputer = SimpleImputer(strategy="median")
imputer.fit(X_train)

# 2. Transform applies those frozen training statistics to both partitions
X_train_imputed = imputer.transform(X_train)
X_test_imputed  = imputer.transform(X_test)  # NEVER call fit or fit_transform here!

If you call fit_transform on X_test, the imputer computes test-set medians, leaking testing bounds into validation features. Test records must be imputed using training statistics, mimicking live inference where future test outcomes are completely unknown.

The All-NaN Silent Column Drop Trap

If a column in your training fold happens to be 100% missing, SimpleImputer cannot compute a mean or median. By default, it silently drops the column, returning an output matrix with fewer columns than input:

bad_df = pd.DataFrame({"a": [1.0, 2.0, 3.0], "b": [np.nan, np.nan, np.nan]})
out = SimpleImputer(strategy="mean").fit_transform(bad_df)
print("Output shape:", out.shape)  # Returns (3, 1) — column 'b' has VANISHED!

If column b is present in the test set, downstream models will crash with dimensional mismatch exceptions. To prevent this, pass keep_empty_features=True in scikit-learn 1.2+ to retain the column populated with zeros.

---

7. Implementation Lab

tip

Implementation Lab: Missing Value Imputation Benchmarks

Trace how MCAR vs. MNAR missingness alters baseline statistical distributions, calculate listwise deletion row volumes across scaling dimensions, and evaluate MAE performances of Simple, KNN, and Iterative (MICE) imputers live on the California housing dataset.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Click "Runtime" > "Run all" or execute cells individually using Shift + Enter.
3. Follow the guided experiments: test the 19% variance loss, benchmark KNN vs. MICE recovery error, and inspect add_indicator=True feature matrices.

Comparative Implementation: Baseline vs. Multivariate Imputation

1. SimpleImputer with MissingIndicator

from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

# Median imputation robust to outliers + binary indicator flag for MAR/MNAR
pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median", add_indicator=True)),
    ("classifier", LogisticRegression(max_iter=1000))
])

pipe.fit(X_train, y_train)
score = pipe.score(X_test, y_test)

2. Multivariate KNNImputer

from sklearn.impute import KNNImputer
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

# Distance-weighted nearest neighbors imputation (requires scaling!)
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("imputer", KNNImputer(n_neighbors=5)),
    ("classifier", LogisticRegression(max_iter=1000))
])

pipe.fit(X_train, y_train)
score = pipe.score(X_test, y_test)

---

8. Common Mistakes

1. Imputing before asking WHY

• The Misconception: "Our dataset has missing values, so let's call SimpleImputer(strategy='mean') and proceed."
• Wrong Thinking: Assuming all missing data can be solved with substitution.
• The Right Principle: Diagnose whether data gaps are MCAR, MAR, or MNAR first. Under MNAR, the observed mean is already biased by 25%. Mean substitution produces a confidently wrong model. Use add_indicator=True to let the model learn from missingness.

2. Scaling Before Imputing

• The Misconception: "Standardize features first, then impute missing values."
• Wrong Thinking: Calling StandardScaler().fit_transform(X) on matrices containing NaNs.
• The Right Principle: Always impute first, scale second. Standardizing standard deviations is corrupted by missing coordinates, and many scalers fail on NaN entries.

3. Blind Listwise Deletion

• The Misconception: "Only 10% of values are missing per column, so calling df.dropna() is harmless."
• Wrong Thinking: Forgetting that row loss compounds exponentially across columns via $(1 - p)^k$.
• The Right Principle: On an 8-column dataset with 10% missingness per column, calling dropna() discards 57% of your data. Use imputation rather than discarding clean features from partially missing rows.

4. fit_transform on Test Data

• The Misconception: "Call imputer.fit_transform(X_test) so the test set is filled using its own mean."
• Wrong Thinking: Re-calculating substitution statistics on the validation partition.
• The Right Principle: Imputers must learn substitution statistics strictly on X_train. The test set must be transformed using training statistics via imputer.transform(X_test) to prevent data leakage.

---

9. Summary

Statistical Foundations

• Rubin's Taxonomy: MCAR (recoverable), MAR (multivariate recoverable), MNAR (unrecoverable bias).
• The Variance Collapse: Mean substitution replaces points with constants, reducing standard deviation by ~19%.
• Deletion Compounding: Listwise deletion keeps $(1 - p)^k$ rows; 8 columns at 10% missingness loses 57% of data.

Operational Imputation Rules

• Strategy Selection: Use median for skewed metrics; use most_frequent for binary/categorical fields.
• Correlation Prerequisite: KNNImputer requires r > 0.3; otherwise it performs worse than guessing the mean.
• Preserve Signal: Use add_indicator=True to retain missingness patterns under MAR/MNAR.
• Test Isolation: Fit imputers on X_train; apply to X_test via .transform().

info

Key Takeaways

• No Imputer Fixes MNAR: Imputing with the observed mean under MNAR cements a 25% bias. Preserve the signal with add_indicator=True.
• Mean Imputation Distorts Variance: Replaces diverse observations with identical constants, artificially shrinking confidence intervals by ~19%.
• The Categorical Mean Trap: Mean substitution on binary integer columns produces impossible values like 0.714286. Always use most_frequent.
• Fit on Train, Transform on Test: Never call fit_transform on test data. Wrap imputers and models in a unified Pipeline to enforce test isolation.

Next in this section: Encoding Categorical Data - transforming strings and categories into model-ready numbers.

See also: Loading and Preparing Data · The Toolkit and the Pipeline

---

10. Active Recall Flashcards

Attempt each conceptual diagnostic question first, then click to reveal the underlying derivation and explanation.

1. [THEORY] Define Donald Rubin's three missing data mechanisms (MCAR, MAR, MNAR) and state whether statistical imputation can recover the true mean in each.

• MCAR (Missing Completely at Random): Missingness is completely independent of both observed and unobserved data ($P(M \mid Y_{obs}, Y_{mis}) = P(M)$). The surviving sample is unbiased. Imputation can recover the true mean.
• MAR (Missing at Random): Missingness depends on other observed features in the table, but not the missing variable itself ($P(M \mid Y_{obs}, Y_{mis}) = P(M \mid Y_{obs})$). Multivariate imputation (KNN / MICE) can recover the mean by conditioning on those observed columns.
• MNAR (Missing Not at Random): Missingness depends directly on the unobserved missing value itself ($P(M \mid Y_{obs}, Y_{mis}) \neq P(M \mid Y_{obs})$). The observed distribution is systematically biased. No statistical imputer can recover the true mean.

2. [ANALYZE] In the California district income benchmark, why did applying mean imputation under MNAR produce an unrecoverable 25% bias?

Under MNAR, the top 30% of highest incomes were hidden. The observed mean of the remaining 70% was 2.877 against the true population mean of 3.851. Mean substitution computes the average over observed data, filling all 600 missing gaps with 2.877. Every imputed value is systematically too low, locking in an unrecoverable 25.3% downward bias.

3. [THEORY] Why does mean imputation cause standard deviation to collapse by ~19% under MCAR? What operational risks does this introduce?

• Why It Collapses: Substituting 600 missing entries with a single constant value ($\mu$) means those 600 entries contribute exactly zero squared deviation ($(x_i - \mu)^2 = 0$) to the variance formula, artificially deflating standard deviation from 1.936 to 1.573.
• Operational Risks: Understated variance leads to artificially narrow confidence intervals and inflated $t$-statistics, creating false statistical significance. Furthermore, inter-feature correlations are diluted because constants cannot co-vary.

4. [THEORY] State the listwise deletion compounding formula for $k$ features with missing rate $p$. What percentage of rows is discarded on an 8-column dataset with 10% missingness per column?

• Compounding Formula: $\text{Share of complete rows retained} = (1 - p)^k$.
• Calculation for $k=8, p=0.10$: $(1 - 0.10)^8 = 0.90^8 \approx 0.4304 \quad (43.0\% \text{ kept})$
• Percentage Discarded: Exactly $100\% - 43.0\% = \mathbf{57.0\%}$ of all rows are discarded. Listwise deletion penalizes wide tables exponentially.

5. [ANALYZE] A junior engineer runs SimpleImputer(strategy='mean') on a binary column employed: [1, 0]. What value is imputed, why is it dangerous, and what should be used instead?

• Imputed Value: The mean of the binary indicators, resulting in a fractional value like 0.714286.
• Why Dangerous: 0.714286 is a non-existent state (nobody is 71.4% employed). The matrix remains numeric so no error is raised, but the fractional value distorts decision boundaries and linear weights.
• Correct Strategy: Use strategy="most_frequent" (mode imputation), which guarantees that filled values match valid categorical states.

6. [THEORY] Under what mathematical condition does KNNImputer outperform SimpleImputer? What happens when this condition is violated?

• Required Condition: High correlation between the missing feature and surrounding columns (r > 0.35).
• When Violated ($r \to 0$): In the California benchmark on an uncorrelated feature ($r = 0.03$), KNNImputer achieved an error of 1.0566 — worse than guessing the mean (0.9840). When features do not co-vary, KNN averages neighbors across irrelevant metric space, injecting distance noise.

7. [ANALYZE] Why is add_indicator=True in SimpleImputer considered "the closest thing to a free win" under MAR or MNAR?

Under MAR and MNAR, whether a patient or customer declined to provide an answer is itself an informative predictor. Standard imputation fills the gap but destroys the missingness signal. add_indicator=True appends a binary column (1.0 if missing, 0.0 if observed), allowing downstream models to learn patterns from the missingness itself even if the imputed point estimate is imperfect.

8. [THEORY] Explain the scikit-learn fit / transform contract for imputers. What error occurs if fit_transform is called on X_test?

• The Contract: .fit() computes substitution metrics ($\mu, \text{median}$) strictly from training rows X_train. .transform() substitutes those frozen training statistics into missing cells on both X_train and X_test.
• Error on Test: Calling fit_transform(X_test) recomputes medians over test rows, causing Data Leakage. The test set influences its own preprocessing, artificially inflating validation scores.

9. [ANALYZE] Describe the all-NaN column trap in SimpleImputer. How does it cause silent downstream pipeline crashes, and how is it resolved?

• The Trap: If a feature column in X_train happens to be 100% missing, SimpleImputer cannot compute a mean or median. By default, it silently drops the column, returning a matrix with shape $(N, D-1)$.
• Downstream Crash: If that column is present in X_test, imputer.transform(X_test) or the downstream model will throw a shape mismatch error because the training pipeline dropped a column.
• Resolution: Pass keep_empty_features=True to SimpleImputer (introduced in scikit-learn 1.2+), which retains the empty column populated with zeros.

10. [PROG] Write a leak-free scikit-learn pipeline chaining median imputation with missing indicator flags and a LogisticRegression classifier.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

# 1. Load & 2. Split
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42, stratify=data.target
)

# 3. Build Unified Pipeline (Impute -> Scale -> Classify)
pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median", add_indicator=True)),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000))
])

# 4. Fit & 5/6. Evaluate
pipeline.fit(X_train, y_train)
print(f"Leak-Free Test Accuracy: {pipeline.score(X_test, y_test):.4f}")