Missing Values

Topic — SimpleImputer(strategy="mean") is two lines and it always "works" — no error, no warning, a full table at the end. This page is about what those two lines actually did to your data, when they were the right call, and how to tell the difference.

The central claim, which the measurements below support: why a value is missing matters more than which function you use to fill it.

---

First, why is it missing?

Missingness Taxonomy: MCAR vs. MAR vs. MNAR

In statistical modeling, missing data is structured into three primary mechanisms defined by Donald Rubin (1976):

• MCAR (Missing Completely at Random): Data loss is entirely independent of any observed or unobserved variable. The missing rows represent a perfectly unbiased random slice of your population (e.g., a laboratory container dropped and shattered). Imputing with column averages is statistically valid.
• MAR (Missing at Random): The probability of a data point being missing depends on other observed, recorded variables, but not the missing value itself. For example, younger survey respondents are less likely to report salary details; but if we control for Age, the missingness within Age classes is completely random. Information can be effectively reconstructed using multivariate predictors (e.g. KNNImputer).
• MNAR (Missing Not at Random): The probability of data missing depends directly on the unobserved missing value itself. For example, high-income earners refuse to report salary because of their high wealth. This creates severe bias; the missingness carries its own signal, and no statistical imputer can recover the true mean.

🧠 Interactive Checkpoint: Classifying Data Gaps

Scenario: A blood pressure tracking sensor automatically stops recording metrics when a patient's temperature spikes above 104°F (and temperature is recorded separately in the same table). What is the statistical missingness mechanism of the blood pressure column?

Statisticians classify missingness into three mechanisms. This isn't a theoretical nicety — the mechanism decides whether imputation can work at all.

Mechanism	Means	Example	Can imputation recover it?
MCAR — Missing Completely At Random	Missingness is unrelated to anything, observed or not	A lab machine failed on random samples	Yes, in principle
MAR — Missing At Random	Missingness depends on other observed columns	Older patients skip a question, and you recorded age	Yes, using those columns
MNAR — Missing Not At Random	Missingness depends on the missing value itself	The highest earners decline to state income	No — the information is gone

The names are unhelpful (MAR isn't "random" in any everyday sense), but the distinction is sharp:

• Under MCAR, the rows with data look like the rows without. Your sample is smaller but unbiased.
• Under MAR, the rows differ — but in ways other columns can explain, so a model can compensate.
• Under MNAR, the rows differ for reasons nothing in your data captures. No technique fixes this, because the systematic difference is invisible.

The difference, measured

Take a complete column — median income across 2,000 California districts — remove 30% of it two different ways, and compare against the truth you still have:

Output

   scenario                                mean     std     corr
   complete truth                         3.851   1.936   -0.097
   MCAR 30% gone, before imputing         3.815   1.885   -0.120
   MCAR + mean imputation                 3.815   1.573   -0.101
   MNAR top 30% hidden, before            2.877   0.831   -0.069
   MNAR + mean imputation                 2.877   0.696   -0.056

Under MCAR the surviving data is trustworthy. Mean 3.815 against a true 3.851, std 1.885 against 1.936. Losing 30% of values at random cost almost nothing in accuracy, because what remained was a fair sample.

Under MNAR it is not. Hiding the top 30% of incomes gives an observed mean of 2.877 against a true 3.851 — a 25% underestimate. And imputing with the mean cannot help, because the mean you have available is already the wrong mean. You fill 600 gaps with 2.877 and every one is confidently, systematically too low.

danger

No imputer can fix MNAR

Look at the two MNAR rows: imputation left the mean at 2.877 and pushed the std further from the truth. Applying a more sophisticated imputer would not have helped — KNN and Iterative imputers learn from the observed values, and every observed value here is from the bottom 70%.

When missingness is MNAR, the honest options are: model the missingness itself (see add_indicator below), collect the data differently, or state the limitation. Silently imputing produces a confident, biased model.

Diagnosing the mechanism is a domain question, not a statistical one — the same conclusion as the sentinel values on the previous page. A useful test: could the value itself plausibly be the reason it's absent? Income, weight, symptom severity, salary expectations — usually yes. Sensor readings dropped by a network fault — usually no.

---

What mean imputation costs

Return to the MCAR rows, the case where imputation is legitimate:

	mean	std
Complete truth	3.851	1.936
MCAR, before imputing	3.815	1.885
MCAR + mean imputation	3.815	1.573

The mean survived. The standard deviation fell from 1.936 to 1.573 — a 19% loss.

This is not a bug, it's arithmetic. You replaced 600 varied values with 600 copies of one number. Those 600 rows now contribute zero variance. The column's spread is understated, and so is anything computed from it.

Consequences worth knowing:

• Variance and standard deviation shrink, so confidence intervals are too narrow and significance is overstated
• Correlations are diluted — a block of identical values can't co-vary with anything
• The distribution grows a spike at the mean that does not exist in reality
• The more you impute, the worse all three get — at 50% missing, half your column is one number

note

Mean imputation is a point estimate pretending to be data

SimpleImputer fills the single most likely value and discards all uncertainty. The model then treats an imputed 3.815 with exactly the same confidence as a measured 3.815.

This is what multiple imputation exists to fix: impute several times with different plausible values, fit a model to each, and combine — so the uncertainty survives into the results. IterativeImputer is scikit-learn's single-imputation approximation of that idea.

---

Deletion, and its arithmetic

The alternative to filling gaps is dropping rows — listwise deletion, df.dropna(). It's honest in a way imputation isn't: nothing is invented. It's also more expensive than people expect, because a row is discarded if any column is missing.

For k columns each independently missing a fraction p, the share of complete rows is (1 − p)ᵏ:

Output

2. LISTWISE DELETION — dropna() on 8 columns
     missing per column  rows kept (theory)  measured
                    1%              92.3%     91.6%
                    5%              66.3%     65.0%
                   10%              43.0%     42.6%

8 columns at 5% missing each — a level nobody would call a problem — costs you 35% of your rows. At 10% each you keep 43%. The measured figures track the formula closely, so this is predictable rather than bad luck.

The exponent is what hurts: the loss compounds across columns, so wide datasets are punished hardest.

	Deletion	Imputation
Invents data?	No	Yes
Keeps sample size?	No — loses 1 − (1−p)ᵏ	Yes
Biased under MCAR?	No, just smaller	No
Biased under MNAR?	Yes	Yes
Distorts variance?	No	Yes
Reasonable when	Few missing, plenty of rows, MCAR	Missing spread across many columns

Neither is safe under MNAR. Deletion under MNAR quietly removes exactly the rows that differ, which is the same bias by another route.

Two other options worth naming:

• Drop the column. If a feature is 80% missing, it carries little and imputing it fabricates most of it. Often the right call.
• Treat missing as a category. For categorical data, "Unknown" is a legitimate value rather than a gap to be filled. This is strategy="constant".

---

Choosing a strategy

SimpleImputer offers four:

strategy	Fills with	Use for
"mean"	Column mean	Numeric, roughly symmetric, no wild outliers
"median"	Column median	Numeric that is skewed or has outliers
"most_frequent"	The mode	Categorical, or low-cardinality integers
"constant"	fill_value	When missing has meaning, or for text

Prefer median over mean more often than you'd think. The mean is pulled by outliers, and one extreme value drags every imputed cell with it. On the previous page's data, AveOccup had a maximum 442× its median — a mean imputation there fills gaps with a number inflated by a handful of extremes. Median is unaffected.

Mean on a categorical column produces nonsense

This is the most common strategy error, and it's silent:

cat = pd.DataFrame({"employed": [1, 1, 0, 1, 1, np.nan, 1, 0]})
SimpleImputer(strategy="mean").fit_transform(cat)
SimpleImputer(strategy="most_frequent").fit_transform(cat)

Output

mean imputation gives: 0.714286
most_frequent gives:   1

employed is a yes/no fact. 0.714286 is not a possible value of it — nobody is 71% employed. The array is still numeric so nothing complains, and a model will happily consume a column containing a value that cannot exist.

The trap is that 0 and 1 look numeric. A column being stored as a number does not make it a quantity — the same test as the postcode case on the previous page: would arithmetic on this mean anything?

---

When a smarter imputer is worth it

SimpleImputer ignores every other column. Two alternatives don't:

• KNNImputer — find the k most similar rows by their other features, and average their values for the missing one
• IterativeImputer — model each column as a function of the others, and iterate

The natural assumption is that these must be better. Here is the honest measurement. Take complete data, hide 20% of one column, impute, and compare against the values we hid. Error is normalised so that 0.0 is perfect recovery and 1.0 is no better than guessing the column mean:

Output

3. RECOVERY ERROR vs THE VALUES WE HID  (0 = perfect, 1 = no better than the mean)

   predictable column  (r = 0.83)
     SimpleImputer(mean)        1.4752
     KNNImputer(5)              0.7255
     IterativeImputer           0.7276

   unpredictable column  (r = 0.03)
     SimpleImputer(mean)        0.9840
     KNNImputer(5)              1.0566
     IterativeImputer           0.9840

The result reverses depending on one thing: how well the other columns predict the missing one.

When the column is predictable (r = 0.83) — average bedrooms, given average rooms — KNNImputer cut the error roughly in half, 1.4752 → 0.7255. Real, substantial, worth the extra cost.

When it isn't (r = 0.03) — house age, given average occupancy — KNNImputer was worse than the mean (1.0566 vs 0.9840). It averaged neighbours that were neighbours in an irrelevant space, adding noise. IterativeImputer scored 0.9840, identical to the mean, because its regression found no usable signal and correctly fell back.

tip

Check the correlation before reaching for a fancy imputer

df.corr()[column_with_gaps].abs().sort_values(ascending=False)

If the missing column's best correlation with another feature is weak, KNNImputer and IterativeImputer have nothing to work with, and the median is both cheaper and safer. The sophistication pays in proportion to the correlation, and not otherwise.

Two further cautions from the same experiment. When all five columns were masked at once, plain mean beat both — with gaps everywhere, KNN's neighbours are themselves incomplete. And IterativeImputer scored far worse than the mean on heavily skewed columns (AveOccup, skew 44.7), because its linear model extrapolates catastrophically into a long tail.

Imputer	Cost	Good when	Bad when
mean / median	Trivial	Weak correlations; a quick baseline	You had usable signal and ignored it
KNNImputer	Expensive — distances between all rows	Strong correlations; few gaps	Many gaps; irrelevant features; large data
IterativeImputer	Expensive — iterated regressions	Strong linear structure	Heavy tails; extrapolation

Always measure rather than assume, using exactly the method above: hide values you have, and see which imputer recovers them.

---

Keeping the fact that it was missing

If missingness is informative — which is precisely the MAR and MNAR cases — then whether a value was missing is itself a feature, and imputation destroys it.

add_indicator=True keeps it:

imp = SimpleImputer(strategy="mean", add_indicator=True).fit(tr)

Output

without indicator: 2 columns
with indicator:    4 columns  <- 2 extra flags

 age  salary  age_missing  salary_missing
25.0 50000.0          0.0             0.0
30.0 72500.0          0.0             1.0
35.0 70000.0          0.0             0.0
32.5 80000.0          1.0             0.0
40.0 90000.0          0.0             0.0

You get the filled column and a binary flag per column recording where the gaps were. The model can then learn "customers who didn't state income behave differently" — a pattern that survives even when the imputed number itself is wrong.

This is the closest thing to a free win on this page. Under MNAR it's often the only part of the signal you can legitimately keep.

---

The fit / transform contract

Imputation is the clearest possible illustration of the rule from The Toolkit and the Pipeline: fit learns, transform applies.

fit computes the statistics and stores them:

imp = SimpleImputer(strategy="mean").fit(train)
print(imp.statistics_)

Output

imp.statistics_ (learned from train) = [3.25e+01 7.25e+04]

transform fills gaps using those numbers — including on the test set:

Output

test set after transform:
 age  salary
32.5 72500.0
99.0 10000.0
-> the test row was filled with TRAIN means, not its own

The test row's NaNs became 32.5 and 72500 — the training means. That's correct and it's the whole point. If transform recomputed the mean from the test set, the test set would have influenced its own preparation, and its score would no longer estimate performance on unseen data.

danger

fit_transform on train, transform on test

X_train = imp.fit_transform(X_train)   # learn AND apply
X_test   = imp.transform(X_test)       # apply only — never fit_transform here

Calling fit_transform on the test set is data leakage. It raises no error, changes the numbers only slightly, and inflates your score. The most reliable defence is a Pipeline, which makes the mistake structurally impossible.

An all-NaN column silently disappears

A genuine trap:

bad = pd.DataFrame({"a": [1.0, 2.0, 3.0], "b": [np.nan] * 3})
out = SimpleImputer(strategy="mean").fit_transform(bad)

Output

input shape  (3, 2)   columns ['a', 'b']
output shape (3, 1)   <- column 'b' has VANISHED
statistics_ = [ 2. nan]  (nan means 'unusable')

There is no mean of nothing, so SimpleImputer drops the column and returns fewer columns than it received. No warning.

This bites when a column happens to be entirely missing in your training slice but present in test — the shapes then disagree, and the error surfaces somewhere unrelated. Check imp.statistics_ for nan entries, or use keep_empty_features=True to retain the column.

---

Common Mistakes

Here are the most common conceptual pitfalls when handling missing datasets, compared with the correct engineering principles.

🚨 Mistake 1: Imputing before asking WHY

• The Pitfall: Blindly applying an imputation strategy (such as mean-fill) without diagnosing the underlying missingness mechanism.
• ❌ Wrong Thinking: "It has missing values, let's call SimpleImputer(strategy='mean') and move on."
• ✅ The Right Principle: Establish whether data gaps are MCAR, MAR, or MNAR first. Under MNAR (non-random drops), mean imputation is heavily biased and cannot recover the truth. You must use add_indicator=True or collect data differently.

🚨 Mistake 2: Scaling Before Imputing

• The Pitfall: Scaling or standardizing feature coordinates before performing your missing value imputation.
• ❌ Wrong Thinking: Standard scaling all data first, leaving NaN blocks intact.
• ✅ The Right Principle: Always impute missing values first, and scale second. Standardizing standard deviations is heavily distorted by missing indices, and subsequent imputation can skew normalized margins.

🚨 Mistake 3: Blind Deletion

• The Pitfall: Routinely dropping incomplete rows using .dropna() (listwise deletion) on wide datasets.
• ❌ Wrong Thinking: "Only 10% is missing in each column, let's drop them."
• ✅ The Right Principle: Calculate rows kept first: $1 - (1 - p)^k$. For 8 columns at 10% missingness each, listwise deletion silently drops 57% of your entire dataset! Use imputation solvers instead of throwing away massive data volumes.

🚨 Mistake 4: Estimator fit leakage

• The Pitfall: Calling fit_transform on your test or deployment dataset inputs.
• ❌ Wrong Thinking: imputer.fit_transform(X_test)
• ✅ The Right Principle: Call strictly imputer.transform(X_test). The imputer must learn substitution statistics (like column means or medians) solely from the training partition to avoid data leakage.

---

Summary

🔍 Missingness Taxonomies

• MCAR: Completely random gaps. Safe to drop or mean-fill.
• MAR: Explained by observed variables. Reconstruct using multivariate models (KNN/Iterative).
• MNAR: Gaps carry their own signal. Severe bias risk. Use missingness indicators.

🤖 sklearn Imputation solvers

• SimpleImputer: Fills columns with mean, median (skewed data), or most frequent (categorical).
• KNN / MICE: Predict missing values from correlated columns. KNN performs worse than mean if correlation is weak.

info

📌 Key Takeaways: Missing Values Imputation

• 📉 Imputation Shrinks Variance: Mean imputation preserves column averages but crushes standard deviation (by 19% in our California trial). Imputed values contribute zero variance and dilute correlations.
• 📐 Listwise Deletion scales poorly: Dropping incomplete rows via dropna scales exponentially with column counts. 8 columns at 10% missingness drops 57% of your dataset.
• 📊 Use Skew-Appropriate Solvers: Prefer median for highly skewed fields, and most_frequent for categorical columns. Filling binary outputs with mean can yield impossible values (like 0.714286).
• 🔌 Test-Set Imputing Rule: Fit imputers solely on the training partition (imp.fit(X_train)). Retain and transform test entries strictly using those pre-learned metrics to ensure no data leakage.

Next in this section: Encoding Categorical Data — turning Country into something a model can consume, which is the other half of why X was still object

See also: Loading and Preparing Data for finding the gaps in the first place · The Toolkit and the Pipeline for the fit/transform rule

---

Run It Yourself

tip

Lab Exercise: Missing Value Imputation

Trace how MCAR vs. MNAR missingness alters baseline statistical distributions, calculate listwise deletion row volumes across scaling dimensions, and evaluate MAE performances of Simple, KNN, and Iterative (MICE) imputers.

How to run the lab:

1. Click the "Open In Colab" badge above.
2. Run each cell sequentially (Shift + Enter).
3. Experiment with varying KNN neighbors or inducing custom skewed distributions.

Practice Questions

Test your understanding by working each problem out first, then click the card to reveal the detailed solution. Questions are tagged by type: [THEORY], [PROG], [OUT], or [ANALYZE].

Mechanisms

❓ M1. [THEORY] Define MCAR, MAR and MNAR, and give an example of each.

• MCAR (Missing Completely at Random): The probability of missingness is entirely independent of any observed or unobserved variables.
  • Example: A blood sample tube breaks in transit at random; its results are missing.
• MAR (Missing at Random): The probability of missingness is systematically related to other observed variables, but not to the missing value itself.
  • Example: Male patients are less likely to disclose their weight, but within the male subgroup, missingness does not depend on how heavy they actually are.
• MNAR (Missing Not at Random): The probability of missingness directly depends on the missing value itself (unobserved data).
  • Example: Individuals with high incomes systematically refuse to report their income on a survey.

❓ M2. [THEORY] For which mechanisms can imputation recover the information, and for which can it not? Explain why.

• Can recover (MCAR and MAR): Under MCAR/MAR, missing values can be estimated using correlation structures or subgroup patterns from other observed variables. Under MAR, modeling the conditional distribution using observed predictors allows us to reconstruct values unbiasedly.
• Cannot recover (MNAR): Under MNAR, the missingness has an unobserved bias linked to the missing values themselves. Since data is missing because of what the value would have been, the remaining observed data lacks the critical tail of the distribution, making the true distribution unrecoverable without external modeling.

❓ M3. [ANALYZE] A survey asks for income and 30% decline. Which mechanism is most likely, and what does that imply about imputing the mean?

• Most likely mechanism: MNAR (Missing Not at Random), as high earners and very low earners systematically refuse to report income.
• Implication for mean imputation: Imputing the global mean of the observed 70% will severely bias the dataset. It crushes variance, under-represents income inequality, and distorts downstream model coefficients because the non-respondents' actual mean differs from the observed mean.

❓ M4. [ANALYZE] Give a practical test for suspecting MNAR, and explain why the question cannot be answered from the data alone.

• Practical test: Split your data into "missing" and "observed" subsets for the target variable and compare distributions of other related, fully-observed variables.
• Why it can't be answered from data alone: You can never mathematically prove MNAR solely from the dataset because the missing values themselves are unobserved. Proving MNAR requires external domain knowledge, validation audits, or follow-up physical surveys of a random subset of non-respondents to see if their actual values differ systematically.

❓ M5. [ANALYZE] Under MNAR, listwise deletion is sometimes described as "at least honest". Argue against that.

While listwise deletion avoids "inventing" false data, it is not "honest" because it creates a heavily biased sample, completely excluding specific subgroups (e.g., the wealthy or the critically ill). This causes severe selection bias and invalidates the generalizability of any downstream estimator, hiding structural bias under the guise of a complete dataset.

The cost of imputation

❓ C1. [OUT] Complete truth had mean 3.851, std 1.936. After MCAR removal plus mean imputation: mean 3.815, std 1.573. Explain why the mean survived and the std did not.

• Why mean survived: Because the missingness was completely at random (MCAR), the observed subset is a representative sample of the true population. The average of the observed values is an unbiased estimator of the true mean.
• Why std did not: Mean imputation fills missing coordinates with the exact center point (3.815). This artificially spikes the peak of the distribution at the mean while adding zero variance for the imputed subset, causing the standard deviation to shrink drastically (from 1.936 to 1.573).

❓ C2. [THEORY] Explain, in terms of what imputed rows contribute, why mean imputation must shrink variance.

Variance is defined as the average squared deviation from the mean: $\sigma^2 = rac{1}{N}\sum(x_i - ar{x})^2$. Each imputed value $x_{imputed}$ is set exactly to the mean $ar{x}$. Therefore, its contribution to the sum of squared deviations $(x_{imputed} - ar{x})^2$ is exactly $0$. While the sum of squared deviations remains unchanged, the denominator $N$ increases, mathematically forcing the variance to shrink.

❓ C3. [THEORY] Name three consequences of variance shrinkage for downstream analysis.

1. Hypothesis Testing Distortion: Artificially narrow standard errors lead to inflated t-statistics and false-positive p-values.
2. Correlation Underestimation: Shrinking variance attenuates the covariance between variables, suppressing correlation coefficients ($r$) towards zero.
3. Confidence Interval Narrowing: Artificially tight intervals overstate the precision of our estimates, misleading stakeholders.

❓ C4. [ANALYZE] "The mean is unchanged, so mean imputation is safe." Rebut this using the measured figures.

While the mean remains stable (~3.8), the standard deviation drops by 18.7% (from 1.936 to 1.573). This distortion destroys the shape of the data distribution, making it highly unsafe for any downstream model that relies on feature variance (like PCA, linear regression, or tree-based splits).

❓ C5. [THEORY] What is mean imputation failing to represent that multiple imputation captures?

Mean imputation fails to represent uncertainty. It treats an estimated guess as a known, concrete fact. Multiple Imputation (MI) captures this uncertainty by generating multiple different imputed datasets with variation, reflecting the true predictive distribution of the missing values.

Deletion

❓ D1. [THEORY] Give the formula for the fraction of rows surviving dropna() with k columns each missing a fraction p.

Assuming the missingness is independent across columns, the probability of a row having no missing values in any of the $k$ columns is: $ext{Surviving Fraction} = (1 - p)^k$

❓ D2. [OUT] With 8 columns at 10% missing each, what fraction of rows is kept? Show the calculation.

• Formula: $(1 - 0.10)^8 = 0.90^8$
• Calculation: $0.90^8 pprox 0.430467$
• Result: Only 43.05% of the rows are kept (56.95% of the dataset is discarded).

❓ D3. [ANALYZE] A dataset has 40 columns, each 2% missing. Would you use dropna()? Justify with the arithmetic.

• Arithmetic: $(1 - 0.02)^{40} = 0.98^{40} pprox 0.4457$.
• Decision: No. Even with a tiny 2% missingness rate per column, you would discard over 55% of your rows (1 - 0.446 = 0.554). This massive loss of sample size and statistical power means dropna() is highly unsuitable here.

❓ D4. [THEORY] Name two situations where dropping the whole column is better than either imputing or dropping rows.

1. Extremely High Missingness: If a column is missing more than 60–70% of its values and lacks high-quality predictive proxies.
2. Non-Predictive / Unusable Features: When the column has high missingness and the feature itself is a duplicate or is theoretically irrelevant to the target.

Strategies

❓ S1. [THEORY] List the four SimpleImputer strategies and the data type each suits.

1. mean: Suitable for symmetric, normally distributed numeric data.
2. median: Suitable for skewed or outlier-heavy numeric data.
3. most_frequent: Suitable for categorical/ordinal features.
4. constant: Suitable for creating a placeholder category (like "missing") or a safe default value.

❓ S2. [OUT] Mean imputation on a 0/1 column produced 0.714286. Explain why this is invalid and what should have been used.

• Why invalid: A value of 0.714286 is mathematically impossible for a binary categorical/boolean feature (e.g. a patient cannot be "71.4% employed").
• What should have been used: strategy="most_frequent" (mode) or strategy="constant" to preserve the discrete, binary nature of the column.

❓ S3. [ANALYZE] Why should median be preferred to mean on a column whose maximum is 442× its median?

A maximum value that is 442 times the median indicates an extremely right-skewed distribution with heavy right-tail outliers. The mean is highly sensitive to outliers and will be pulled far to the right, whereas the median is robust to extreme values and accurately represents the central tendency of the bulk of the data.

❓ S4. [THEORY] A column stores 0 and 1. What single question determines whether mean is a legitimate strategy?

"Is this column a continuous probability/proportion, or is it a categorical/binary label?" If it represents a continuous proportion (e.g., click-through probability), the mean is a valid coordinate. If it represents a hard label (e.g., survived or died), the mean produces non-existent, invalid coordinates.

Smarter imputers

❓ K1. [OUT] Interpret this result: why does the ranking reverse?

   predictable column  (r = 0.83)
     SimpleImputer(mean)        1.4752
     KNNImputer(5)              0.7255
   unpredictable column  (r = 0.03)
     SimpleImputer(mean)        0.9840
     KNNImputer(5)              1.0566

• Predictable column: Strong correlations ($r = 0.83$) allow KNN to use neighboring samples to make accurate, low-error reconstructions, outperforming the naive mean.
• Unpredictable column: Because there is zero signal/correlation ($r = 0.03$), KNN is averaging random noise from arbitrary neighbors. Since the mean is the mathematically optimal guess for minimizing squared errors on a noisy distribution, KNN's arbitrary neighbor averages result in a higher error than the global mean, reversing the ranking.

❓ K2. [THEORY] Explain how KNNImputer fills a value, and why weak correlations make it perform worse than the mean.

• How it works: For a row with a missing value, it finds the $k$ nearest rows (neighbors) based on Euclidean distance across the other non-missing columns, and averages their values.
• Why weak correlations hurt: If there are no meaningful correlations, Euclidean distance measures meaningless noise. The neighbors found are mathematically random. Averaging a small number of random neighbors ($k=5$) introduces high variance and arbitrary values, yielding worse predictions than the global mean (which uses all $N$ data points to minimize overall MSE).

❓ K3. [OUT] IterativeImputer scored 0.9840 on the unpredictable column — identical to the mean. Explain what happened inside it.

IterativeImputer models each feature with missing values as a function of all other features (running sequential regression models). When it evaluated the unpredictable column, it found no predictive relationships ($R^2 pprox 0$). Consequently, the regression model's coefficients collapsed to zero, and the model defaulted back to predicting the global mean, showing its robustness to noise.

❓ K4. [ANALYZE] Describe the procedure for measuring which imputer is best on your own data, without needing any external ground truth.

1. Take a complete subset of your data (rows with no missing values).
2. Artificially mask a random 10-20% of the target column's values as NaN (inducing synthetic missingness).
3. Apply each imputer candidate to fill in these synthetic NaNs.
4. Compute the Root Mean Squared Error (RMSE) or Mean Absolute Error (MAE) between the imputed values and the original hidden ground truth values.
5. Select the imputer with the lowest reconstruction error.

❓ K5. [ANALYZE] Why did IterativeImputer do much worse than the mean on a column with skew 44.7?

An extreme skew of 44.7 indicates highly non-normal data with a long tail. By default, IterativeImputer uses linear regression estimators (like BayesianRidge), which assume normally distributed residuals. In the presence of extreme skew, the linear model gets severely distorted by heavy-tailed values, producing massive, unrealistic out-of-bounds predictions.

❓ K6. [THEORY] What single check should you run before choosing KNNImputer over median?

A correlation/feature relationship check. Calculate the correlation matrix or run a quick mutual information score. If there is no linear or non-linear correlation structure in your features, KNNImputer will perform worse than a simple median while taking significantly more computational time.

fit and transform

❓ F1. [THEORY] What does fit compute for a SimpleImputer, and where is it stored?

• What it computes: The designated statistic (e.g., mean, median, or mode) for each column in the training dataset.
• Where it is stored: In the imputer's learned parameters as public-facing attributes ending with an underscore, specifically statistics_.

❓ F2. [OUT] A test row's NaNs were filled with 32.5 and 72500, which are the training means. Explain why using the test set's own means would be wrong.

Using the test set's statistics introduces Data Leakage. In a real deployment, you may receive a single test row, which has no "test set mean". A model must treat test data as unseen, applying the exact same transformation parameters learned from the training distribution to ensure consistency and prevent information leakage.

❓ F3. [THEORY] Which method do you call on training data, and which on test data?

• Training data: Call .fit_transform() (or .fit() followed by .transform()).
• Test data: Call .transform() strictly (never call .fit() or .fit_transform() on test or validation data).

❓ F4. [OUT] An input of shape (3, 2) produced output of shape (3, 1). What happened, why, and what would you inspect to detect it?

• What happened: One of the two columns was completely dropped.
• Why: That column was entirely NaN in the training data, so SimpleImputer could not calculate a mean/median and dropped it by default.
• What to inspect: Inspect the .statistics_ or feature_names_in_ attributes of the fitted imputer, and check df.isna().sum() on your input training data.

❓ F5. [ANALYZE] Describe a concrete scenario in which the dropped-column behaviour causes a failure that surfaces far from its cause.

During training, column B is entirely NaN and gets silently dropped, so the imputer outputs a 4-column matrix instead of 5. The downstream classifier fits on this 4-column array. During production, column B is populated. When transform() is called, it outputs all 5 columns. The classifier receives a 5-column matrix and throws a cryptic ValueError: query data dimension mismatch or crashes deep inside an optimization layer because of a training-time column mismatch.

Applying it

❓ P1. [PROG] For a DataFrame, print each column's missing count and percentage, sorted worst first.

import pandas as pd
missing_count = df.isna().sum()
missing_pct = 100 * df.isna().mean()
audit_df = pd.DataFrame({"count": missing_count, "percentage": missing_pct})
print(audit_df.sort_values(by="count", ascending=False))

❓ P2. [PROG] Write a function that hides 20% of a chosen complete column at random, imputes with mean, median and KNNImputer, and prints the RMSE of each against the hidden truth.

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.metrics import root_mean_squared_error

def compare_imputers(df, column):
    truth = df[column].copy()
    mask = np.random.rand(len(df)) < 0.2
    
    # Create masked df
    df_masked = df.copy()
    df_masked.loc[mask, column] = np.nan
    
    # Prepare helper for metrics
    def get_rmse(imputed_series):
        return root_mean_squared_error(truth[mask], imputed_series[mask])
        
    # 1. Mean
    mean_imp = SimpleImputer(strategy="mean")
    df_mean = df_masked.copy()
    df_mean[[column]] = mean_imp.fit_transform(df_masked[[column]])
    
    # 2. Median
    med_imp = SimpleImputer(strategy="median")
    df_med = df_masked.copy()
    df_med[[column]] = med_imp.fit_transform(df_masked[[column]])
    
    # 3. KNN
    numeric_cols = df_masked.select_dtypes(include=[np.number]).columns
    knn_imp = KNNImputer(n_neighbors=5)
    df_knn = pd.DataFrame(knn_imp.fit_transform(df_masked[numeric_cols]), columns=numeric_cols)
    
    print(f"Mean RMSE:   {get_rmse(df_mean[column]):.4f}")
    print(f"Median RMSE: {get_rmse(df_med[column]):.4f}")
    print(f"KNN RMSE:    {get_rmse(df_knn[column]):.4f}")

❓ P3. [PROG] Impute with add_indicator=True and print the resulting column count before and after.

from sklearn.impute import SimpleImputer
import pandas as pd

print("Before:", df.shape[1])
imp = SimpleImputer(strategy="mean", add_indicator=True)
imputed_array = imp.fit_transform(df)
print("After:", imputed_array.shape[1])

❓ P4. [PROG] Compute, for a given DataFrame, the fraction of rows that dropna() would keep, and compare it against (1−p)ᵏ using the mean missing rate as p.

import numpy as np
import pandas as pd

k = df.shape[1]
actual_keep = df.dropna().shape[0] / len(df)

# Calculate average missingness rate across all cells
p = df.isna().mean().mean()
expected_keep = (1 - p) ** k

print(f"Actual Keep Fraction:   {actual_keep:.4f}")
print(f"Expected Keep ((1-p)^k): {expected_keep:.4f}")

❓ P5. [PROG] Build a Pipeline containing a SimpleImputer and a StandardScaler, fit it on train, and explain in a comment why this arrangement prevents leakage.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler

# Bundle both preprocessing steps in a Pipeline.
# When pipeline.fit(X_train) is executed, the pipeline fits the imputer first,
# transforms the training data, then fits the scaler on the imputed data.
# During transform() (on test data), it applies the exact same learned training
# statistics (training means/medians and scale factors) without referencing test set data,
# completely eliminating risk of test-set leakage.
pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

pipeline.fit(X_train)

❓ P6. [ANALYZE] You have a medical dataset where blood_pressure is 35% missing, and you learn the reading was skipped when patients were too unwell. State the mechanism, say whether you would impute, and describe what you would do instead.

• Mechanism: MNAR (Missing Not at Random), because missingness directly relates to the unobserved patient's state (unwell patients skipped reading).
• Impute or not: Do not apply naive imputation alone (like mean/median), as this masks the critical condition and severely underestimates risk.
• What to do instead: Use add_indicator=True to capture the missingness pattern as a binary indicator feature (e.g. blood_pressure_missing). This feature acts as a powerful proxy for "critically ill" that the downstream estimator can learn to weigh heavily.

Quick self-check

🧠 1. What do MCAR, MAR and MNAR stand for?

• MCAR: Missing Completely at Random.
• MAR: Missing at Random.
• MNAR: Missing Not at Random.

🧠 2. Which mechanism makes imputation impossible, and why?

MNAR, because the missingness is systematically linked to the unobserved values themselves, making it impossible to reconstruct the true distribution solely from observed data.

🧠 3. Mean imputation preserves the mean. What does it damage?

It severely crushes the variance and standard deviation of the feature, attenuates correlation coefficients, and distorts downstream hypothesis tests.

🧠 4. By how much did the standard deviation fall in the measured example?

It fell from 1.936 to 1.573 (an 18.7% reduction).

🧠 5. With 8 columns at 5% missing each, what fraction of rows survives dropna()?

$(1 - 0.05)^8 = 0.95^8 pprox 0.6634$ (only 66.3% of the rows survive).

🧠 6. When should you use median instead of mean?

When the numeric feature is highly skewed or contains extreme outliers.

🧠 7. What is wrong with 0.714286 as an imputed value for employed?

employed is a binary categorical column ($0$ or $1$). A continuous fraction like 0.714286 is invalid and conceptually meaningless.

🧠 8. What decides whether KNNImputer beats the mean?

The presence of predictive correlations between the features. If correlations are strong, KNN performs well; if they are near-zero, KNN performs worse than the mean.

🧠 9. What does IterativeImputer do when there is no signal to find?

Its underlying regression models learn coefficients close to zero and default back to predicting the global mean, making it robust to noise.

🧠 10. What does add_indicator=True preserve, and when does it matter most?

It preserves the pattern of missingness by creating a companion binary column. This is critical for MNAR scenarios where missingness carries predictive value.

🧠 11. Which of fit_transform and transform belongs on the test set?

Only transform belongs on the test set.

🧠 12. What happens to a column that is entirely NaN when fit is called?

It cannot be fit because there are no non-null values to calculate statistics from; the column is silently dropped during transformation.

\n