Loading and Preparing Data

Topic — Loading a CSV looks like the one step that can't go wrong. It is in fact where the most damaging errors enter, because they enter silently — no exception, no warning, just columns that are quietly the wrong type and missing values that were never counted as missing.

This page is a defensive routine: what pd.read_csv decides on your behalf, how to find out what it decided, and how to get X and y out the other side in a state you can trust.

---

The imports

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

Alias	Library	Role here
np	NumPy	Arrays — what the data becomes underneath
plt	Matplotlib's Pyplot module	Plotting
pd	Pandas	Loading and holding the data as a table

As close to universal as Python conventions get. Deviating makes your code harder for anyone else — including future you — to skim.

---

read_csv is not a neutral loader

Columnar (Parquet) vs. Row-Based (CSV/JSON) Loader Design

In modern big data engineering, choosing the appropriate file serialization layout is critical.

• Row-Based formats (CSV/JSON): Store entries sequentially on disk row by row (A1, B1, C1, A2, B2, C2...). This is optimal for write-intensive operations or simple text editing. However, querying a single feature requires loading the entire table into memory.
• Columnar formats (Parquet): Partition and store entries column by column (A1, A2... B1, B2...). This allows analytical engines to scan and load only the requested features, maximizing disk-to-memory throughput. Parquet features Snappy compression, built-in strongly-typed metadata schemas, and statistics per column, enabling ultra-fast audits before any data-parsing loop begins.

A CSV file is text. Every value in it is a string until something decides otherwise, and that something is pd.read_csv, making a series of inferences per column that it does not report.

Here is a deliberately realistic file — the kind you actually receive:

messy.csv

id,Country,Age,Salary,Region,Purchased
1,France,44,72000,EU,No
2,Spain,27,48000,EU,Yes
3,Germany,30,54000,EU,No
4,Spain,38,"61,000",EU,No
5,Germany,40,,EU,Yes
6,France,35,58000,EU,Yes
7,Spain,N/A,52000,EU,No
8,France,48,79000,EU,Yes
9,Germany,-1,83000,EU,No
10,France,37,67000,EU,Yes
11,France,?,999999,EU,No
10,France,37,67000,EU,Yes

Load it the obvious way and inspect what you got:

naive = pd.read_csv("messy.csv")
print(naive.dtypes)
print(naive.isna().sum().sum())

Output

id            int64
Country      object
Age          object
Salary       object
Region       object
Purchased    object
dtype: object

2

Two things just happened, both bad, neither announced.

Age and Salary are object, not numbers. They contain digits in every row you care about, but one ? in Age and one "61,000" in Salary were enough for pandas to give up on a numeric type and store the whole column as Python strings. Any arithmetic downstream either raises something confusing or — worse — succeeds and concatenates strings.

It reported 2 missing values. There are more than that, as the next section shows.

danger

Print .dtypes immediately after every load

This is a one-line habit that catches a whole class of silent failure:

df = pd.read_csv("data.csv")
print(df.dtypes)          # ← do this every single time

A column you believe is numeric showing as object means something non-numeric is hiding in it. You want to discover that now, not three steps later when a model reports a nonsensical score.

---

The missingness pandas cannot see

Pandas recognises a fixed list of strings as missing. It is worth knowing exactly what's on it:

print(len(pd._libs.parsers.STR_NA_VALUES))
print(sorted(pd._libs.parsers.STR_NA_VALUES))

Output

19

'', '#N/A', '#N/A N/A', '#NA', '-1.#IND', '-1.#QNAN', '-NaN', '-nan',
'1.#IND', '1.#QNAN', '<NA>', 'N/A', 'NA', 'NULL', 'NaN', 'None', 'n/a',
'nan', 'null'

That's why the N/A in row 7 became NaN automatically. Now note what is not on the list:

Encoding of "missing"	Caught by default?
Empty cell, NA, N/A, null, None, NaN	✅ Yes
?	❌ No
-, --, .	❌ No
unknown, missing, not recorded, TBD	❌ No
-1 as "no value"	❌ No
999, 9999, 999999 as "no value"	❌ No
0 meaning "not measured"	❌ No

The last three are sentinel values, and they are the dangerous category. A -1 in an age column is a perfectly valid number as far as any library is concerned. It will be averaged, scaled, and fed to a model as though someone were minus one year old.

warning

No library can find sentinels for you

Whether -1 in Age means "missing" or is a genuine data error, and whether 999999 in Salary is a real high earner or a placeholder, is not inferable from the data. It requires knowing how the data was collected.

This is the part of preprocessing that cannot be automated, and it's the reason a data dictionary or a conversation with whoever produced the file is worth more than any technique on this page.

Once you know, you declare them:

df = pd.read_csv("messy.csv",
                 na_values=["?", "-1", "999999"],   # domain knowledge, not automatic
                 thousands=",")
print(df.dtypes)
print(df.isna().sum().sum())

Output

id             int64
Country       object
Age          float64
Salary       float64
Region        object
Purchased     object
dtype: object

5

Age and Salary are numeric now, and the missing count went from 2 to 5. Same file, same rows — the three extra were always missing, and were previously being treated as data.

The read_csv parameters worth knowing

Parameter	What it does	When you need it
na_values=	Extra strings/values to treat as missing	Sentinels, ?, unknown
thousands=","	Strip digit-group separators	"61,000" staying numeric
dtype={...}	Force a column's type instead of inferring	IDs with leading zeros, codes
usecols=[...]	Read only these columns	Wide files; less memory
parse_dates=[...]	Parse to real datetimes	Any date column
index_col=	Use a column as the index	Natural keys
encoding=	File's text encoding	UnicodeDecodeError on load
skiprows=, nrows=	Skip preamble; read a sample	Reports with header junk
chunksize=	Iterate in chunks	Files larger than memory

🧠 Interactive Checkpoint: Messy CSV Loader

Scenario: You are handed a text file separated by semicolons (;), where missing records are written as the word unknown, and the column account_id contains leading zeros (e.g., 00938). Which set of parameters will load this cleanly?

note

dtype={"zip": str} — the leading-zero trap

A postcode column containing 01234 will be inferred as int64 and become 1234. The leading zero is gone and cannot be recovered. Anything that is an identifier rather than a quantity — postcodes, phone numbers, account numbers, product codes — should be read as str, even though it looks numeric.

Ask of each column: would arithmetic on this mean anything? If not, it isn't a number.

---

A pre-flight audit

Rather than remembering to check five things, check them all at once. This function is small, and running it after every load costs nothing:

def audit(df, name):
    print(f"AUDIT — {name}: {df.shape[0]} rows x {df.shape[1]} columns")

    dups = int(df.duplicated().sum())
    print(f"   duplicate rows:   {dups}")

    const = [c for c in df.columns if df[c].nunique(dropna=False) <= 1]
    print(f"   constant columns: {const if const else 'none'}")

    for c in df.columns:
        note = ""
        if df[c].dtype == object:
            coerced = pd.to_numeric(df[c], errors="coerce")
            if coerced.notna().sum() > 0.5 * df[c].notna().sum():
                note = "LOOKS NUMERIC but is object"
        elif int((df[c] < 0).sum()):
            note = f"{int((df[c] < 0).sum())} negative value(s)"
        print(f"   {c:<12}{str(df[c].dtype):>9}"
              f"{int(df[c].isna().sum()):>9}{df[c].nunique():>8}   {note}")

Run on the naive load:

Output

AUDIT — naive read: 12 rows x 6 columns
   duplicate rows:   1   <- investigate
   constant columns: ['Region']   <- no predictive value

   column          dtype  missing  unique   note
   id              int64        0      11
   Country        object        0       3
   Age            object        1      10   LOOKS NUMERIC but is object
   Salary         object        1      10   LOOKS NUMERIC but is object
   Region         object        0       1
   Purchased      object        0       2

Four findings from one call:

Finding	Why it matters
1 duplicate row	Row 10 appears twice. Duplicates inflate whatever they contain, and if they straddle a train/test split the same record is in both
Region is constant	Every value is EU. It cannot possibly help a model — it carries zero information
Age looks numeric but isn't	The pd.to_numeric(..., errors="coerce") trick: if most values convert cleanly, the column was meant to be numeric
id has 11 unique values in 12 rows	Consistent with the duplicate; also a column to drop, since a row identifier is never a feature

The id column deserves a note of its own. An identifier is often correlated with the target purely because of how the data was ordered or collected, so a model will happily use it and score well — on data that includes those exact ids, and never again. Drop identifiers before modelling.

What the audit does not tell you

It flags Region as constant, but it cannot tell you that a column is constant only in this sample. It flags negatives, but not that 999999 is a placeholder. It counts duplicates, but not whether they're a genuine repeat measurement or a join gone wrong. The audit narrows what you have to think about; it doesn't replace the thinking.

---

Separating features from target

With a trustworthy table, split it into X (the features) and y (the target) — the vocabulary from Types of Learning.

clean = df.drop(columns=["id", "Region"]).drop_duplicates()

X_df = clean.iloc[:, :-1]     # all rows, all columns except the last
y_df = clean.iloc[:, -1]      # all rows, only the last column

Output

X (11, 3) (2-D)    y (11,) (1-D)
X columns: ['Country', 'Age', 'Salary']

Note the order: audit and clean first, then split. Dropping the identifier, the constant column and the duplicate row before splitting means those problems never reach a model.

iloc[row_selector, column_selector] selects by position. : means everything along that axis; -1 is the last element as in any Python sequence, so :-1 is "everything up to but excluding the last".

Expression	Selects
iloc[:, :-1]	all rows, every column but the last — the features
iloc[:, -1]	all rows, the last column only — the target
iloc[2, 1]	one cell, by position

---

loc and iloc — two differences, not one

Everyone learns that iloc is positional and loc is label-based. The second difference is the one that causes bugs:

s = pd.Series([10, 20, 30, 40, 50])
print(list(s.iloc[1:3]))
print(list(s.loc[1:3]))

Output

[20, 30]
[20, 30, 40]

iloc excludes its stop; loc includes it. Same-looking slice, different number of elements.

	iloc	loc
Indexes by	Integer position	Label
Stop value	Excluded (like a Python list)	Included
[1:3] gives	2 items	3 items
Survives sorting/filtering	No — position changes	Yes — label follows the row

loc includes the stop because a label isn't necessarily part of an ordered sequence — pandas can't know what comes "before" the label "Salary", so excluding it would be meaningless. The inconsistency is a consequence of the two indexers answering genuinely different questions.

Both agree on df.iloc[2, 1] and df.loc[2, "Age"] in a freshly loaded DataFrame only because the index is still the default 0, 1, 2, …. Sort or filter the frame and they diverge immediately: loc[0] still finds the row labelled 0, wherever it now sits, while iloc[0] finds whatever is now first.

Rule of thumb: iloc when you mean the Nth row; loc when you mean the row named N. After any filtering, prefer loc, or call .reset_index(drop=True) to make position and label agree again.

---

NumPy slicing

Once you're in arrays rather than DataFrames, slicing follows NumPy's rules — which come up constantly in the encoding and scaling steps ahead, where you operate on specific column ranges.

sample = np.array([[10, 20, 30, 40],
                    [50, 60, 70, 80],
                    [90, 100, 110, 120]])
print(sample[:, 1:3])

Output

[[ 20  30]
 [ 60  70]
 [100 110]]

Read sample[:, 1:3] in two halves: : selects all rows; 1:3 selects columns 1 and 2 — the stop is excluded, exactly as in a Python list. That expression, all rows and the numeric feature columns only, is precisely the slice the next page hands to SimpleImputer so it touches Age and Salary without touching Country.

One asymmetry worth knowing: an out-of-range slice is silently clipped, while an out-of-range index raises.

sample[:, 1:99]     # fine — returns columns 1 to 3
sample[:, 99]       # IndexError

---

Should you convert to NumPy at all?

The conventional line is X = df.iloc[:, :-1].values. It works, and it costs you things worth keeping.

.values versus .to_numpy()

Both convert. .to_numpy() is the current recommendation — .values predates it, has fuzzier behaviour around extension dtypes, and is effectively legacy. Prefer .to_numpy() in new code.

The shape rule is the same either way, and it's where X being 2-D and y being 1-D originates:

Converting	Gives	Shape
A DataFrame (many columns)	2-D array	(rows, columns)
A Series (one column)	1-D array	(rows,)

So X, sliced from all-but-one column, is a DataFrame → 2-D. And y, sliced from one column, is a Series → 1-D. The shape convention every model expects is a consequence of this, decided before you ever call .fit.

What conversion costs, measured

Mixed types collapse to object.

print(X_df.to_numpy().dtype)      # object

A single array can hold only one type. X mixes 'France' with 44.0, so NumPy falls back to storing generic Python object pointers. That is not just inelegant:

n = 300_000
num = np.arange(n, dtype=float)
obj = num.astype(object)
# time num.sum() vs obj.sum()

Output

sum of 300,000 floats — float64 array:    0.10 ms
sum of 300,000 floats — object  array:    2.91 ms
object is 28x slower

28× slower on identical numbers. A float64 array is one contiguous block that compiled code walks directly; an object array is a block of pointers, each chased to a separate Python object.

Column names are thrown away.

kept = StandardScaler().fit(num_df)             # a DataFrame
lost = StandardScaler().fit(num_df.to_numpy())  # an array

Output

fitted on DataFrame  -> feature_names_in_ = ['Age', 'Salary']
fitted on .to_numpy() -> has feature_names_in_? False

Fitted on a DataFrame, scikit-learn records feature_names_in_. It then validates that anything you later pass has the same columns in the same order — catching a whole category of bug where columns get reordered between training and prediction. Convert to NumPy and you lose that check, plus every error message that would have named the offending column.

Where category dtype wins

For a text column with few distinct values, staying in pandas and using category is dramatically cheaper than either object or one array:

Output

200,000 rows, 3 distinct values
object   dtype: 12,600,353 bytes
category dtype:    200,429 bytes
reduction:           98.4%

98.4% less memory. category stores each distinct string once and keeps a compact array of integer codes — which is, not coincidentally, exactly what label encoding does by hand on the next page but one.

The recommendation

Keep DataFrames as long as you can. Modern scikit-learn accepts them throughout, preserves feature names, and gives better errors. Convert with .to_numpy() only where something genuinely requires an array, and convert after encoding, when the frame is all-numeric and the result won't be object.

The .values idiom you'll see in most tutorials isn't wrong — it predates good DataFrame support and has simply been overtaken.

---

Common Mistakes

Here are the most common conceptual pitfalls when loading and preparing datasets, compared with the correct engineering principles.

🚨 Mistake 1: Silent Coercion

• The Pitfall: Blindly assuming a column parsed as floating-point or integer because it looks numeric.
• ❌ Wrong Thinking: Assuming Age is float64 because the rows look numeric.
• ✅ The Right Principle: Always execute df.dtypes immediately after load. A single corrupt character (like a "?" or comma ",") converts the entire column into object dtype, which will fail mathematical operations.

🚨 Mistake 2: ID Loss

• The Pitfall: Allowing Pandas to parse postcodes, IDs, or account numbers as numeric metrics.
• ❌ Wrong Thinking: ID = 01234 parsed as integer 1234.
• ✅ The Right Principle: Force these identifiers to string format explicitly using dtype={"zip": str} in your load call. Once leading zeros are discarded by numerical inferencing, they are lost.

🚨 Mistake 3: Slicing Exclusivity

• The Pitfall: Expecting loc[1:3] to behave exactly like iloc[1:3] and exclude the stop index.
• ❌ Wrong Thinking: Expecting loc[1:3] to return 2 items.
• ✅ The Right Principle: loc is label-inclusive and returns 3 items. iloc is integer-exclusive and returns 2 items.

🚨 Mistake 4: Premature Array Conversion

• The Pitfall: Converting to NumPy too early using .values or .to_numpy().
• ❌ Wrong Thinking: .values on the raw DataFrame as soon as the file is loaded.
• ✅ The Right Principle: Stay in Pandas DataFrames until you've split your data and completed categorical encoding. Mixed types collapse to object in NumPy, which is 28x slower than standard float64.

---

Summary

📂 Data Loader Checklist

• pd.read_csv: Delimiters (sep), separators (thousands), types (dtype), chunks (chunksize).
• Parquet columns: Columnar layout, type guarantees, built-in metadata schemas, ultra-fast pre-flight audits.

🔍 Audit & Index Verification

• Audit blocks: Scan for duplicate records, inspect zero-variance constant columns.
• Index keys: loc[1:3] (inclusive stop) vs iloc[1:3] (exclusive stop).

info

📌 Key Takeaways: Loading & Preparing Data

• 🕵️ dtypes Fail Silently: A single invalid character (like a "?" or thousands separator ",") transforms columns into object strings. Always inspect df.dtypes immediately after every loading loop.
• ⚠️ Danger of Sentinels: Numerical markers (such as -1 or 999999) bypass standard missingness parser checks. Only domain knowledge and na_values parameter declarations can capture them defensively.
• 🧱 Postpone Array Conversion: Avoid calling .to_numpy() or .values right after load. Converting mixed string-metric features yields object matrix layouts, measured at 28× slower than float64 operations.
• 🔗 Keep DataFrame validation: Training on DataFrames preserves feature_names_in_ schemas on estimators, allowing scikit-learn to catch column reordering bugs automatically.

Next in this section: Missing Values — imputing the NaNs this page has now correctly counted

See also: Types of Learning for the X/y vocabulary · The Toolkit and the Pipeline for where loading sits in the seven-step pipeline

---

Run It Yourself

tip

Lab Exercise: Loading Data Defensively in Pandas

Verify the silent hazards of type inference, build custom data audit pipelines to capture duplicated rows and zero-variance constant columns, and trace how loc vs. iloc slice models on messy dataset inputs.

How to run the lab:

1. Click the "Open In Colab" badge above.
2. Run each cell sequentially (Shift + Enter).
3. Experiment with removing thousands separators and checking coerced object arrays.

Practice Questions

Test your understanding by working each problem out first, then click the card to reveal the detailed solution. Questions are tagged by type: [THEORY], [PROG], [OUT], or [ANALYZE].

What loading decides for you

❓ L1. [THEORY] Every value in a CSV file is text. Explain what pd.read_csv does about that, and what it tells you about the decisions it made.

• What pandas does: It runs statistical data-type inferencing across columns sequentially. It attempts to parse columns as integers (int64), floats (float64), or booleans, defaulting to raw string objects (object) if any non-numeric token is found.
• Where it tells you: It reports absolutely nothing during loading. You must explicitly execute df.dtypes immediately after every load to verify the decisions it made.

❓ L2. [ANALYZE] A column of 10,000 ages contains one ?. State the resulting dtype, and explain what happens when you later try to compute the mean.

• Resulting dtype: object.
• What happens to mean: Pandas cannot execute mathematical operations on string objects. Attempting to call .mean() on the column will raise a TypeError: must be real number, not str, halting your modeling pipeline.

❓ L3. [THEORY] Pandas recognises 19 strings as missing by default. Name five, and name four common representations of missingness that are not on the list.

• Five recognized defaults: "", "#N/A", "NaN", "null", "n/a".
• Four common unrecognized entries: "?", "-", "unknown", -1 or 999999.

❓ L4. [THEORY] What is a sentinel value? Give two examples, and explain why no library can detect them automatically.

• Sentinel Value: A valid numerical coordinate used as a coded marker indicating a missing record (e.g. -1 for age, or 999999 for salary).
• Why libraries can't detect them: They are mathematically valid numbers. Only domain knowledge and custom manual declarations via na_values can flag them as missingness rather than metrics.

❓ L5. [ANALYZE] A postcode column contains 01234. Explain what read_csv will do to it by default, why the damage is irreversible, and the test you should apply to decide a column's type.

• Naive parsing behavior: It strips the leading zero and casts the postcode as a float or integer (1234).
• Why irreversible: Once written to RAM as 1234, there is no automatic way to distinguish whether the original was 01234, 001234, or simply 1234.
• The test: If performing mathematical arithmetic (such as calculating the mean or standard deviation) on a column is meaningless, the column is a category and must be read as a string (dtype={col: str}).

Auditing

❓ A1. [THEORY] Name four things a pre-flight audit should check, and say what each would catch.

1. Dtypes audit: Catches numeric columns masked as string object due to corrupt characters.
2. Duplicate check: Catches duplicate rows that can skew splits and train/test boundaries.
3. Constant columns check: Catches zero-variance features (e.g., all rows are "North") that provide no predictive value.
4. Anomaly boundaries check: Catches out-of-bounds metrics (e.g. negative ages -1 or out-of-scale salaries 999999).

❓ A2. [THEORY] Explain the pd.to_numeric(col, errors="coerce") trick for detecting columns that should be numeric but aren't.

It attempts to cast strings into numerical floats. By setting errors="coerce", any non-numeric tokens (like ?) are forced into NaN. If auditing shows that more than half of the converted column contains valid numbers (not NaN), the column should be numeric and contains noise characters.

❓ A3. [ANALYZE] Why should an id column be dropped before modelling, even when it correlates strongly with the target?

IDs are unique identifiers (100% cardinality). An estimator will fit weights directly to the unique ID integer keys, achieving 100% training accuracy through overfitting, but will fail completely on any unseen test identifiers.

❓ A4. [ANALYZE] A constant column is described as having "no predictive value". Explain why, and say whether it is harmful or merely useless.

• Why: A column with zero variance contains identical values for all classes. It offers no discriminative power.
• Harmful or useless: Both. While useless mathematically, it is structurally harmful because it consumes memory, slows down training epochs, and can introduce dummy-variable collinearity issues.

❓ A5. [ANALYZE] Give two reasons duplicate rows matter, one of which concerns the train/test split.

1. Data Leakage: If duplicate rows exist, splitting the data randomly will land identical copies in both train and test sets, inflating test accuracy artificially.
2. Metric Distortion: Duplicate rows bias training optimization steps by over-weighting specific coordinates.

loc, iloc and slicing

❓ I1. [OUT] For s = pd.Series([10, 20, 30, 40, 50]), give the output of s.iloc[1:3] and s.loc[1:3], and explain why they differ.

• s.iloc[1:3]: [20, 30] (returns 2 items, stop index 3 is exclusive).
• s.loc[1:3]: [20, 30, 40] (returns 3 items, stop index key 3 is inclusive).

❓ I2. [THEORY] State the two differences between loc and iloc.

1. Stop exclusivity: iloc excludes the stop offset; loc includes the stop key.
2. Referencing system: iloc strictly references integer offset locations ($0$-based); loc strictly references named index label keys.

❓ I3. [ANALYZE] Explain why it is reasonable for loc to include its stop value while iloc excludes it.

• For iloc: Offset slicing relies on mathematical lengths ($0$-based offsets), where stop - start equals number of items.
• For loc: Slicing is categorical label-based (e.g. loc["Jan":"Mar"]). If the stop label was excluded, the user would need to know the next chronological label (which might not exist or be hard to specify), making categorical indexing highly unintuitive.

❓ I4. [THEORY] After df = df.sort_values("Age"), which of loc and iloc still refers to the same row as before? What would make them agree again?

• Which still refers: loc (it tracks the literal, unique row index label key, wherever it moves).
• How to agree: Reset the index row keys using df.reset_index(drop=True).

❓ I5. [OUT] For a 4-column array, what do sample[:, 1:99] and sample[:, 99] each do?

• sample[:, 1:99]: Standard slicing. Slices columns from index 1 to the end cleanly without raising errors (slices are tolerant of out-of-bounds markers).
• sample[:, 99]: Direct indexing. Attempts to retrieve column offset 99, throwing an out-of-bounds IndexError because the index doesn't exist.

Converting to NumPy

❓ N1. [THEORY] Explain why X comes out 2-D and y comes out 1-D, in terms of DataFrames and Series.

• X (DataFrame): Represents multiple feature coordinates (a 2-D tabular matrix, matching scikit-learn's (n_samples, n_features) expectation).
• y (Series): Represents a single target output sequence (a 1-D vector, matching (n_samples,)).

❓ N2. [THEORY] Why should .to_numpy() be preferred over .values in new code?

.values is a legacy attribute with inconsistent behaviors depending on the underlying data types. .to_numpy() is an explicit, modern method that allows you to specify data types and memory configurations safely.

❓ N3. [OUT] Summing 300,000 floats took 0.10 ms as float64 and 2.91 ms as object. Explain the ratio in terms of how each is stored in memory.

• float64: Stored as a single, contiguous block of raw numeric bytes in memory, running computations in compiled C layers.
• object: Stored as an array of pointers pointing to fragmented, individual Python float wrappers scattered in RAM, requiring slow, repeated Python runtime interpreter lookups (unboxing) for each addition.

❓ N4. [OUT] A StandardScaler fitted on a DataFrame has feature_names_in_; one fitted on an array does not. Name a concrete bug the first can catch and the second cannot.

Column Reordering: If deployment features are passed in a different column order (e.g. passed as [Salary, Age] instead of [Age, Salary]), the DataFrame-fitted scaler will immediately catch the mismatch via feature_names_in_ and raise an exception. The array-fitted scaler will process the raw matrix silently, standardizing salaries with age statistics and producing garbage predictions.

❓ N5. [OUT] A 200,000-row text column with 3 distinct values took 12,600,353 bytes as object and 200,429 as category. Explain the 98.4% reduction, and name the encoding technique this resembles.

• Explanation: Instead of replicating heavy string bytes 200,000 times, category stores unique strings once in a lookup key dictionary, and maps the 200,000 rows as tiny 1-byte integer keys.
• Encoding resembled: Label Encoding (Ordinal Encoding).

❓ N6. [ANALYZE] Most tutorials write X = df.iloc[:, :-1].values as the very next line after loading. Give three things this costs, and state when conversion should happen.

• Three costs:
  1. Silent mixed-type casting: Prompts numeric metrics to collapse into expensive object dtypes if any string feature is present.
  2. Loss of metadata: Strips column names, destroying scikit-learn's feature_names_in_ structural validation.
  3. Data leakage hazard: Preprocessing scaling is applied to the full array before train/test partition splits are established.
• When it should happen: At the very end of the pipeline, strictly after categorical encoding is complete and splits have been established.

Applying it

❓ P1. [PROG] Load messy.csv so that ?, -1 and 999999 are missing and "61,000" parses as a number. Print dtypes and the total missing count.

import pandas as pd
df = pd.read_csv("messy.csv", na_values=["?", "-1", "999999"], thousands=",")
print(df.dtypes)
print("Total missing:", df.isna().sum().sum())

❓ P2. [PROG] Write a function that returns the list of columns whose dtype is object but where more than half the non-null values convert cleanly with pd.to_numeric.

def find_coerced_numeric_cols(df):
    cols = []
    for c in df.select_dtypes(include='object').columns:
        non_null_count = df[c].notna().sum()
        if non_null_count == 0:
            continue
        coerced = pd.to_numeric(df[c], errors='coerce')
        if coerced.notna().sum() > 0.5 * non_null_count:
            cols.append(c)
    return cols

❓ P3. [PROG] Print every duplicated row in messy.csv, including the first occurrence.

import pandas as pd
df = pd.read_csv("messy.csv")
print(df[df.duplicated(keep=False)])

❓ P4. [PROG] Load messy.csv reading only Country, Age and Purchased, with Country as a category dtype. Print the memory used by each column.

import pandas as pd
df = pd.read_csv("messy.csv", usecols=["Country", "Age", "Purchased"], dtype={"Country": "category"})
print(df.memory_usage(deep=True))

❓ P5. [ANALYZE] You receive a 2 GB CSV that will not fit in memory. Name three read_csv parameters that help and explain what each does.

1. usecols: Discards irrelevant feature columns immediately, loading only required features.
2. nrows: Limits row scans (e.g., loads first 10,000 rows to establish schemas).
3. chunksize: Streams the 2 GB file in smaller iteration blocks (e.g., 10,000 rows at a time) to prevent memory crashes.

Quick self-check

🧠 1. What should you print immediately after every read_csv?

df.dtypes and df.isna().sum().

🧠 2. How many strings does pandas treat as missing by default? Is ? one of them?

Pandas treats 19 strings as missing by default. "?" is not on the list.

🧠 3. What is a sentinel value, and why can't a library find them for you?

A valid numeric code (such as -1 or 999999) representing missingness. Libraries can't find them because they are mathematically valid coordinates.

🧠 4. Why must a postcode column be read as str?

To prevent numeric inferencing from permanently stripping leading zeros (e.g. 01234 casting to integer 1234).

🧠 5. Name three things a pre-flight audit checks.

Duplicate rows, constant columns (variance ≤ 0), and numeric columns masked as objects.

🧠 6. Why drop an id column even when it predicts well?

Because unique ID integers act as highly overfit memorization indices that do not generalize to unseen test instances.

🧠 7. What are the two differences between loc and iloc?

• iloc excludes the stop offset; loc includes it.
• iloc references integer offsets; loc references index label keys.

🧠 8. How many elements does loc[1:3] return? iloc[1:3]?

loc[1:3] returns 3 elements. iloc[1:3] returns 2 elements.

🧠 9. Why does X end up 2-D and y 1-D?

Because X is a multi-column DataFrame (matrix), while y is a single-column Series (vector).

🧠 10. Why is an object array so much slower than a float64 one?

Because object is an array of memory pointers requiring slow, dynamic Python unboxing lookups, while float64 stores values in continuous, uniform memory blocks.

🧠 11. What does converting to NumPy cost you that staying in pandas doesn't?

Strips column name metadata, losing scikit-learn's feature_names_in_ column alignment validation checks.

🧠 12. What is the right moment to convert to an array?

At the very end of preprocessing, after splits are completed and all string columns have been encoded.

\n