Encoding Categorical Data

Topic - Two chapters ago, the feature matrix $X$ emerged with dtype('object') because text columns like Country sat beside numeric metrics. This chapter solves that. The easy part is converting text into numbers; the challenging part is doing so without asserting false mathematical relationships to the model.

Every encoding scheme involves a fundamental three-way trade-off between dimensionality, assumed ordering, and data leakage risk. No single method optimizes all three.

---

1. Why Encoding is Needed at All

Most machine learning algorithms — especially linear models, support vector machines, and neural networks — compute predictions using vector dot products ($w^T x$) and matrix inversions. Text tokens like 'France' or 'Tokyo' have no intrinsic mathematical meaning; algorithms cannot calculate dot products over strings.

Beyond mathematical necessity, numerical representations are orders of magnitude more computationally efficient. As measured in earlier chapters, an unencoded Python object array sums 28x slower and consumes 98.4% more memory than an optimized categorical or numeric array.

---

Interactive Live Encoder Sandbox

Toggle between encoding schemes to see how the categorical sequence ["Red", "Green", "Blue", "Green"] transforms in memory:

Category	Label Code
Red	0
Green	1
Blue	2
Green	1

Dimensional Footprint: 1 column, compact integers (0, 1, 2)

Mathematical Implication: Asserts Blue > Green > Red, and Blue is twice Green.

---

2. Label Encoding & What It Asserts

Label encoding assigns a distinct integer to each unique category ($0, 1, 2, \dots$). It inputs one column and outputs one column, making it the cheapest possible transformation.

from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
y = le.fit_transform(["No", "Yes", "No", "Yes"])  # Returns array([0, 1, 0, 1])

For a binary target vector y, label encoding is mathematically sound. However, applying label encoding to nominal features (such as Color: [Red, Green, Blue] -> [0, 1, 2]) forces three false mathematical claims onto the data:

1. Magnitude: Blue ($2$) is greater than Red ($0$).
2. Distance: Green ($1$) sits exactly halfway between Red and Blue.
3. Ratio: Blue ($2$) is mathematically twice Green ($1$).

For genuinely ordinal data (low < medium < high), these ordering claims reflect physical reality. For nominal categories (Paris, Tokyo, London), they are completely fabricated.

---

How Much Does It Cost? It Depends Entirely on the Model

To measure the empirical penalty of false ordering, consider an experiment across six cities where true target outcomes alternate along alphabetical order (the default numbering sequence used by encoders):

Target Response per City

0 = Chennai (+2.5)   1 = Delhi (-2.0)    2 = Jaipur (+2.4)
3 = Kolkata (-1.8)   4 = Mumbai (+2.2)   5 = Pune (-2.1)

Because positive and negative effects alternate at every consecutive integer, no single linear cut or shallow threshold can isolate positive outcomes:

Model Architecture	Accuracy with Label Encoding	Accuracy with One-Hot Encoding	Realized Performance Gain
Logistic Regression	0.6737	0.9837	+31.00 points
Decision Tree (depth = 1)	0.6637	0.9837	+32.00 points
Random Forest (unrestricted)	0.9837	0.9837	+0.00 points (Zero Penalty)
Majority Class Baseline	0.5063	-	-

---

Interactive Model Encoding Cost Visualizer

Select a model to observe how architectural capacity determines whether label encoding damages accuracy:

Logistic Regression (Linear Model)

Severe Accuracy Penalty

Label Encoded Accuracy

67.4%

One-Hot Accuracy

98.4%

Accuracy Penalty

-31.0%

Architectural Reason: A linear model fits a single weight coefficient w. It can only express monotonic relationships ("higher integer -> higher probability"). Against alternating positive/negative cities, a single straight line fails completely, performing barely above a random baseline.

tip

The Architectural Rule Stated Properly:

Label encoding nominal data is safe for deep, unrestricted tree ensembles (Random Forests, Gradient Boosting) and heavily damaging for linear models, SVMs, k-Nearest Neighbors, and neural networks. When in doubt, one-hot encoding is the safest default because it asserts zero false orderings for any model family.

LabelEncoder (Targets) vs. OrdinalEncoder (Features)

A common scikit-learn API mistake is running LabelEncoder in a loop across input feature columns:

• LabelEncoder: Documented strictly for 1-D targets (y). It expects a 1-D sequence and cannot participate in ColumnTransformer or standard feature pipelines.
• OrdinalEncoder: Designed for 2-D feature matrices (X). Integrates directly into scikit-learn Pipeline and ColumnTransformer workflows.

# For genuinely ordinal features, explicitly state the category hierarchy:
from sklearn.preprocessing import OrdinalEncoder

# BAD: Alphabetical default makes 'high'=0, 'low'=1, 'medium'=2 (meaningless)
naive_oe = OrdinalEncoder().fit(sizes)

# GOOD: Explicit hierarchical order preserved: low=0 < medium=1 < high=2
clean_oe = OrdinalEncoder(categories=[["low", "medium", "high"]]).fit(sizes)

---

3. One-Hot Encoding & The Dummy Variable Trap

One-hot encoding creates a binary column ($0$ or $1$) for each unique category. Every category becomes an orthogonal basis vector equidistant from every other category:

$\text{France} \to [1, 0, 0], \quad \text{Germany} \to [0, 1, 0], \quad \text{Spain} \to [0, 0, 1]$

Applying One-Hot Encoding with ColumnTransformer

Real datasets contain mixed numeric and categorical columns. Use ColumnTransformer to target categorical features while leaving numerical columns intact:

from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.preprocessing import OneHotEncoder

ct = ColumnTransformer(
    transformers=[
        ("cat", OneHotEncoder(handle_unknown="ignore"), make_column_selector(dtype_include=object))
    ],
    remainder="passthrough"  # Retain numerical features unchanged
)

The Dummy Variable Trap & drop="first"

One-hot encoding introduces exact mathematical redundancy across columns. For any given sample, the sum across all $k$ one-hot indicator columns is identically equal to 1:

$\sum_{j=1}^k x_j = 1 \implies x_k = 1 - \sum_{j \neq k} x_j$

Because any single column can be predicted perfectly from the remaining $k-1$ columns, the feature matrix $X$ lacks full column rank. In unregularized linear regression (Ordinary Least Squares), this creates perfect multicollinearity, rendering the normal equation matrix $(X^T X)$ singular and non-invertible.

# Resolve the trap by dropping the first category:
ohe = OneHotEncoder(drop="first", sparse_output=False)

The dropped category becomes the baseline reference category (represented as all-zeros). Every remaining regression coefficient represents an offset relative to this reference baseline.

Linear Models (OLS)

• Requirement: Mandatory to set drop="first".
• Reason: Eliminates the dummy variable trap, restoring full column rank to the matrix so $(X^T X)^{-1}$ can be inverted reliably.

Regularized Linear Models (Ridge / Lasso)

• Requirement: Optional (drop=None is standard).
• Reason: The $L_2$ or $L_1$ penalty term ($\lambda I$) prevents matrix singularity, allowing symmetric coefficient regularization across all categories.

Tree-Based Models (Random Forest / XGBoost)

• Requirement: Do NOT drop (drop=None).
• Reason: Dropping a column forces decision trees to make multiple splits across $k-1$ columns just to isolate the dropped category, wasting tree depth.

---

4. What Cardinality Costs: Dense vs. Sparse Memory

The fundamental weakness of one-hot encoding is dimensionality explosion on high-cardinality columns. If a feature contains $k$ unique categories, it creates $k$ new columns.

Consider encoding a single column across $50,000$ rows:

Memory Footprint per Cardinality Tier

Categories (k)   One-Hot Cols    Dense Memory (MB)   Sparse Memory (MB)
             3              3              1.2 MB               0.8 MB
            10             10              4.0 MB               0.8 MB
            50             50             20.0 MB               0.8 MB
           256            256            102.4 MB               0.8 MB
         1,000          1,000            400.0 MB               0.8 MB  <- 500x difference

In dense format, 1,000 categories consumes 400 MB. In sparse format (scipy.sparse.csr_matrix), it consumes only 0.8 MB flat regardless of cardinality — a 500x memory reduction!

---

Interactive Cardinality Memory & Sparsity Calculator

Adjust the dataset row count and feature cardinality to observe the memory divergence between dense and sparse matrix representations:

Number of Rows (N): 50,000

Unique Categories (k): 500

Dense Memory Allocation

190.7 MB

N x k x 8 bytes

Sparse Memory (CSR)

0.76 MB

Stores only non-zero coordinates

Sparsity Factor: Sparse matrix storage provides a 251x memory reduction. Calling .toarray() or setting sparse_output=False converts clean 0.76 MB allocations into 190.7 MB of memory bloat, risking immediate Out-Of-Memory container termination.

High-Cardinality Alternatives: Binary & Frequency Encoding

• Binary Encoding: Converts integer category codes into binary bit representations ($\lceil \log_2 k \rceil$ columns). 256 categories require only 8 binary columns instead of 256 one-hot columns. However, bit representations group unrelated categories arbitrarily.
• Frequency Encoding: Replaces each category with its empirical frequency ($P(\text{category})$). Encodes into exactly 1 column without asserting false orderings, serving as a clean non-leaking baseline.

---

5. Target Encoding & The Leak Inside It

Target encoding replaces each category with the average target outcome for that category:

$\hat{x}_c = \frac{1}{|S_c|} \sum_{i \in S_c} y_i$

It maps high-cardinality features into exactly 1 column, directly capturing target correlations. However, computing category target means across your dataset creates the most lethal data leak in machine learning.

The Pure-Noise Experiment

In an experiment with 2,000 rows, 500 categories (~4 rows each), and a completely random target (true theoretical score $\approx 0.50$):

Target Encoding on Pure Noise

Method                                              CV Accuracy Score
Naive Global Target Mean (Leaked)                   0.7095  <- FAKE ACCURACY
Scikit-Learn TargetEncoder (Cross-Fitted)           0.4935  <- HONEST EVALUATION

Naive target encoding manufactured 21 points of fake accuracy out of pure noise. Because each category had only ~4 rows, each category's mean was essentially a direct copy of those 4 target values, smuggling $y$ directly into $X$.

---

Interactive Target Leakage Simulator

Compare naive global target encoding against scikit-learn's out-of-fold cross-fitting mechanism:

CV Score on Random Noise

0.7095

Production Score on Live API

0.5012

Leakage Status

Catastrophic Leak

The Trap: Global target encoding incorporates row $i$\'s own target into its feature calculation before cross-validation folds are created. Cross-validation cannot catch the leak because all folds are contaminated, leading to catastrophic production failure.

---

Interactive Checkpoint: Target Leakage

Why does naive target encoding result in a heavily overfitted 0.71+ cross-validation score on completely random target labels?

---

6. Unseen Categories at Inference Time

In production, live requests frequently introduce categories that never appeared during training:

# Training data has: ['Chennai', 'Delhi', 'Mumbai']
# Production request arrives with: 'Kolkata'

By default, scikit-learn encoders crash:

Default Exception on Unseen Category

ValueError: Found unknown categories ['Kolkata'] in column 0 during transform

Production Resiliency Settings

To prevent production pipeline crashes:

• OneHotEncoder(handle_unknown="ignore"): Encodes unseen categories as an all-zeros vector, representing "none of the known categories". This maintains identical column counts without crashing.
• OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1): Encodes unseen categories with a designated out-of-bounds marker (such as -1).
• LabelEncoder: Has no parameter for unseen categories, reinforcing that it belongs strictly on closed target sets ($y$).

---

7. Choosing an Encoding Scheme

---

8. Implementation Lab

tip

Implementation Lab: Categorical Encoding Benchmarks

Benchmark accuracy impacts of label vs. one-hot encoding on linear models, inspect the 500x memory explosion when converting sparse representations to dense arrays, and simulate target leakage live on pure noise.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Click "Runtime" > "Run all" or execute cells individually using Shift + Enter.
3. Follow the guided experiments: test drop="first", verify memory with sparse_output=True, and measure target leakage on random labels.

Comparative Implementation: Production Pipeline vs. Naive Script

1. Production Cross-Fitted Pipeline

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import TargetEncoder
from sklearn.linear_model import LogisticRegression

# Safe out-of-fold target encoding inside pipeline
model = make_pipeline(
    TargetEncoder(target_type="binary"),
    LogisticRegression()
)
model.fit(X_train, y_train)

2. Naive Manual Target Leakage (DANGEROUS)

# DANGEROUS: Calculates category means over the whole dataset
# prior to splitting, leaking future target info into training rows
global_means = df.groupby("city")["target"].mean()
df["city_encoded"] = df["city"].map(global_means)

---

9. Common Mistakes

1. Target Leakage

• The Misconception: "Calculating category means over the whole dataset before splitting is fine as long as I evaluate with cross-validation."
• Wrong Thinking: Standard cross-validation cannot catch leaks that occurred before the split. Naive target encoding scores ~0.71 on pure random noise.
• The Right Principle: Always use scikit-learn's cross-fitted TargetEncoder and place it inside a Pipeline so that target statistics are strictly isolated within training folds.

2. Nominal Labeling

• The Misconception: "Encoding Red=0, Green=1, Blue=2 is safe for any model because they are just numbers."
• Wrong Thinking: Asserting artificial linear ordering costs linear models over 30 points of accuracy.
• The Right Principle: Nominal data (no real order) should be one-hot encoded by default. Use label/ordinal encoding strictly for deep tree ensembles or genuinely ordinal features.

3. API Misuse (LabelEncoder on Features)

• The Misconception: "I can run LabelEncoder in a loop over my feature matrix X."
• Wrong Thinking: LabelEncoder expects a 1-D sequence for target labels ($y$) and breaks ColumnTransformer integration.
• The Right Principle: Always use OrdinalEncoder for 2-D features ($X$).

4. Dense Memory Blowups

• The Misconception: "One-hot encoding is always safe because the raw CSV is small."
• Wrong Thinking: Converting high-cardinality one-hot outputs to dense arrays (sparse_output=False or .toarray()) causes a 500x memory explosion.
• The Right Principle: Keep the default sparse_output=True. Most scikit-learn models consume sparse matrices natively with zero overhead.

---

10. Summary

Encoder Selection

• Ordinal Features: Use OrdinalEncoder(categories=[[...]]); explicitly state the sequence to avoid alphabetical distortion.
• Nominal Features (Low k): Use OneHotEncoder(handle_unknown="ignore") with sparse output enabled.
• Nominal Features (High k): Use TargetEncoder() inside a Pipeline to prevent data leakage via cross-fitting.

Production Guardrails

• Unseen Categories: Set handle_unknown="ignore" to output all-zeros rather than crashing during live inference.
• Dummy Variable Trap: Use drop="first" strictly for unregularized linear regression; retain all columns for trees.
• Pipeline Integrity: Never hand-compute target encodings. Encapsulate all transformations in a Pipeline.

info

Key Takeaways

• False Ordering Penalty: Label encoding nominal data degrades linear models (−31 points accuracy) and shallow trees (−32 points) because they cannot map alternating signs.
• Capacity Solves Ordering: Deep tree ensembles suffer zero penalty from false orderings because non-linear splits route around the integer mapping perfectly.
• Target Encoding Danger: Naive target encoding manufactures 21 points of fake accuracy (0.71 vs 0.50) on pure noise; cross-fitted TargetEncoder correctly scores 0.49.
• Sparse by Default: One-hot encoding high cardinality requires sparse matrices; converting to dense arrays causes a 500x memory explosion.

Next in this section: Feature Scaling and Transformation - putting numeric features on comparable mathematical footing.

See also: Types of Learning · Loading and Preparing Data · Missing Values

---

11. Active Recall Flashcards

Attempt each conceptual diagnostic question first, then click to reveal the underlying derivation and explanation.

1. [THEORY] Why must categorical data be encoded, and why is numerical representation more efficient?

• Mathematical Requirement: Algorithms built on mathematical operations (linear regression, neural networks, SVMs) require numbers for vector dot products and matrix operations. String tokens have no intrinsic mathematical meaning.
• Computational Efficiency: Numerical arrays store homogeneous types in contiguous memory blocks. String object arrays require individual memory pointer lookups, running 28x slower and using 98.4% more memory.

2. [THEORY] What three false mathematical assertions does label encoding nominal data Red=0, Green=1, Blue=2 make?

1. Magnitude: Asserts that Blue ($2$) is greater than Red ($0$).
2. Distance: Asserts that Green ($1$) sits exactly halfway between Red and Blue.
3. Ratio: Asserts that Blue ($2$) is mathematically twice Green ($1$).

3. [ANALYZE] Why does label encoding nominal data damage linear models (-31 points accuracy) but leave unrestricted trees unaffected?

• Linear Models: A linear model fits a single weight $w$, forcing a monotonic relationship between the integer code and the target. When category effects alternate along alphabetical codes, a single linear hyperplane cannot separate them.
• Unrestricted Trees: Given sufficient depth, decision trees can split the integer axis repeatedly (e.g. code <= 0.5, then 1.5 < code <= 2.5), isolating each category individually and completely overcoming the artificial ordering.

4. [THEORY] What is the difference between LabelEncoder and OrdinalEncoder in scikit-learn?

• LabelEncoder: Strictly designed for 1-D target labels (y). Takes a 1-D sequence and cannot participate in ColumnTransformer.
• OrdinalEncoder: Designed for 2-D feature matrices (X). Supports multiple columns, handles unseen categories, and integrates seamlessly into scikit-learn pipelines.

5. [THEORY] What is perfect multicollinearity (the dummy variable trap) and how does drop="first" fix it?

• The Trap: Across all $k$ one-hot columns, the sum for any row is identically 1 ($\sum x_j = 1$). One column can be predicted perfectly from the other $k-1$ columns ($x_k = 1 - \sum_{j \neq k} x_j$). The feature matrix lacks full rank, making $(X^T X)$ singular and non-invertible in Ordinary Least Squares.
• The Fix: drop="first" removes the first column, keeping $k-1$ columns. The dropped category becomes the baseline reference category (encoded as all-zeros), restoring full matrix rank.

6. [ANALYZE] For which models is drop="first" necessary, and for which is it counterproductive?

• Necessary: Unregularized linear regression (Ordinary Least Squares) to prevent matrix singularity.
• Counterproductive: Tree-based models (Random Forests, Gradient Boosting). Dropping a column forces decision trees to make multiple splits across $k-1$ features just to isolate the dropped category.

7. [ANALYZE] Why does one-hot encoding memory stay flat at 0.8 MB for sparse output while exploding to 400 MB for dense output at 1,000 categories?

• Dense Output: Allocates an $N \times k$ float64 array ($50,000 \times 1,000 \times 8\text{ bytes} \approx 400\text{ MB}$), storing millions of zeros explicitly.
• Sparse Output (CSR): One-hot rows contain exactly one 1 and $k-1$ zeros. Sparse format stores only the non-zero coordinates and column indices ($50,000 \times 16\text{ bytes} \approx 0.8\text{ MB}$), remaining completely flat regardless of category count.

8. [THEORY] Describe how naive target encoding introduces target leakage, and how cross-fitting prevents it.

• Naive Leakage: Computes category target means across all rows prior to splitting. Row $i$'s encoded feature incorporates its own target $y_i$, allowing models to achieve fake 0.71+ accuracy on pure random noise.
• Cross-Fitting Fix: Scikit-learn's TargetEncoder computes out-of-fold means: row $i$'s replacement value is computed strictly from rows $j \neq i$ within training partitions, preserving strict validation isolation.

9. [THEORY] What happens by default when an encoder meets an unseen category at prediction time, and how do you resolve it?

• Default Behavior: Raises a ValueError: Found unknown categories in column 0, crashing production pipelines.
• Resolution: Set handle_unknown="ignore" in OneHotEncoder, which encodes unseen categories as all-zeros (representing "none of the known categories"), keeping matrix shapes stable without crashing.

10. [ANALYZE] How many columns does binary encoding produce for 256 categories, and what is its main drawback?

• Columns Produced: $\lceil \log_2(256) \rceil = 8$ columns (compared to 256 for one-hot).
• Main Drawback: Binary bit groupings are completely arbitrary. Bit 3 groups together every category whose integer code happens to have that bit set, creating meaningless composite features that confuse linear models and tree splits.