Types of Learning

Topic — The single most consequential architectural decision in machine learning is determined by one question: Do you have target labels ($y$)? The presence, absence, or scarcity of ground-truth labels determines your learning paradigm - supervised, unsupervised, semi-supervised, or self-supervised - and dictates which algorithms are mathematically valid.

---

1. Feature Matrices vs. Target Vectors

In machine learning, data is represented by an input matrix $X$ and an optional target vector $y$:

              ┌──────────── Features (X) ────────────┐   ┌─ Target (y) ─┐
              Clump      Uniformity   Marginal            Diagnosis
              thickness  of cell size adhesion
    sample 1      5            1          1               benign
    sample 2      5            4          5               benign
    sample 3      3            1          1               malignant

Mathematical Symbol	Industrial Term	Linear Algebra Shape	Description
$X$	Feature Matrix	2-D: $(N \times d)$	All measured predictor variables across $N$ samples and $d$ features.
$y$	Target Vector	1-D: $(N,)$	Ground-truth labels or continuous outcomes we wish to predict.
Sample	Row / Instance	1-D: $(1 \times d)$	A single observed entity (e.g. one patient or one transaction).
Feature	Attribute / Predictor	Column	One measured property of that entity.

tip

The Whole Paradigm Fork in One Question

• Do you have a target column $y$ for all rows? $\implies$ Supervised Learning.
• Do you have NO target column $y$? $\implies$ Unsupervised Learning.
• Do you have a target column $y$ for only a small subset of rows? $\implies$ Semi-Supervised Learning.

---

2. Interactive Feature Subtype Classifier

Machine learning features split into numeric (quantitative) and categorical (qualitative) types, each branching into two crucial sub-types:

Feature Data Subtype Classifier

Select a real-world variable to inspect its mathematical properties and Scikit-Learn encoding strategy:

Customer Satisfaction ("Low", "Medium", "High")

Primary Type:
Categorical

Specific Subtype:
Ordinal

Arithmetic Meaningful?
No (cannot average)

Recommended Encoder:
OrdinalEncoder (0, 1, 2)

Statistical Rule: Ordinal categories have a real mathematical sequence, but the distance between steps is not fixed or measurable. Low/Medium/High can be safely encoded as integers 0, 1, 2.

The Target Rule Worth Memorising

When setting up your data architecture:

The data used to make predictions - the features ($X$) - can be a mixture of numeric and categorical variables. But the output - the target ($y$) - alone dictates the technique: if $y$ is continuous numeric, it is Regression; if $y$ is categorical, it is Classification.

---

3. The Positive Class Convention in Binary Classification

In binary classification ($y \in \{0, 1\}$), the two classes are mathematically asymmetric:

• Positive Class ($1$): The specific outcome you are actively searching for (e.g. presence of disease, fraudulent charge, spam email, customer cancellation).
• Negative Class ($0$): The baseline, ordinary, default "nothing to report" state (e.g. healthy, legitimate, non-spam, retained).

danger

The "Positive != Good" Trap and Library Defaults

1. "Positive" has nothing to do with being socially or clinically good. In medical oncology, having a malignant tumor is the positive class (a positive test result).
2. Scikit-Learn defaults to alphanumeric sorting. In Scikit-Learn's built-in Breast Cancer dataset, label 1 is assigned to benign (healthy) because "benign" comes before "malignant" alphabetically.
3. If you compute recall_score() without overriding pos_label, your model measures how well it finds healthy patients, completely missing the clinical question ("how well do we catch cancer?"). Always verify your label mapping.

---

4. Interactive Label Supervision Spectrum

Adjust the label availability slider to observe how machine learning paradigms transition from pure unsupervised exploration to fully supervised learning:

Label Supervision Spectrum Explorer

Move the slider to simulate varying levels of available ground-truth labels:

Label Availability: 10%Partially Labeled

Semi-Supervised Learning

10% Supervised

Operational Mechanism:
A small pool of verified labels (1-20%) bootstraps a large pool of unlabeled data via self-training or pseudo-labeling.

Validation Feasibility:
Evaluated against the holdout labeled test set. Success relies strictly on the cluster assumption.

Production Case: Medical radiology: 50 expert-annotated tumor scans combined with 5,000 unannotated MRI scans.

---

5. The Four Learning Paradigms in Depth

1. Supervised Learning

• Data Requirement: Every single training row contains input features $X$ and ground-truth answer $y$.
• The Workflow: Fits mathematical weights by minimizing the residual loss between predicted $\hat{y}$ and true $y$.
• Key Advantage: Direct, objective validation on an isolated test set ($R^2$, Accuracy, ROC-AUC).
• Core Branches:
  1. Classification: Target is categorical ($y \in \{\text{Cat}, \text{Dog}\}$).
  2. Regression: Target is continuous ($y \in \mathbb{R}$).

2. Unsupervised Learning

• Data Requirement: Contains only input features $X$. No target vector $y$ exists.
• The Workflow: Discovers geometric patterns, high-density manifolds, and clusters.
• Core Families:
  1. Clustering: Grouping rows into unnamed clusters (KMeans, DBSCAN).
  2. Dimensionality Reduction: Compressing wide feature spaces (PCA, t-SNE).
  3. Density Estimation: Modeling the probability density $P(X)$ to flag anomalies.
• Evaluation Challenge: Has no objective accuracy. Groupings require domain expert sign-off.

3. Semi-Supervised Learning

• Data Requirement: A small subset of labeled data $L$ combined with a massive pool of unlabeled data $U$.
• The Self-Training Loop:
  1. Train a model on the small labeled set $L$.
  2. Predict on the unlabeled set $U$.
  3. Select the most confident predictions and pseudo-label them.
  4. Retrain on the expanded dataset and repeat.
• The Cluster Assumption: Assumes data points close to each other in feature space share the same class label. If this fails, self-training amplifies its own errors!

4. Self-Supervised Learning (SSL)

• The Modern Paradigm: The foundation of modern Large Language Models (LLMs) and vision transformers.
• Mechanism (The Data is its own Label):
  • Masked Modeling (e.g. BERT): Hides tokens in sentences and trains the network to predict the masked word.
  • Autoregressive Generation (e.g. GPT): Predicts the next token in sequence.
• Representational Learning: Produces high-dimensional embeddings that are subsequently fine-tuned with a tiny supervised dataset.

---

6. Implementation Lab

tip

Lab Exercise: Three Paradigms on the Breast Cancer Dataset

An interactive, fully functional Google Colab / Jupyter Notebook is available to run Supervised, Starved Supervised (30 labels), Semi-Supervised (SelfTrainingClassifier), and Unsupervised (KMeans) on the exact same 426-patient dataset live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Run cells sequentially to benchmark accuracy across varying label counts.

Key Experiments to Run:

• Experiment 1 (The Starved Supervised Baseline): Compare fully supervised accuracy ($95.8\%$, 426 labels) against starved supervised ($91.8\%$, 30 labels) to measure the value of 396 additional annotations.
• Experiment 2 (Semi-Supervised Lift Test): Run SelfTrainingClassifier on the 30 labels plus 396 unlabeled rows and observe why semi-supervised lift is small ($92.1\%$) on well-separated datasets.
• Experiment 3 (Unsupervised Agreement vs. ARI): Observe KMeans achieving $90.8\%$ raw agreement with zero labels, but check Adjusted Rand Index ($0.664$) to see why raw accuracy on clusters is misleading.

---

7. Common Mistakes

1. Believing Positive Means Good

• The Misconception: "The positive class in binary classification represents the desirable medical or business outcome."
• Wrong Thinking: Designating healthy patients as the positive class.
• The Right Principle: The positive class is the target condition you are actively seeking to detect (cancer, fraud, churn). It is named after a positive test result, not moral or medical goodness.

2. Integer Encoding Nominal Categories

• The Misconception: "We can encode colors as Red=0, Green=1, Blue=2 for linear regression."
• Wrong Thinking: Imposing false ordinal relationships on nominal categories.
• The Right Principle: Only Ordinal variables (Low/Med/High) possess a natural rank. Nominal variables must be One-Hot encoded; otherwise, models assume Blue is twice as large as Green!

3. Measuring Accuracy on Unsupervised Clusters

• The Misconception: "Our KMeans clustering model achieved 91% accuracy."
• Wrong Thinking: Comparing arbitrary cluster IDs ($0, 1$) directly to true classes with standard classification accuracy.
• The Right Principle: Clustering has no predefined ground-truth labels. Cluster numbers are arbitrary. Use the Adjusted Rand Index (ARI) to evaluate pair-wise clustering alignment while correcting for random chance.

4. Semi-Supervised Learning is Free Accuracy

• The Misconception: "Adding unlabeled data to our training set will always improve model score."
• Wrong Thinking: Expecting self-training to consistently lift performance.
• The Right Principle: Semi-supervised learning only succeeds when data conforms to the cluster assumption. If early pseudo-labels are incorrect, self-training reinforces its own errors and actively degrades accuracy.

---

8. Summary

Feature Subtypes

• Numeric Continuous: Infinite precision scale (Height, Temperature).
• Numeric Discrete: Countable integers (Bedrooms, Ticket count).
• Categorical Nominal: Unordered labels (Color, Zip code). Needs One-Hot.
• Categorical Ordinal: Natural ranking (Low/Med/High). Integer encodable.

The Four Paradigms

• Supervised: 100% labeled $(X, y)$. Classification & Regression.
• Unsupervised: 0% labels. Discovers clusters, projections (KMeans, PCA).
• Semi-Supervised: Few labels + abundant unlabeled data (Self-training).
• Self-Supervised: Pre-text prediction tasks on raw data (LLMs, BERT).

info

Key Takeaways

• Target Alone Decides: Features ($X$) can be a mixture of numeric and categorical data; the target variable ($y$) alone dictates whether the problem is Classification or Regression.
• Watch pos_label: Scikit-Learn defaults to alphabetical label ordering, which frequently maps the negative class to 1. Always set pos_label explicitly.
• Clustering != Classification: Clustering proposes unnamed geometric segments; it cannot be evaluated using classification accuracy.

Next Section: Reinforcement Learning - learning through environment states, actions, rewards, and policy optimization without a static dataset.

---

9. Active Recall

Test your understanding by answering first, then clicking to reveal the underlying principles.

Interactive Checkpoint Quiz

Interactive Checkpoint: The Nominal Encoding Trap

A data scientist encodes postal Zip Codes (e.g. 90210, 10001) as standard continuous integers in a linear regression model. What is the fundamental statistical error?

Review Flashcards

1. [THEORY] Why is the "positive class" in medical diagnostics designated as having the disease?

• Positive Test Detection: "Positive" refers to a positive detection outcome (the condition was detected), not moral goodness.
• Asymmetric Priority: Having cancer is the rare, consequential event. Downstream evaluation metrics (like Precision, Recall, and F2-Score) are formulated around the positive class to measure how reliably critical cases are captured.

2. [ANALYZE] In Scikit-Learn's built-in Breast Cancer dataset, label 1 is benign. What dangerous bug occurs if you evaluate recall without checking?

• The Inverted Evaluation: Scikit-Learn treats label 1 as the default positive class.
• Calling recall_score(y_test, y_pred) evaluates the model's ability to find healthy (benign) patients!
• A high score ($96\%$) gives false confidence, while the critical life-or-death clinical question ("how many malignant tumors were caught?") remains completely unmeasured. Always specify pos_label=0 or re-map labels.

3. [THEORY] What is the "cluster assumption" in semi-supervised learning, and what happens when it fails?

• The Cluster Assumption: Assumes that data points lying close together in feature space or within the same high-density cluster share the same class label.
• Failure Consequence: If classes overlap or clusters do not align with target boundaries, self-training assigns pseudo-labels to incorrect points with high confidence, amplifying errors and producing accuracy worse than trained on the small labeled set alone.

4. [MATH] Why is standard accuracy misleading when evaluating KMeans clustering, and what metric should be used instead?

• Arbitrary Indexing & Chance Agreement: Clustering assigns arbitrary IDs (Cluster 0, Cluster 1) with no concept of ground-truth names. Standard accuracy does not account for label permutation or random alignment.
• The Solution: Use the Adjusted Rand Index (ARI). ARI evaluates whether pairs of samples are placed in the same or different clusters and corrects for chance agreement, ensuring random clustering scores exactly 0.0.

5. [THEORY] How does Self-Supervised Learning (SSL) differ from traditional Unsupervised Learning?

• Unsupervised Learning: Seeks to discover flat geometric clusters (KMeans) or low-dimensional projections (PCA) directly from raw features without predictive supervision.
• Self-Supervised Learning: Automatically constructs predictive pretext tasks from the data itself (e.g. masking words in BERT or predicting the next token in GPT). It trains via supervised backpropagation against internal data structure to learn deep universal representations.