Types of Learning

Topic — The previous page kept reaching the same fork: do you have labels? This page is that fork, in full. It is the single most consequential fact about a dataset, because it decides which paradigm you're in — and therefore which techniques are even available to you.

Before the paradigms, two pieces of vocabulary that everything else is stated in.

---

Features and attributes

Take a cancer dataset. The column names across the top — Clump thickness, Uniformity of cell size, Uniformity of cell shape, Marginal adhesion, and so on — are called attributes or features. The two words mean the same thing.

The standard layout, which almost every dataset you meet will follow:

              ┌──────────── features (X) ────────────┐   ┌─ target (y) ─┐
              Clump      Uniformity   Marginal            Class
              thickness  of cell size adhesion
    sample 1      5            1          1               benign
    sample 2      5            4          5               benign
    sample 3      3            1          1               malignant
       ...

Term	Also called	Meaning
Sample	row, instance, observation, record	One thing you're making a prediction about
Feature	attribute, column, variable, predictor, input	One measured property of that thing
Target	label, class, output, response, ground truth	The answer you want to predict
X	feature matrix	All features — 2-D, samples × features
y	target vector	The answers — 1-D, one per sample

The X / y naming is a near-universal convention in code, capital X because it's a matrix and lowercase y because it's a vector.

tip

The whole paradigm question in one sentence

Do you have a y column? If yes $\to$ supervised. If no $o$ unsupervised. If you have one for only some rows $\to$ semi-supervised.

---

Numeric and categorical data

When dealing with machine learning, the two most commonly used kinds of data are numeric and categorical.

	Numeric	Categorical
Values	Numbers on a scale	Names from a set
Example	5.1, 23, 98.6	benign, red, Tamil Nadu
Arithmetic meaningful?	Yes — averages, differences	No
Ordering meaningful?	Yes	Only sometimes

Each splits further, and these four sub-types are what later decisions actually turn on:

Type	Sub-type	Meaning	Example
Numeric	Continuous	Any value in a range	Height 170.4 cm
Numeric	Discrete	Countable whole numbers	3 bedrooms
Categorical	Nominal	No natural order	red, green, blue
Categorical	Ordinal	Has an order, but no fixed spacing	low < medium < high

The nominal/ordinal distinction matters more than it looks. low/medium/high can sensibly become 0/1/2 because the order is real. Doing the same to red/green/blue invents a claim that green sits between red and blue and that blue is "three times" red — which is why encoding choices are a topic of their own later.

🧠 Interactive Checkpoint: Classify Your Data

Scenario 1: A store customer leaves a rating of "Highly Satisfied", "Satisfied", or "Dissatisfied". What subtype of data is this?

Scenario 2: You are tracking standard 5-digit US Zip Codes (e.g. 90210) in a user registry. What data type is this?

The rule worth memorising

When dealing with classification problems:

The data used to make predictions — the features — can be either numeric or categorical. But the output, the class, is always categorical.

This gives a clean two-way grid. Look only at the target to name the technique:

Target is...	Technique	Example
Numeric	Regression	Predicting a house price
Categorical	Classification	Predicting benign vs malignant

Features can be a mixture of both in either case. The target alone decides.

---

Positive and negative class

In binary classification the two classes are not treated symmetrically. One is designated the positive class and the other the negative class.

• Positive class — the outcome you are looking for, or trying to detect
• Negative class — the ordinary, default, "nothing to report" outcome

Task	Positive class	Negative class
Disease detection	Has the disease	Healthy
Spam filtering	Spam	Legitimate mail
Fraud detection	Fraudulent	Legitimate
Churn prediction	Will leave	Will stay

Two things about this that surprise people:

"Positive" has nothing to do with good. Having cancer is the positive class. It's positive in the sense of a positive test result — the thing was found.

It is a choice, not a property of the data. Nothing stops you designating "healthy" as positive. The convention is to pick the rare and consequential class, because everything downstream — precision, recall, and the whole confusion matrix — is defined relative to the positive class. Flip the choice and every one of those numbers changes meaning.

danger

The default is frequently wrong

Tools pick a positive class for you, and their guess follows label order, not your intent. In scikit-learn's built-in breast cancer data, label 1 is benign — so out of the box, the "positive class" is not having cancer. Every recall figure you compute is then answering "how well do we find healthy people?", which is almost never the question.

Always check which class is 1, and set pos_label explicitly when it isn't the one you mean.

---

The Learning Paradigms

	Supervised	Unsupervised	Semi-supervised	Self-supervised (Modern)
Labels	Every row (human-labelled)	None	A few rows	None (derived from the data itself)
You are asking	"Predict this $y$ target"	"What structure exists?"	"Predict cheaply"	"Learn representations"
Feedback	Yes — compare prediction to $y$	No	Partial	Yes — prediction of masked tokens
Techniques	Classification, regression	Clustering, reduction	Hybrid co-training	Masking, Autoregressive modelling
Evaluation	Straightforward	Hard, subjective	Partly	Structured (loss on mask)
Main cost	Manual Annotation	None	A small annotation pool	High pre-training compute

---

Supervised learning

If you have labels in your data, it's supervised learning.

The flow is:

   labelled data  ──►  model training  ──►  prediction

You teach the model with pre-defined data — you already know the right answers, and you show them to the model so it can learn the relationship between features and answer. The name is literal: there is a supervisor, and the supervisor is your label column.

Using the cancer data: you give the model Clump thickness, Uniformity of cell size and the rest, together with whether each sample turned out benign or malignant. It learns the mapping. Then you hand it a new sample with no answer attached, and it predicts.

The defining advantage: you can tell whether it worked. Compare predictions against known answers and you get a number. This sounds obvious, and it's the thing unsupervised learning cannot do.

The two types of supervised learning

	Classification	Regression
Predicts	A category	A continuous value
Target type	Categorical	Numeric
Output example	malignant	52.4
"Close" counts?	No	Yes
Example	Which disease is this?	What will this house sell for?

That's the entire taxonomy of supervised learning. Every supervised algorithm is doing one of these two things.

---

Unsupervised learning

The model works on its own to discover information.

We do not supervise the model — we let it work on its own to discover insights that may not be visible to the human eye. There is no label column, so there is no notion of a right answer to check against. The model reports structure it finds; whether that structure is useful is a judgement you make.

Three families of task:

Clustering

Grouping data points or objects that are somehow similar.

Clustering uncovers patterns and structures within data by grouping similar items together. It does three distinct jobs:

Job	What it gives you
Discovering structure	What kinds of thing are in here?
Summarisation	Describing a huge dataset as a handful of representative groups
Anomaly detection	Points fitting no cluster well are outliers

Example: a bank's desire to segment its customers based on certain characteristics. Nobody defines the segments in advance — the algorithm proposes them, and the bank interprets them afterwards.

That last job is worth noticing: clustering doubles as anomaly detection by highlighting data points that don't fit well into any cluster.

Dimensionality reduction

Reducing the number of features or variables in a dataset while retaining its essential information.

Fewer columns, same meaning. Makes models faster, sometimes more accurate, and makes plotting possible.

Density estimation

Figuring out how the data is spread out or distributed, which lets you identify patterns and anomalies.

Once you know what "normal" looks like as a distribution, anything improbable under it is by definition unusual — which is another route to anomaly detection.

---

Supervised vs unsupervised

	Supervised	Unsupervised
Label column	Required	Absent
Goal	Predict a known quantity	Discover unknown structure
Supervision	A supervisor exists	None
Evaluation	Straightforward — compare to truth	Hard, often subjective
Guidance	You state what to learn	The model decides what's interesting
Risk	Needs expensive labels	May find structure you don't care about
Output	A prediction per new sample	A description of the data
Techniques	Classification, regression	Clustering, dimensionality reduction, density estimation

note

Why unsupervised learning is genuinely harder

Not harder to run — harder to know if you succeeded. A classifier is 94% accurate or it isn't. A clustering has no accuracy; you get groups, and deciding whether they're the right groups is a judgement call. This is why unsupervised results usually need a domain expert to sign off, and why "we clustered the data" is a weaker claim than it sounds.

---

Semi-supervised learning

Semi-supervised learning falls between supervised and unsupervised learning.

The training data is a combination of both labelled and unlabelled data — it uses a large amount of unlabelled data along with a small amount of labelled data to improve learning accuracy.

Why it exists

The main goal is to make the most out of the available data when labelling is expensive or time-consuming.

That constraint is the whole motivation, and it is extremely common:

• A radiologist must examine each scan — hours of specialist time per hundred images
• Unlabelled scans, meanwhile, accumulate for free

So it is commonly used where unlabelled data is abundant, such as image classification or text classification tasks. Scraping a million unlabelled images is easy; paying for a million labels is not.

Example algorithm: Semi-Supervised Support Vector Machines.

How it actually works

The common mechanism is self-training, and it's simple enough to state in four steps:

1. Train a model on the few labelled rows
2. Predict on the unlabelled rows
3. Take the predictions it's most confident about and treat them as if they were real labels
4. Retrain on the enlarged set, and repeat

The bet being made is the cluster assumption: points close together in feature space probably share a label. When that holds, unlabelled data reveals the shape of the classes and helps. When it doesn't, step 3 injects confident mistakes and compounds them.

warning

Semi-supervised learning is not free accuracy

In the worked demo below, 30 labels plus 396 unlabelled rows scored 0.921 against 0.918 for the same 30 labels alone — a gain of 0.003, essentially nothing.

That is a real and typical result. Semi-supervised methods help when the unlabelled data reveals structure the labelled points miss; when the classes are already well separated, there is nothing left to reveal. In poorly configured cases it is actively worse than discarding the unlabelled data, because self-training amplifies its own early errors.

Treat it as something to try and measure, never as a default.

---

Self-Supervised Learning (The Modern Paradigm)

Self-supervised learning (SSL) uses the internal structure of the data itself to construct pre-text labels, bypassing manual annotation entirely.

The core principle: The data is its own label. The model takes an unlabelled sample, hides or "masks" a portion of it, and is trained to predict the missing piece.

       unlabelled text ──► hide "mask" words ──► predict "mask" (loss on prediction)

This is the standard paradigm behind modern foundational models and Large Language Models (LLMs).

How it actually works

1. Masked Prediction (e.g. BERT): The sentence "The quick brown [MASK] jumps over the lazy dog" is fed to the model. The model learns language semantics by predicting that the masked word is fox.
2. Autoregressive Next-Token Prediction (e.g. GPT): The model is given a prompt "Once upon a..." and must predict the single most probable next token (time).
3. Representational Learning: By completing billions of these "pre-text" tasks, the model develops deep internal representations of text, pixels, or audio.
4. Fine-Tuning: The model is then "fine-tuned" on a tiny supervised dataset for a specific task (e.g., sentiment analysis, classification) with extremely high accuracy, requiring a fraction of the labels a standard supervised model would need.

Feature	Unsupervised	Self-Supervised
Core Goal	Discover flat clusters or latent features	Build a general representation of syntax/features
Output	Group IDs, principal components	A high-dimensional representation vector (embeddings)
Direct Application	Customer segmentation, image compression	Foundational LLMs (GPT, Llama), computer vision backbone

---

Choosing between them

Do you have a target column?
│
├─ Yes, for every row ──────────────► SUPERVISED
│                                     │
│                                     ├─ target numeric ─────► regression
│                                     └─ target categorical ─► classification
│
├─ Yes, but only for a few rows ────► SEMI-SUPERVISED
│    (and labelling more is expensive; measure against
│     plain supervised on the few you have)
│
└─ No ──────────────────────────────► Can we derive labels from data?
                                      │
                                      ├─ Yes (predict masked data) ─► SELF-SUPERVISED
                                      │                               (Foundational pre-training)
                                      │
                                      └─ No ─────────────────────────► UNSUPERVISED
                                                                       │
                                                                       ├─ want groups ──────► clustering
                                                                       ├─ too many columns ► dimensionality reduction
                                                                       └─ want the shape ──► density estimation

One practical note: if you have no labels but need predictions, the answer is usually not unsupervised learning — it's go and get some labels. Unsupervised learning answers a different question, not a cheaper version of the same one.

---

Common Mistakes

Here are the most common conceptual pitfalls when mapping business problems to learning paradigms, compared with the correct engineering principles.

🚨 Mistake 1: Positive = Good

• The Misconception: "The 'positive' class in binary classification refers to the good or desirable outcome."
• ❌ Wrong Thinking: Designating the healthy class as positive simply because health is positive/good.
• ✅ The Right Principle: The positive class is the target outcome you are actively looking for or trying to detect (e.g., disease, spam, fraud), which is often rare and consequential. It has nothing to do with being clinically or socially good.

🚨 Mistake 2: Default Class

• The Misconception: "We can just accept whichever default positive class the ML library picks for us."
• ❌ Wrong Thinking: Assuming the tool naturally knows which class is of interest, leading to incorrect calculations for recall/precision.
• ✅ The Right Principle: ML libraries typically sort labels alphanumerically and pick the last one or the first one depending on the library. In scikit-learn's breast cancer dataset, label 1 is benign (not having cancer), which is usually the opposite of what you want to evaluate. Always check your label mapping and set pos_label explicitly.

🚨 Mistake 3: Feature vs. Target

• The Misconception: "If our features are numeric, we must use regression. If our features are categorical, we must use classification."
• ❌ Wrong Thinking: Attempting to choose the machine learning technique based on the types of the features.
• ✅ The Right Principle: Features can be any mixture of numeric and categorical data. The target variable alone determines the technique: if the target is continuous numeric, it is a Regression problem; if the target is categorical, it is a Classification problem.

🚨 Mistake 4: Regressor Output

• The Misconception: "A regression model can output categories if we train it on numeric encodings of those categories."
• ❌ Wrong Thinking: Training a regressor on encoded target classes (e.g., benign=0, malignant=1) and expecting it to output discrete categories or valid categorical decisions directly without custom thresholding.
• ✅ The Right Principle: Classification outputs are discrete categorical labels. Regression outputs are continuous numeric values. Attempting to use a standard regression model directly for classification outputs introduces incorrect assumptions of continuity and distance.

🚨 Mistake 5: Encoding Nominal

• The Misconception: "We can encode nominal categorical categories like colors as sequential numbers (red=0, green=1, blue=2) to feed into our model."
• ❌ Wrong Thinking: Converting non-ordered categorical values (Nominal) directly to integers, which invents a mathematical order and scale that doesn't exist.
• ✅ The Right Principle: Sequential integer encodings are only suited for Ordinal data (which has a natural rank/sequence, like low/medium/high). For Nominal data (no natural order), you must use techniques like one-hot encoding to avoid introducing spurious mathematical relationships (e.g., green is mathematically between red and blue).

🚨 Mistake 6: Clustering vs. Classification

• The Misconception: "Clustering and classification are the same because they both group things together."
• ❌ Wrong Thinking: Confusing unsupervised grouping of similar data with supervised prediction of pre-defined category labels.
• ✅ The Right Principle: Classification is a supervised task predicting pre-defined, human-labeled target categories ($y$). Clustering is an unsupervised task that finds natural, unlabeled groupings based purely on spatial or feature similarity in the data.

🚨 Mistake 7: Clustering Accuracy

• The Misconception: "We can evaluate the performance of our clustering model by measuring its classification accuracy."
• ❌ Wrong Thinking: Computing standard accuracy score directly against cluster IDs, assuming clusters map perfectly and statically to true classes without correction.
• ✅ The Right Principle: Unsupervised clustering has no pre-defined ground truth labels during training. Since cluster IDs are arbitrary (e.g., Cluster 0 and Cluster 1), standard accuracy is inappropriate and misleading. Instead, use metrics like Adjusted Rand Index (ARI) that measure pairwise similarity and correct for agreement by chance.

🚨 Mistake 8: Semi-Supervised Lift

• The Misconception: "Using semi-supervised learning with extra unlabelled data will always improve model accuracy over using only the labelled data."
• ❌ Wrong Thinking: Assuming more data always yields a performance lift in any setting.
• ✅ The Right Principle: Semi-supervised learning only provides accuracy gains if the dataset conforms to the cluster assumption (points close together share a label). If the classes are already well-separated or don't follow the assumption, self-training can introduce confident mistakes and actively degrade accuracy.

🚨 Mistake 9: Labels Substitute

• The Misconception: "Since we don't have target labels, we will use clustering to perform classification."
• ❌ Wrong Thinking: Using unsupervised models as a direct, cheaper substitute for a supervised classification task.
• ✅ The Right Principle: Unsupervised learning answers a different question (discovering unknown structure) rather than predicting a specific known label. If you absolutely need a classifier, clustering is rarely a substitute—you must obtain a subset of labeled training data.

🚨 Mistake 10: Matrix Shapes

• The Misconception: "It doesn't matter if the target vector y is structured as a 1-D array or a 2-D single-column matrix."
• ❌ Wrong Thinking: Passing y as a 2-D column matrix (N, 1) into ML libraries that expect a 1-D vector (N,).
• ✅ The Right Principle: In standard ML libraries (like scikit-learn), the feature matrix X is always 2-D (samples × features), while the target vector y is strictly 1-D (samples,). Mixing these up leads to shape mismatches, runtime warnings, or silent broadcasting bugs.

---

Summary

🤖 Supervised & Semi-Supervised

• Supervised: Full manual label column $y$. Evaluated directly via target accuracy/loss metrics.
• Semi-Supervised: A tiny labeled subset plus abundant cheap unlabeled data. Uses self-training loops.

📊 Unsupervised & Self-Supervised

• Unsupervised: No label column. Finds latent customer clusters, projections, or distributions. Hard/subjective to validate.
• Self-Supervised: No manual labels; masks portions of the raw sample and learns to predict it. Powers modern LLM pre-training.

info

📌 Key Takeaways: Types of Learning

• 📏 Features vs. Target: Features ($X$) are standard multi-dimensional matrices, while targets ($y$) are 1-D vectors. The target alone determines the technique.
• 📍 Nominal vs. Ordinal: Nominal categories have no numeric weight or natural sequence (e.g. color), while Ordinal ratings have order (e.g. low/med/high) and can be scaled.
• ⚖️ The Positive Class is a Choice: Always check binary outputs. Library defaults default to alphanumeric sorting, which routinely makes the negative class the positive target. Set pos_label explicitly.
• 🧪 Unlabelled Lift is Not Guaranteed: Semi-supervised learning only yields accuracy lifts if data fits the cluster assumption. Otherwise, it compounds early errors.

Next in this section: Reinforcement Learning — the paradigm that has no dataset at all · The Toolkit and the Pipeline

See also: Applications and Major Techniques for the eight techniques these paradigms unlock · What is Machine Learning? for why labels beat hand-written rules

---

Run It Yourself

tip

Lab Exercise: Three Learning Paradigms on One Dataset

Run Supervised, Starved Supervised, Semi-Supervised, and Unsupervised models live on the exact same 426-patient Breast Cancer dataset. Tweak label visibility settings to watch how accuracy scales.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Run each cell sequentially (Shift + Enter).
3. Tweak parameters (N_LAB to 5, 50, or 100) to trace learning efficiency curves.

---

Practice Questions

Test your understanding by working each problem out first, then click the card to reveal the detailed solution. Questions are tagged by type: [THEORY], [PROG], [OUT], or [ANALYZE].

Features and data types

❓ F1. [THEORY] In a cancer dataset with columns Clump thickness, Uniformity of cell size, Marginal adhesion and Class, identify the features and the target. What is the other name for "feature"?

• Features (Attributes): Clump thickness, Uniformity of cell size, and Marginal adhesion. These are the measured properties of each sample used to make predictions.
• Target (Label/Class): Class (e.g., benign or malignant). This is the outcome or answer we want to predict.
• Other names for feature: Attribute, column, variable, predictor, or input.

❓ F2. [THEORY] Distinguish nominal from ordinal categorical data, and give an example of each. Why can only one of them be sensibly encoded as 0, 1, 2?

• Nominal Categorical Data: Categories with no natural order or ranking. Example: Colors (red, green, blue) or ZIP codes.
• Ordinal Categorical Data: Categories that have a natural sequential ranking, but without mathematically fixed or measurable spacing between them. Example: Education level (high school < bachelors < PhD) or customer satisfaction ratings (low < medium < high).
• Why only one can be encoded as 0, 1, 2: Ordinal data can be encoded sequentially (e.g., low=0, medium=1, high=2) because the mathematical order is preserved. Nominal data cannot be sensibly encoded this way because doing so invents a false ranking (e.g., encoding red=0, green=1, blue=2 falsely claims that green is between red and blue, and that blue is "twice" as large as green, which biases distance-based ML models).

❓ F3. [THEORY] State the rule about feature and output types in classification problems.

• The Rule: The features (inputs) used to make predictions can be either numeric or categorical (or a mix of both). However, the output (target class) in classification problems is always categorical.

❓ F4. [THEORY] A dataset's features are all numeric and its target is the text pass/fail. Which technique is this, and which column determined your answer?

• Technique: Classification (specifically binary classification).
• Determining column: The target column containing pass/fail. Because the target contains discrete categorical labels, the technique is classification, regardless of whether the features are numeric.

❓ F5. [THEORY] Why is X written with a capital letter and y with a lowercase one?

• X (Capital): Denotes a 2-D matrix (samples × features) holding all input features. In linear algebra, matrices are represented with uppercase letters.
• y (Lowercase): Denotes a 1-D vector (one target value per sample). In linear algebra, vectors are represented with lowercase letters.

The positive class

❓ C1. [THEORY] Define the positive and negative class. For disease detection, spam filtering and churn prediction, state the positive class in each.

• Positive Class: The rare, consequential, or specific outcome you are actively looking for or trying to detect.
• Negative Class: The ordinary, baseline, default, or "nothing to report" outcome.
• Disease Detection: Positive class = Has the disease / Negative class = Healthy.
• Spam Filtering: Positive class = Spam / Negative class = Legitimate mail.
• Churn Prediction: Positive class = Will leave (churn) / Negative class = Will stay.

❓ C2. [ANALYZE] "Positive class" sounds like it means the good outcome, but in disease detection the positive class is having the disease. Explain the sense in which it is "positive", and why the convention picks the rare class.

• The sense of "positive": It refers to a positive test result—meaning the condition or entity was successfully found or detected, not that the outcome is socially or medically desirable.
• Why convention picks the rare class: The class we want to detect is typically the rare and highly consequential one. Downstream evaluation metrics (such as precision and recall) are defined around the positive class to measure how effectively the model detects these critical events without raising too many false alarms.

❓ C3. [ANALYZE] In scikit-learn's breast cancer data, label 1 is benign. Explain precisely what goes wrong if you compute recall without noticing this, and what you would change.

• What goes wrong: If label 1 represents benign (healthy), then standard binary classification metrics like recall will evaluate the model's ability to correctly find healthy patients. A recall score of 95% would mean "95% of healthy patients were found," while leaving the critical clinician question—"how well do we detect malignant tumors?"—completely unmeasured.
• What you would change: When loading or formatting the data, you should map malignant to 1 and benign to 0, or explicitly set the parameter pos_label=0 in scikit-learn metrics functions (such as recall_score(..., pos_label=0)) to specify that malignant is the positive class.

The learning paradigms

❓ P1. [THEORY] Name the four learning categories and state, for each, what the training data contains.

1. Supervised Learning: Every training sample contains both features ($X$) and a manually assigned target label ($y$).
2. Unsupervised Learning: The training data contains only features ($X$) with absolutely no target labels ($y$).
3. Semi-Supervised Learning: A combination of a small pool of labeled samples ($X$ and $y$) and a large pool of unlabeled samples (only $X$).
4. Self-Supervised Learning: Unlabeled raw data samples (only $X$), where target labels are generated automatically from the data itself by hiding or masking a portion of each sample.

❓ P2. [THEORY] Draw the supervised learning flow in three stages, and explain what "teach the model with pre-defined data" means.

• Flow Chart:

  Labelled Training Data ──► Model Training (Fitting) ──► Target Predictions on New Data

• "Teach with pre-defined data": This means the algorithm is shown both the input features and the correct ground-truth answers (labels) simultaneously. The algorithm learns the mapping or relationships between the features and the answers by minimizing its prediction errors against these known answers.

❓ P3. [THEORY] Name the two types of supervised learning and give the target type and an output example for each.

1. Classification:
   • Target Type: Categorical (discrete classes).
   • Output Example: malignant, spam, or class_1.
2. Regression:
   • Target Type: Numeric (continuous scale).
   • Output Example: 250000.0 (house price) or 37.2 (temperature).

❓ P4. [THEORY] Name the three families of unsupervised task, with a one-line description of each.

1. Clustering: Grouping similar data points together based purely on feature distances or spatial distributions.
2. Dimensionality Reduction: Compressing the number of features (columns) in a dataset while preserving its core variance and underlying structure.
3. Density Estimation: Modeling the underlying statistical probability distribution of the data to identify normal vs. abnormal regions.

❓ P5. [THEORY] Clustering does three distinct jobs. Name all three, and explain how one of them makes clustering usable for anomaly detection.

• The Three Jobs:
  1. Discovering Structure: Revealing natural, hidden subgroups within the unlabeled data.
  2. Summarisation: Describing huge datasets by using a few representative cluster centroids.
  3. Anomaly Detection: Highlighting data points that do not fit well into any cluster.
• Usability for Anomaly Detection: Points that lie far away from any cluster centroid (low cluster membership or high reconstruction error) represent anomalous outliers. By setting a distance threshold, you can identify these unusual points automatically.

❓ P6. [ANALYZE] Explain why unsupervised learning is harder to evaluate than supervised learning. Is it harder to run?

• Why it's harder to evaluate: Supervised learning has a definite, objective ground-truth label for each row, making it easy to calculate exact metrics (like 94% accuracy). Unsupervised learning has no target labels or ground truth. The model proposes groups or structures, but determining whether those groups are useful or meaningful is subjective and usually requires domain experts.
• Is it harder to run: No, running unsupervised algorithms (e.g., KMeans.fit()) is computationally simple and requires the same or fewer lines of code as supervised learning. The difficulty is entirely in validation and interpretation.

❓ P7. [THEORY] Why does semi-supervised learning exist? Name the constraint that motivates it and two task areas where it is commonly used.

• Why it exists: To leverage massive amounts of cheap, easily accessible unlabeled data to improve prediction accuracy without incurring the cost of labeling everything.
• Motivating Constraint: Labeling data is expensive, slow, or requires rare specialist time (e.g., a radiologist labeling cancer scans), whereas unlabeled data is abundant and free.
• Two Task Areas: Image classification and text classification (or document analysis) where scraping web pages or images is free, but paying humans to label them is expensive.

❓ P8. [THEORY] Describe the four steps of self-training, and name the assumption it depends on.

• The Four Steps of Self-Training:
  1. Train a baseline model using only the small pool of labeled data rows.
  2. Predict labels for the large pool of unlabeled rows.
  3. Select the predictions that the model is most confident about and add them (with their predicted labels) to the labeled training pool.
  4. Retrain the model on the expanded labeled dataset, and repeat the process iteratively.
• The Cluster Assumption: This process assumes that points close to each other in the feature space are highly likely to share the same target class label. If this assumption fails, the model will confidently self-label incorrectly and amplify its errors.

Reading the output

❓ O1. [OUT] From the demo table output: How much accuracy did 396 extra labels buy? What did semi-supervised gain over starved supervised? What does row 4 achieving 0.908 with zero labels tell you?

• Accuracy bought by 396 labels: Going from starved supervised (30 labels, 0.918 accuracy) to fully supervised (all 426 labels, 0.958 accuracy) only bought 0.040 (or 4.0 percentage points) of accuracy.
• Semi-supervised gain: Gained only 0.003 (0.3 percentage points) over the starved supervised baseline (0.921 vs 0.918).
• What Row 4 tells us: Achieving 0.908 with KMeans (using zero labels) indicates that the two classes (benign and malignant) are highly separable in the standardized feature space. The natural structure of the data itself is strong enough that clustering almost perfectly recovers the actual classes without any human annotations.

❓ O2. [ANALYZE] A colleague concludes from row 4 that labelling was a waste of effort. Give two reasons this conclusion is unsafe.

1. Dataset Simplicity / Separability: This is an exceptionally easy, separable dataset. On harder real-world problems, unsupervised models often perform poorly or fail to align with the labels you care about, while supervised accuracy remains high.
2. Label Alignment & Evaluation: Without labels, you have no objective way of knowing which features KMeans clustered on, or whether the clusters map to "cancer vs healthy" or some other pattern (e.g. scanner type, patient age group). Furthermore, without labels, you cannot compute accuracy or prove to stakeholders that the system is safe to deploy.

❓ O3. [ANALYZE] The starved and semi-supervised rows are averaged over 15 random label draws. Explain why reporting a single draw would have been misleading.

• With a small label sample size (only 30 rows out of 426), a single random split can vary wildly by chance. One draw might accidentally pick highly representative points, resulting in a very high accuracy. Another draw might pick unrepresentative outliers, resulting in terrible accuracy. Reporting a single draw would capture random noise instead of the true, expected performance of the algorithm. Averaging over 15 random seeds smooths out this sampling variance.

❓ O4. [OUT] KMeans reported agreement 0.908 but an adjusted Rand index of 0.664. Both describe the same clustering. Why do they differ, and which is the more honest figure?

• Why they differ: Standard agreement (accuracy) does not account for random chance; even a random binary clustering can achieve 50% agreement by luck. The Adjusted Rand Index (ARI) measures pairwise similarity and mathematically corrects for random alignment, resetting the expected score of a random clustering to exactly 0.0.
• Which is more honest: Adjusted Rand Index (0.664) is much more honest and rigorous because it corrects for chance and handles arbitrary cluster indexing properly.

Applying it

❓ A1. [PROG] Load load_breast_cancer and print the number of samples, the number of features, and how many rows belong to each class.

You can run the following standard Python script to load the data and extract these statistics:

import numpy as np
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X, y = data.data, data.target

print(f"Number of samples: {X.shape[0]}")
print(f"Number of features: {X.shape[1]}")
print(f"Class counts: malignant={np.sum(y == 0)}, benign={np.sum(y == 1)}")

❓ A2. [PROG] Fit a KMeans with n_clusters=2 on the standardised features without using y, then print its agreement with the true labels under both possible cluster-to-class mappings.

You can standardize the features, fit the clustering model, and compute accuracy for both orientations using the following code:

from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import accuracy_score

# Load and scale
data = load_breast_cancer()
X_scaled = StandardScaler().fit_transform(data.data)

# Fit unsupervised KMeans
km = KMeans(n_clusters=2, random_state=0, n_init=10).fit(X_scaled)
labels = km.labels_

# Evaluate both mappings
acc_direct = accuracy_score(data.target, labels)
acc_inverted = accuracy_score(data.target, 1 - labels)

print(f"Direct Mapping Accuracy: {acc_direct:.3f}")
print(f"Inverted Mapping Accuracy: {acc_inverted:.3f}")
print(f"Best Clustering Agreement: {max(acc_direct, acc_inverted):.3f}")

❓ A3. [PROG] Build a semi-supervised setup: keep 20 labels, set the rest to -1, fit a SelfTrainingClassifier, and print test accuracy.

The following code trains and evaluates a semi-supervised self-training classifier:

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.pipeline import make_pipeline

# Load and split
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.25, random_state=0, stratify=data.target
)

# Mask labels: keep only 20, set rest of train labels to -1
rng = np.random.RandomState(42)
masked_y = np.copy(y_train)
unlabeled_indices = rng.permutation(len(y_train))[20:]
masked_y[unlabeled_indices] = -1

# Create self-training pipeline
base_model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
self_training = SelfTrainingClassifier(base_model)

# Fit and score
self_training.fit(X_train, masked_y)
acc = self_training.score(X_test, y_test)
print(f"Semi-Supervised Test Accuracy (20 labels): {acc:.3f}")

❓ A4. [ANALYZE] You have 500,000 unlabelled product reviews and budget to label 500. Name the paradigm, describe your plan, and state how you would tell whether the unlabelled reviews helped at all.

• Paradigm: Semi-supervised learning or Self-supervised learning (specifically, pre-training a transformer using SSL on all 500k reviews and fine-tuning on the 500 labeled ones).
• Plan:
  1. Scrape all 500,000 unlabeled reviews.
  2. Pay domain experts or use high-quality workers to annotate exactly 500 reviews with target labels (e.g. sentiment classification: Positive/Negative).
  3. Split the 500 labeled reviews into a training set (e.g. 350) and a clean, isolated validation/test set (e.g. 150).
  4. Train a baseline supervised model (e.g. Logistic Regression on TF-IDF or fine-tuned BERT) on only the 350 training labels and evaluate it on the 150 test labels.
  5. Use a semi-supervised algorithm (like SelfTrainingClassifier) or a self-supervised model (pre-trained language representations from the 500,000 reviews, fine-tuned on the 350 labeled reviews) and train the classifier.
  6. Evaluate this final model on the exact same 150 test labels.
• How to tell if unlabeled reviews helped: Compare the test accuracy of the baseline supervised model (trained on only 350 labels) against the test accuracy of the semi-supervised/self-supervised model. If the semi-supervised/self-supervised model's test accuracy is significantly and robustly higher, the unlabeled reviews successfully helped.

Quick self-check

❓ 1. What is another word for "feature"?

Attribute, predictor, column, variable, or input.

❓ 2. What do X and y conventionally hold, and what shape is each?

• X holds the feature matrix and has a 2-D shape (samples × features).
• y holds the target vector (ground truth answers) and has a 1-D shape (samples,).

❓ 3. What are the two most common kinds of data?

Numeric and Categorical data.

❓ 4. In classification, can features be categorical? Can the output be numeric?

• Features: Yes, features can be categorical.
• Output: No, the classification output (target class) must always be categorical.

❓ 5. Which column decides whether a task is regression or classification?

The target column (y) alone decides. If the target is continuous numeric, it is regression. If it is categorical, it is classification.

❓ 6. Does "positive class" mean the good outcome?

No. "Positive" refers to a positive detection result (the presence of the condition being searched for, like spam, fraud, or disease), not whether the outcome is socially or medically good.

❓ 7. What single question separates supervised from unsupervised learning?

Do you have a target column (y) in your training data? (If yes, it's supervised. If no, it's unsupervised).

❓ 8. Name the two types of supervised learning.

Classification and Regression.

❓ 9. Name the three families of unsupervised task.

Clustering, Dimensionality Reduction, and Density Estimation.

❓ 10. What are the three jobs clustering performs?

1. Discovering Structure (identifying underlying groupings).
2. Summarisation (representing huge datasets by a few key groups).
3. Anomaly Detection (identifying outliers that do not fit standard clusters).

❓ 11. Why does semi-supervised learning exist?

Because manually labeling data is expensive or time-consuming (requiring specialist labor), whereas unlabeled data is extremely abundant and cheap.

❓ 12. Which paradigm is hardest to evaluate, and why?

Unsupervised learning, because there is no ground-truth target label to compare predictions against, making evaluation subjective and dependent on domain expertise.