What is Machine Learning?

Topic — Every other page in this section is a technique. This page is the argument for why those techniques exist at all: there is a class of problem that ordinary programming genuinely cannot solve, and this is what it looks like.

The one-sentence definition

Machine learning is the field of study that gives computers the ability to learn without being explicitly programmed. — Arthur Samuel, 1959

The load-bearing phrase is "without being explicitly programmed". Everything on this page is an unpacking of those five words.

Tom Mitchell (1997) sharpened it into a definition you can actually test a system against:

A computer program learns from experience E with respect to task T and performance measure P, if its performance at T, as measured by P, improves with E.

Three things must be nameable, or it isn't learning. Click the tabs below to explore how they interact mathematically and operationally:

🎯 Task (T)

• The Operational Definition: The specific problem, action, or prediction you want the computer to solve.
• In our Cats-and-Dogs lab below: Labeling an animal photo as either a cat, dog, or bird.
• Mathematical Note: The task is represented mathematically as a mapping function from input features to categorical targets: $f(X) \to Y$.

📊 Experience (E)

• The Operational Definition: The historical, empirical data that the system is allowed to analyze and train on.
• In our Cats-and-Dogs lab below: 10 already-labeled animal rows containing their physical measurements.
• Mathematical Note: Experience is represented as a dataset $D = \{(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)\}$ drawn from an underlying data distribution.

📈 Performance (P)

• The Operational Definition: The objective mathematical metric used to measure whether the system has actually "learned" or improved.
• In our Cats-and-Dogs lab below: The fraction (accuracy) of new, unseen animals labeled correctly.
• Mathematical Note: Performance must be measured on a separate test set to ensure we evaluate generalization rather than memorization.

tip

Use this as a checklist

If you can't name T, E and P for something, it is not a machine learning problem yet — and that is usually the real reason a project stalls. P is the one people skip, and it is the one that decides whether you ever find out you succeeded.

---

What "explicitly programmed" means

In traditional programming, a developer writes code giving specific instructions that tell the computer exactly what to do in every situation.

Take a calculator. You explicitly code how to handle addition, subtraction, multiplication and division:

def calculate(a, b, op):
    if op == "+":
        return a + b
    if op == "-":
        return a - b
    if op == "*":
        return a * b
    if op == "/":
        return a / b

This is a complete and correct program, and it will be correct forever. Note why that works:

• The rules of arithmetic were known before the program was written
• There are four cases, and they are all listed
• A human could state each rule precisely in one line

Traditional programming is the right tool whenever those three conditions hold. Most software ever written satisfies them. Machine learning is for the problems that don't.

---

The task that breaks explicit programming

Here is the scenario that makes the problem concrete.

You have a dataset of images of animals — say cats and dogs — and you want to build an application that can recognise and tell them apart.

The first thing you must do is interpret each image as a set of features:

• Does the image show the animal's eyes? If so, what size?
• Does it have ears?
• What about a tail?
• How many legs?
• Does it have wings?

So an image becomes a row of numbers. That step is unavoidable in either approach — it's the subject of the data preprocessing notes later on. The question is what you do next.

The rule-based attempt

Traditionally, we had to write down rules or methods to make the application "intelligent" enough to detect the animals:

def what_animal(has_wings, num_legs, weight_kg):
    if has_wings:
        return "bird"
    if num_legs == 4 and weight_kg < 10:
        return "cat"
    if num_legs == 4 and weight_kg >= 10:
        return "dog"
    return "unknown"

It was a failure. Not "inefficient" — it did not work. There are three separate reasons, and they compound.

---

Why the rule-based attempt failed

1. It needed too many rules

The features above give six questions. Suppose each has just three possible answers (no / small / large). The number of distinct feature combinations you'd have to cover is:

3 × 3 × 3 × 3 × 3 × 3  =  3⁶  =  729

729 rules — for six coarse features and three animals. Add one more feature and it becomes 3⁷ = 2187. Add fur texture, ear shape, and tail length and you are past 19,000.

This is the combinatorial explosion, and it is not an engineering problem you can outwork. The rules grow exponentially in the number of features while your ability to write them grows, at best, linearly.

2. It was highly dependent on the current dataset

Look again at weight_kg < 10 in the rule above. Where did 10 come from? A human guessed it. It was chosen by looking at the cats and dogs that happened to be in the dataset at the time.

So the moment reality supplies:

• a Chihuahua at 2.5 kg — a dog, under the line → classified cat
• a Maine Coon at 11 kg — a cat, over the line → classified dog

...the rule is wrong, and no amount of care in writing it would have helped. The threshold encodes an accident of the sample, not a fact about animals.

3. It did not generalise to new samples

Points 1 and 2 combine into the fatal one. A rule set is only ever correct on the cases its author imagined. The world keeps producing cases nobody imagined. Since the program cannot revise itself, every new animal is a new bug report — and each fix risks breaking an earlier rule.

danger

The real cost isn't accuracy, it's maintenance

A rule-based system's error rate doesn't just start high — it grows over time as the world drifts away from the dataset the rules were tuned on. The only repair is a human rewriting rules forever.

---

How machine learning does it instead

Machine learning lets us build a model that looks at all the feature sets and their corresponding animal types, and learns the pattern of each animal.

Nothing is written down by hand. The model is produced by a machine learning algorithm, and it detects the animal without being explicitly programmed to do so.

The shift is in what the human supplies:

	Traditional programming	Machine learning
Human writes	the rules	the examples
Computer receives	rules + data	data + answers
Computer produces	answers	the rules (the model)
To handle new cases	rewrite the rules	add data, retrain
Thresholds like 10 kg	guessed by a human	derived from the data

Read the third row twice. It is the whole inversion: in machine learning, the rules are the output, not the input.

The four-year-old analogy

Machine learning follows the same process a four-year-old child uses to learn, understand and differentiate animals. Nobody hands a child 729 rules. The child sees animals, is told what they are, gets some wrong, and adjusts.

So machine learning algorithms, inspired by the human learning process, iteratively learn from data and allow machines to find hidden insights.

"Iteratively" matters — it is the mechanism behind gradient descent, covered later. Learning is not one calculation; it is a loop of guess → measure the error → adjust.

---

Seeing the difference in code

Same task, same features, two approaches.

Compare the implementation code for both paradigms:

❌ Traditional Rules (Hand-Written)

The hand-written rules rely on human intuition (e.g., guessing that "body weight" represents the major difference):

def rule_based(row):
    has_wings, legs, weight, snout, claws = row
    if has_wings:
        return "bird"
    if weight < 10:          # the human's guess: "small means cat"
        return "cat"
    return "dog"

✅ Learned Model (Scikit-Learn)

The learned model is given absolutely no rules—only the raw dataset matrix X_train and target vectors y_train to fit against:

from sklearn.tree import DecisionTreeClassifier

# Model derives all feature splits and decision boundaries autonomously
tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)

Now ask what the model taught itself:

Output

|--- retractable_claws <= 0.50
|   |--- has_wings <= 0.50
|   |   |--- class: dog
|   |--- has_wings >  0.50
|   |   |--- class: bird
|--- retractable_claws >  0.50
|   |--- class: cat

It never used weight. Given the same five features, the algorithm found that retractable_claws separates cats from dogs perfectly, and discarded the feature the human built their entire rule set around.

On four animals neither approach had seen:

Output

  animal         truth   rules    tree
  cat   6.0 kg   cat     cat      cat
  dog   8.0 kg   dog     cat  X   dog
  cat  13.0 kg   cat     dog  X   cat
  bird  1.2 kg   bird    bird     bird

  hand-written rules  50%
  learned tree        100%

The two failures are exactly the Chihuahua and Maine Coon cases predicted above — the mid-size dog under the 10 kg line, and the heavy cat over it.

note

What this demo does and doesn't prove

It does show that an algorithm can identify a better feature than a human's first instinct, and that hand-tuned thresholds fail on new data.

It does not show that ML always wins. The training set was deliberately built to contain a heavy cat and a light dog, so weight genuinely couldn't separate them. Had it contained only small cats and large dogs, the tree would have happily split on weight and failed the same way the rules did — an early glimpse of why representative data and honest validation matter more than the choice of algorithm.

---

Why now, if the idea is from the 1960s?

The core ideas are old — Samuel's self-improving checkers program was 1959, Rosenblatt's perceptron 1958. So why did machine learning only become popular recently?

Three things changed, and all three were necessary:

	What changed	Why it mattered
1	The rise of big data and the internet	Learning needs E. The internet made labelled examples abundant for the first time — text, images, clicks, transactions
2	Improved computation power	Training is an iterative loop over the whole dataset. GPUs made runs that would have taken months finish in hours
3	New algorithms and techniques to better handle and learn from data	Better optimisation, regularisation and architectures — the methods that let large models train stably

The useful takeaway: the bottleneck was never the idea. It was E and the compute to consume it. That is also why "get more/better data" so often beats "try a fancier algorithm" in practice.

---

AI vs ML vs DL

These three get used interchangeably in conversation and they are not interchangeable. They nest, strictly:

🤖 Artificial Intelligence

Any system that behaves autonomously without direct human intervention (including rule-based expert systems).

📊 Machine Learning

A subset of AI that learns patterns and rules directly from data using statistical models (e.g. Linear Regression, Decision Trees).

🧠 Deep Learning

A subset of ML that learns hierarchical representations automatically using multi-layered networks (e.g. Transformers, Large Language Models, Generative AI).

	Artificial Intelligence	Machine Learning	Deep Learning
Scope	The whole field	Subset of AI	Subset of ML
Defining idea	A system does its own task without human intervention	Learns from data using statistical tools	Hierarchical representation learning using multi-layered networks
Typical use	End-to-end autonomous behaviour	Tabular prediction, forecasting, classification	Perception and generation — images, audio, Transformers, LLMs, GenAI
Needs hand-designed features?	-	Usually yes	No, it automatically learns features and representations
Data needed	-	Works on modest datasets	Very large datasets (requires mass scale)
Example	Self-driving car; Netflix recommending a title	Predicting loan default from applicant data	GPT-4 generating text; recognizing a face in a photo
Covered in these notes	As context	The whole focus	No

Note that AI does not require learning at all. A rule-based chess engine is AI. That is why the what_animal function above was a legitimate attempt at AI — a bad one, but AI.

🧠 Interactive Checkpoint: Testing Your Taxonomy

A rule-based expert system (like a classical chess engine that contains no statistical learning) belongs in which category?

Why the job titles don't map to different fields

Whether you are an ML developer, a DL developer, a CV (computer vision) developer or an AI engineer — at the end of the day, you are building an AI application. The labels describe which layer of the diagram you spend your time in, not different professions.

Worked example of the layers cooperating: Netflix → movie genre → recommendation. Deep learning may extract features from artwork and trailers; machine learning predicts what you'll watch from your history; the AI application is the fact that a page of recommendations appears with nobody at Netflix choosing it for you.

---

Common Mistakes

Here are the most common conceptual pitfalls when starting out in Machine Learning, compared with the correct engineering principles.

🚨 Mistake 1: Does ML Write Code?

• The Misconception: "Machine Learning means the computer writes its own programming code to solve the problem."
• ❌ Wrong Thinking: Expecting the model to emit a .py file or a set of explicit rule structures.
• ✅ The Right Principle: The computer fits numeric parameters (weights and thresholds) of a statistical model whose mathematical architecture (like a Decision Tree or Neural Network) was chosen and restricted by you.

🚨 Mistake 2: Rules vs. ML

• The Misconception: "Since Machine Learning is modern, we should use it to build everything—even calculators or tax estimators."
• ❌ Wrong Thinking: Applying a model where the rules are 100% known and countable (e.g., standard addition or legal tax brackets).
• ✅ The Right Principle: If the rules are known, static, and finite, just write the code. Machine Learning is for domains where the rules are either too complex to state (images, speech) or are dynamically changing.

🚨 Mistake 3: Features & Scaling

• The Misconception: "Adding more features and columns to a dataset always improves model accuracy."
• ❌ Wrong Thinking: Dumping uncleaned columns into the model, assuming the algorithm will sift out the noise.
• ✅ The Right Principle: Features multiply the dimension of the feature space exponentially ($3^6 \to 3^7$). Too many features relative to your sample size (E) leads to the curse of dimensionality and severe overfitting.

---

Summary

🤖 Definitions & Core Frameworks

• Conceptual: Learning without being explicitly programmed (Samuel, 1959).
• Operational: Performance at T, measured by P, improves with experience E (Mitchell, 1997).
• The Inversion: In traditional programming, rules are the input. In machine learning, rules are the output.

🚨 Why Traditional Rules Fail

• Combinatorial Explosion: Rules grow exponentially ($3^6 = 729$), making manual rule sets unscalable.
• Sample Dependency: Hard-coded thresholds (like weight_kg < 10) encode accidents of the current sample, failing on edge cases (Maine Coons, Chihuahuas).
• Maintenance Trap: In a drifting world, every new sample becomes a new bug report.

info

📌 Key Takeaways: What is Machine Learning?

• 🎯 The Inversion: Traditional programming runs on hand-written rules. In Machine Learning, rules are the output, derived autonomously from data.
• 🖼️ Preprocessing != Learning: Translating raw inputs (like photos) into a matrix of numerical features is required for both approaches—feature extraction itself is not the statistical "learning" step.
• 🔄 The Iterative Loop: ML algorithms do not learn in a single step; they learn through a continuous optimization loop: Guess $\to$ Measure Error $\to$ Adjust.
• 📦 Nested Taxonomy: Deep Learning is a nested subset of Machine Learning, which is a nested subset of Artificial Intelligence ($DL \subset ML \subset AI$).

Next in this section: Applications and Major Techniques — the eight things ML actually does, and where each gets used · Types of Learning — how E differs by paradigm · The Toolkit and the Pipeline — the code that implements it

---

Run It Yourself

tip

Lab Exercise: Rules vs. Learning

An interactive, fully-functional Google Colab / Jupyter Notebook is available to experiment with this rules-vs-learning contest live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook instantly in your browser.
2. Click "Run all" or execute cells individually using Shift + Enter to see the output.

Things to try inside the script:

• Delete retractable_claws from the features list and re-fit. The tree will be forced back onto body weight—watch it fail exactly like the human's rules did!
• Remove the Maine Coon and Chihuahua rows from training, then re-fit and test. This demonstrates the unrepresentative-sample failure—the single most common real-world ML bug!
• Explore feature importances: Print tree.feature_importances_ alongside FEATURES to see the discarded features score exactly 0.0.

---

Practice Questions

Test your understanding by working each problem out first, then click the card to reveal the detailed solution. Questions are tagged by type: [THEORY], [PROG], [OUT], or [ANALYZE].

Definitions

❓ A1. [THEORY] State the definition of machine learning, and explain what the phrase "without being explicitly programmed" rules out.

• Arthur Samuel's Definition (1959): "Machine learning is the field of study that gives computers the ability to learn without being explicitly programmed."
• What it rules out: It rules out writing hand-crafted, hard-coded conditional logic or decision matrices (e.g., if-else loops) by humans. Instead of a human dictating the boundaries (rules), the algorithm learns the boundary weights directly from data (experience E).

❓ A2. [THEORY] Mitchell's definition names T, E and P. Identify all three for the task "predict whether a bank customer will default on a loan".

• Task (T): Classify or predict whether a customer will default (Yes/No) or predict their probability of defaulting on a loan.
• Experience (E): Historical customer profile databases, financial logs, credit histories, and known past loan repayment outcomes.
• Performance Measure (P): Classification evaluation metrics—such as Precision (avoiding false alarms on trustworthy clients) or Recall (capturing as many default risks as possible to avoid financial losses).

❓ A3. [THEORY] Give two examples of software that should NOT be built with machine learning, and justify each.

1. A Simple Calculator / Tax Estimator: The rules of arithmetic and standard tax brackets are 100% known, finite, and static. Using ML introduces unnecessary statistical error to a problem that can be solved with perfect, explicit if-else code.
2. User Authentication / Password Validator: Passwords must match exactly. Introducing a statistical model (which predicts a probability of a match) would introduce false positives, severely compromising security. Explicit string matching is mandatory.

❓ A4. [THEORY] Why is a rule-based chess engine still considered AI, despite containing no learning?

• The Taxonomy: Artificial Intelligence is the broad umbrella field that describes any system capable of acting autonomously without human intervention.
• The Chess Engine: A rule-based chess engine uses hand-written heuristics, minimax trees, and search algorithms to make intelligent chess decisions autonomously. It solves an intelligent problem without human help, satisfying the definition of AI, even though it contains no statistical learning (ML).

Why rules fail

❓ B1. [THEORY] A feature set has 6 features, each taking 3 possible values. How many distinct combinations exist? Show the calculation, then state what happens when a 7th feature is added.

• Calculation: $3^6 = 729$ distinct combinations.
• When adding a 7th feature: It multiplies the space by 3, resulting in $3^7 = 2,187$ combinations.
• The Lesson: This illustrates combinatorial explosion. Writing rules manually grows linearly with effort, but the state-space of rules grows exponentially with features, making explicit programming unscalable.

❓ B2. [ANALYZE] The rule "if weight_kg < 10: return 'cat'" classified the training data correctly. Explain precisely why it is nonetheless a defective rule, and name two real animals that break it.

• Why it is defective: The threshold (10 kg) is a hard-coded human guess tuned exclusively to the accidents of the small, local sample dataset. It represents overfitting to a sample's weight distribution, not an invariant biological fact.
• Two breakers:
  1. Maine Coon Cat: Can easily weigh 11+ kg (classified as a dog under the rule).
  2. Chihuahua Dog: Weighs only 2.5 kg (classified as a cat under the rule).

❓ B3. [THEORY] List the three reasons the rule-based approach to animal recognition failed, and explain which of the three is the direct consequence of the other two.

1. Needed too many rules (combinatorial explosion).
2. Highly dependent on current dataset (fragile, guessed thresholds).
3. Complete failure to generalize to new samples (the ultimate consequence).

• The Consequence: The third failure (failure to generalize) is the direct consequence of 1 & 2. Because the rules are too sample-dependent and cannot realistically scale to cover the variety of the real world, the model breaks the moment it meets new data.

❓ B4. [ANALYZE] "We can fix the rule-based system by hiring more developers to write more rules." Argue against this using the growth rate of the rule count.

• The Argument: Hiring more developers is a linear solution to an exponential problem. Developer rule-writing capabilities scale linearly ($O(N)$ with budget/headcount), while the feature state-space scales exponentially ($O(k^d)$ where $d$ is features). The exponential growth will always outpace linear capacity, causing the rule set to fall behind and collapse under technical debt.

The inversion

❓ C2. [OUT] A decision tree trained on five features produces this rules output: claws > 0.5 → cat, has_wings > 0.5 → bird. Which features did the model actually use, which did the human pick, and what does this show?

• Features Used: The model chose retractable_claws and has_wings and ignored body weight entirely.
• The Human's Choice: The human rule-based approach relied heavily on weight_kg.
• The Lesson: This shows that human intuition regarding feature importance is often flawed and biased towards what looks obvious, whereas machine learning algorithms can analyze the complete covariance of features and discover optimal, non-obvious classification parameters.

❓ C4. [ANALYZE] In the worked demo the hand-written rules scored 80% on training data but 50% on unseen data, while the tree scored 100% on both. Explain what the gap between the two columns measures, and why the rules' gap is the more worrying result.

• What the gap measures: It measures the generalization error (the drop in performance between data the model saw during training vs. new, unseen data).
• Why the rules' gap is worrying: The rules' performance dropped from 80% to 50% (random guess levels!). This massive drop indicates that the rules did not capture true features—they merely memorized an accident of the training sample. The tree's 100%/100% performance indicates true, robust generalization.

---

Quick self-check

Test your knowledge with these rapid-fire active recall questions.

🧠 1. Define machine learning in one sentence.

The field of study that gives computers the ability to learn from data without being explicitly programmed.

🧠 2. What does "explicitly programmed" mean?

A human developer manually writing hard-coded logic and conditional loops (like if-else blocks) to decide outputs.

🧠 3. In Mitchell's definition, what are T, E, and P?

• T (Task): The job you want the system to perform.
• E (Experience): The historical data/examples the model learns from.
• P (Performance): The metric used to evaluate how well the system does the task.

🧠 4. Why did the rule-based animal classifier fail?

Because it suffered from combinatorial explosion, used fragile guessed thresholds, and completely failed to generalize on unseen dogs and cats.

🧠 5. In machine learning, are the rules an input or an output?

Rules are the output of a machine learning pipeline (whereas they are the input of traditional programming).