Reinforcement Learning
Topic — The three paradigms so far all begin the same way: a table of data arrives, and you check whether it has a label column. Reinforcement learning begins with no table at all. The agent generates its own data by acting, and finds out afterwards how that went.
That difference is not a detail. It changes what "training data" even means.
The definition
Reinforcement learning is a feedback-based technique in which an agent learns to behave in an environment by performing actions and seeing the results of those actions.
The two words carrying the weight are feedback-based and actions.
- For each good action, the agent gets positive feedback
- For each bad action, the agent gets negative feedback, or a penalty
Since there is no labelled data, the agent is bound to learn by its experience only. Nobody ever tells it the correct action. It is told, after the fact, how much reward it collected — and it must work backwards from that to figure out what it should have done.
The primary goal of the agent is to improve its performance by getting the maximum positive rewards.
The vocabulary
Reinforcement learning has more moving parts than the other paradigms, and the terms are used precisely.
| Term | Meaning | In the grid demo below |
|---|---|---|
| Agent | The learner and decision-maker | The thing walking around |
| Environment | Everything outside the agent, which it acts upon | The 4×4 grid and its rules |
| State | The situation the agent is currently in | Which square it stands on |
| Action | A choice available to the agent | Move up, down, left or right |
| Reward | The numeric feedback for an action | +10 jackpot, −10 pit, −1 per step |
| Policy | The agent's strategy — state → action | "On this square, go down" |
| Episode | One run from start to a terminal state | Start until it hits goal or pit |
| Return | Total reward accumulated over an episode | The episode's score |
| Value | Expected return from a state onwards | How promising this square is |
The policy is the actual output of reinforcement learning. Where supervised learning produces a model that maps features to a prediction, reinforcement learning produces a policy that maps situations to behaviour.
The loop
Everything in reinforcement learning is this cycle, repeated:
┌─────────────────────────────────────┐
│ │
▼ │
┌──────────┐ action a ┌────┴─────┐
│ │ ────────────────────────► │ │
│ AGENT │ │ ENVIRON │
│ │ ◄──────────────────────── │ MENT │
└──────────┘ new state s', reward r └──────────┘
1. agent observes the current state
2. agent chooses an action
3. environment returns a new state and a reward
4. agent updates its policy
5. repeat
Two properties of this loop cause all the difficulty:
The agent's actions change the data it sees next. In supervised learning your dataset is fixed; here, a bad early policy means the agent only ever visits bad states and never discovers the good ones. The data distribution depends on the thing being learned.
Reward is often delayed. A chess move that loses the game may look fine when played. The agent receives one signal at the end and must apportion credit across all the moves that led there — the credit assignment problem.
Why it isn't the other three
| Supervised | Unsupervised | Semi-supervised | Reinforcement | |
|---|---|---|---|---|
| Training data | Labelled rows | Unlabelled rows | Both | Generated by acting |
| Feedback | The correct answer | None | Partial | A reward, not an answer |
| Told what's right? | Yes | Never | Sometimes | No — only how good |
| Data fixed in advance? | Yes | Yes | Yes | No |
| Output | A prediction | A description | A prediction | A policy |
| Timing of feedback | Immediate | — | Immediate | Often delayed |
The row that matters most is the third. Reinforcement learning shares "no labels" with unsupervised learning, and people therefore lump them together — but this is wrong:
Supervised learning is told "the answer was malignant".
Unsupervised learning is told nothing at all.
Reinforcement learning is told "that got you −1" — which is neither the answer nor silence.
A reward says how good an action was, never what you should have done instead. That's why the agent must try alternatives to find out: evaluating one action tells it nothing about the others.
The two hard problems
Exploration vs exploitation
At every step the agent faces a dilemma:
- Exploit — take the best action it currently knows about, and collect the reward
- Explore — try something else, which may be worse, but might reveal something better
Exploit too much and the agent locks onto the first decent thing it finds, never discovering that something far better exists. Explore too much and it spends its life taking actions it already knows are bad.
The standard mechanism is ε-greedy: with probability ε take a random action, otherwise take the
best known one. ε is the dial between the two, and the demo below shows exactly what it costs to
set it wrongly in either direction.
Credit assignment
When reward arrives late, which of the fifty actions you took deserves the credit? Reinforcement learning solves this by propagating value backwards: a state is valuable if it leads to valuable states. This is why the agent in the demo eventually learns that the square next to the start is worth something, despite that square paying nothing itself.
Where it's used
Reinforcement learning plays a significant role in the development of autonomous vehicles, also known as self-driving cars. The fit is natural: there is no dataset of correct steering angles for every situation, but there is abundant feedback — stayed in lane, maintained distance, reached the destination, or crashed.
| Domain | The agent | The reward |
|---|---|---|
| Autonomous vehicles | The driving policy | Progress, safety, comfort |
| Game playing | The player | Winning; score |
| Robotics | Motor controller | Task completed, energy used |
| Recommendation | The ranking policy | Clicks, watch time, retention |
| Resource management | Scheduler | Throughput, latency, cost |
The common shape: you can recognise good behaviour, but you cannot write down the correct action for every situation. That is precisely the gap reinforcement learning fills — and it's the same gap that made hand-written rules fail in the first place, one level further up.
Common Mistakes
| # | Mistake | Wrong | Right |
|---|---|---|---|
| 1 | Calling it unsupervised | "no labels, so unsupervised" | It has feedback; unsupervised has none |
| 2 | Treating reward as a label | Reward says what to do | It says how good, not what |
| 3 | Confusing policy with model | "the model predicts" | RL outputs a policy — behaviour |
| 4 | Setting ε to 0 | "always do the best thing" | Locks onto the first decent option |
| 5 | Setting ε to 1 | "explore everything" | Pays maximum cost for no extra gain |
| 6 | Judging progress by reward while learning | Online reward looks bad | Evaluate the greedy policy separately |
| 7 | Expecting immediate feedback | Reward per action tells you enough | Reward is often delayed |
| 8 | Assuming a fixed dataset | Collect data, then train | Actions determine future data |
| 9 | Stopping at the first plateau | "it converged" | It may be in a local optimum |
Summary
| Concept | One line |
|---|---|
| Definition | Feedback-based learning by performing actions and seeing the results |
| Good action | Positive feedback |
| Bad action | Negative feedback or penalty |
| Labels | None — the agent learns by experience only |
| Goal | Maximise cumulative positive reward |
| Output | A policy, mapping states to actions |
| Key dilemma | Exploration vs exploitation |
| Flagship use | Autonomous vehicles |
Key takeaways
- Reinforcement learning has no dataset; the agent generates its own data by acting
- Good actions earn positive feedback, bad ones a penalty
- With no labelled data, the agent learns by its experience only
- Reward tells the agent how good an action was, never what it should have done — this is why it must explore
- It is not unsupervised learning: unsupervised has no feedback, reinforcement has feedback without answers
- The output is a policy — behaviour — not a prediction
- The agent's actions change what data it sees next, so a bad early policy is self-reinforcing
- Reward is often delayed, creating the credit assignment problem
- ε-greedy trades exploration against exploitation, and both extremes are costly
- Measured:
ε ≤ 0.3settled permanently for a policy worth 1.0;ε = 0.5found the one worth 5.0 — five times better, from changing nothing but curiosity ε = 1.0found the same best policy asε = 0.5but paid −12.17 instead of −4.73 while learning; maximum exploration is wasteful, not optimal- An agent can sit on a plateau for 2,000 episodes looking converged, then break through
Next in this section: The Toolkit and the Pipeline — the libraries and the workflow that implement all four paradigms
See also: Types of Learning for the three paradigms this one sits beside · Applications and Major Techniques for where each technique is used
Run It Yourself
No scikit-learn, no dataset — 60 lines of NumPy and an agent that learns a grid it has never been told anything about.
"""Reinforcement learning from scratch — an agent learns a grid by trial and error."""
import numpy as np
GRID = ["S.t.",
".XX.",
"...X",
"...G"]
ROWS, COLS = len(GRID), len(GRID[0])
ACTIONS = ["^", "v", "<", ">"]
MOVES = [(-1, 0), (1, 0), (0, -1), (0, 1)]
REWARD = {"G": 10.0, "t": 2.0, "X": -10.0} # G=jackpot, t=small prize, X=pit
STEP_R = -1.0
def step(state, a):
"""The environment. Given a state and an action, return next state, reward, done."""
r, c = state
dr, dc = MOVES[a]
nr, nc = r + dr, c + dc
if not (0 <= nr < ROWS and 0 <= nc < COLS):
nr, nc = r, c # walls bounce you back
ch = GRID[nr][nc]
if ch in REWARD:
return (nr, nc), REWARD[ch], True
return (nr, nc), STEP_R, False
def run_greedy(Q):
"""Follow the learned policy with no exploration. Returns (total reward, ending cell)."""
s, done, guard, total, r = (0, 0), False, 0, 0.0, 0.0
while not done and guard < 100:
guard += 1
s, r, done = step(s, int(np.argmax(Q[s[0], s[1]])))
total += r
return total, GRID[s[0]][s[1]]
def train(epsilon, episodes=3000, alpha=0.1, gamma=0.95, seed=0, track=False):
rng = np.random.RandomState(seed)
Q = np.zeros((ROWS, COLS, 4))
curve, online = [], []
for ep in range(episodes):
s, total, done, guard = (0, 0), 0.0, False, 0
while not done and guard < 100:
guard += 1
if rng.rand() < epsilon:
a = rng.randint(4) # EXPLORE — try something
else:
a = int(np.argmax(Q[s[0], s[1]])) # EXPLOIT — best known
s2, r, done = step(s, a)
future = 0.0 if done else np.max(Q[s2[0], s2[1]])
Q[s[0], s[1], a] += alpha * (r + gamma * future - Q[s[0], s[1], a])
s, total = s2, total + r
online.append(total)
if track and ep % 250 == 0:
curve.append((ep, run_greedy(Q)[0]))
return Q, np.array(online), curve
def show_policy(Q):
rows = []
for r in range(ROWS):
rows.append(" ".join(
GRID[r][c] if GRID[r][c] in REWARD else ACTIONS[int(np.argmax(Q[r, c]))]
for c in range(COLS)))
return "\n".join(" " + row for row in rows)
print("THE ENVIRONMENT")
for row in GRID:
print(" " + " ".join(row))
print(" S=start G=jackpot (+10) t=small prize (+2) X=pit (-10) step (-1)")
print(" the agent is told NONE of this — no map, no rules, no labels")
Q, online, curve = train(epsilon=0.5, track=True)
print("\nLEARNING PROGRESS — reward the greedy policy would earn, as training proceeds")
for ep, val in curve:
print(f" after {ep:>4} episodes {val:>6.1f}")
print("\nTHE LEARNED POLICY — best action found for each square")
print(show_policy(Q))
print("\nWHAT THE AGENT BELIEVES AT THE START SQUARE")
for a, name in enumerate(ACTIONS):
print(f" Q[start, '{name}'] = {Q[0, 0, a]:>6.2f}")
print(f" -> it picks '{ACTIONS[int(np.argmax(Q[0, 0]))]}', worth {Q[0, 0].max():.2f}")
print(" hand-computed optimum: jackpot path 3.21, small-prize path 0.90")
print("\nEXPLORATION vs EXPLOITATION — same algorithm, only epsilon differs")
print(f" {'epsilon':>8} {'behaviour':<16}{'settles on':>13}{'policy R':>10}{'cost while':>12}")
print(f" {'':>8} {'':<16}{'':>13}{'(final)':>10}{'learning':>12}")
for eps, label in [(0.0, "never explores"), (0.05, "rarely"), (0.1, "sometimes"),
(0.2, "regularly"), (0.3, "often"), (0.5, "half the time"),
(1.0, "always random")]:
Qe, oe, _ = train(epsilon=eps)
total, ending = run_greedy(Qe)
name = {"G": "JACKPOT", "t": "small prize", "X": "pit"}.get(ending, "nowhere")
print(f" {eps:>8.2f} {label:<16}{name:>13}{total:>10.1f}{oe[-500:].mean():>12.2f}")
THE ENVIRONMENT
S . t .
. X X .
. . . X
. . . G
S=start G=jackpot (+10) t=small prize (+2) X=pit (-10) step (-1)
the agent is told NONE of this — no map, no rules, no labels
LEARNING PROGRESS — reward the greedy policy would earn, as training proceeds
after 0 episodes -100.0
after 250 episodes 1.0
after 500 episodes 1.0
after 750 episodes 1.0
after 1000 episodes 1.0
after 1250 episodes 1.0
after 1500 episodes 1.0
after 1750 episodes 1.0
after 2000 episodes 1.0
after 2250 episodes 1.0
after 2500 episodes 5.0
after 2750 episodes 5.0
THE LEARNED POLICY — best action found for each square
v > t ^
v X X ^
> v v X
> > > G
WHAT THE AGENT BELIEVES AT THE START SQUARE
Q[start, '^'] = 2.05
Q[start, 'v'] = 3.21
Q[start, '<'] = 2.05
Q[start, '>'] = 0.90
-> it picks 'v', worth 3.21
hand-computed optimum: jackpot path 3.21, small-prize path 0.90
EXPLORATION vs EXPLOITATION — same algorithm, only epsilon differs
epsilon behaviour settles on policy R cost while
(final) learning
0.00 never explores small prize 1.0 1.00
0.05 rarely small prize 1.0 0.79
0.10 sometimes small prize 1.0 0.44
0.20 regularly small prize 1.0 -0.21
0.30 often small prize 1.0 -1.06
0.50 half the time JACKPOT 5.0 -4.73
1.00 always random JACKPOT 5.0 -12.17
What to notice in that output
The learning curve tells a story in four acts. At episode 0 the greedy policy scores −100.0 —
the agent wanders until the 100-step safety limit, reaching nothing. By episode 250 it scores +1.0:
it has found the small prize and locked on. Then it sits at +1.0 for two thousand episodes,
looking every bit like a converged solution. At episode 2500 it breaks through to +5.0. Anyone who
had stopped training at episode 1000 would have shipped a policy worth a fifth of the achievable
reward, with no indication anything was wrong.
The policy is readable, and it is correct. Trace the arrows from S: down, down, then right along
the bottom row to G, threading between the pits. The agent derived that route from nothing but
rewards.
The learned numbers match theory exactly. Q[start, 'v'] = 3.21 and Q[start, '>'] = 0.90.
Hand-computing the discounted return for the jackpot path gives
−1 − 0.95 − 0.95² − 0.95³ − 0.95⁴ + 0.95⁵ × 10 = 3.21, and for the two-step small-prize path
−1 + 0.95 × 2 = 0.90. The algorithm didn't approximate the right answer, it found it.
Now the epsilon table, which is the real lesson. Every row runs the same algorithm with the same seed for the same 3,000 episodes. Only curiosity differs:
εfrom 0.00 to 0.30 all settle for the small prize — a final policy worth1.0. Not a bug, not bad luck: five different agents, all confidently converged, all permanently wrong.ε = 0.50finds the jackpot — worth5.0, five times better. The only change was a willingness to act against its own best judgement half the time.ε = 1.00finds the same best policy asε = 0.50, but paid−12.17per episode while learning instead of−4.73. So maximum exploration isn't optimal, it's just expensive — there is a sweet spot, and it isn't at either end.
Read the last column as a price. As ε rises, reward during learning falls monotonically:
1.00 → 0.79 → 0.44 → −0.21 → −1.06 → −4.73 → −12.17. That is the literal cost of curiosity. The
ε = 0 agent is the best-performing agent while learning and the worst policy at the end.
ε = 0.5 looks extreme. The reason is the trap: reaching the small prize ends the episode after
two steps, so a greedy agent gets almost no opportunity per episode to see anything further away.
Environments that terminate early on mediocre outcomes are exactly the ones that punish
under-exploration hardest.
Things worth trying:
- Set
episodes=10000withε=0.1. Does patience substitute for curiosity here, or not? - Change the small prize to
+8inREWARD. Now the local optimum is genuinely close to the jackpot — watch how much harder escaping becomes. - Set
gamma=0.5. The agent becomes short-sighted, discounting distant reward heavily, and should prefer the near prize even at highε. - Set
STEP_R = 0.0. With no cost per step the agent has no incentive to find a short path — compare the policy it settles on.
Practice Questions
Work each one out on paper first, then check it by running the code — everything you need is on this page. No answers are included, deliberately.
| Tag | Means |
|---|---|
| [THEORY] | Explain in words. Definitions, reasons, comparisons. |
| [PROG] | Write the program. Complete, runnable code. |
| [OUT] | Read the output. Interpret given results exactly as printed. |
| [ANALYZE] | Argue a position. Weigh a claim, diagnose a failure, justify a trade-off. |
Definitions
R1. [THEORY] Define reinforcement learning in one sentence, and explain what "feedback-based" means in that definition.
R2. [THEORY] What happens for a good action, and what happens for a bad one? What is the agent's primary goal?
R3. [THEORY] Define each of: agent, environment, state, action, reward, policy, episode.
R4. [THEORY] "Since there is no labelled data, the agent is bound to learn by its experience only." Explain what this rules out.
R5. [THEORY] What is the output of reinforcement learning, and how does it differ from the output of supervised learning?
Distinguishing the paradigms
D1. [ANALYZE] A colleague says reinforcement learning is a kind of unsupervised learning because neither has labels. Explain why this is wrong, using the idea of feedback.
D2. [THEORY] Complete the sentence for all three: supervised learning is told ___, unsupervised learning is told ___, reinforcement learning is told ___.
D3. [ANALYZE] Explain why a reward is not a label. Why does this force the agent to explore?
D4. [THEORY] In supervised learning the dataset is fixed before training. Why is this not true in reinforcement learning, and what problem does that create?
D5. [THEORY] What is the credit assignment problem, and why does delayed reward cause it?
Exploration and exploitation
E1. [THEORY] Define exploration and exploitation, and state the risk of doing too much of each.
E2. [THEORY] Describe the ε-greedy strategy. What does ε control?
E3. [OUT] From the demo:
epsilon behaviour settles on policy R cost while
0.00 never explores small prize 1.0 1.00
0.30 often small prize 1.0 -1.06
0.50 half the time JACKPOT 5.0 -4.73
1.00 always random JACKPOT 5.0 -12.17
- Which agent produced the best final policy, and which performed best while learning?
- What did raising ε from 0.30 to 0.50 achieve?
- Why is ε = 1.00 not the best choice, given it also found the jackpot?
E4. [ANALYZE] The last column falls steadily as ε rises: 1.00, 0.79, 0.44, −0.21, −1.06, −4.73, −12.17. Explain what this column is measuring and why it must behave this way.
E5. [ANALYZE] Five agents with ε from 0.00 to 0.30 all converged confidently on a policy worth one fifth of the best available. Explain the mechanism, and say what this implies about trusting a flat learning curve.
Reading the output
O1. [OUT] The greedy policy scored −100.0 after 0 episodes. Given the step reward is −1 and
there is a 100-step safety limit, explain that number.
O2. [OUT] The learning curve sat at +1.0 from episode 250 to 2250, then jumped to +5.0. Name
what the plateau represents and what the jump represents.
O3. [ANALYZE] Someone stops training at episode 1000 because the reward has been flat for 750 episodes. What have they lost, and how could they have detected the problem?
O4. [OUT] The agent learned Q[start, 'v'] = 3.21 and Q[start, '>'] = 0.90. Verify the second
of these by hand, given gamma = 0.95, a step reward of −1, and a small prize of +2 two moves
away.
O5. [OUT] Read the learned policy and write out the full route from S to G:
v > t ^
v X X ^
> v v X
> > > G
Applying it
P1. [PROG] Modify the grid to add a second pit, and retrain. Print the resulting policy.
P2. [PROG] Write a loop that trains with gamma in [0.5, 0.8, 0.95, 0.99] and prints which
terminal state the greedy policy ends at for each. Explain the pattern.
P3. [PROG] Change the small prize from +2 to +8 and find, by experiment, the smallest ε that
still discovers the jackpot.
P4. [ANALYZE] You are building a delivery-route agent. You can simulate routes cheaply but each real delivery is expensive. Explain how this changes how much exploration you can afford, and where you would do it.
P5. [ANALYZE] Explain why autonomous driving suits reinforcement learning, referring to what would be needed to attack it with supervised learning instead.
Quick self-check
- Define reinforcement learning in one sentence.
- What does the agent get for a good action? For a bad one?
- What is the agent's primary goal?
- Name the five core components of a reinforcement learning problem.
- What is a policy?
- Why is reinforcement learning not unsupervised learning?
- What does a reward tell the agent, and what does it not tell it?
- What is the exploration–exploitation dilemma?
- What does ε control in ε-greedy?
- What goes wrong if ε is 0? If ε is 1?
- What is the credit assignment problem?
- Why is a flat learning curve not proof of convergence?