Reinforcement Learning

Topic — The three paradigms so far all begin the same way: a table of data arrives, and you check whether it has a label column. Reinforcement learning begins with no table at all. The agent generates its own data by acting, and finds out afterwards how that went.

That difference is not a detail. It changes what "training data" even means.

---

The definition

Reinforcement learning is a feedback-based technique in which an agent learns to behave in an environment by performing actions and seeing the results of those actions.

The two words carrying the weight are feedback-based and actions.

• For each good action, the agent gets positive feedback
• For each bad action, the agent gets negative feedback, or a penalty

Since there is no labelled data, the agent is bound to learn by its experience only. Nobody ever tells it the correct action. It is told, after the fact, how much reward it collected — and it must work backwards from that to figure out what it should have done.

The primary goal of the agent is to improve its performance by getting the maximum positive rewards.

---

The vocabulary

Reinforcement learning has more moving parts than the other paradigms, and the terms are used precisely.

Term	Meaning	In the grid demo below
Agent	The learner and decision-maker	The thing walking around
Environment	Everything outside the agent, which it acts upon	The 4×4 grid and its rules
State	The situation the agent is currently in	Which square it stands on
Action	A choice available to the agent	Move up, down, left or right
Reward	The numeric feedback for an action	+10 jackpot, −10 pit, −1 per step
Policy	The agent's strategy — state → action	"On this square, go down"
Episode	One run from start to a terminal state	Start until it hits goal or pit
Return	Total reward accumulated over an episode	The episode's score
Value	Expected return from a state onwards	How promising this square is

The policy is the actual output of reinforcement learning. Where supervised learning produces a model that maps features to a prediction, reinforcement learning produces a policy that maps situations to behaviour.

---

🧠 Interactive Checkpoint: Reinforcement Learning Terminology

Scenario 1: An autonomous vehicle's central computer decides to "Apply maximum braking power" after detecting an obstacle. What core RL component does this decision represent?

Scenario 2: The vehicle's camera, LiDAR, and speed sensors report: "Object detected at 10 meters, current vehicle speed is 50 km/h, lane position is centered". What component does this represent?

Scenario 3: The overall driving software of the vehicle, which converts raw sensor readings into specific control commands (steering, accelerating, braking), represents the:

The loop

Everything in reinforcement learning is this cycle, repeated:

              ┌─────────────────────────────────────┐
              │                                     │
              ▼                                     │
        ┌──────────┐         action a          ┌────┴─────┐
        │          │ ────────────────────────► │          │
        │  AGENT   │                           │  ENVIRON │
        │          │ ◄──────────────────────── │   MENT   │
        └──────────┘   new state s', reward r  └──────────┘

        1. agent observes the current state
        2. agent chooses an action
        3. environment returns a new state and a reward
        4. agent updates its policy
        5. repeat

Two properties of this loop cause all the difficulty:

The agent's actions change the data it sees next. In supervised learning your dataset is fixed; here, a bad early policy means the agent only ever visits bad states and never discovers the good ones. The data distribution depends on the thing being learned.

Reward is often delayed. A chess move that loses the game may look fine when played. The agent receives one signal at the end and must apportion credit across all the moves that led there — the credit assignment problem.

---

Why it isn't the other three

	Supervised	Unsupervised	Semi-supervised	Reinforcement
Training data	Labelled rows	Unlabelled rows	Both	Generated by acting
Feedback	The correct answer	None	Partial	A reward, not an answer
Told what's right?	Yes	Never	Sometimes	No — only how good
Data fixed in advance?	Yes	Yes	Yes	No
Output	A prediction	A description	A prediction	A policy
Timing of feedback	Immediate	—	Immediate	Often delayed

The row that matters most is the third. Reinforcement learning shares "no labels" with unsupervised learning, and people therefore lump them together — but this is wrong:

note

Reward is not a label, and it is not nothing

Supervised learning is told "the answer was malignant". Unsupervised learning is told nothing at all. Reinforcement learning is told "that got you −1" — which is neither the answer nor silence.

A reward says how good an action was, never what you should have done instead. That's why the agent must try alternatives to find out: evaluating one action tells it nothing about the others.

---

The two hard problems

Exploration vs exploitation

At every step the agent faces a dilemma:

• Exploit — take the best action it currently knows about, and collect the reward
• Explore — try something else, which may be worse, but might reveal something better

Exploit too much and the agent locks onto the first decent thing it finds, never discovering that something far better exists. Explore too much and it spends its life taking actions it already knows are bad.

The standard mechanism is ε-greedy: with probability ε take a random action, otherwise take the best known one. ε is the dial between the two, and the demo below shows exactly what it costs to set it wrongly in either direction.

Credit assignment

When reward arrives late, which of the fifty actions you took deserves the credit? Reinforcement learning solves this by propagating value backwards: a state is valuable if it leads to valuable states. This is why the agent in the demo eventually learns that the square next to the start is worth something, despite that square paying nothing itself.

---

Where it's used

Reinforcement learning plays a significant role in the development of autonomous vehicles, also known as self-driving cars. The fit is natural: there is no dataset of correct steering angles for every situation, but there is abundant feedback — stayed in lane, maintained distance, reached the destination, or crashed.

Domain	The agent	The reward
Autonomous vehicles	The driving policy	Progress, safety, comfort
Game playing	The player	Winning; score
Robotics	Motor controller	Task completed, energy used
Recommendation	The ranking policy	Clicks, watch time, retention
Resource management	Scheduler	Throughput, latency, cost

The common shape: you can recognise good behaviour, but you cannot write down the correct action for every situation. That is precisely the gap reinforcement learning fills — and it's the same gap that made hand-written rules fail in the first place, one level further up.

Reinforcement Learning from Human Feedback (RLHF)

The most prominent modern application of Reinforcement Learning is not in games or robotics, but in Generative AI — specifically, fine-tuning Large Language Models (LLMs) to align with human values, safety, and utility. This technique is known as Reinforcement Learning from Human Feedback (RLHF).

Why is RLHF necessary? While next-token prediction (Self-Supervised Learning) is powerful, it can produce completions that are toxic, inaccurate, or unhelpful. RLHF fine-tunes the model's behavior by treating the LLM as an Agent and the dialogue session as the Environment.

The RLHF pipeline operates in three key phases:

1. Human Rating of Completions: A prompt is given to multiple copies or runs of the LLM. Humans (or high-quality AI raters) evaluate the generated completions, ranking them based on quality, helpfulness, and safety.
2. Training a Reward Model: These human preference rankings are used to train a separate neural network called the Reward Model. This model learns to map any prompt-completion pair to a single scalar Reward value, representing how much a human would approve of that response.
3. Policy Optimization: The base LLM is treated as a Policy (mapping prompt states to token actions). A policy optimization algorithm — most commonly Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO) — fine-tunes the LLM. The algorithm maximizes the expected reward from the Reward Model while penalizing the LLM if its predictions drift too far from the original base model (using a KL-divergence penalty to ensure stability).

Through RLHF, reinforcement learning serves as the critical bridge that transforms raw text-completion engines into safe, polite, and effective conversational assistants (such as ChatGPT, Claude, and Gemini).

---

Common Mistakes

Here are the most common conceptual pitfalls when implementing and analyzing Reinforcement Learning algorithms, compared with the correct engineering principles.

🚨 1. Unsupervised Fallacy

• The Misconception: "Reinforcement learning is just a type of unsupervised learning because it does not use human labels."
• ❌ Wrong Thinking: Treating RL as if it runs without any feedback, missing the critical difference in objective.
• ✅ The Right Principle: Unsupervised learning receives no feedback at all and is purely exploratory. Reinforcement learning receives numerical feedback (rewards/penalties). It has feedback without answers.

🚨 2. Reward = Label

• The Misconception: "A reward is just another word for a label, telling the model the correct action to take."
• ❌ Wrong Thinking: Believing that a reward of -1 or +10 directly informs the agent of the exact action it should have performed.
• ✅ The Right Principle: A reward says how good an action was, never what the agent should have done instead. Evaluating one action tells it nothing about the other choices, which forces the agent to explore alternatives.

🚨 3. Policy vs. Model

• The Misconception: "Reinforcement learning outputs a standard predictive model just like supervised learning."
• ❌ Wrong Thinking: Expecting the output of RL to be a static mapping of features to predictive target variables.
• ✅ The Right Principle: RL outputs a policy — an active strategy or function mapping situational states to optimal behaviors or actions.

🚨 4. Zero Exploration (ε = 0)

• The Misconception: "We should set the exploration rate ε to 0 from the start so the agent always takes the best-known action."
• ❌ Wrong Thinking: Trying to maximize online rewards immediately, leading to a complete lack of curiosity.
• ✅ The Right Principle: If an agent never explores, it locks onto the first mediocre option it discovers (premature convergence), failing to find far more valuable paths.

🚨 5. Total Exploration (ε = 1)

• The Misconception: "An exploration rate of ε = 1 is optimal because the agent will fully discover the environment."
• ❌ Wrong Thinking: Believing that random action selection leads to an optimal policy without any performance cost.
• ✅ The Right Principle: Setting ε = 1 makes the agent behave completely randomly forever, paying a massive cumulative penalty in rewards while gaining no additional information compared to a balanced decaying epsilon.

🚨 6. Online Reward Progress

• The Misconception: "We must evaluate our agent's overall progress by plotting its online reward accumulated during training."
• ❌ Wrong Thinking: Panicking when the learning curve of an active explorer looks highly unstable or negative.
• ✅ The Right Principle: During learning, exploration actions (random moves) will drag the online score down. To measure true progress, evaluate the greedy policy (setting ε = 0) periodically on separate clean runs.

🚨 7. Immediate Feedback Expectation

• The Misconception: "The environment should reward every single correct choice immediately."
• ❌ Wrong Thinking: Assuming that a chess move or steering adjustment can be graded individually without looking at the final outcome.
• ✅ The Right Principle: Rewards in RL are often highly delayed (e.g., winning/losing a game after hours of play). The agent must solve the credit assignment problem by propagating value backwards through time.

🚨 8. Fixed Dataset Assumption

• The Misconception: "We must gather a complete dataset of states and rewards first, then train our RL agent off-line."
• ❌ Wrong Thinking: Treating RL as a standard batch-supervised process with static inputs.
• ✅ The Right Principle: The agent's actions actively determine the future states it will visit. A bad early policy means the agent never visits optimal states, altering its own data distribution.

🚨 9. Premature Stopping

• The Misconception: "We can stop training as soon as the reward curve flattens out for a few hundred episodes."
• ❌ Wrong Thinking: Confusing a flat learning curve (local optimum or trap) with true, optimal convergence.
• ✅ The Right Principle: RL agents can remain trapped on a plateau (such as a local small prize) for thousands of episodes before a random exploration sequence triggers a breakthrough to the global optimum.

---

Summary

🎯 Core Mechanics

• Definition: Feedback-based learning where an agent learns to behave in an environment through trial and error.
• Labels: Completely absent. The agent receives numerical rewards or penalties after taking actions.
• Goal: Maximize the cumulative positive reward (total return) over a series of actions.

⚙️ Decisions & Outputs

• Output: A policy ($\pi$: state $o$ action) representing the mapping from situational states to behavioral actions.
• Key Dilemma: Exploration (trying new unknown actions) vs. Exploitation (taking the best-known actions).
• Primary Application: Autonomous vehicles, complex gaming (Go, Chess), robotics, and LLM alignment (RLHF).

📌 Key Takeaways: Reinforcement Learning

• No Fixed Dataset: The agent generates its own data dynamically through interactions. Bad early choices can lead to poor training data distributions.
• Evaluation is Delayed: Due to delayed feedback, agents face the credit assignment problem—working backwards to see which early decisions caused the final reward.
• Exploration-Exploitation Tradeoff: Controlled by parameter ε in ε-greedy. Too little exploration leads to local optima; too much leads to wasteful random walking.
• Plateaus are Deceptive: A flat learning curve is not proof of convergence. An agent can sit on a local optimum for 2,000 episodes before discovering a breakthrough.
• Curiosity has a Price: High exploration rate ε increases the cost during learning, but yields a significantly higher final policy return.

Next in this section: The Toolkit and the Pipeline — the libraries and the workflow that implement all four paradigms

See also: Types of Learning for the three paradigms this one sits beside · Applications and Major Techniques for where each technique is used

---

Run It Yourself

💻 Lab Exercise: Q-Learning Gridworld Simulation

Execute a raw, 60-line NumPy implementation of a Q-learning agent navigating a 4x4 Gridworld containing jackpot rewards (+10), small prizes (+2), and deep penalty pits (-10). Tweak exploration rate ε live to observe premature convergence and policy breakthroughs.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Run each cell sequentially (Shift + Enter).
3. Experiment with parameters such as episodes=10000, gamma=0.5, or STEP_R=0.0 to see how the agent adapts.

What to notice in that output

The learning curve tells a story in four acts. At episode 0 the greedy policy scores −100.0 — the agent wanders until the 100-step safety limit, reaching nothing. By episode 250 it scores +1.0: it has found the small prize and locked on. Then it sits at +1.0 for two thousand episodes, looking every bit like a converged solution. At episode 2500 it breaks through to +5.0. Anyone who had stopped training at episode 1000 would have shipped a policy worth a fifth of the achievable reward, with no indication anything was wrong.

The policy is readable, and it is correct. Trace the arrows from S: down, down, then right along the bottom row to G, threading between the pits. The agent derived that route from nothing but rewards.

The learned numbers match theory exactly. Q[start, 'v'] = 3.21 and Q[start, '>'] = 0.90. Hand-computing the discounted return for the jackpot path gives −1 − 0.95 − 0.95² − 0.95³ − 0.95⁴ + 0.95⁵ × 10 = 3.21, and for the two-step small-prize path −1 + 0.95 × 2 = 0.90. The algorithm didn't approximate the right answer, it found it.

Now the epsilon table, which is the real lesson. Every row runs the same algorithm with the same seed for the same 3,000 episodes. Only curiosity differs:

• ε from 0.00 to 0.30 all settle for the small prize — a final policy worth 1.0. Not a bug, not bad luck: five different agents, all confidently converged, all permanently wrong.
• ε = 0.50 finds the jackpot — worth 5.0, five times better. The only change was a willingness to act against its own best judgement half the time.
• ε = 1.00 finds the same best policy as ε = 0.50, but paid −12.17 per episode while learning instead of −4.73. So maximum exploration isn't optimal, it's just expensive — there is a sweet spot, and it isn't at either end.

Read the last column as a price. As ε rises, reward during learning falls monotonically: 1.00 → 0.79 → 0.44 → −0.21 → −1.06 → −4.73 → −12.17. That is the literal cost of curiosity. The ε = 0 agent is the best-performing agent while learning and the worst policy at the end.

tip

Why this environment needs so much exploration

ε = 0.5 looks extreme. The reason is the trap: reaching the small prize ends the episode after two steps, so a greedy agent gets almost no opportunity per episode to see anything further away. Environments that terminate early on mediocre outcomes are exactly the ones that punish under-exploration hardest.

Things worth trying:

1. Set episodes=10000 with ε=0.1. Does patience substitute for curiosity here, or not?
2. Change the small prize to +8 in REWARD. Now the local optimum is genuinely close to the jackpot — watch how much harder escaping becomes.
3. Set gamma=0.5. The agent becomes short-sighted, discounting distant reward heavily, and should prefer the near prize even at high ε.
4. Set STEP_R = 0.0. With no cost per step the agent has no incentive to find a short path — compare the policy it settles on.

---

Practice Questions

Test your understanding by working each problem out first, then click the card to reveal the detailed solution. Questions are tagged by type: [THEORY], [PROG], [OUT], or [ANALYZE].

Definitions

❓ R1. [THEORY] Define reinforcement learning in one sentence, and explain what "feedback-based" means in that definition.

• Definition: Reinforcement learning is a feedback-based paradigm in which an autonomous agent learns to make a sequence of decisions in an environment by performing actions and receiving corresponding numeric rewards or penalties.
• "Feedback-based" meaning: Unlike supervised learning where a teacher supplies the correct answer, the RL agent is only told how good or bad its action was after the fact via a reward signal. It must learn by experience which actions are optimal without explicit instruction.

❓ R2. [THEORY] What happens for a good action, and what happens for a bad one? What is the agent's primary goal?

• Good Action: The agent receives positive feedback (a positive numeric reward, such as +10).
• Bad Action: The agent receives negative feedback or a penalty (such as -10).
• Primary Goal: To maximize its cumulative reward (total return) over time through its experience.

❓ R3. [THEORY] Define each of: agent, environment, state, action, reward, policy, episode.

• Agent: The learner and decision-making entity.
• Environment: Everything outside the agent that the agent interacts with and acts upon.
• State: The current, instantaneous situation or configuration of the agent and environment.
• Action: A choice or physical output available to the agent in its current state.
• Reward: The instantaneous scalar numeric feedback returned by the environment evaluating the last action.
• Policy: The agent's overall strategy or mapping function from states to actions ($\pi(s) o a$).
• Episode: A single trial or run from the initial starting state to a terminal ending state.

❓ R4. [THEORY] "Since there is no labelled data, the agent is bound to learn by its experience only." Explain what this rules out.

• This rules out passive supervised learning from a fixed historical dataset containing human-labeled correct answers (targets $y$). There is no teacher showing the model the correct action for any given situation, meaning the agent cannot simply "mimic" answers and must instead actively interact with the environment to collect its own feedback loop.

❓ R5. [THEORY] What is the output of reinforcement learning, and how does it differ from the output of supervised learning?

• Output of RL: A policy — a behavioral strategy mapping situations (states) to optimal actions.
• Difference from Supervised: Supervised learning outputs a predictive model mapping features to static predictions (a class label in classification or a continuous number in regression). RL's policy is about active decision-making and behaviors over sequential states rather than single static predictions.

Distinguishing the paradigms

❓ D1. [ANALYZE] A colleague says reinforcement learning is a kind of unsupervised learning because neither has labels. Explain why this is wrong, using the idea of feedback.

• The colleague is incorrect because they ignore the presence of feedback.
• Unsupervised learning has absolutely no labels and no feedback at all; it simply discovers structural groupings or patterns in static data.
• Reinforcement learning has feedback (rewards or penalties) evaluating every action. While it doesn't have ground-truth answers (labels) telling it what it should have done, it does receive evaluative signals telling it how well it did, which drives learning.

❓ D2. [THEORY] Complete the sentence for all three: supervised learning is told ___, unsupervised learning is told ___, reinforcement learning is told ___.

• Supervised learning is told the correct answer (ground-truth label $y$).
• Unsupervised learning is told nothing at all (only input features $X$).
• Reinforcement learning is told how good its chosen action was (the scalar reward $r$).

❓ D3. [ANALYZE] Explain why a reward is not a label. Why does this force the agent to explore?

• Why a reward is not a label: A label is the explicit "correct" answer representing the optimal target. A reward is merely a scalar evaluation of the chosen action. It does not tell the agent what other actions would have achieved or whether an untried action is better.
• Why it forces exploration: Because receiving a reward of -1 or +2 for an action gives the agent no information about the alternative actions. To find out if a different action would yield a higher reward (like +10), the agent must actively explore those alternatives.

❓ D4. [THEORY] In supervised learning the dataset is fixed before training. Why is this not true in reinforcement learning, and what problem does that create?

• Why it's not true in RL: The dataset is generated dynamically by the agent's actions. The state the agent visits next depends entirely on the action it takes now.
• The resulting problem: If the agent has a bad early policy, it may only ever visit low-reward, poor states and never discover high-reward regions. The data distribution depends entirely on the behavior being learned, making training unstable or highly biased toward early choices.

❓ D5. [THEORY] What is the credit assignment problem, and why does delayed reward cause it?

• Credit Assignment Problem: The challenge of determining which specific action(s) among hundreds or thousands of steps taken during an episode were responsible for the final outcome.
• Why delayed reward causes it: When the reward is delayed (e.g., winning/losing a chess game at the very end), there is no immediate feedback for individual intermediate moves. The agent must work backwards to apportion credit or blame to the early decisions that set up the victory or defeat.

Exploration and exploitation

❓ E1. [THEORY] Define exploration and exploitation, and state the risk of doing too much of each.

• Exploitation: Taking the best-known action based on current knowledge to maximize immediate reward.
  • Risk of too much: The agent gets trapped in a local optimum (premature convergence) and never discovers much better global paths.
• Exploration: Trying random or untried actions to discover more about the environment.
  • Risk of too much: The agent spends too much time taking known bad actions, accumulating massive penalties and never leveraging what it has learned.

❓ E2. [THEORY] Describe the ε-greedy strategy. What does ε control?

• ε-greedy strategy: An action-selection mechanism where the agent selects a completely random action with probability $\epsilon$ (explores) and selects the action with the highest estimated value with probability $1 - \epsilon$ (exploits).
• What ε controls: The parameter $\epsilon$ (typically between 0 and 1) acts as the dial balancing exploration and exploitation.

❓ E3. [OUT] From the demo: Which agent produced the best final policy, and which performed best while learning? What did raising ε from 0.30 to 0.50 achieve? Why is ε = 1.00 not the best choice, given it also found the jackpot?

• Best final policy: The agents with $\epsilon = 0.50$ and $\epsilon = 1.00$ (both settled on the jackpot worth 5.0).
• Best performance while learning: The agent with $\epsilon = 0.00$ (average score of 1.00 while learning, avoiding all negative penalties).
• What raising $\epsilon$ from 0.30 to 0.50 achieved: It increased curiosity enough to escape the local optimum (small prize) and discover the global jackpot.
• Why $\epsilon = 1.00$ is not the best choice: Because even after finding the jackpot path, it continued to act 100% randomly during training, accumulating a massive average cost of -12.17 per episode. It is highly wasteful.

❓ E4. [ANALYZE] The last column falls steadily as ε rises: 1.00, 0.79, 0.44, −0.21, −1.06, −4.73, −12.17. Explain what this column is measuring and why it must behave this way.

• What it measures: The average reward accumulated while learning (specifically, during the last 500 training episodes).
• Why it behaves this way: Because higher $\epsilon$ forces the agent to make random actions more frequently. Even if the agent knows the perfect path, it is structurally forced to make random moves (which often lead to pits or walls, incurring step penalties), pulling the training average down. This represents the literal cost of curiosity.

❓ E5. [ANALYZE] Five agents with ε from 0.00 to 0.30 all converged confidently on a policy worth one fifth of the best available. Explain the mechanism, and say what this implies about trusting a flat learning curve.

• The Mechanism: The nearby small prize terminated the episode after only two steps. Because the agents had low exploration rates, they quickly found this small prize and exploitation dominated. Since the episode ended immediately, they had no opportunity to explore the states further down the grid where the jackpot was located.
• Implication: A flat learning curve is not proof of optimal convergence. An agent can look perfectly stabilized and converged, but be locked in a severe local optimum.

Reading the output

❓ O1. [OUT] The greedy policy scored −100.0 after 0 episodes. Given the step reward is −1 and there is a 100-step safety limit, explain that number.

• At episode 0, the Q-table is initialized to all zeros. Followed greedily, the agent has no preferences and simply walks randomly (or takes default actions). Since it fails to find the small prize or a pit, it runs into the 100-step safety cutoff, accumulating a step reward of -1 for each of the 100 steps, resulting in exactly -100.0.

❓ O2. [OUT] The learning curve sat at +1.0 from episode 250 to 2250, then jumped to +5.0. Name what the plateau represents and what the jump represents.

• The Plateau: Represents the local optimum where the agent's greedy policy has locked onto the nearby small prize (return +1.0).
• The Jump: Represents a policy breakthrough where a random exploration step finally found the path to the jackpot, propagating high Q-values backward and updating the greedy policy to the global optimum (return +5.0).

❓ O3. [ANALYZE] Someone stops training at episode 1000 because the reward has been flat for 750 episodes. What have they lost, and how could they have detected the problem?

• What they lost: They lost the optimal policy (worth 5.0 instead of 1.0 — a 5x improvement).
• How to detect the problem: They could run separate exploratory diagnostics, monitor the Q-value variances, or test higher exploration rates ($\epsilon$) to see if the flat line was due to premature convergence.

❓ O4. [OUT] The agent learned Q[start, 'v'] = 3.21 and Q[start, '>'] = 0.90. Verify the second of these by hand, given gamma = 0.95, a step reward of −1, and a small prize of +2 two moves away.

• The state starting with action > leads directly to the small prize t in exactly 2 moves.
• Step 1: Takes action >, receives step reward -1.
• Step 2: Takes the next action to land on t, receives reward +2, terminating the episode.
• Applying the discount factor $\gamma = 0.95$: $ext{Discounted Return} = -1 + \gamma imes 2 = -1 + 0.95 imes 2 = -1 + 1.90 = 0.90$
• The calculated value matches the agent's learned expectation of 0.90 exactly.

❓ O5. [OUT] Read the learned policy and write out the full route from S to G.

Given the learned policy map:

   v  >  t  ^
   v  X  X  ^
   >  v  v  X
   >  >  >  G

• Start at S (0,0). Policy is v (go down) $o$ lands on (1,0).
• At (1,0), policy is v (go down) $o$ lands on (2,0).
• At (2,0), policy is > (go right) $o$ lands on (2,1).
• At (2,1), policy is v (go down) $o$ lands on (3,1).
• At (3,1), policy is > (go right) $o$ lands on (3,2).
• At (3,2), policy is > (go right) $o$ lands on (3,3) which is G (jackpot).
• Full Route: (0,0) → (1,0) → (2,0) → (2,1) → (3,1) → (3,2) → (3,3) [G] (avoiding all pits X).

Applying it

❓ P1. [PROG] Modify the grid to add a second pit, and retrain. Print the resulting policy.

To add an extra pit, you would modify the environment's GRID representation. For example, placing an extra pit X at coordinate (0,1):

GRID = ["SXt.",   # Added 'X' at index 1
        ".XX.",
        "...X",
        "...G"]

Retraining with this grid forces the agent to learn a policy that navigates down immediately from start, avoiding the new pit X at the top.

❓ P2. [PROG] Write a loop that trains with gamma in [0.5, 0.8, 0.95, 0.99] and prints which terminal state the greedy policy ends at for each. Explain the pattern.

The following python snippet outlines the loop:

for g in [0.5, 0.8, 0.95, 0.99]:
    Q, _, _ = train(epsilon=0.5, gamma=g, episodes=3000)
    total, ending = run_greedy(Q)
    print(f"Gamma: {g:.2f} -> Settles on: {ending} (Return: {total:.1f})")

• Pattern Explanation: Lower values of $\gamma$ (like 0.5) make the agent short-sighted, prioritizing the immediate small prize t even if a jackpot is nearby. Higher values of $\gamma$ (like 0.95 or 0.99) make the agent far-sighted, willing to take multiple steps of -1 penalty to secure the high-value jackpot G.

❓ P3. [PROG] Change the small prize from +2 to +8 and find, by experiment, the smallest ε that still discovers the jackpot.

With a small prize of +8, the local optimum is extremely attractive. An agent using standard low curiosity will easily lock onto it. Under the hood, you can run a grid search over $\epsilon$:

REWARD["t"] = 8.0 # Update reward in script
for eps in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7]:
    Q, _, _ = train(epsilon=eps, episodes=5000)
    _, ending = run_greedy(Q)
    print(f"Epsilon {eps:.1f} found ending: {ending}")

Typically, you will find that a higher minimum $\epsilon$ (such as 0.4 or 0.5) is required to break the powerful pull of the local +8 reward.

❓ P4. [ANALYZE] You are building a delivery-route agent. You can simulate routes cheaply but each real delivery is expensive. Explain how this changes how much exploration you can afford, and where you would do it.

• How it changes exploration: You cannot afford real-world exploration because taking random actions (e.g., driving into a river or making a 500-mile detour) is extremely expensive and dangerous.
• Where you would do it: You must perform 100% of your exploration inside a high-fidelity simulator (cheap, safe, and fast). Once the agent learns a robust policy in simulation, you deploy it to the real world with exploration disabled ($\epsilon = 0$) or set to a highly restricted safety-fallback mode.

❓ P5. [ANALYZE] Explain why autonomous driving suits reinforcement learning, referring to what would be needed to attack it with supervised learning instead.

• Why RL suits it: Because we can easily write down feedback rules (rewards for staying in lane, matching speed limits; severe penalties for crashes), but we cannot write down the correct physical control commands for every infinite scenario.
• What Supervised would need: Supervised learning would require millions of hours of human driving data (features: camera feeds; targets: steering angles/brake pressures). This is called Behavioral Cloning, but it suffers from severe distribution shift: if the model makes a tiny error, it gets off-track into states never seen in the training data, where it has no labels and fails catastrophically.

Quick self-check

❓ 1. Define reinforcement learning in one sentence.

A feedback-based learning paradigm where an agent learns to make a sequence of decisions in an environment by performing actions and receiving rewards or penalties.

❓ 2. What does the agent get for a good action? For a bad one?

A numeric reward for a good action, and a penalty (or negative reward) for a bad action.

❓ 3. What is the agent's primary goal?

To maximize the cumulative positive reward (total return) over time.

❓ 4. Name the five core components of a reinforcement learning problem.

Agent, Environment, State, Action, and Reward.

❓ 5. What is a policy?

The agent's strategy mapping states to actions ($\pi(s) o a$).

❓ 6. Why is reinforcement learning not unsupervised learning?

Because reinforcement learning receives evaluative numerical feedback (rewards/penalties), whereas unsupervised learning receives no feedback at all.

❓ 7. What does a reward tell the agent, and what does it *not tell it?

It tells the agent how good the chosen action was, but it never tells it what the correct action was or what other choices would have achieved.

❓ 8. What is the exploration–exploitation dilemma?

The choice between trying new actions to find better strategies (exploration) vs taking the best-known action to secure immediate reward (exploitation).

❓ 9. What does ε control in ε-greedy?

The probability $\epsilon$ of choosing a completely random exploratory action instead of exploiting the best-known option.

❓ 10. What goes wrong if ε is 0? If ε is 1?

• If $\epsilon = 0$: The agent never explores, locking permanently onto the first mediocre option it finds.
• If $\epsilon = 1$: The agent acts completely randomly forever, accumulating massive penalties and never utilizing its learning.

❓ 11. What is the credit assignment problem?

The difficulty of determining which specific past action(s) caused a delayed reward received at the end of an episode.

❓ 12. Why is a flat learning curve not proof of convergence?

Because the agent could be trapped on a plateau (local optimum) for thousands of episodes before a random exploration breakthrough reveals the global jackpot.