Reinforcement Learning

Topic - Supervised, unsupervised, and semi-supervised learning all begin with an existing table of data. Reinforcement learning begins with no dataset at all. An autonomous agent actively interacts with an external environment, collects numerical feedback, and learns optimal sequential behavior through trial and error.

This architectural shift fundamentally redefines what training data means: the dataset is generated dynamically by the agent's own actions.

---

1. The Core Concept

Reinforcement learning is a feedback-based learning paradigm in which an agent learns to make optimal decisions in an environment by performing actions and observing the resulting rewards or penalties.

• Positive Feedback (Reward): Encourages the agent to reinforce and repeat the chosen behavior.
• Negative Feedback (Penalty): Discourages the agent from selecting that action in similar situations.
• Zero Ground Truth Labels: The agent is never told the correct action. It is only informed, after taking an action, of the numerical score it collected. It must work backwards from scalar feedback to deduce an optimal strategy.

The ultimate goal of the agent is to maximize its cumulative positive reward (total return) over time.

Core Vocabulary

Reinforcement learning operates on a formal mathematical vocabulary where terms carry exact technical definitions:

Term	Technical Meaning	In the 4x4 Gridworld Simulation
Agent	The decision-making entity and learner	The autonomous navigation algorithm
Environment	The external system the agent interacts with	The 4x4 maze, walls, and state transition rules
State ($s$)	The instantaneous observation or configuration	The current coordinate $(x, y)$ occupied on the grid
Action ($a$)	A discrete or continuous decision available	Movement: Up, Down, Left, or Right
Reward ($r$)	Scalar feedback returned by the environment	+10 jackpot, +2 small prize, −10 pit, −1 per step
Policy ($\pi$)	Strategy mapping situations to actions: $\pi(s) \to a$	The complete navigation map across all 16 cells
Episode	One full trial from start to a terminal state	Navigation run until hitting the goal, a pit, or 100 steps
Return ($G$)	Cumulative discounted reward accumulated in an episode	$G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}$
Value ($V$ or $Q$)	Expected long-term return from a state or state-action pair	How promising a specific square or action is long-term

The policy ($\pi$) is the actual output of reinforcement learning. Where supervised learning produces a model mapping features to static predictions, reinforcement learning produces a policy mapping environmental states to sequential behaviors.

---

Interactive Checkpoint: Reinforcement Learning Terminology

Scenario 1: An autonomous vehicle central computer decides to "Apply maximum braking pressure" after detecting an obstacle. What core RL component does this decision represent?

Scenario 2: The vehicle sensors report: "Object detected at 10 meters, speed is 50 km/h, road is wet". What component does this represent?

Scenario 3: The overall driving software that maps any arbitrary sensor input into steering, throttling, and braking commands represents the:

---

2. The Agent-Environment Loop

Every reinforcement learning problem operates through a continuous, closed-loop cycle of observation, action, and evaluation:

The Two Primary Algorithmic Challenges

Two structural realities distinguish reinforcement learning from standard machine learning:

1. Actions Alter Future Data Distributions: In supervised learning, the dataset is stationary and fixed before training begins. In reinforcement learning, the agent's actions dictate which states it visits next. If an early policy is poor, the agent wanders exclusively through low-reward states, never discovering high-value regions. The training distribution is non-stationary and coupled to the policy being learned.
2. Delayed Feedback & Credit Assignment: Rewards are rarely immediate. An opening move in chess may determine a victory forty turns later, but the game-winning reward (+1) arrives only at checkmate. Apportioning credit or blame to intermediate actions taken long before the terminal state is called the Credit Assignment Problem.

Comparative Paradigm Breakdown

Dimension	Supervised	Unsupervised	Semi-Supervised	Reinforcement
Training Data	Pre-existing labelled rows $(X, y)$	Pre-existing unlabelled rows $(X)$	Both labelled and unlabelled	No dataset: generated via active trial
Feedback Type	Explicit target answers	No feedback	Partial target answers	Evaluative scalar reward ($r$), not an answer
Told What is Right?	Yes	Never	Sometimes	No - only told how good or bad an action was
Data Distribution	Fixed prior to training	Fixed prior to training	Fixed prior to training	Dynamic: determined by the policy actions
Core Objective	Minimize prediction loss	Discover latent clusters/patterns	Generalize with sparse labels	Maximize cumulative return ($G_t$)
Learned Artifact	Predictive model ($\hat{y} = f(X)$)	Cluster or density structure	Semi-supervised classifier	Behavioral policy ($\pi: s \to a$)

note

A Reward is Not a Label, and it is Not Nothing

• Supervised learning is told: "The true answer was Malignant."
• Unsupervised learning is told: Nothing at all.
• Reinforcement learning is told: "Taking that step earned you a score of -1."

A reward evaluates how well the agent performed, but never reveals what the agent should have done instead. Evaluating action A yields zero information about alternative actions B or C. The agent is forced to actively explore unknown alternatives to discover superior policies.

---

3. Mathematical Formulation: Q-Learning

To solve the credit assignment problem and determine optimal policies, reinforcement learning algorithms estimate the Action-Value Function $Q(s, a)$, representing the expected cumulative reward of taking action $a$ in state $s$ and following policy $\pi$ thereafter.

In tabular Q-Learning, updates follow the Bellman Optimality Equation:

$Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$

Mathematical Decomposition

Click through the tabs below to dissect the components of the Bellman update rule:

1. Current Estimate Q(s, a)

• What it represents: The current stored expectation of the total reward achievable by executing action $a$ in state $s$.
• Algorithmic Role: Acts as the baseline anchor. When the table is initialized, these values are typically set to zero.

2. Learning Rate (α)

• What it represents: The step size parameter ($\alpha \in (0, 1]$) determining how aggressively new empirical observations overwrite prior beliefs.
• Algorithmic Role: Setting $\alpha = 0.1$ ensures stable, gradual updates in stochastic environments. Setting $\alpha = 1.0$ completely replaces the old estimate with the most recent experience.

3. TD Target (r + γ max Q)

• What it represents: The Temporal Difference (TD) Target — the immediate reward $r$ observed, plus the discounted value of the best possible action in the subsequent state $s'$.
• Algorithmic Role: Provides the newly observed reality. Because it uses the agent's own estimates to update itself, Q-learning is a bootstrapping algorithm.

4. Discount Factor (γ)

• What it represents: The discount rate ($\gamma \in [0, 1)$) defining how heavily the agent values future rewards compared to immediate ones.
• Algorithmic Role: When $\gamma \to 0$, the agent is short-sighted, caring only about immediate payoffs. When $\gamma \to 1$, the agent is far-sighted, willingly enduring negative step penalties to secure distant high-value jackpots.

5. TD Error (Target - Current)

• What it represents: $\delta = \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$ — the discrepancy between the expected return and the observed outcome.
• Algorithmic Role: Drives policy improvement. If the TD error is positive, taking action $a$ yielded a better outcome than expected, increasing $Q(s, a)$.

---

4. The Exploration-Exploitation Dilemma

At every decision point, an agent faces a fundamental trade-off:

• Exploitation: Choose the best action known from current memory to maximize immediate score.
  • Failure Mode: If the agent exploits too early, it locks onto the first mediocre reward it finds (premature convergence) and never discovers that a 5x larger jackpot exists elsewhere.
• Exploration: Choose an unvisited or random action to uncover new environmental dynamics.
  • Failure Mode: If the agent explores too much, it wastes time taking actions it already knows are disastrous (such as walking into pits), driving its learning score deeply negative.

The ε-Greedy Strategy

The industry-standard solution is the $\epsilon$-greedy selection rule:

$a = \begin{cases} \operatorname{argmax}_a Q(s, a) & \text{with probability } 1 - \epsilon \quad \text{(Exploit)} \\ \text{Random action from } \mathcal{A} & \text{with probability } \epsilon \quad \text{(Explore)} \end{cases}$

The parameter $\epsilon \in [0, 1]$ acts as a tuning dial between greed and curiosity.

---

Interactive 4x4 Gridworld & Exploration Visualizer

Observe how changing the exploration rate $\epsilon$ determines whether an agent locks onto the nearby small prize $(t, +2)$ or navigates around the penalty pits $(X, -10)$ to reach the distant jackpot $(G, +10)$:

[AGENT](0,0)

.(0,1)

t (+2)(0,2)

.(0,3)

.(1,0)

X (-10)(1,1)

X (-10)(1,2)

.(1,3)

.(2,0)

.(2,1)

.(2,2)

X (-10)(2,3)

.(3,0)

.(3,1)

.(3,2)

G (+10)(3,3)

ε = 0.50 (Balanced Curiosity)

Willingness to explore 50% of the time enables the agent to endure negative step penalties, bypass the local small prize, circumnavigate the pits, and discover the +10 jackpot.

Terminal Outcome

Global Optimum Discovered

Final Policy Value

+5.00

The Empirical Cost of Curiosity

Every row in the benchmark below executes the exact same Q-learning algorithm with identical random seeds for 3,000 training episodes on the 4x4 Gridworld. Only the exploration parameter $\epsilon$ differs:

Exploration Rate ($\epsilon$)	Strategy Classification	Average Reward While Learning	Value of Final Learned Policy	Realized Outcome
$\epsilon = 0.00$	Pure Exploitation	$+1.00$	$+1.00$	Permanently locked onto small prize $t$
$\epsilon = 0.10$	Low Exploration	$+0.79$	$+1.00$	Trapped on local plateau
$\epsilon = 0.30$	Moderate-Low	$+0.44$	$+1.00$	Trapped on local plateau
$\epsilon = 0.50$	Balanced Curiosity	$-4.73$	$+5.00$	Discovers optimal Jackpot route (5x payoff)
$\epsilon = 1.00$	Pure Randomness	$-12.17$	$+5.00$	Finds jackpot, but pays massive exploration penalty

Key Insights from the Empirical Data:

1. Low $\epsilon$ Yields Confident, Permanent Failure: Agents with $\epsilon \in [0.00, 0.30]$ all converge confidently on a policy worth $+1.00$. This is not bad luck; it is structural. Because reaching the small prize terminates the episode in 2 steps, an agent lacking curiosity never receives the opportunity to explore the lower quadrants.
2. Curiosity Has an Inescapable Price: Inspect the Reward While Learning column. As $\epsilon$ rises, score during training falls monotonically: $+1.00 \to +0.79 \to +0.44 \to -4.73 \to -12.17$. The $\epsilon = 0.00$ agent looks like the top performer during training, yet delivers the worst policy at test time.
3. Flat Learning Curves are Deceptive: An agent can sit on a flat reward plateau for 2,000 consecutive episodes looking perfectly converged, only to experience an explosive breakthrough when an exploratory sequence unlocks the global jackpot.

---

5. Real-World Applications & The RLHF Pipeline

Reinforcement learning excels in complex, continuous control domains where writing explicit analytical rules is impossible, but evaluating behavioral outcomes is straightforward:

Domain	The Learning Agent	The Environment	Objective Reward Signal
Autonomous Vehicles	Motion planning controller	Highway traffic, lane bounds, pedestrians	Progress made, lane centering, comfort, crash penalties
Robotic Locomotion	Actuator torque controller	Physical terrain, gravity, friction	Distance covered, energy efficiency, balance preservation
Resource Scheduling	Cloud compute orchestrator	Kubernetes cluster workloads	CPU utilization, response latency, operational server costs
Quantitative Trading	Order execution algorithm	Financial order book dynamics	Net profit, Sharpe ratio, drawdown risk penalties
Generative AI Alignment	Large Language Model (LLM)	Conversational dialogue turns	Human helpfulness, truthfulness, and safety scores

Deep-Dive: Reinforcement Learning from Human Feedback (RLHF)

The most transformative modern application of reinforcement learning is not in robotics or video games, but in Generative AI — fine-tuning Large Language Models (LLMs) to align with human intent, safety, and utility.

Pre-trained LLMs trained solely on next-token prediction (Self-Supervised Learning) mimic text found across the internet, making them prone to hallucinations, toxicity, or unhelpful answers. RLHF transforms raw foundation models into reliable assistants through a 3-stage alignment process.

---

Interactive 3-Stage RLHF Pipeline Stepper

Step through the pipeline that bridges raw token prediction to aligned conversational assistants:

Stage 1: Supervised Fine-Tuning & Human Ranking

Key Entity: Human / AI Raters

Operational Input: Prompt: "How do I build a simple solar dehydrator?"

Mechanism: The base model generates 4 candidate completions. Human or high-capability AI raters rank them from best to worst based on helpfulness, accuracy, and safety: [Completion B > Completion A > Completion C > Completion D].

Outcome: Curated preference dataset of pairwise ranked completions (A preferred over B).

---

6. Implementation Lab

tip

Implementation Lab: Q-Learning Gridworld Simulation

Execute a standalone NumPy implementation of a Q-learning agent navigating a 4x4 Gridworld containing jackpot rewards (+10), small prize traps (+2), and terminal penalty pits (-10). Experiment with varying exploration rates $\epsilon$ live to observe premature convergence and breakthroughs.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the notebook.
2. Click "Runtime" > "Run all" or execute cells individually using Shift + Enter.
3. Follow the guided experiments to test parameters such as episodes=10000, gamma=0.5, or STEP_R=0.0.

Comparative Implementation (Production vs. From Scratch)

Compare a high-level environment loop against the core vectorized Q-learning update step written in NumPy from scratch:

1. Production Environment Loop (Gymnasium)

import gymnasium as gym

# Initialize environment
env = gym.make("FrozenLake-v1", is_slippery=False)
state, info = env.reset()

# Execution loop
for step in range(100):
    # Select action using policy or epsilon-greedy
    action = env.action_space.sample() 
    next_state, reward, terminated, truncated, info = env.step(action)
    
    if terminated or truncated:
        state, info = env.reset()
    else:
        state = next_state
        
env.close()

2. Tabular Q-Learning (NumPy from Scratch)

import numpy as np

def update_q_table(Q, s, a, r, s_next, alpha=0.1, gamma=0.95):
    """
    Executes a single Bellman Optimality update:
    Q(s, a) <- Q(s, a) + alpha * [r + gamma * max_a' Q(s', a') - Q(s, a)]
    """
    best_future_q = np.max(Q[s_next])
    td_target = r + gamma * best_future_q
    td_error = td_target - Q[s, a]
    
    # In-place update
    Q[s, a] += alpha * td_error
    return Q

---

7. Common Mistakes

1. Unsupervised Fallacy

• The Misconception: "Reinforcement learning is just an advanced form of unsupervised learning because neither method uses ground-truth labels."
• Wrong Thinking: Assuming that absence of labels means absence of feedback.
• The Right Principle: Unsupervised learning receives zero feedback and simply clusters input features $X$. Reinforcement learning receives numerical feedback signals (rewards/penalties) that explicitly direct learning. It has evaluation without explicit answers.

2. Reward = Label

• The Misconception: "A reward of -1 tells the agent that its chosen action was incorrect, acting like a training label."
• Wrong Thinking: Believing scalar feedback reveals what the agent should have done.
• The Right Principle: A reward evaluates how good an action was, but never reveals what the optimal action was. Testing action A tells the agent nothing about actions B or C, which forces continuous exploration.

3. Policy vs. Predictive Model

• The Misconception: "Reinforcement learning outputs a predictive regression or classification model."
• Wrong Thinking: Looking for a static function that predicts target variables from fixed features.
• The Right Principle: RL produces a policy ($\pi$) — an active decision function mapping dynamic environmental states to optimal sequential actions.

4. Epsilon Extremes (ε = 0 vs. ε = 1)

• The Misconception: "Setting ε = 0 guarantees maximum performance because the agent always picks its best action."
• Wrong Thinking: Prioritizing immediate exploitation from the start.
• The Right Principle: With $\epsilon = 0$, the agent suffers premature convergence, locking onto the first mediocre reward it encounters. Conversely, $\epsilon = 1.0$ wanders randomly forever without leveraging its learned value function.

5. Trusting Flat Learning Curves

• The Misconception: "The training curve has been flat for 1,000 episodes, so the agent has fully converged to the optimal policy."
• Wrong Thinking: Confusing a local plateau with global convergence.
• The Right Principle: RL agents can remain trapped on sub-optimal plateaus for thousands of episodes before random exploration uncovers a superior path. Always test policy variance and probe with higher exploration before halting training.

6. Evaluating Online Training Scores

• The Misconception: "Our agent learning curve is negative, so the algorithm is failing to learn."
• Wrong Thinking: Judging an agent by the rewards it collects while actively exploring.
• The Right Principle: Exploratory actions intentionally choose suboptimal moves to gather information, pulling online scores down. To measure true capability, periodically test the greedy policy ($\epsilon = 0$) on separate clean evaluation runs.

---

8. Summary

Core Mechanics

• No Pre-existing Dataset: The agent generates its own experience dynamically through environmental interaction.
• Evaluative Feedback: Numerical rewards/penalties tell the agent how well it performed, but never provide the correct answer.
• Objective: Maximize cumulative discounted return ($G_t$) across episodes.

Engineering Realities

• Output Artifact: Produces a policy ($\pi: s \to a$) mapping states to sequential behavioral choices.
• The Exploration Dilemma: High exploration ($\epsilon$) lowers training rewards but is mandatory to discover global optima.
• Primary Domains: Robotics, autonomous driving, resource scheduling, and LLM alignment (RLHF via PPO/DPO).

info

Key Takeaways

• Reward != Label: Rewards evaluate the selected action but reveal nothing about untried alternatives. Exploration is mathematically mandatory.
• The Cost of Curiosity: High exploration lowers immediate scores during training, but yields 5x higher policy values by escaping local traps.
• Credit Assignment Problem: Delayed terminal rewards require temporal difference bootstrapping (Q-learning) to propagate value backwards to early choices.
• Stationarity Assumption Broken: Actions change future observations; poor early policies alter the training distribution.

Next in this section: The Toolkit and the Pipeline - the libraries and workflow that implement all four paradigms.

---

9. Active Recall Flashcards

Attempt each conceptual diagnostic question first, then click to reveal the underlying derivation and explanation.

1. [THEORY] Define reinforcement learning and explain why it is fundamentally distinct from unsupervised learning.

• Definition: Reinforcement learning is a feedback-based paradigm where an agent learns to make a sequence of decisions in an environment by executing actions and receiving scalar rewards or penalties.
• Distinction from Unsupervised Learning: Unsupervised learning receives zero feedback and groups static input features $X$. Reinforcement learning receives explicit numerical feedback that evaluates the quality of each action, driving iterative policy improvement.

2. [ANALYZE] Why is a reward not equivalent to a label? How does this distinction force exploration?

• Why Reward != Label: A label in supervised learning tells the model the exact ground-truth answer it should have predicted. A reward merely states how good or bad the chosen action was.
• Why It Forces Exploration: Receiving a reward of -1 for action A gives the agent zero information about what would happen if it chose action B or C. To determine if action B yields +10, the agent must actively explore it.

3. [THEORY] What is the credit assignment problem, and why does delayed reward create it?

• Credit Assignment Problem: The challenge of determining which specific past action(s) among hundreds or thousands of steps were responsible for a final outcome.
• Why Delayed Reward Causes It: When feedback arrives only at the end of an episode (e.g., winning or losing a chess match at turn 60), intermediate moves receive zero immediate feedback. Algorithms must propagate value backwards through time to assign credit to early tactical moves.

4. [ANALYZE] In supervised learning the dataset is fixed prior to training. Why is this not true in RL, and what danger does it create?

• Why It Is Not True: The agent generates its own training data dynamically. Which state the agent visits next depends directly on the action it selects now.
• The Danger: If an early policy is poor, the agent will only ever visit low-reward states and never encounter high-reward regions. The training distribution is non-stationary and tightly coupled to the policy being trained.

5. [THEORY] Formulate the Bellman update rule for tabular Q-learning and define the Temporal Difference (TD) target.

• Update Rule: $Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$
• TD Target: The term $r + \gamma \max_{a'} Q(s', a')$, representing the immediate reward $r$ plus the discounted value of the optimal action in the subsequent state $s'$.

6. [ANALYZE] Explain the exploration-exploitation dilemma and the consequences of setting ε = 0 vs. ε = 1 in ε-greedy selection.

• The Dilemma: Balancing between trying new actions to discover better strategies (exploration) versus selecting the best-known action to maximize immediate reward (exploitation).
• Consequence of $\epsilon = 0$: The agent exploits 100% of the time, suffering from premature convergence by locking onto the first mediocre reward it finds.
• Consequence of $\epsilon = 1$: The agent acts completely randomly forever, accumulating massive penalties and failing to execute a learned strategy.

7. [OUT] In the empirical benchmark, why did agents with ε between 0.00 and 0.30 all converge on a policy worth only +1.0 instead of the +5.0 jackpot?

• Mechanism: Reaching the nearby small prize ($t, +2$) terminated the episode after only two steps. Because exploration was low ($\le 0.30$), the agents rapidly discovered this local reward and greedy exploitation dominated. Because the episode terminated immediately upon touching $t$, the agent never had an opportunity to explore the lower quadrants where the $+10$ jackpot lay.

8. [ANALYZE] Why does average reward during training fall as exploration rate ε increases? What is this phenomenon called?

• Why It Falls: Higher $\epsilon$ forces the agent to take random actions more frequently. Even when the agent has learned which actions are disastrous, exploration forces it to test them, stepping into penalty pits and incurring step costs.
• The Phenomenon: This is the literal Cost of Curiosity. The best-performing agent during learning ($\epsilon = 0$) delivers the worst final policy, while the agent that explores ($\epsilon = 0.5$) pays an immediate training cost to secure a 5x superior final policy.

9. [THEORY] Why is a flat learning curve over hundreds of episodes not proof of optimal convergence in reinforcement learning?

• Because the agent can easily be trapped on a local optimum plateau. Until an exploratory sequence of actions breaches the local barrier and stumbles upon the global reward, the value function will show zero variance and remain flat.

10. [ANALYZE] Explain why autonomous vehicle navigation suits reinforcement learning better than supervised behavioral cloning.

• Supervised Behavioral Cloning Limitations: Requires recording human drivers and mapping sensor inputs directly to steering angles. If the model makes a minor compounding error, it enters an out-of-distribution state never seen in human demonstrations, where it has no training labels and fails catastrophically.
• Reinforcement Learning Fit: Engineers can easily formulate the objective feedback rules (positive reward for progress, lane centering; negative penalties for collisions), allowing the agent to explore and learn self-correcting policies across infinite continuous scenarios.