Teaching "Learning from Mistakes" to Architect mode (ages 15–18)

Hook & Warm-Up

No mascot, no games — treat this like the opening of a technical seminar. Read the app's hook line as written:

"From tabular Q-learning to deep RL with PPO and SAC, reinforcement learning has enabled superhuman performance in games and increasingly in real-world control. Let's examine the theoretical foundations and modern algorithms."

Frame the stakes before diving in: "Reinforcement learning is the technique behind AlphaGo, most of modern robotics research, and — less obviously — behind how conversational AI tools like ChatGPT are fine-tuned after their initial language training. If any of you are considering AI, robotics, or ML engineering as a career path, RL is one of the harder and more actively-researched subfields, and today is your first real exposure to why." Write today's roadmap on the board: value-based methods (Q-learning, DQN), policy-based methods (REINFORCE, PPO), actor-critic methods, and RLHF.

Main Activity

This age band gets the full technical picture the app itself only summarizes. Work through it in four parts, building each on the last.

Part 1: The loop and the credit assignment problem

Re-derive the RL loop precisely: an agent in state s takes action a, the environment returns a reward r and a new state s'. The agent's objective is to maximize cumulative reward over time, not just the immediate reward — which immediately raises a hard problem: "If you win a chess game 40 moves after a mistake, which of those 40 moves actually caused the loss?" This is the credit assignment problem — figuring out which past actions in a sequence are responsible for a reward that arrived much later, especially when rewards are sparse (only given rarely, like "win" or "lose" at the very end of a game rather than after every move). Note that this is exactly what makes RL harder than supervised learning: supervised learning gets an immediate, per-example correct answer; RL sometimes has to wait dozens or thousands of steps to find out if an early decision was any good.

Part 2: Value-based methods — Q-learning to DQN

Introduce the Q-function precisely: Q(s, a) estimates the total expected future reward of taking action a in state s, then acting optimally afterward. Classic (tabular) Q-learning stores one value per state-action pair and updates it as the agent gains experience — which works fine for small problems (a grid, a simple board), but breaks down for anything with a huge or continuous state space, like a video frame from an Atari game or a robot's sensor readings. DeepMind's Deep Q-Network (DQN), published in 2015, solved this by using a neural network to approximate Q(s, a) instead of a lookup table, letting the same approach scale to raw pixel input. Two tricks made DQN stable enough to actually work: experience replay (storing past transitions and training on random samples from that memory, rather than only the most recent experience, which breaks harmful correlations between consecutive steps) and a separate, slowly-updated target network used to compute training targets, which prevents the training process from chasing its own rapidly-shifting predictions.

Part 3: Policy-based methods — REINFORCE to PPO, and actor-critic

Contrast the two big families directly: value-based methods (Q-learning, DQN) learn to estimate how good actions are, then pick the best one; policy-based methods instead directly learn a parameterized policy — a function from state to a probability distribution over actions — and adjust its parameters to make good actions more likely. The simplest version, REINFORCE, nudges the policy toward actions that led to high reward, but its updates are noisy and training can be unstable if a single update moves the policy too far in one step. Proximal Policy Optimization (PPO), from OpenAI, fixes this by clipping how much the policy is allowed to change in a single update — the "proximal" in the name — which trades a little bit of optimization speed for much more stable training, and is a major reason it became one of the most widely used RL algorithms in practice, including at OpenAI and DeepMind. Actor-critic methods (and more advanced variants like Soft Actor-Critic, SAC) combine both families: an "actor" network picks actions like a policy-based method, while a "critic" network estimates values like a Q-learning-style method, and the critic's estimate is used to make the actor's updates lower-variance and more sample-efficient. Sample efficiency matters enormously in practice: DQN and PPO can need millions of environment interactions to train well, which is fine in a fast video-game simulator but can be prohibitively slow or expensive when the "environment" is a physical robot.

Part 4: RLHF — reinforcement learning meets language models

Tie the whole lesson to something students have almost certainly used: "The reason a chatbot's responses feel helpful and appropriately cautious, rather than just statistically likely, is largely reinforcement learning." Walk through RLHF concretely: after a language model is pretrained on large amounts of text, human raters are shown multiple candidate responses to the same prompt and rank them by preference. That preference data trains a separate reward model to predict which responses humans would prefer. The original language model is then further trained with reinforcement learning (commonly a PPO-style algorithm) to produce responses that score highly according to that learned reward model. Note explicitly for the class: this is the same state-action-reward loop from Part 1, just with "state" as the conversation so far, "action" as the next words generated, and "reward" coming from a model trained on human preferences instead of a game score.

Close Part 4 with the same caution that applies to every reward-driven system in this lesson: a reward model trained on human preference rankings is still a proxy for "what humans actually want," not the thing itself. If raters systematically prefer confident-sounding answers over answers that honestly express uncertainty, the resulting model can learn to sound more confident than its actual accuracy warrants — a real, documented failure mode in RLHF-trained systems, and the same basic reward-hacking pattern discussed under reward shaping in the discussion questions below, just showing up in language instead of a game score: the model is optimizing exactly what the reward model scores highly, which is not perfectly identical to what the human raters actually wanted. Emphasize that this isn't a hypothetical concern for a future version of AI; it's an active area of research for the exact tools students may already use.

Discussion

• Why is the credit assignment problem harder in RL than in supervised learning? Give a concrete example of a decision with a long delay before its consequence becomes clear — it doesn't have to be about AI.
• DQN needed both experience replay and a target network to train stably. What do you think would go wrong if a neural-network-based Q-learning system trained only on its most recent experience, with no target network?
• Why might a company choose PPO over more sample-efficient methods like SAC for training a large language model, even if PPO needs more data?
• If RLHF trains a model to produce responses a reward model scores highly, what happens if the reward model itself has blind spots or biases from the human raters who trained it? What would that failure look like in a chatbot's behavior?
• Reward shaping (giving partial credit along the way, not just at the end) can speed up training but risks the agent optimizing the shaped reward instead of the real goal. Can you design a reward-shaping scheme for a self-driving car that you suspect could be exploited, and explain how?
• Sparse rewards (only scoring at the very end of a long task) make credit assignment harder but are less prone to being gamed than densely shaped rewards. Is that tradeoff worth it? Under what circumstances would you pick one over the other?
• If a company fine-tunes a language model with RLHF using raters from one country or cultural background, what kinds of bias might that introduce into the model's behavior for users elsewhere?

Quiz Walkthrough

Wrap-Up & Extension

Close with: "Everything from AlphaGo to the chatbot fine-tuning pipeline behind tools you already use runs on the same underlying loop you learned today: try, get scored, adjust. The algorithms (DQN, PPO, SAC) are just increasingly sophisticated ways of making that loop stable and efficient enough to work on hard, real-world problems. If this interested you, reinforcement learning is a legitimate specialization inside AI/ML careers — it's a smaller field than general machine learning engineering, which can actually make it easier to stand out with a focused project."

Extension activity — Train a Tiny RL Agent (30–45 minutes, needs internet access): Have students (individually or in pairs) find and read through a beginner-level tutorial for training a simple RL agent in a free, browser- or Python-based environment (a classic starting point is a "CartPole" or "Frozen Lake" style environment, widely used as a first exercise in RL courses). They don't need to write the code from scratch — reading and annotating an existing tutorial, identifying where the state, action, reward, and policy live in the code, is enough for this session. Have each pair present: what was the agent's state and action space, what was the reward, and did they notice any sign the agent found an unintended shortcut? This is a strong candidate for a portfolio entry — "read and explained a real RL training pipeline" is a concrete, defensible line for a student interested in an AI-adjacent college application or internship.