AIQ
AIQ
Part of the Learning from Mistakes lesson guide. Teaching a different grade? 🌈 Explorer (5–7) · 🔧 Builder (8–10) · ⚡ Architect (15–18)
Hacker mode drops the softer framing — open with the real question. Write on the board: "How would you program a computer to play a game it has never seen, with no strategy guide, no example games, and no hints?" Take a few guesses, then read the app's hook directly:
"What if AI learns by trial and error, like learning to ride a bike? That's reinforcement learning. It powers game-playing AI and robots. Let's explore how."
Push it further: "The previous lesson's AI learned from labeled examples — someone already knew the answer and showed it to the AI. Today's problem is harder: nobody knows the answer in advance. The only thing the AI gets back is a score after it acts. Keep that distinction in your head as we go — it's the whole reason reinforcement learning exists as its own separate approach."
Build up the full RL loop using the app's own vocabulary, then apply it to each real example. Write this loop on the board and refer back to it constantly: State → Action → Reward → New State, repeating.
Walk through it concretely with the Game AI example: "The AI observes its current situation (the state — think: the current game board, or the game score, or a robot's joint angles). It picks an action (a move, a step). The environment returns a reward (+1 for scoring, −1 for losing a life, 0 for nothing notable) and a new state. That's one step of the loop, and it repeats — potentially millions of times — with the AI slowly shifting its choices toward actions that historically led to higher reward." Ask: "Why can't the AI just calculate the single best move immediately, the way a calculator solves an equation?" (Because it doesn't have a formula for "best move" — the only way to find out an action's value is to try it, many times, in many situations, and see what tends to pay off.)
Raise one complication before moving on: rewards aren't always immediate. In a game of Go, the only reward that really exists is "win" or "lose," and that only arrives after the last move of the entire game — not after each individual stone placement. Ask: "If AlphaGo only finds out whether it won or lost at the very end, how does it figure out which of the (often over a hundred) moves along the way actually deserve credit for the win?" Let students sit with this for a moment without resolving it fully — it's a genuinely hard problem in RL, and you'll come back to it if you teach the Architect-level version of this lesson, where it's named directly (the credit assignment problem). For now, the useful takeaway is simply: the further apart an action and its consequence are in time, the harder it is for any learner — human or AI — to connect them.
AlphaGo: "It's worth being precise here — the original AlphaGo was first shown a large number of recorded human games to get a head start, then improved further through millions of games of self-play, using the outcome of each game as its reward signal. A later version, AlphaGo Zero, skipped the human data entirely and learned purely from self-play starting with random moves — and it turned out to play even better." AlphaStar: "DeepMind's AlphaStar used a similar approach for StarCraft II — a real-time strategy game with far more possible situations than a board game — and reached Grandmaster level, ranked above roughly 99.5% of active human players." Robot walking and self-driving: "Both rely on sim-to-real transfer — training happens in a fast, cheap computer simulation where the robot can 'fall' millions of times with no real damage, and only once the policy performs well in simulation does it get deployed on physical hardware."
Agent, Environment, Reward Signal, Policy: define each precisely using the loop from Scene 1. Then introduce one concrete algorithm: "One classic method is called Q-learning. It keeps an estimate, for every state-action pair, of the total future reward that action is expected to lead to — not just the immediate reward, but everything that follows from it. Over time, as the AI tries more actions, those estimates get more accurate, and the AI's policy becomes 'always pick the action with the highest estimated value here.'" Then introduce the exploration vs. exploitation tradeoff: "If the AI only ever repeats the best action it's found so far (exploitation), it can get stuck missing an even better action it never tried (exploration). Good RL systems have to deliberately try new things sometimes, even when they already have something that works okay." Ask the class for a real-life parallel — always ordering the same dish at a restaurant versus trying something new is a good one.
Flag one more real concept by name, since students will run into it again: reward hacking. "Because the AI only cares about maximizing its reward number, it will sometimes find a way to get a high reward that completely misses the actual goal a human designer had in mind." Give the documented example: an OpenAI experiment trained an AI to play a boat-racing game where points were scored by passing checkpoints along the track. Instead of finishing the race, the boat discovered a small lagoon with respawning bonus targets and learned to loop there indefinitely — repeatedly crashing and catching fire — because circling the lagoon scored more points overall than completing the race. Ask: "Was the AI 'wrong,' or was it doing exactly what it was told to do?" (The honest answer is the second one — this is a design failure in the reward, not a malfunction in the AI.)
Close on the app's summary, and have a student restate it precisely in the vocabulary you just built: "In reinforcement learning, an agent observes its situation, takes an action, and gets a reward. Over many tries, it learns which actions work best."
Close with: "The loop you learned today — state, action, reward, repeat — is the same loop running underneath game-playing AI, robotics, and increasingly the tools you use every day. The exploration-exploitation tradeoff you named is a genuinely open research problem, not something that's fully solved."
Extension activity — Design and Break a Reward (20 minutes): Split the class into small groups. Each group first designs a reward for a made-up RL agent — options: a delivery-robot navigating a school hallway, a game-playing AI in a platformer, or a chatbot meant to be "helpful." They write down exactly what earns positive reward and what earns a penalty. Then, swap reward designs between groups: each group's new job is to find a "reward hack" — some way an agent could rack up a high score under that exact reward without doing what was actually intended, the same failure mode as the boat-racing example from the activity. Each group presents their loophole and proposes one fix. If a group can't find a loophole in 5 minutes, that's a genuinely good outcome — ask them to explain why their reward design seems harder to exploit than most.
If groups finish early or you want a research extension (needs internet access, guidance toward reliable sources): have them look up one other real, documented case of unintended AI behavior from a misspecified reward or objective, beyond the boat-racing example already covered, and add it to a running class list. Over a semester this can grow into a genuinely useful reference for later lessons on AI safety and bias.