AIQ
AIQ
Part of the Train Your Own Model lesson guide. Teaching a different grade? 🌈 Explorer (5–7) · 🔧 Builder (8–10) · 💻 Hacker (11–14)
Open with the lesson's real hook line: "Building real AI goes beyond training a model. You need data pipelines, testing, and deployment. Let's walk through the production ML pipeline."
Ask the class: "If I asked you to guess what percentage of an ML engineer's time at a real company goes into the actual model-training step — the part everyone talks about — what would you guess?" Take guesses; most will overestimate significantly. Say: "It's widely cited in ML engineering circles that training itself is often only around 5-10% of total effort, with data engineering and infrastructure work dominating the rest. Today we're going to build the small demo version of that pipeline, and talk honestly about everything the classroom version skips."
Set expectations for rigor: "This is a genuinely fast-moving, actively debated area of software engineering — MLOps practices are still maturing industry-wide. I'll be precise about what's established practice versus what's a widely-cited estimate rather than a hard law, and I'd rather flag that distinction honestly than overstate certainty."
Frame today's structure: "We'll build a working classifier the same way you did earlier in World 6, but at each step, we'll also discuss what a production team would add on top — versioning, monitoring, deployment testing — that our classroom demo doesn't need at this scale, but a real system absolutely does."
If your class includes students weighing a career in software or ML engineering, add this framing: "A lot of what makes someone valuable on a real ML team isn't knowing the fanciest model architecture — it's understanding this entire pipeline well enough to know where things actually break in practice, and building the boring infrastructure that keeps a model reliable months after launch. That's a very different, and arguably more employable, skill set than just 'knowing AI.'"
Use the AI Playground build as the concrete anchor, and layer production-engineering concepts onto each of the lesson's three scenes.
Flag one relevant design decision before starting: "This entire demo runs client-side — the photo capture, the feature extraction, and the comparison all happen in your browser, with nothing uploaded to a server. That's not just a privacy nicety for a K-12 app; it's also a real trade-off worth naming: it means zero server cost and zero data-handling liability, at the price of every device redoing the same feature-extraction work locally instead of a shared server doing it once for everyone. A production system serving many users would likely make the opposite trade-off."
Have students build a small classifier as before (transfer learning on top of a pretrained MobileNet, comparing feature vectors), but frame the collection step in production terms: "What we just did casually — capture some photos, label them — is, at scale, an entire discipline called data engineering: building pipelines to collect, clean, label, and store training data reliably, often from many sources and many contributors over time." Introduce data versioning: "If you retrain your model next week with 10 new photos added, do you know exactly which photos were used to produce today's model versus next week's? At production scale, you need to — reproducibility requires being able to say exactly which dataset trained exactly which model, so a bug can be traced back to a specific data change." Ask pairs to name one thing that would be different if they had to guarantee this for their own tiny model.
Clarify precisely what the app does versus what a production system does: "Our Playground uses transfer learning plus a nearest-neighbor comparison — instant, and appropriate for a browser demo with a handful of examples per category. A production image classifier serving millions of requests would more likely fine-tune or fully train a neural network, track experiments across many runs, and evaluate multiple candidate models before choosing one to ship. The 5-10% figure from the hook refers to that heavier kind of training work — and even then, it's a minority of the total engineering effort, because data pipelines, serving infrastructure, and monitoring all have to exist and stay reliable around it."
Introduce CI/CD for ML: "In regular software, continuous integration means new code gets automatically tested before it ships. Applied to ML, that means a newly trained model candidate gets automatically evaluated against a fixed test set and quality thresholds before it's allowed to replace the model currently running in production — preventing an accidental quality regression from silently shipping."
Note one genuine complication worth naming honestly: "ML testing is harder than regular software testing in one important way. Regular code either passes a test or it doesn't — deterministic, repeatable. A model's accuracy on a test set is a statistic, not a pass/fail fact, and it can shift slightly between training runs even on identical data, due to randomness in how the process starts. Good ML teams account for this by requiring a new model to clear its quality bar by a meaningful margin, not just barely edge out the old one, and by testing against multiple slices of data — not just one overall accuracy number that could hide poor performance on an important subgroup."
Have pairs test their model with new examples as before, but reframe what happens next in production terms: "In our classroom version, deploying just means we keep using the model we trained. In production, deployment means the model starts serving real user traffic — and that's genuinely a beginning, not an ending." Introduce A/B testing: "Rather than trusting that a new model is better just because it scored higher on a fixed test set, production teams often run it alongside the current model on a slice of real traffic and directly compare real-world performance differences between the two versions before fully switching over."
Introduce ML observability as the ongoing job after deployment: "A model doesn't stay accurate forever just because it worked at launch. Teams monitor for data drift (the real-world inputs starting to look statistically different from training data — think about seasonal changes, new slang, new camera hardware), prediction drift (the model's output distribution shifting over time), and straightforward performance degradation. Ask: "If our classroom model was trained on photos from this room's lighting, and someone tried using it in a much darker room next month, is that data drift? Why or why not?" (Guide toward: yes — the real inputs it now sees have shifted away from what it was trained on, and accuracy would likely drop even though nothing about the model itself changed.)
Close the scene by naming what monitoring actually requires as an engineering investment, not just a good intention: "None of this happens automatically just because someone cares about it. A real observability setup means logging model inputs and outputs continuously, computing statistics over them on a schedule, comparing those statistics against a training-time baseline, and alerting a human when they diverge meaningfully. That's real infrastructure work, running for as long as the model stays in production — a large part of why the 5-10% training-effort figure from the hook holds up: a deployed model needs care for its entire operational life, not just at launch."
Close with: "The model you built today in a browser tab and a production system serving millions of people run on the same fundamental idea — learn from labeled examples, test before you trust it, watch it after it ships. What separates a classroom demo from a production system isn't a smarter algorithm; it's everything we discussed today around that core idea: versioning, automated testing, careful rollout, and ongoing monitoring."
Extension activity — Design the Production Version: In small groups, have students take the classifier they built earlier in class and sketch (on paper or a shared doc) what it would take to run it as a real product feature for, say, 100,000 daily users. Require each group to address: where new training data would come from on an ongoing basis and how it would be versioned; what automated test would need to pass before a retrained model could replace the live one; how they'd structure an A/B test comparing old vs. new model versions; and what specific metric they'd monitor post-launch to catch drift early. Have groups present their design and defend one decision they made under questioning from the rest of the class.
Push the strongest groups further with a follow-up constraint: "Now assume your company has a tight budget and can only build two of the four things you just designed — data versioning, automated pre-deployment testing, A/B testing infrastructure, or drift monitoring — this quarter. Which two would you prioritize, and what specific risk are you accepting by deferring the other two?" There's no single correct answer, but a strong response should name a concrete failure mode for each deferred piece — for example, skipping drift monitoring means a real-world shift (users' cameras or lighting conditions changing from what training data assumed) could silently degrade the product for weeks before anyone notices.