Everything a teacher needs to deliver this lesson — pick your grade's script below once you've read the background.
Teacher Background
You don't need a programming or statistics background to teach this lesson — the underlying idea is closer to "you are what you eat" than to anything mathematical. Most modern AI systems, including the kind students use every day (voice assistants, photo apps, recommendation feeds, chatbots), are not programmed with explicit rules for every situation. Instead, they are shown enormous numbers of examples — called training data — and they gradually adjust themselves to find patterns in those examples. A spam filter isn't told "these exact words mean spam"; it's shown thousands of emails that humans already labeled spam or not-spam, and it learns which patterns tend to show up in each. This is the whole reason data quality matters so much: the AI has no independent way to know what's true or fair. Everything it "knows" comes from what it was shown.
"Garbage in, garbage out" is an old idea from computing generally (it predates AI by decades — it originally described any computer program producing bad output from bad input), but it applies to AI with special force, because AI systems generalize from their training data to situations they've never seen. If the training data is inaccurate, incomplete, too small, or skewed, the AI doesn't just get a few answers wrong — it can systematically get a whole category of people or situations wrong, and it will do so confidently, because it has no way of knowing what it wasn't shown.
Key point: When a student asks "couldn't they just fix the AI?", the important idea to land is that the AI is not separately broken from its data — for this class of system, the data is the AI's knowledge. Fixing the problem usually means fixing or expanding the data, not tweaking a formula.
Bias is the concept that carries the most weight in this lesson, and it's worth being precise about it in front of students, because the everyday meaning of the word ("being unfair" or "having a personal opinion") and the data-science meaning are related but not identical. In this lesson, bias means the training data does not fairly or accurately represent the population the AI will be used on. That can happen in more than one way: selection bias is when certain groups are simply missing or underrepresented in the data (a face-recognition system trained mostly on lighter skin tones will be less accurate on darker skin tones, not because of malice, but because it saw far fewer of those examples); historical bias is when the data accurately reflects the past, but the past itself was unfair, so the AI learns to repeat that unfairness going forward.
The lesson leans on three real, well-documented cases, and it's worth knowing the actual story behind each one, since students often ask follow-up questions:
- Amazon's hiring tool (reported in 2018). Amazon built an experimental AI tool to screen job applicants' resumes. It was trained on resumes submitted to the company over the previous ten years, a period during which the tech industry — and Amazon's applicant pool — was heavily male-dominated. The tool taught itself that male candidates were preferable, and it downgraded resumes that included the word "women's" (as in "women's chess club captain") and graduates of certain all-women's colleges. Amazon scrapped the tool once this was discovered. The lesson here isn't "the engineers were sexist" — it's that ten years of historical hiring data reflected an existing imbalance, and the AI faithfully learned and amplified that imbalance.
- Face recognition and skin tone. Multiple independent research studies — most notably from MIT Media Lab researchers around 2018 — tested commercial face-analysis systems and found they were considerably less accurate at classifying darker-skinned faces, and especially darker-skinned women, than lighter-skinned faces. The consistent explanation across these studies was that the training datasets used to build these systems were dominated by lighter-skinned faces, so the systems had far fewer examples to learn from for other skin tones.
- Loan and credit decisions. Some automated lending systems, trained on decades of historical loan data, have been found to replicate patterns of discrimination against certain neighborhoods or racial groups that existed in that historical data (echoing older, explicitly discriminatory practices like redlining). The AI isn't "deciding" to discriminate — it's finding and repeating a pattern that was already present in the data it learned from.
If a student asks "so is it the AI's fault or the data's fault?" — the most accurate answer is that for these systems, that's a false distinction. The AI has no knowledge or judgment apart from its training data. Responsibility sits with the people who chose, collected, and reviewed that data before training the system on it — which is exactly why the "fixing data problems" part of this lesson (auditing data, adding diverse examples, testing for bias, using diverse teams) is about human decisions, not a purely technical fix.
Note for older grades: this lesson is a deliberate stepping stone toward two ideas students will meet again later in AIQ — that AI can be "confidently wrong" (it doesn't know what it doesn't know), and that responsible AI development requires ongoing human oversight, not a one-time fix. You don't need to over-explain this now; it's enough that students leave with "good data in, good AI out" as a durable mental model they can build on.
Common Misconceptions
"Bias in AI means the AI has opinions or feelings, like being mean to certain people."
Bias in this context is a technical, statistical idea: the training data doesn't fairly or accurately represent everyone the AI will be used on. The AI has no opinions or intentions at all — it's a pattern-matching system that repeats whatever imbalance existed in the examples it learned from. This distinction matters because it points to the actual fix: better, more representative data, not "teaching the AI to be nicer."
"If an AI makes a biased decision, it's a bug that a quick software patch can fix."
For AI systems that learn from data (as opposed to traditional rule-based software), the "knowledge" the system has largely comes from its training data. Fixing bias typically means auditing and expanding the dataset — finding out which groups or situations are underrepresented and adding accurate, representative examples — not just adjusting a formula in the code. It is a data-and-process problem as much as a technical one.
"More data is always better, no matter what it looks like."
Quantity alone doesn't fix a data problem. A dataset can be enormous and still be biased if it overrepresents some groups and underrepresents others — adding a million more examples of the same skewed kind doesn't help. What matters is whether the data is accurate, complete, and representative of everyone the AI will encounter, not just how much of it there is.
"Biased AI outcomes mean the people who built it were deliberately prejudiced."
In the documented real-world cases this lesson discusses (like Amazon's hiring tool), the bias came from historical data patterns that existed before the AI project even started — ten years of a male-dominated applicant pool, for example — not from any single person's intent. That doesn't make the outcome acceptable, but it does change the fix: the responsibility is to actively audit and correct for that historical pattern, not to look for an individual villain.
AIQ