AIQ AIQ
World 5: Is AI Fair? · Lesson 5.1.1

When AI Isn't Fair

Everything a teacher needs to deliver this lesson — pick your grade's script below once you've read the background.

Learning Objectives

This lesson (badge: "Bias Buster" ⚖️) introduces the idea that AI systems can treat people unfairly — not because anyone programmed them to be mean, but because of how they learn. By the end of the lesson, depending on age band, a student should be able to:

Teacher Background

You do not need a computer science background to teach this lesson well. You need one idea, held firmly: most of the AI this lesson is about does not follow rules a programmer wrote line by line. It learns by example. Show it thousands of resumes that got hired in the past, and it learns what a "hireable" resume looks like. Show it thousands of faces, and it learns what a face looks like. That single fact — learning from examples rather than following fixed rules — is where every kind of AI bias in this lesson comes from.

If the examples an AI learns from are lopsided, what it learns will be lopsided too, and it will not know that anything is wrong. A well-known real case: a large tech company built an internal tool to screen job resumes automatically, training it on ten years of resumes the company had actually hired from. Because the tech industry has historically hired far more men than women, the tool learned to prefer resumes that looked like the ones it had seen — and it began downgrading resumes that mentioned things like "women's chess club captain," simply because that word pattern rarely appeared in the resumes that got hired in the past. Nobody told it to penalize the word "women's." It found that pattern in the data on its own.

A student's first instinct is often "so the programmers were sexist/racist." Correct this gently but clearly: intent has nothing to do with it. The AI has no opinions about people. It is a very good pattern-finder, and it will happily find and repeat an unfair pattern that was already sitting in the historical data — a pattern that reflects decisions humans made in the past, long before the AI existed.

Face recognition is the second example the lesson uses, and it follows the exact same logic. A landmark 2018 MIT study called Gender Shades, led by researcher Joy Buolamwini, tested several commercial face-analysis systems and found they were far more accurate at identifying lighter-skinned men than darker-skinned women — some systems were wrong for darker-skinned women roughly a third of the time. The cause, again, was the training data: the photo sets used to build these systems were overwhelmingly of lighter-skinned faces, so the systems simply had far less practice recognizing everyone else.

The lesson's third example — AI used in lending — points at something called proxy variables, and this is the idea worth spending the most time on with older students, because it is the one that trips people up. A bank cannot legally use race to decide who gets a loan. But a ZIP code, in a country with a history of segregated neighborhoods, can carry almost the same information as race without ever mentioning it. An AI system trained to predict loan risk from ZIP code, school attended, or even a person's first name can end up making decisions that split along racial or economic lines — a pattern researchers and journalists have called digital redlining, after the older, human practice of literally drawing red lines on maps around neighborhoods banks would not lend to.

Students — especially in the 11–18 range — often propose "just delete the race and gender columns" as the fix. It feels obvious and it is not enough. Proxy variables are exactly why: the information can still get through a side door. This is a good moment to ask the class, "what other piece of information might quietly tell you someone's income, neighborhood, or background, without ever asking directly?"

The good news, and where the lesson ends, is that none of this is unfixable — it just takes deliberate effort, because fairness does not happen by default. Three things reduce it in practice: training AI on data that actually represents the range of people it will be used on; running bias audits — deliberately testing a system's accuracy separately for different groups before it ships, not just checking one overall accuracy number; and building the system with a team of people who bring different backgrounds and experiences, because a team that all looks the same tends to share the same blind spots. For the oldest students, it is worth adding that this has become a legal and organizational discipline as well as a technical one — companies now run "AI impact assessments" before and during deployment, and there is a body of law (built originally around hiring and housing, not AI) that treats a policy as discriminatory if it disproportionately harms a protected group, whether or not that was the intent.

Why this belongs in a Philippine classroom: AI-driven hiring screens, credit-scoring tools, and insurance pricing algorithms are already used by companies operating here, and that will only grow. The habit this lesson is trying to build — asking "whose examples trained this, and who might it be leaving out?" — is a literacy skill your students will use long after this specific lesson, on tools that do not exist yet.

One more thing worth knowing before you teach this: the app deliberately does not use the words "wrong" or "failed" anywhere in this lesson, in line with AIQ's general approach to feedback for kids. If a student's own guess or opinion during discussion turns out to miss the mark, echo that same spirit — "not quite, let's look at it again" rather than "no, that's wrong." It keeps the emotional register of the lesson consistent with the app itself, especially for the younger age bands.

Materials & Prep

No special materials or advance setup are required. Each student (or pair, for younger grades) needs a device with a browser to work through the AIQ lesson itself — phone, tablet, or computer, whatever the class normally uses. A shared screen or projector is helpful for the opening hook and for group discussion, but not essential. A few age-band scripts below suggest an optional low-tech prop (a physical die for Explorer mode's fairness demonstration, index cards for a sorting activity in Builder mode) — all are things a typical classroom already has, and every activity works fine without them if they are not available.

Common Misconceptions

"AI is biased because the people who built it are racist or sexist."
Almost always, no one involved intended any of this. The bias comes from the historical data the AI learned from — data that reflects decisions and patterns that already existed in society, often decades before the AI was built. The fix is about the data and the process, not about accusing the builders of bad intent.
"If you just delete the race or gender column from the data, the AI becomes fair."
Other pieces of information — a ZIP code, a school, sometimes even a first name — can act as a proxy for race, gender, or income, quietly carrying the same signal through the back door. Removing the obvious column is a start, not a finish.
"Bias is only a problem for 'serious' AI, like hiring or loan systems — a recommendation app or chatbot can't really be biased."
Any AI trained on real-world data can inherit whatever imbalance is in that data, including recommendation systems, search results, and generative tools. The stakes are higher when the decision affects a job or a loan, but the underlying mechanism is identical everywhere.
"There is one correct mathematical definition of 'fair,' and once engineers find it, the problem is solved." (most relevant for 15–18)
Researchers have proposed several reasonable-sounding fairness definitions — for example, giving every group an equal chance of a positive outcome, versus giving every group the same error rate. It has been mathematically proven that, except in special cases, a system cannot satisfy every one of these definitions at once. Choosing which definition matters most for a given system is a human values decision, not something a formula can settle on its own.

Pick your grade's script

← All lesson plans ← AI and Jobs Privacy & AI →