AIQ
AIQ
Part of the AI Perception Challenge lesson guide. Teaching a different grade? 🌈 Explorer (5–7) · 🔧 Builder (8–10) · ⚡ Architect (15–18)
Skip the recap-and-quiz framing this age group has seen twice already in this world — open with the claim itself, since it's the one worth arguing about:
"Modern AI doesn't just see OR hear — it does both at once. Multi-modal AI combines senses. Let's explore how AI mixes vision, sound, and text."
Ask directly: "Up to this lesson, we've treated vision, hearing, and reading as three separate skills an AI can have. Why might it actually be harder to build one system that does two of these at once, instead of just running two separate simple systems side by side?" Let a few answers land without correcting them yet — you're setting up the "fusion" idea in the activity below, and the honest answer (different data types, different formats, different timing) is more convincing after they've tried to guess it themselves.
Then name the term precisely, since this class can handle a real definition instead of a soft one: "A model is multi-modal when it's built to take in more than one type — or 'modality' — of data, like images and text, and process them together in a way that lets information from one inform the other. It's not two AIs bolted together with a translator in between; the goal is one system that reasons across both at once."
Ground the term in something concrete before moving on: hold up a phone and ask, "If I show this phone's camera a math problem written on a whiteboard and ask it out loud, 'what's the answer,' how many different types of data did that one request just involve?" Walk them to the count: an image (the whiteboard), your spoken question (audio), and the text written on the board that has to be read out of the image (embedded text within vision). A system that handles this in one request — image plus voice — is doing genuinely multi-modal work, not just running three separate apps back to back.
Revisit the same twelve real-world examples from earlier in this world, but this time sort them by a harder question: single-modality or multi-modal? Put them up one at a time and have students argue for a classification before you confirm it.
Mostly single-modality (vision only, or text only): License plate readers (vision only — read characters from an image), cancer-detection photo screening (vision only), Google News article grouping (text only — NLP comparing article content), review-summary tools (text only). These are useful to name explicitly as single-modality, because the temptation at this age is to call anything "AI" a multi-modal system by default.
Push on the license-plate example specifically, since it looks like it might combine vision with something else but usually doesn't: "The garage reads your plate with a camera and then checks it against a database of who's paid. Is checking a database a second modality?" Guide the class to the distinction: looking up a value in a database is a simple, deterministic lookup — not a second type of perception. The AI work is entirely in reading the characters off the plate image; everything downstream of that is ordinary software, not a second sense being fused with the first.
Genuinely multi-modal: Voice assistants — "Siri, Alexa, and Google use speech recognition + natural language understanding — TWO AIs together!" Push further than the app's own framing: ask what data type speech recognition outputs (text) and point out that this makes it a pipeline (audio → text → meaning) rather than one model natively handling both audio and text at once — a real and useful distinction most consumer voice assistants use a pipeline, while a smaller number of newer models are trained to process raw audio directly. Content moderation at scale on platforms like Instagram or TikTok increasingly combines vision (scanning images and video frames) with text (captions, comments) in a single review pipeline. Video captioning — a system that watches a video and generates a text description of what's happening — is a clean multi-modal example even though it wasn't one of the twelve named products: it has to convert visual information into language, aligning what's seen with what's said.
Have the class attempt to classify one more example on their own before you introduce VQA formally: self-checkout kiosks that use a camera to identify produce (so a customer doesn't have to type in a PLU code for a banana) but still require the customer to type or tap for anything the camera can't confidently identify. Ask whether this counts as multi-modal — the honest answer is that it's borderline: the camera and the touchscreen aren't combining information about the same decision, they're two separate, non-overlapping paths to the same outcome, which is a meaningfully different design from a system that needs both inputs together to answer one question.
Introduce Visual Question Answering (VQA) as the concrete task that defines "genuinely multi-modal" for this lesson: a VQA system is shown an image and asked a question about it in text — "how many people are in this photo?" — and has to produce a text answer that's only correct if it actually used information from the image. Ask: "Why can't a text-only AI do this task at all, no matter how good it is at language?" (Because the answer depends entirely on information that only exists in the image — there's no text input containing it.)
Close the activity by naming why fusion is hard, which is the substance behind the quiz's hardest question: an image is a grid of pixel values with no inherent order in time; a sentence is a sequence of words where order matters a great deal; audio is a continuous waveform sampled thousands of times per second. A model has to translate all three into some shared numerical representation before it can combine them — and getting that translation right, so that information from one modality usefully affects processing of another, is an active area of AI research, not a solved problem.
Make this tangible with a quick thought experiment: "If I show you a five-second video of someone laughing, at what exact moment does the 'happy' information exist — in the picture, in the sound, or in both together, and does it matter which frame lines up with which second of audio?" Push toward the answer that timing alignment itself is a fusion problem: a system combining video and audio has to line up which visual frame corresponds to which slice of sound, and getting that synchronization wrong can make the combined output actively worse than using either modality alone.
It's worth pointing out to the class why "matching" is a clever training signal: nobody had to hand-write a label for every object in every photo, the way earlier image-recognition systems required. The internet already has billions of images with captions attached — alt text, social media captions, photo descriptions — so the "labels" were free, just noisy and inconsistent. That's part of why this approach scaled so well compared to earlier, hand-labeled datasets, and it's a good bridge into why data availability, not just algorithm design, drives a lot of progress in this field.
Close with: "You now have the vocabulary that separates 'AI that can see' and 'AI that can hear' from 'AI that reasons across both at once' — and you've seen why that combination is a genuinely hard problem, not just a feature checkbox. Multi-modal systems are where most current AI research investment is actually going, precisely because single-modality AI hits a ceiling on what it can be used for."
Extension activity (20–25 minutes): In pairs, have students design (on paper, no coding required) a multi-modal app idea for their own school — for example, an app that photographs a broken piece of classroom equipment and writes a maintenance report, or one that listens to a class discussion and generates a text summary with each speaker's key points. Require them to answer three questions in writing: which modalities does it need, what would a single-modality version of the same idea be unable to do, and what's one way the system could be fooled or fail (tying back to the "Glitch Finder" lesson).
If time allows, have pairs present their idea to another pair and let the listening pair try to poke a hole in it specifically by proposing an edge case where one of the modalities would give misleading information — a photo taken in bad lighting, background noise during the class discussion, handwriting on the broken equipment that's hard to read. This mirrors how real engineering teams stress-test a multi-modal product idea before building it, and reinforces that naming the modalities is the easy half of the design process.