AIQ
AIQ
Part of the AI Perception Challenge lesson guide. Teaching a different grade? 🌈 Explorer (5–7) · 🔧 Builder (8–10) · 💻 Hacker (11–14)
This group has already built the vocabulary in earlier World 2 lessons — vision, speech recognition, NLP, sensors, adversarial failure modes. Treat this lesson as the point where those pieces get named as a single subfield. Open with:
"Modern AI combines vision, language, and audio into single powerful models. Let's examine multi-modal AI and how it understands the world."
Frame the stakes precisely: "Every major AI lab — OpenAI, Google, Anthropic, Meta — has shifted its flagship models from single-modality (text-only, or vision-only) to natively multi-modal in the last few years. That's not a marketing trend; it reflects a real technical shift in how these models are built, from separate specialist models chained together to single architectures trained on mixed data from the start. We're going to look at how that actually works, not just what it can do."
Quick diagnostic before moving on — ask the class to state, in one sentence, the difference between a pipeline approach (e.g., speech-to-text, then a separate text model) and a model that's natively multi-modal (trained end-to-end on multiple data types together). If they can't articulate this distinction cleanly, spend two minutes on it now — the rest of the lesson depends on it.
Worth naming up front, since this class can handle nuance: this lesson stays at the level of publicly documented architecture patterns and confirmed research (CLIP's published paper, Google's public descriptions of Gemini). Where a specific commercial system's internals aren't publicly disclosed, say so explicitly rather than presenting inference as fact — that distinction between "documented" and "widely inferred from similar public work" is itself part of what this lesson is teaching.
Structure this as a technical walkthrough of one real system in depth, rather than another pass through the twelve consumer examples this class has already sorted twice in earlier lessons.
1. Contrastive learning and CLIP. Explain the training setup concretely: CLIP (OpenAI, 2021) was trained on hundreds of millions of image–text pairs scraped from the internet — a photo and its caption, essentially. The training objective, contrastive learning, doesn't ask the model to generate anything; it asks the model to produce a numerical vector (an "embedding") for the image and a separate vector for the text, such that a matching image–text pair end up close together in that shared vector space, and non-matching pairs end up far apart. Once trained, this lets CLIP do something it was never explicitly taught: given a photo, compare its embedding against the embeddings of a list of text labels it's never seen paired with that photo before, and pick the closest one — this is zero-shot classification, and it's the direct payoff of aligning two very different data types (pixels, words) into one shared mathematical space.
Worth pausing on the practical upshot for students who ask "why does this matter": zero-shot classification means CLIP-style models can be deployed on a completely new labeling task — say, sorting photos into categories a company invents next week — without collecting a single new labeled training example for that task. That's a meaningfully different economic and engineering proposition than earlier computer vision, which typically required a labeled dataset built specifically for each new category set.
2. From CLIP to conversational multi-modal models. A model like GPT-4V or Gemini takes the "shared representation" idea further: rather than just comparing embeddings, the image is broken into patches, each patch converted into a token-like vector, and those visual tokens are fed into the same Transformer architecture that processes text tokens — in the same sequence, attended to by the same mechanism. That's the technical answer behind why these models can answer a free-form question about a photo instead of only picking from a fixed label list: the image isn't a separate lookup step, it's part of the same token stream the model reasons over. Be honest with the class here: OpenAI hasn't published GPT-4V's full architecture publicly, so specifics are inferred from published research on similar public architectures (e.g., LLaVA, Flamingo) rather than confirmed for GPT-4V exactly — a good moment to model intellectual honesty about the limits of what's publicly verifiable in this field. Gemini is documented by Google as natively multi-modal from the start of training — text, images, audio, and video processed within one model rather than bolted-on afterward.
A brief but important aside on terminology precision: OpenAI's public CLIP paper reports zero-shot performance on standard benchmarks like ImageNet that was competitive with earlier fully-supervised models trained specifically on that dataset — a genuinely notable result at the time it was published, and a good concrete number to cite if a student asks "how good is zero-shot, actually, compared to a model built just for the task." Encourage students who want the exact figures to look up the original 2021 paper rather than a secondhand summary.
3. Zero-shot and emergent cross-modal transfer. Tie this back to scale: as these models are trained on more parameters and more diverse multi-modal data, they start succeeding at tasks nobody explicitly trained them for — describing an unusual object they've never been labeled on, or reasoning about a chart they've never seen a template of. Be precise with the class about what this does and doesn't imply: it's evidence that scale plus data diversity produces useful generalization across modalities, not evidence of understanding, consciousness, or human-like perception. That's a real distinction actively debated in the AI research community, and it's the correct, defensible answer if this comes up on the quiz or in discussion — resist any framing that treats emergent capability as proof of "real" understanding.
4. Where fusion still breaks. Close the technical portion by connecting back to "Glitch Finder": multi-modal models inherit and compound the failure modes of each modality they combine. A model can hallucinate a plausible-sounding but false caption for an image, misread text embedded within an image, or give inconsistent answers about the same image depending on how a question is phrased — because combining modalities doesn't eliminate the pattern-matching nature of any of the underlying components, it just makes the failure surface larger and harder to audit.
Make this concrete with a documented category of attack: adversarial images crafted specifically to make a vision-language model output a chosen, attacker-controlled caption rather than an accurate one — a direct extension of the "sticker on a stop sign" problem from Glitch Finder, but against a model that also has to produce coherent language, not just a single classification label. This is a live area of AI safety research, and it's a fair thing to tell the class explicitly hasn't been fully solved by anyone.
It's worth flagging the distinction implicit across these four questions: the first two are about understanding multi-modal input (aligning an image with text, or reading both together), while the last one is about generating multi-modal output — producing new images, audio, or video, not just interpreting existing ones. Generation raises the harder version of the consistency problem, since nothing forces a generated image and a generated caption to agree with each other the way a real photograph and a human-written caption naturally would.
Close with: "You've now traced the actual technical arc of this entire world — from a single pixel grid, to a single sound wave, to a single sentence, to models that hold all of it in one shared representation and reason across it. That arc is also, right now, the frontier of where AI research funding and effort are concentrated — which means what you just learned isn't a settled, historical topic. It's still actively being figured out."
Extension activity (25–30 minutes, written or discussion-based): Have students find one recent (last 12 months) credible news or research article about a multi-modal AI system — a new model release, a documented failure case, or a research paper summary — and write a short analysis: what modalities does it combine, is it a pipeline or natively multi-modal (and how can they tell from the source), and what's one claim in the source they should treat skeptically without independent verification. Require at least one primary or technical source, not only a press release, to practice distinguishing marketing claims from documented technical detail.
A good check for rigor before students submit: ask them to identify one sentence in their chosen source that states a capability without evidence, and rewrite it as a properly hedged claim (for example, changing "the model understands images" to "the model was trained to predict text conditioned on image input, and performs well on benchmark X"). This is the same skill professional AI researchers and journalists have to apply constantly, and it's a stronger takeaway from this lesson than memorizing any single model's name.