Teaching "How Computers Read" to Architect mode (ages 15–18)

Hook & Warm-Up

Open at the level this age band can actually engage with technically:

"From bag-of-words to Transformer architectures, NLP has evolved from statistical frequency analysis to contextual language understanding. Let's examine modern language representation and generation."

Ask: "Who here has used a large language model — ChatGPT, Claude, Gemini, whatever's on your phone? Now: can any of you actually describe, mechanically, what happens between you hitting 'send' and text appearing on screen?" Most students will describe the experience, not the mechanism. Say: "That gap — between using something fluently and understanding how it works — is exactly what today closes. We're going past 'it's smart' into tokenization, embeddings, attention, and the actual architectural difference between models like BERT and GPT."

Frame the stakes plainly, since this age band is old enough for it to matter directly: "Some of you will use these systems for schoolwork, some of you may end up building on top of them professionally, and all of you will need to evaluate what they get right and wrong for the rest of your lives. None of that is possible with 'it's magic' as your mental model — it requires knowing, at least at a working level, what's actually computing the output."

Main Activity

Treat this as a technical walkthrough building toward the Transformer, using the lesson's three scenes as checkpoints rather than the whole content.

Tokenization. Before any model sees text, it must be tokenized. Introduce Byte-Pair Encoding (BPE) and WordPiece as the two dominant subword tokenization schemes: both start from individual characters and iteratively merge pairs into single tokens, building a vocabulary that balances coverage against size — common words end up as single tokens, rare or novel words get split into smaller familiar pieces. They differ in the merge rule: BPE merges whichever adjacent pair occurs most frequently in the training corpus; WordPiece instead merges whichever pair most increases the likelihood of the training data under the resulting vocabulary — a related but distinct criterion, closer to "which merge helps the model predict the text best" than raw frequency counting. This is why a model can process a word it's never seen before: it falls back to subword fragments it has seen.

Where this breaks down: morphologically rich, code-switched languages. BPE and WordPiece vocabularies are learned from whatever corpus trains them, and the large public tokenizers most production models use (GPT's, for instance) are trained on overwhelmingly English-dominated web text. That has a direct, measurable consequence for Tagalog. English affixation is almost entirely prefixes and suffixes tacked onto a stable root. Tagalog also uses infixation — inserting an affix inside the root itself, not just at its edges (e.g., "sulat," write, becomes "sumulat," wrote, by inserting "-um-" after the first consonant) — plus reduplication and layered circumfixes, so a single root can surface as a large family of distinct surface forms: "aral" (study) yields "mag-aaral" (student/will study), "pinag-aralan" (studied, pored over), "nakakapagpabago" ("able to cause change," root "bago" with several stacked affixes). A tokenizer's merge rules are learned by frequency (BPE) or likelihood-gain (WordPiece) over its training corpus — and if that corpus is mostly English, these specific Tagalog character sequences never occurred often enough to earn their own merged token, so the tokenizer falls back to smaller, sometimes near-character-level fragments for them, while common English words compress into one or two tokens each. The measurable result is a worse token fertility ratio (tokens consumed per unit of meaning) for Tagalog than for English on the same tokenizer — which costs more compute at inference and spreads a word's meaning across more sequence positions for self-attention to reassemble, making the model's job measurably harder for the exact same amount of underlying information.

Layer code-switching on top of that, because it's the norm rather than the exception in Philippine text — SMS, social media, and everyday classroom speech constantly mix English and Tagalog inside a single sentence, often with an English root taking Tagalog grammatical affixes directly (e.g., "ni-reply," replied, fuses the Tagalog past-tense marker onto the English loanword "reply"; "i-submit mo" instructs someone to submit something, again fusing a Tagalog directional/imperative affix onto an English root). This means "language" isn't even a fixed, sentence-level property a tokenizer or a downstream model can assume — the expected token statistics can flip mid-sequence, and pretraining corpora that are effectively a mix of a large monolingual-English pile and a much smaller monolingual-Tagalog pile contain comparatively little of this specific code-switched pattern, so it's the most under-represented case of all three. Be precise with students about the takeaway: this is a training-data imbalance problem, not evidence that Tagalog is intrinsically harder to process — a tokenizer and model trained predominantly on Tagalog text would show the exact same fragmentation problem in reverse, applied to English's own less-common inflected forms.

Embeddings and positional encoding. Each token maps to a learned vector — a word embedding — in a high-dimensional space where geometric proximity approximates semantic similarity. The king − man + woman ≈ queen result is the famous illustration, but present it with the caveats it deserves: it comes from an older generation of static word embeddings (word2vec/GloVe — one fixed vector per word, predating the Transformer), it's more fragile than the popular framing suggests (it typically only holds if "king," "man," and "woman" are excluded from the nearest-neighbor search, and doesn't generalize reliably to other analogies), and it is not the same mechanism as the contextual embeddings a Transformer produces, where self-attention recomputes a token's vector per-sentence — so "bank" gets a different vector in "river bank" versus "savings bank." Both kinds are learned from raw text with no explicit supervision on concepts like gender or royalty, but only the contextual kind is what BERT and GPT, below, actually use. One subtlety worth adding for this age band: because a Transformer processes all tokens in parallel rather than sequentially, it has no inherent notion of word order — positional encoding is added to each token's embedding specifically to inject that ordering information back in.

Self-attention and multi-head attention. This is the architectural core. Self-attention lets each token compute a weighted combination of every other token's representation, with weights determined by learned relevance — effectively, "how much should interpreting this token depend on that token." Walk through the classic disambiguation example on the board: "The trophy didn't fit in the suitcase because it was too big." Ask students to resolve what "it" refers to, then ask what would need to happen computationally for a model to get that right — the answer is that the representation for "it" needs to draw information from "trophy" specifically, across several intervening words, which is exactly what a learned attention weight between those two tokens accomplishes. Multi-head attention runs several of these attention computations in parallel with different learned parameters, letting the model capture different types of relationships simultaneously (say, syntactic dependency in one head and coreference in another). Layer normalization stabilizes training by normalizing activations between these sublayers, letting the very deep stacks used in real models train reliably.

BERT vs. GPT. Present these as two applications of the same Transformer building block, diverging on one key design choice: BERT is trained bidirectionally — it can condition on tokens both before and after the one it's predicting (via masked-language-model pretraining, where random tokens are hidden and the model learns to reconstruct them using surrounding context), which produces representations well-suited to classification and understanding tasks. GPT is trained autoregressively — each token is predicted using only preceding context, never future tokens — which is exactly the constraint that makes it usable for open-ended generation, since at inference time there is no "future" to condition on yet; you can't condition on text that doesn't exist yet. Close with fine-tuning: a pre-trained model, already holding broad language patterns from massive unlabeled text, can be adapted to a specific task using a comparatively small labeled dataset, which is why building a useful NLP system today rarely means training a model from scratch. It's worth naming the practical implication directly: pretraining on huge unlabeled text corpora is extremely expensive and done by a small number of well-resourced labs, but fine-tuning a pre-trained model is comparatively cheap, which is exactly what's made specialized NLP applications accessible to far more developers than a decade ago.

Worth flagging as an aside for this age band: the Transformer paper that introduced this architecture was titled "Attention Is All You Need" — the name itself was a claim that self-attention alone, without the recurrence or convolution earlier architectures relied on, was sufficient to achieve state-of-the-art results. That framing is a useful shorthand for what made the architecture such a significant departure from what came before it.

Run the practice round fast, then use it as a springboard: for each AI example (spam filter, auto-translate, autocomplete, chatbot), ask which of BERT-style or GPT-style behavior it more resembles — spam filtering and translation-quality scoring look like classification (BERT-family), autocomplete and chatbots look like generation (GPT-family), even though real production systems often blend both.

Discussion

• "Positional encoding exists because attention itself has no sense of word order. What would break in a sentence's meaning if that information were removed entirely?" (Push toward a concrete example: "the dog bit the man" and "the man bit the dog" contain identical tokens with identical attention relationships available — only word order distinguishes their meaning.)
• "Why does GPT's autoregressive constraint make it well suited for generation, but a worse fit than BERT for tasks like sentiment classification?" (Classification benefits from seeing the entire input at once before deciding; generation, by definition, produces output incrementally with no full input to condition backward from.)
• "Fine-tuning adapts a pre-trained model with a small labeled dataset instead of training from scratch. What does that imply about who can realistically build specialized NLP applications today versus ten years ago?" (The barrier to entry has dropped enormously — a small team can fine-tune an existing foundation model rather than needing the compute and data to pretrain one.)
• "We said Tagalog's infixes and layered affixes cause worse token fertility than English on the same tokenizer, purely because of training-data imbalance. If a company wanted to genuinely fix this rather than just work around it, what would actually have to change — the tokenizer, the training data, or something else? Would 'just add more Tagalog data' alone solve it?" (Push toward: more monolingual Tagalog text helps the affixation problem some, but the tokenizer's vocabulary itself would need to be re-learned or extended to include Tagalog-specific merges, and the code-switching problem specifically needs code-switched training examples, not just more of each language separately.)
• "If embeddings are learned purely from patterns in human-written text, what happens to biases already present in that text? Is there a clean fix?" (There isn't a fully clean fix — mitigation techniques exist, like debiasing embeddings post-hoc or curating training data, but none eliminate the underlying issue, since the model is fundamentally learning statistical regularities in whatever text it's given.)
• "Multi-head attention lets different heads capture different relationship types. Why might a single attention mechanism not be enough?" (A single attention pattern would have to average across every kind of relationship — syntax, coreference, topic — at once; separate heads let the model represent several distinct relationship types in parallel without them interfering.)
• "Few-shot prompting lets a model perform a new task from just a few in-context examples, without any weight updates. What does that suggest about what 'learning' means for these models, compared to how a traditional ML classifier is trained?"

Quiz Walkthrough

Wrap-Up & Extension

Close with: "Every fluent sentence a language model produces is the output of tokenization, learned embeddings, layered self-attention, and either bidirectional or autoregressive training — nothing more mystical than that, and nothing less genuinely sophisticated either. Understanding the mechanism is what lets you evaluate these systems critically instead of either worshipping or dismissing them."

Optional add-on — measure it directly: If students have internet access, have them open a public tokenizer visualizer (OpenAI's tokenizer tool is the simplest) and paste in three parallel sentences: a plain English sentence, its Tagalog translation using a couple of heavily-affixed words, and a natural Taglish version they'd actually text a friend. Have them count tokens per sentence and per word for each, then explain the pattern in terms of what was covered above — token fertility should visibly worsen for Tagalog and worsen further for the Taglish version, which is the training-data-imbalance argument made concrete instead of theoretical.

Extension activity — architecture comparison table: Have students research and build a short comparison table (BERT vs. GPT vs. a third model of their choice — T5, LLaMA, or a current model they use) across: training objective, typical use case, context handling (bidirectional vs. autoregressive), and one real product built on it. Have a few students present their table and defend why a company might pick one architecture family over another for a specific product (say, a resume-screening tool vs. a customer service chatbot).

For a group with extra time or a strong technical baseline, add a second layer to the exercise: ask each group to identify one thing that could realistically go wrong if their chosen architecture were deployed for their chosen product without safeguards — a BERT-based resume screener trained on biased historical hiring data, or a GPT-based chatbot that confidently states an incorrect return policy. Require the failure mode to be traceable to something covered today (embeddings encoding real-world bias, or autoregressive generation optimizing for plausibility rather than truth) rather than a generic "AI can be wrong" statement. This extends the 10–15 minute core lesson into a full period and connects the architecture concepts to real engineering trade-offs and responsibilities, which is the level this age band should be operating at.