Teaching "How Computers Hear" to Architect mode (ages 15–18)

Hook & Warm-Up

Open with the app's own hook line, which is dense on purpose — let it feel like the start of a real technical lecture, not a simplified recap:

"From Mel-frequency cepstral coefficients to attention-based encoder-decoder architectures, speech recognition has evolved from statistical methods to end-to-end deep learning. Let's examine the modern audio ML pipeline."

Ask: "Has anyone worked with audio data, even informally — music production, a voice-memo app, a Python audio library? What's the actual data type you're working with when you load a sound file?" (A digital waveform: a sequence of amplitude samples over time.) Then frame the session: "We're going to trace the real pipeline modern systems like Whisper use, from raw waveform to text, and where older statistical methods hit their limits."

If your class has already covered convolutional neural networks in "How Computers See," draw the parallel explicitly here: "Vision went from hand-crafted edge detectors to CNNs learning their own features. Audio went through almost the exact same transition, just a bit later and with a slightly different intermediate representation — the mel-spectrogram instead of a raw pixel grid. Watch for that pattern today."

Main Activity

Use the lesson's three scenes as checkpoints, but teach at the level the hook sets — this band can handle the real terminology, including where the app's own simplified language undersells the mechanism.

Scene 1 — Sound Waves 🌊. Establish the signal chain formally: sound pressure waves are transduced by a microphone into a continuous analog voltage signal; an analog-to-digital converter samples that signal at a fixed rate — 44.1 kHz is the CD-audio standard the lesson uses, though it's worth noting most speech models actually downsample to 16 kHz, since speech energy is concentrated below 8 kHz and the Nyquist rate for that is 16 kHz — capturing amplitude at each instant as a quantized number. From there, explain why raw waveforms are a poor direct input for a recognition model: they're extremely high-dimensional and don't expose frequency structure directly. The classic solution, worth naming explicitly since the hook mentions it, is the mel-spectrogram: a short-time Fourier transform (STFT) converts short overlapping windows of the waveform into frequency-domain slices, and the mel scale reweights those frequencies to match human pitch perception (roughly logarithmic, denser resolution at lower frequencies). The result — a 2D time-by-mel-frequency image — is the actual input to most modern ASR (automatic speech recognition) models, not the raw wave. Quantization is worth a sentence too, if a student asks about bit depth: each amplitude sample also gets rounded to a fixed number of discrete levels (16-bit audio gives 65,536 possible values per sample), which is a separate axis from sample rate — sample rate is "how often," bit depth is "how precisely each measurement is recorded."

Scene 2 — Speech Recognition 🗣️. This is where to unpack the historical shift the hook names. Older statistical pipelines used hand-engineered features (MFCCs — mel-frequency cepstral coefficients, a compressed representation of the mel-spectrogram) feeding Hidden Markov Models (HMMs) combined with Gaussian Mixture Models, requiring a separate pronunciation dictionary and language model. Modern systems replace this with end-to-end deep learning: an encoder (often a Transformer) processes the full mel-spectrogram sequence, and a decoder generates text directly, trained jointly rather than as separate hand-tuned stages. Two specific technical problems are worth naming, since the quiz asks about them directly: alignment — audio frames and output text tokens don't correspond one-to-one, since a single phoneme can span many frames — is solved by approaches like CTC (Connectionist Temporal Classification), which allows the model to output a probability distribution over an expanded label set (including a "blank" token) at every frame and collapse repeats during decoding; and data efficiency — fully-supervised training needs enormous amounts of transcribed audio, which is expensive — is addressed by self-supervised pre-training methods like wav2vec 2.0, which mask segments of the raw audio representation and train the model to predict them from context, learning useful audio representations from unlabeled audio before any transcripts are involved. On-device wake-word detection (the "Hey Siri" case) deliberately uses a much smaller, cheaper model — typically a small CNN or RNN optimized for low power draw and continuous operation — trading generality for the ability to run constantly without meaningfully affecting battery life; only once triggered does audio route to a larger model, frequently cloud-hosted.

Scene 3 — Sound AI 🎵. Frame each example by what kind of ML problem it actually is: Shazam is nearest-neighbor search over a fingerprint index built from spectral peak constellations, not a neural sequence model — a useful contrast, since it shows not every "sound AI" problem needs deep learning; Merlin's bird-ID is a supervised audio classification task, almost certainly using a CNN over spectrogram-like features trained on labeled species recordings; Nest's sound-event detection is likewise a classifier distinguishing categories (bark vs. glass breaking) from short audio segments; AirPods' active noise cancellation is fundamentally a real-time signal processing problem (generating a phase-inverted waveform), though recent implementations increasingly use learned models to predict the cancellation signal adaptively, blending the two paradigms.

Tie it back to Whisper explicitly, since the app's summary names it implicitly through the era it describes: Whisper is trained on roughly 680,000 hours of multilingual, multitask audio using large-scale weak supervision (unfiltered internet audio-transcript pairs, rather than a small curated dataset), which is a major reason for its robustness across accents, background noise, and languages compared to earlier systems trained on smaller, cleaner datasets. Then let students work through the in-app practice round independently.

Discussion

• "Why might weak supervision on 680,000 hours of noisy internet audio outperform a smaller, carefully curated, professionally transcribed dataset?" (Scale and diversity of accents, noise conditions, and languages can outweigh per-example label quality, up to a point.)
• "CTC solves an alignment problem. Can you think of another domain — outside speech — where inputs and outputs don't line up one-to-one, and a similar trick might apply?" (Handwriting recognition, video captioning, some sequence-to-sequence translation problems.)
• "Self-supervised pre-training (wav2vec) doesn't need transcripts. What does that trade off against fully supervised training, and why does that tradeoff matter for languages with little transcribed data?" (Guide toward: it enables low-resource languages to benefit from pre-training on unlabeled audio that does exist, even without expensive transcription.)
• "Why does it make engineering sense to keep wake-word detection on a small on-device model instead of just running the same large model used for full transcription all the time?" (Power, latency, and privacy — the large model literally couldn't run continuously on a phone's battery.)
• "Shazam doesn't use a neural network the way modern speech recognition does. Does that make it less 'AI'? What definition of AI does your answer depend on?"
• "Sample rate (how often) and bit depth (how precisely) are two independent design choices in a digitization pipeline. If you were designing a cheap, low-power wake-word chip, which one would you be more willing to cut, and why?" (There's no single right answer — the point is reasoning about the tradeoff explicitly: coarser bit depth mainly loses fidelity in quiet moments, while a lower sample rate can start losing entire frequency ranges that speech recognition depends on.)
• "Shazam's fingerprinting approach and modern encoder-decoder ASR are both, in a sense, matching patterns against something learned in advance. Where does the boundary between 'classical signal processing' and 'AI' actually sit, if it sits anywhere?" (There's no clean answer — a productive discussion should surface that a fixed, human-designed matching algorithm (Shazam's) and a model that learned its own decision boundaries from training data (ASR's) are doing meaningfully different things even when both get labeled "AI" colloquially.)

Quiz Walkthrough

Wrap-Up & Extension

Close with: "The shift from MFCCs and HMMs to end-to-end Transformer encoder-decoders is one of the clearest examples in all of AI of hand-engineered features being replaced by features the model learns for itself — the same shift that happened in computer vision when CNNs replaced hand-crafted edge detectors. The engineering choices around where a model runs — on-device versus cloud, small versus large — are just as important as the model architecture itself."

Extension activity — "Pipeline Trade-off Memo": Have students write a short (half-page) design memo for a hypothetical product: a real-time captioning feature for a video call app used on low-end Android phones with unreliable internet in the provinces. The memo should specify and justify: what runs on-device versus in the cloud, what sample rate and feature representation (raw waveform, MFCC, or mel-spectrogram) makes sense given power constraints, whether a CTC-style or full encoder-decoder architecture fits better given the need for real-time (streaming) output rather than waiting for a full utterance, and what happens to accuracy or latency if connectivity drops mid-call. This stretches the 10–15 minute core lesson into a full 40–45 minute period and forces the trade-offs discussed abstractly into a concrete, locally-relevant engineering decision. Push students to be specific rather than hand-wavy about the streaming constraint in particular — a non-streaming encoder-decoder that waits for a full utterance before producing any text is a poor fit for live captions, since captions that lag several seconds behind a live speaker defeat the point of the feature, which is exactly why CTC-style frame-synchronous decoding (or streaming variants of the encoder-decoder architecture) matters for this specific use case even though a full encoder-decoder can be more accurate overall on non-real-time transcription.