Ep 1127 GitHub 4:28 w/ Pippa & Tyler

Natural Language Processing before LLMs a Hugging Face Space by dlouapre

Pippa and Tyler explore a Hugging Face Space by dlouapre that teaches classical NLP — n-grams, TF-IDF, word2vec, RNNs — through hands-on exercises using IMDB and WikiText-2. They discuss why pre-LLM fundamentals still matter for building intuition about tokenization, context windows, and transfer learning, and how the course structure (starter code, folded hints, checkpoints) mirrors the way teams should evaluate modern LLM components.

Embed this episode

Paste this on any site — the player is a self-contained iframe with no cookies or trackers.

<iframe src="https://sandrise.io/exploring-next/embed/1127"
  width="100%" height="180" style="max-width:640px;border:0;border-radius:12px;overflow:hidden"
  title="Exploring Next — Episode 1127 audio player"
  loading="lazy" allow="autoplay" referrerpolicy="strict-origin-when-cross-origin"></iframe>
Embed & API docs →
Script Nemotron 3 Ultra 550B A55B Voice Inworld TTS 2

Transcript

Pippa Okay, so there's this Hugging Face Space — dlouapre's "NLP before LLMs" — and it's basically a whole course chapter disguised as a notebook. First lecture of a master's course in Paris. And the hook is: here's why you should spend a few hours on techniques that LLMs supposedly made obsolete.

Tyler Right. The genealogy argument. You don't study n-grams because you'll deploy them — you study them because the same two questions drove the field for decades: how do you handle variable-length text input, and how do you transfer learning across tasks when language is fundamentally the same everywhere?

Pippa Exactly. And the course makes it concrete immediately — two datasets, two running tasks. IMDB movie reviews for sentiment classification, fifty thousand labeled reviews. WikiText-2 for language modeling, two million words from Wikipedia with a thirty-three-thousand word vocab. Both load straight from the Hub.

Tyler Yeah. stanfordnlp/imdb and Salesforce/wikitext. Plus TinyShakespeare and text8 as side datasets. The structure is deliberate: starter code loads the data, hints and solutions are folded so you don't peek, checkpoints show expected outputs. Every exercise is thirty minutes max, a few dozen lines.

Pippa And the constraint I love: do it without a coding agent. Which… honestly feels like the same discipline as writing your own evals instead of trusting a leaderboard. You have to feel where the representation breaks.

Tyler Mm-hm. The first exercise is literally unigram and bigram Markov models on WikiText-2. Count words, normalize by corpus size, that's your p(x_i). Then bigram: p(x_i | x_{i-1}) = count(x_{i-1}, x_i) / count(x_{i-1}). You see immediately why context length matters — a bigram only remembers one word back.

Pippa Right, and then the course walks you forward: TF-IDF for classification, word2vec for dense embeddings, RNNs and LSTMs for sequence modeling. Each step solves a limitation of the previous one. Vocabulary size explodes with n-grams. TF-IDF loses word order. Word2vec fixes semantics but still no context. RNNs handle sequence but vanish gradients.

Tyler The through-line that clicks for me: next-word prediction IS classification. Just… classification over a vocabulary of tens of thousands instead of two sentiment classes. Same conditional probability math. The Space even shows the chain rule decomposition explicitly — p(x1 x2 x3 x4) = p(x1) p(x2|x1) p(x3|x1 x2) p(x4|x1 x2 x3). That's the autoregressive generation loop in four lines.

Pippa And the translation section — there's this great visual of the same sentence in English, French, German, Turkish. French keeps English order, German moves verbs to clause ends, Turkish nearly reverses everything. That word alignment problem is exactly why attention replaced fixed positional assumptions. You see the transformer coming before it arrives.

Tyler Oh interesting. So the course doesn't just teach old techniques — it stages the problems that made each next technique necessary. By the time you hit the transformer chapter, you've felt the pain of fixed context, vanishing gradients, alignment mismatch.

Pippa Yeah. Research taste, basically. The author calls it that explicitly — you develop taste by seeing which ideas survived and why. And there's a reading list: a few dozen seminal papers you 'must have seen once in your life if you want to be serious about the topic.'

Tyler Right. So Build Next is straightforward: open the Space, their write-up Load the datasets. Run the first n-gram exercise. Twenty minutes, thirty lines. You'll have a working language model that generates garbage — but you'll understand exactly why it's garbage, and what each subsequent chapter adds to fix it.

Pippa And the mental model to carry forward: every LLM component you evaluate today — tokenizer, embedding layer, attention head, context window — is an answer to one of these classical problems. The scale changed. The questions didn't.

Tyler Solid place to start. Or restart, depending on how long you've been at this.

Pippa Exactly. Thanks for the walk-through, Tyler.