Topic

Sequence Modeling

6 episodes

  1. Ep 1127

    Natural Language Processing before LLMs a Hugging Face Space by dlouapre

    Pippa and Tyler explore a Hugging Face Space by dlouapre that teaches classical NLP — n-grams, TF-IDF, word2vec, RNNs — through hands-on exercises using IMDB and WikiText-2. They discuss why pre-LLM fundamentals still matter for building intuition about tokenization, context windows, and transfer learning, and how the course structure (starter code, folded hints, checkpoints) mirrors the way teams should evaluate modern LLM components.

  2. Ep 860

    Attention Is All You Need

    Onyx and Echo dig into the original Transformer paper — what problem it solved, how scaled dot-product attention and multi-head attention actually work, why dropping recurrence was the real lever, and whether the architectural choices have held up across nearly a decade of subsequent work.

  3. Ep 766

    Overview: Fine tuning on Execution Traces

    We keep throwing around fine-tuning on execution traces, so we finally define it from the ground up. We’re talking about training a model on the steps, not just the answer, and why that changes what it learns.

  4. Ep 757

    Overview: Sequence Modeling

    We slow down and finally define sequence modeling, the idea underneath next-token prediction, language models, and a surprising amount of modern AI. We keep it grounded in one picture: covering the next word and training a model to guess what belongs there.

  5. Ep 636

    Overview: State Management in Language Models

    We finally do the episode we keep circling back to: state management in language models. We walk through the idea from the ground up, using the cache-and-notes picture to show why models don’t have to recompute everything every token, and where that trade-off starts biting.

  6. Ep 630

    Overview: Autoregressive Generation

    We finally slow down and make autoregressive generation click: the whole thing is just a model writing one token, then using what it wrote to choose the next one. We keep the focus on the loop, the trade-offs, and why that one-step-at-a-time setup is still the backbone of modern language models.