Topic
Sequence Modeling
6 episodes
-
Natural Language Processing before LLMs a Hugging Face Space by dlouapre
Pippa and Tyler explore a Hugging Face Space by dlouapre that teaches classical NLP — n-grams, TF-IDF, word2vec, RNNs — through hands-on exercises using IMDB and WikiText-2. They discuss why pre-LLM fundamentals still matter for building intuition about tokenization, context windows, and transfer learning, and how the course structure (starter code, folded hints, checkpoints) mirrors the way teams should evaluate modern LLM components.
-
Attention Is All You Need
Onyx and Echo dig into the original Transformer paper — what problem it solved, how scaled dot-product attention and multi-head attention actually work, why dropping recurrence was the real lever, and whether the architectural choices have held up across nearly a decade of subsequent work.
-
Overview: Fine tuning on Execution Traces
We keep throwing around fine-tuning on execution traces, so we finally define it from the ground up. We’re talking about training a model on the steps, not just the answer, and why that changes what it learns.
-
Overview: Sequence Modeling
We slow down and finally define sequence modeling, the idea underneath next-token prediction, language models, and a surprising amount of modern AI. We keep it grounded in one picture: covering the next word and training a model to guess what belongs there.
-
Overview: State Management in Language Models
We finally do the episode we keep circling back to: state management in language models. We walk through the idea from the ground up, using the cache-and-notes picture to show why models don’t have to recompute everything every token, and where that trade-off starts biting.
-
Overview: Autoregressive Generation
We finally slow down and make autoregressive generation click: the whole thing is just a model writing one token, then using what it wrote to choose the next one. We keep the focus on the loop, the trade-offs, and why that one-step-at-a-time setup is still the backbone of modern language models.