GitHub FareedKhan Dev/train LLM From scratch: A straightforward method for training your LLM, from downloading data to generating text.
A single-GPU end-to-end LLM training guide lands on GitHub—hand-written PyTorch, pretrain to chat in one repo, plus full RLHF. Vince is giddy; Ava wants to know which corner of the GPU shelf this actually runs on. Build Next shows the exact CLI to kick it off on a T4.
Transcript
Vince Okay, okay—so there’s this repo that’s basically a one-way ticket to a PhD position if you follow the README closely enough. Train LLM From Scratch, written in plain PyTorch, no transformers, no peft, no trl—just raw attention and hand-written loops.
Ava Right. And it magically fits a billion parameters on a single GPU—sure.
Vince That’s not the half of it—this is end-to-end. Raw text to tokenized pile to base model to SFT to reward model to PPO to GRPO to chat interface. One repo, one idea.
Ava One idea is a loop: tokenize, next-token prediction, tweak loss, repeat until it does what you want. Still needs hardware.
Vince Fair—but the student appeal is huge. If you’re trying to land a PhD spot, showing you trained your own model from scratch is … kind of perfect?
Ava Stop—this is a teaching repo, not a product. Still excited to see it.
Vince I’m more than excited—I’m giddy. Look at the GPU table: 13M model fits on a T4, a billion needs an A100. That’s not a bug, that’s a feature. It’s honest about the ladder.
Ava On a free Colab T4? You’ll max out around two billion. A billion-parameter run is still fantasy land.
Vince Yeah, yeah, but the point is you can START on a T4. For most of us, that’s the only step that matters. Tomorrow you need a bigger card, you rent time or save up—still better than chasing GPUs as a student.
Ava Okay, okay. And the 5090 finally pushes into the billion-parameter zone—if you’re willing to drop two grand and pull three-fifty watts.
Vince Exactly. Hardware gate, not model gate. That’s the clarity I love.
Ava Sure. Now, the post-training half—the SFT, reward model, DPO, PPO, GRPO—all hand-coded inside one tiny transformer? That’s the part that actually interests me.
Vince I see. So it’s not just pretraining—it’s the full RLHF pipeline baked into one repo. Researchers get a compact stack to hack, not just a toy.
Ava Compact, yes. Maintainable? Only if the dual config systems don’t bite you.
Vince Right—two configs: plain Python constants for legacy pretrain, JSON plus CLI overrides for everything else. It’s editable, import-friendly, and it ships smoke tests that finish on a CPU in seconds.
Ava Smoke tests are the lifeboat. Still—two config systems means two places to mis-read the README.
Vince True, but the smoke configs exist for exactly that reason. Run a five-second check, then scale up.
Ava Fair. Now the data side—four streams, r50k_base, HDF5 files, loss masks that train only on assistant tokens. Okay, I’ll admit it: the peek at the first ten token IDs is cute.
Vince Ten integers that decode straight back to text—yeah, that’s the sausage. The whole thing is tokenization plus loss masking plus ‘keep tweaking the loss.’
Ava If you squint. Still—the loss mask is the bit that keeps the model from parroting prompts.
Vince Agreed. And the color-coded legend in the README means every block has a job, not just a name—nice touch for students.
Ava …Still a teaching tool. But if we’re handing out gold stars, I’ll give one for the chat template.
Vince Oh, come on—this is the rare repo that actually lets you go full loop. Pretrain base, SFT it, build a reward model, run PPO or GRPO, then talk to it. All on your own GPU.
Ava On your own big-ish GPU. Or one you rent.
Vince Exactly. Now—where does this sit in the race? It’s not a frontier model, it’s a training lab. Open-weight snapshot as of June 2026: DeepSeek V4 Flash leads the pack, Z.ai’s GLM-5.2 is pushing agentic coding, and the frontier tier is still Claude/GPT/ Gemini.
Ava So this is four-to-twelve weeks behind the top closed systems, not miles behind. Still useful pressure.
Vince And useful for students and open labs who want to hack post-training without buying a data-center cluster.
Ava …I’ll grant you that. Still—remember when you swore off GraphQL?
Vince Oh, that’s right, you dragged that up. Yes, I still stand by it. But this? This is different.
Ava Okay, okay. Build Next: how do we actually run this?
Vince Clone the repo, editable install—pick the extras you need. git clone the link in the show notes cd train-llm-from-scratch pip install -e "." if you’re doing training, or "." if you want the full tour.
Ava And for a 13M pretrain run on a T4:
Vince python scripts/prepare_pretrain_data.py --split train --num_shards 1 --out data/pile_train.h5 python scripts/train_transformer.py
Ava Then chat it with the Streamlit panel.
Vince There you go. Now go update your PhD application and tell them you trained an LLM from scratch on one GPU.
Ava …Good luck with that. I’m filing this under “teaching aid that will outlive most PhD timelines.”