Topic
Cross Attention
5 episodes
-
Overview: Reranking
We finally slow down on reranking, the search-and-RAG move we keep name-dropping whenever retrieval quality comes up. We build it from the basic intuition, then get into why the second scoring pass helps, where it breaks, and why it is still very much alive in current systems.
-
Attention Is All You Need
Onyx and Echo dig into the original Transformer paper — what problem it solved, how scaled dot-product attention and multi-head attention actually work, why dropping recurrence was the real lever, and whether the architectural choices have held up across nearly a decade of subsequent work.
-
Concurrent Image Understanding and Generation: Self Correcting Coupled Markov Jump Processes
Fern and Lintel dig into a new paper on doing image understanding and image generation at the same time, inside one decoding loop. The hook is simple: most systems either describe first and draw later, or they run both sides in parallel without letting the latest text and image decisions correct each other mid-step. This paper tries to fix that with a coupled masked-diffusion sampler that can both coordinate and backtrack.
-
Overview: Attention Mechanism
We finally slow down and explain the attention mechanism from the ground up: why models need selective focus, how query-key-value attention works, and why it became the engine under transformers, long context, and hybrid attention systems.
-
nvidia/Nemotron TwoTower 30B A3B Base BF16 · Hugging Face
Justy and Cody dig into NVIDIA’s Nemotron-TwoTower-30B-A3B-Base-BF16 and whether block-wise diffusion decoding is a real systems win or just a benchmark-shaped detour. Cody is skeptical about the headline throughput claim and the way the model compares itself to a single autoregressive baseline, while Justy focuses on who actually benefits from faster generation without a big quality drop. They land on cautious interest: interesting infrastructure idea, but not a universal replacement for standard decoding.