Topic
Llama 3 1
3 episodes
-
Many Shot CoT ICL: Making In Context Learning Truly Learn
Justy and Cody dig into a paper arguing that long-context chain-of-thought prompting behaves less like stuffing a prompt with relevant examples and more like teaching the model during inference. They unpack why many-shot tricks from classification break on reasoning, why semantic retrieval stops helping, and how the paper’s Curvilinear Demonstration Selection tries to order examples like a smooth mini-curriculum.
-
Google AI breakthrough means chatbots use six times less memory during conversations without compromising performance
Google's TurboQuant compresses AI working memory (the KV cache) by up to 6x in real time using two novel techniques — PolarQuant and QJL — without degrading model performance. Justy and Cody dig into what this actually means for inference costs, who benefits first, and why the 'DeepSeek moment' framing is both apt and a little overblown.
-
New KV cache compaction technique cuts LLM memory 50x without accuracy loss
MIT researchers developed Attention Matching, a KV cache compaction technique that achieves 50x memory reduction in LLMs without accuracy loss, solving a critical bottleneck for enterprise applications handling long contexts.