Topic
Tensor Parallelism
2 episodes
-
The efficient frontier of LLM inference
Baseten's 'Efficient Frontier of LLM Inference' argues that inference engineering involves two fundamentally different types of techniques: those that let you trade off between latency and throughput along an existing efficiency curve, and those that push the entire curve outward, creating universal gains. The article maps concrete techniques (batch sizing, parallelism strategies, quantization, kernel optimization, speculative decoding, prefill/decode disaggregation) onto this framework and shows that the frontier is jagged and empirically discovered, not smooth.
-
Kimi K3 Is Here: Efficient Day 0 Support on vLLM
Vince and Ava unpack Moonshot AI's Kimi K3, a 2.8‑trillion‑parameter multimodal MoE, and its day‑zero support in vLLM. They walk through the model’s hybrid attention, the engineering tricks that make a 1 M‑token context feasible, the practical deployment recipe, and how it stacks up against other frontier models.