Topic
Calibration
8 episodes
-
AREX: Towards a Recursively Self Improving Agent for Deep Research
Pippa and Tyler dig into AREX, a recursively self-improving deep research agent from BAAI that alternates between an inner search loop and an outer constraint-verification loop — and discuss whether that architecture is genuinely novel or a smarter repackaging of ideas the field already had.
-
Overview: Calibration
We finally slow down and make calibration click: what it means for a model’s confidence to match reality, how you measure that, and why it matters when you actually want to trust the thing. We keep it grounded in the systems we’ve been circling for ages, because calibration is everywhere once you start looking.
-
Introducing TabFM: A zero Shot foundation model for tabular data
Justy and Cody examine TabFM, Google Research’s zero-shot foundation model for tabular classification and regression. They unpack its hybrid row-column attention design, synthetic-data training, TabArena evidence, the trade-off between out-of-the-box convenience and tuned ensembles, and whether BigQuery integration could make this genuinely useful in everyday data workflows.
-
Thinking Machines open sources first multimodal language model, Inkling, focused on low cost and 'resistance to censorship'
Inkling, Thinking Machines' open-source multimodal MoE model (975B total / 41B active parameters), lands as a broad, balanced generalist with a standout feature: a controllable 'thinking effort' knob to dial cost vs. performance from 0.2 to 0.99. Enterprises get native text+image+audio fusion, Apache 2.0 weights, and a lighter Inkling-Small preview, but benchmarks show it trails specialized open and closed models on coding and pure reasoning, while remaining competitive on multimodality and agentic workflows. The episode debates whether the real win is the runtime control surface (Tinker platform) and a cautious, non-censoring epistemics posture — not the headline parameters.
-
LLM Evaluation Frameworks Compared: How to Actually Measure What Your Model Does MachineLearningMastery
Fern and Lintel dig into a comparison of RAGAS, DeepEval, and Promptfoo, landing on the article’s real argument: eval frameworks are less about novel metrics than about where evaluation fits in your workflow, and the dangerous part is trusting LLM-as-a-judge without auditing its biases. They like the article’s practical split between RAG scoring, CI gates, and prompt red-teaming, but push on where the examples are a little too toy-like and where teams can overread framework choice as the main problem instead of test-set design and human calibration.
-
Large language models often prioritize Western moral values, overlooking other cultures
A research paper finds LLMs tend to mirror Western moral priorities when asked to roleplay citizens of 48 countries, and two hosts discuss what this actually means for users, products, and culture.
-
Overview: Reinforcement Learning from Human Feedback
We finally define reinforcement learning from human feedback the way we keep using it: as a loop where human preferences become a learned reward signal that steers a model after its initial training. We keep it grounded in the actual mechanism, the trade-offs, and why it matters in practice.
-
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
Jessica and Cathy dig into RL with Metacognitive Feedback (RLMF): a post-training loop that rewards models not just for correct answers, but for accurately judging how well they did—improving both task performance and the faithfulness of uncertainty expressions. They explain the mechanism (metacognitive data selection and metacognitive advantage scaling), discuss trade-offs, and debate whether this is still research-only or actually shippable.