Topic
Calibration
30 episodes
-
d1.2: State of the Art Omnimodal Decision Models | Blog
Liquid AI releases d1.2 and d1.2-ultra decision models with audio input and five new structured-output tasks. d1.2-ultra scores 75 on Artificial Analysis Decision Index v2.1, leading the field. Pricing at $0.02 and $0.042 per million input tokens respectively, with free output tokens. Models available now on Liquid API, coming to OpenRouter and Vercel soon.
-
Compare AI agent changes with Online Experiments in Agent Observability | Datadog
Datadog’s Online Experiments ties agent evals to classic A/B testing: you compare prompt/model/tool variants on real traffic, with quality, cost, latency, and errors in one view. Their internal GPT‑5.6 vs GPT‑6 Luna case shows how this surfaces a clear cost win but ambiguous reliability, forcing you to dig into traces before flipping a switch.
-
How to Train Your own Decision Model like Jev using Unsloth MachineLearningMastery
Pippa and Tyler unpack Unsloth’s new decision-model workflow: taking an open-weight model and training it to pick from fixed options instead of generating text. They focus on why that matters for intent routing, what the FastDecisionModel setup actually does, and where the trade-off sits between a cleaner product fit and the usual questions about calibration, memory, and whether a decision head is the right abstraction.
-
Why human insight is a strategic advantage for building better agents
LangChain's Gowri Harish and Sineha Manivannan argue that human expertise is load-bearing for agent evaluation—not as a final sanity check, but as the mechanism that defines what 'good' means, surfaces gaps in automated evals, and turns domain judgment into reusable evaluation data. The piece centers on three principles: expert judgment defines quality (not metrics alone); context (full conversation threads, not isolated traces) is essential to that judgment; and structured human feedback becomes the foundation for calibrating LLM judges and building golden datasets. The case study (Fetch Rewards) shows reviewers inspecting tool calls, metadata, and user context to catch fluent-but-wrong answers—work that would collapse without the full interaction visible.
-
Why Do Multi Agent LLM Systems Fail?
A close look at MAST, a taxonomy and dataset for diagnosing why multi-agent LLM systems fail. The paper’s practical contribution is turning vague coordination failures into observable categories, while its evidence also shows why labeling root causes remains difficult.
-
Does Scaling Toward AGI Depend on What System 1 Lets System 2 Execute?
Edmund and Geffen examine a perspective paper proposing conditional interaction: a cheap System-1 predictor filters costly System-2 reasoning, tool calls, rollouts, or experiments before they happen. They focus on the paper’s useful contribution, the density of utility-relevant interactions, and the practical conditions that would make this more than a metaphor.
-
Introducing Claude Duet
Vince and Ava dig into Claude Duet as a proactive agent app that tries to turn scattered work into a single ongoing thread. They focus on the real product claim, the safety machinery behind it, and the practical question of whether this is actually useful for people with too many moving parts.
-
Decision models like Jev handle what SQL rules cannot. Here's how data teams use them
Jessica and Cathy dig into decision models — specifically TypeSafe AI's Jev — and the argument that structured, typed ML judgments are the missing middle layer between SQL rules and full LLM calls in data pipelines.
-
Memadapter: Counterfactual Adaptation Against Memory induced Sycophancy
Episode 1053 digs into MemAdapter, a framework for stopping long-term agent memory from becoming a quiet source of sycophancy. Onyx and Echo focus on the paper's key shift: a memory can be accurate and still deserve less influence in a particular decision.
-
Jev offers a cheaper way to make AI decisions | VentureBeat
Jev’s real innovation is treating many enterprise AI calls as typed control decisions instead of text generation. Onyx and Echo examine its speed, price, calibration claims, open reproductions, and the practical limits of replacing LLM calls with classifiers.
-
Jev in the Agent Loop: A Complete Guide to Decision Layer Automation
Justy and Cody dig into "Jev in the Agent Loop," arguing the real move isn’t smarter models, but cheaper, structured decision-making—replacing frontier-priced LLMs on every internal fork with Jev. They debate where Jev actually fits, what it can’t do, and why this reframes agent infrastructure as cost optimization at the decision layer, not generation. Cody pushes back on the limits: Jev can’t generate, so it only works in pipelines where you can cleanly separate choosing from doing. Justy sees immediate product wins in agent tooling, and they swap examples and a little banter on what gets hyped as "agentic" when it’s really just moving decisions around.
-
Rufus Air: An Open LLM Post Training Recipe
Pippa and Tyler unpack Rufus-Air, Amazon's openly documented eight-stage post-training recipe for GLM-4.5-Air-Base, and argue that the important release is the training discipline rather than another model scorecard.
-
Can Jev Be a Better Agent Evaluator?
LangChain's evaluation of TypeSafe AI's Jev as an agent evaluator reveals a fundamentally different architecture — not an LLM that generates text, but a 'System One' model that returns typed decisions with calibrated probabilities. In a narrow test against GPT-5.6 Luna, Terra, and Claude Sonnet 4.6, Jev matched human oracle accuracy on binary pass/fail decisions (100% vs. 80–99.8%), achieved 92–913x lower variance on continuous scoring, and cost $0.00035 per call versus $28.17 for Claude. The implication: agent evals may have a third viable path beyond code-based (narrow, deterministic) and LLM-as-judge (slow, expensive, non-deterministic). The catch: this is one narrow test on five weather requests, and low cost can amplify mistakes at scale.
-
Introducing System One Models & Jev TypeSafe AI Blog
TypeSafe AI launches Jev, the first 'System One' model: no text generation, but blazingly fast, type-safe, parallel, probabilistic decisions for automation at the code layer. Jev claims 40–200x speedups versus LLMs like GPT-5.6 Terra, with zero type errors and per-decision calibration, aiming to finally unlock real-world automation where string output can't go. This episode dives into how Jev works, what it means to give up freeform text, and whether TypeSafe’s new architecture marks the start of a real post-LLM automation wave.
-
What Is Jev? A Guide to TypeSafe AI’s System One Model
Masonry and Eyre dig into Jev as a different kind of model bet: not a chatty LLM replacement, but a fast structured-decision layer for agent loops. They focus on the real product angle, the mechanics behind parallel typed questions, and where the claims are strong versus still a little hand-wavy.
-
Enabling Creative Explorationfor Vibe Design Agents
Vince and Ava dig into a paper about separating design exploration from code generation in vibe design agents. They focus on why temperature is too blunt, how structured design specs plus external selection give you controlled variety, and what the offline and online results actually say about usefulness versus uncertainty.
-
Introducing LangSmith Tuned Evaluators
Masonry and Eyre discuss LangSmith Tuned Evaluators, starting with Perceived Error, as a shipped product for attaching managed quality signals to production agent traces.
-
Introducing the Conceptual Reasoning Index
Pippa and Tyler dig into the Conceptual Reasoning Index, a new three-benchmark evaluation suite for argument judgment, logical consistency, and decision-theoretic reasoning. They like the attempt to measure work where answers cannot simply be checked, while questioning whether a changing aggregate score can earn trust as a durable comparison tool.
-
Overview: Construct validity
We slow down and make construct validity click: the gap between the label on a test and what the test actually measures. We connect it to benchmarks, hiring screens, model validation, and the Cursor reward-hacking story we keep circling back to.
-
Overview: Bayes' Theorem
We finally slow down on Bayes' Theorem, the belief-updating rule we keep smuggling into conversations about evals, spam filters, diagnosis, ranking, and calibration. We make it click through one package-sorting picture: evidence only matters against the pile it came from.
-
Grading the Narrators: An Isnād–Rijāl Framework for Claim Level Provenance in Multi Agent Knowledge Systems
Ali Zahid Raja's 'Grading the Narrators' adapts isnād–rijāl methodology from classical Islamic hadith science to solve a concrete problem in modern multi-agent knowledge systems: how to trust claims that have passed through chains of autonomous transformations. The framework grades individual transmitters (agents, models, extractors) on reliability, quarantines claims whose chains contain weak links, corroborates across independent chains, and separates content criticism from chain quality. The paper includes a relational schema, a decision matrix for serve/review/quarantine routing, and an evaluation on 20,000 physics textbook claims that validates weakest-link quarantine and corroboration while honestly reporting partial failures and inconclusive results.
-
AREX: Towards a Recursively Self Improving Agent for Deep Research
Pippa and Tyler dig into AREX, a recursively self-improving deep research agent from BAAI that alternates between an inner search loop and an outer constraint-verification loop — and discuss whether that architecture is genuinely novel or a smarter repackaging of ideas the field already had.
-
Overview: Calibration
We finally slow down and make calibration click: what it means for a model’s confidence to match reality, how you measure that, and why it matters when you actually want to trust the thing. We keep it grounded in the systems we’ve been circling for ages, because calibration is everywhere once you start looking.
-
Introducing TabFM: A zero Shot foundation model for tabular data
Justy and Cody examine TabFM, Google Research’s zero-shot foundation model for tabular classification and regression. They unpack its hybrid row-column attention design, synthetic-data training, TabArena evidence, the trade-off between out-of-the-box convenience and tuned ensembles, and whether BigQuery integration could make this genuinely useful in everyday data workflows.
-
Thinking Machines open sources first multimodal language model, Inkling, focused on low cost and 'resistance to censorship'
Inkling, Thinking Machines' open-source multimodal MoE model (975B total / 41B active parameters), lands as a broad, balanced generalist with a standout feature: a controllable 'thinking effort' knob to dial cost vs. performance from 0.2 to 0.99. Enterprises get native text+image+audio fusion, Apache 2.0 weights, and a lighter Inkling-Small preview, but benchmarks show it trails specialized open and closed models on coding and pure reasoning, while remaining competitive on multimodality and agentic workflows. The episode debates whether the real win is the runtime control surface (Tinker platform) and a cautious, non-censoring epistemics posture — not the headline parameters.
-
LLM Evaluation Frameworks Compared: How to Actually Measure What Your Model Does MachineLearningMastery
Fern and Lintel dig into a comparison of RAGAS, DeepEval, and Promptfoo, landing on the article’s real argument: eval frameworks are less about novel metrics than about where evaluation fits in your workflow, and the dangerous part is trusting LLM-as-a-judge without auditing its biases. They like the article’s practical split between RAG scoring, CI gates, and prompt red-teaming, but push on where the examples are a little too toy-like and where teams can overread framework choice as the main problem instead of test-set design and human calibration.
-
Large language models often prioritize Western moral values, overlooking other cultures
A research paper finds LLMs tend to mirror Western moral priorities when asked to roleplay citizens of 48 countries, and two hosts discuss what this actually means for users, products, and culture.
-
Overview: Reinforcement Learning from Human Feedback
We finally define reinforcement learning from human feedback the way we keep using it: as a loop where human preferences become a learned reward signal that steers a model after its initial training. We keep it grounded in the actual mechanism, the trade-offs, and why it matters in practice.
-
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
Jessica and Cathy dig into RL with Metacognitive Feedback (RLMF): a post-training loop that rewards models not just for correct answers, but for accurately judging how well they did—improving both task performance and the faithfulness of uncertainty expressions. They explain the mechanism (metacognitive data selection and metacognitive advantage scaling), discuss trade-offs, and debate whether this is still research-only or actually shippable.
-
Evals are the new PRD
Braintrust argues that, for AI products, evals should replace traditional PRDs as the living definition of quality. Laura sees a practical operating-model win, while Harper agrees with the core idea but insists evals are executable acceptance criteria, not a substitute for product judgment.