Topic
Calibration
18 episodes
-
Can Jev Be a Better Agent Evaluator?
LangChain's evaluation of TypeSafe AI's Jev as an agent evaluator reveals a fundamentally different architecture — not an LLM that generates text, but a 'System One' model that returns typed decisions with calibrated probabilities. In a narrow test against GPT-5.6 Luna, Terra, and Claude Sonnet 4.6, Jev matched human oracle accuracy on binary pass/fail decisions (100% vs. 80–99.8%), achieved 92–913x lower variance on continuous scoring, and cost $0.00035 per call versus $28.17 for Claude. The implication: agent evals may have a third viable path beyond code-based (narrow, deterministic) and LLM-as-judge (slow, expensive, non-deterministic). The catch: this is one narrow test on five weather requests, and low cost can amplify mistakes at scale.
-
Introducing System One Models & Jev TypeSafe AI Blog
TypeSafe AI launches Jev, the first 'System One' model: no text generation, but blazingly fast, type-safe, parallel, probabilistic decisions for automation at the code layer. Jev claims 40–200x speedups versus LLMs like GPT-5.6 Terra, with zero type errors and per-decision calibration, aiming to finally unlock real-world automation where string output can't go. This episode dives into how Jev works, what it means to give up freeform text, and whether TypeSafe’s new architecture marks the start of a real post-LLM automation wave.
-
What Is Jev? A Guide to TypeSafe AI’s System One Model
Masonry and Eyre dig into Jev as a different kind of model bet: not a chatty LLM replacement, but a fast structured-decision layer for agent loops. They focus on the real product angle, the mechanics behind parallel typed questions, and where the claims are strong versus still a little hand-wavy.
-
Enabling Creative Explorationfor Vibe Design Agents
Vince and Ava dig into a paper about separating design exploration from code generation in vibe design agents. They focus on why temperature is too blunt, how structured design specs plus external selection give you controlled variety, and what the offline and online results actually say about usefulness versus uncertainty.
-
Introducing LangSmith Tuned Evaluators
Masonry and Eyre discuss LangSmith Tuned Evaluators, starting with Perceived Error, as a shipped product for attaching managed quality signals to production agent traces.
-
Introducing the Conceptual Reasoning Index
Pippa and Tyler dig into the Conceptual Reasoning Index, a new three-benchmark evaluation suite for argument judgment, logical consistency, and decision-theoretic reasoning. They like the attempt to measure work where answers cannot simply be checked, while questioning whether a changing aggregate score can earn trust as a durable comparison tool.
-
Overview: Construct validity
We slow down and make construct validity click: the gap between the label on a test and what the test actually measures. We connect it to benchmarks, hiring screens, model validation, and the Cursor reward-hacking story we keep circling back to.
-
Overview: Bayes' Theorem
We finally slow down on Bayes' Theorem, the belief-updating rule we keep smuggling into conversations about evals, spam filters, diagnosis, ranking, and calibration. We make it click through one package-sorting picture: evidence only matters against the pile it came from.
-
Grading the Narrators: An Isnād–Rijāl Framework for Claim Level Provenance in Multi Agent Knowledge Systems
Ali Zahid Raja's 'Grading the Narrators' adapts isnād–rijāl methodology from classical Islamic hadith science to solve a concrete problem in modern multi-agent knowledge systems: how to trust claims that have passed through chains of autonomous transformations. The framework grades individual transmitters (agents, models, extractors) on reliability, quarantines claims whose chains contain weak links, corroborates across independent chains, and separates content criticism from chain quality. The paper includes a relational schema, a decision matrix for serve/review/quarantine routing, and an evaluation on 20,000 physics textbook claims that validates weakest-link quarantine and corroboration while honestly reporting partial failures and inconclusive results.
-
AREX: Towards a Recursively Self Improving Agent for Deep Research
Pippa and Tyler dig into AREX, a recursively self-improving deep research agent from BAAI that alternates between an inner search loop and an outer constraint-verification loop — and discuss whether that architecture is genuinely novel or a smarter repackaging of ideas the field already had.
-
Overview: Calibration
We finally slow down and make calibration click: what it means for a model’s confidence to match reality, how you measure that, and why it matters when you actually want to trust the thing. We keep it grounded in the systems we’ve been circling for ages, because calibration is everywhere once you start looking.
-
Introducing TabFM: A zero Shot foundation model for tabular data
Justy and Cody examine TabFM, Google Research’s zero-shot foundation model for tabular classification and regression. They unpack its hybrid row-column attention design, synthetic-data training, TabArena evidence, the trade-off between out-of-the-box convenience and tuned ensembles, and whether BigQuery integration could make this genuinely useful in everyday data workflows.
-
Thinking Machines open sources first multimodal language model, Inkling, focused on low cost and 'resistance to censorship'
Inkling, Thinking Machines' open-source multimodal MoE model (975B total / 41B active parameters), lands as a broad, balanced generalist with a standout feature: a controllable 'thinking effort' knob to dial cost vs. performance from 0.2 to 0.99. Enterprises get native text+image+audio fusion, Apache 2.0 weights, and a lighter Inkling-Small preview, but benchmarks show it trails specialized open and closed models on coding and pure reasoning, while remaining competitive on multimodality and agentic workflows. The episode debates whether the real win is the runtime control surface (Tinker platform) and a cautious, non-censoring epistemics posture — not the headline parameters.
-
LLM Evaluation Frameworks Compared: How to Actually Measure What Your Model Does MachineLearningMastery
Fern and Lintel dig into a comparison of RAGAS, DeepEval, and Promptfoo, landing on the article’s real argument: eval frameworks are less about novel metrics than about where evaluation fits in your workflow, and the dangerous part is trusting LLM-as-a-judge without auditing its biases. They like the article’s practical split between RAG scoring, CI gates, and prompt red-teaming, but push on where the examples are a little too toy-like and where teams can overread framework choice as the main problem instead of test-set design and human calibration.
-
Large language models often prioritize Western moral values, overlooking other cultures
A research paper finds LLMs tend to mirror Western moral priorities when asked to roleplay citizens of 48 countries, and two hosts discuss what this actually means for users, products, and culture.
-
Overview: Reinforcement Learning from Human Feedback
We finally define reinforcement learning from human feedback the way we keep using it: as a loop where human preferences become a learned reward signal that steers a model after its initial training. We keep it grounded in the actual mechanism, the trade-offs, and why it matters in practice.
-
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
Jessica and Cathy dig into RL with Metacognitive Feedback (RLMF): a post-training loop that rewards models not just for correct answers, but for accurately judging how well they did—improving both task performance and the faithfulness of uncertainty expressions. They explain the mechanism (metacognitive data selection and metacognitive advantage scaling), discuss trade-offs, and debate whether this is still research-only or actually shippable.
-
Evals are the new PRD
Braintrust argues that, for AI products, evals should replace traditional PRDs as the living definition of quality. Laura sees a practical operating-model win, while Harper agrees with the core idea but insists evals are executable acceptance criteria, not a substitute for product judgment.