Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
Jessica and Cathy dig into RL with Metacognitive Feedback (RLMF): a post-training loop that rewards models not just for correct answers, but for accurately judging how well they did—improving both task performance and the faithfulness of uncertainty expressions. They explain the mechanism (metacognitive data selection and metacognitive advantage scaling), discuss trade-offs, and debate whether this is still research-only or actually shippable.
Transcript
Jessica —okay, so I read this thing where the model keeps confidently asserting nonsense and the paper’s fix is ‘teach the model to know when it doesn’t know’—which I’m down for, because that’s the whole vibe of Exploring Next, but—
Cathy Yeah.
Jessica —but Cathy, come on, you’ve been saying agents hallucinate with high confidence since day one. Isn’t this just ‘RLHF but with a second loss term’?
Cathy Mm-hm.
Jessica ExplainingNext energy right there—
Cathy I didn’t say it was wrong. I said it’s the kind of thing we’ve all been poking at since ChatGPT shipped.
Jessica Right, but this paper claims they solved faithful calibration—meaning the model’s expressed uncertainty actually matches its internal belief. That’s a step up from ‘don’t hallucinate’—it’s ‘don’t fake conviction’.
Cathy Sure. And the paper’s title is ‘Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs’—so they’re leaning into the metacognition framing.
Jessica Which, okay, I’m here for— agents that say ‘I’m not sure’ when they’re not sure feel safer in advisory roles. But I’ve heard ‘trust the model to self-evaluate’ before and it usually ends in ‘the model is overconfident about being overconfident.’ How is this different?
Cathy Because they don’t just reward the right answer; they reward the right self-assessment. If the model rates itself a five out of ten but gets the answer wrong, that’s a negative signal. If it rates itself a two but nails it—also a negative signal. Only when the rating and the outcome align do they get the positive bump.
Jessica Okay, that’s not ‘pick the answer the human liked’—it’s ‘train the model to know if the human would have liked it.’
Jessica …which is the adult way to ship new training paradigms.
Cathy …okay, fine. But the compute isn’t nothing—you’re running the model once to judge its own performance, then again to produce the output, then a third pass to rewrite the uncertainty string. All so the user can read ‘probably’ instead of a percentage.
Jessica And if it keeps your most expensive reviewer from hallucinating into a million-dollar mistake? Worth it. Trade-offs are part of the product.
Cathy Fine. I grant you the engineering trade-off is smaller than the philosophical one—if the model’s self-judgment is fundamentally flawed, the whole thing’s a house of cards.
Jessica Then start small—on a domain you control. If it works, scale. If it crumbles, you learned something.
Cathy …which is the adult way to ship new training paradigms.
Jessica Alright, but tell me—what do you actually think? Is this a real mechanism or just another research toy with a catchy name?
Cathy —so I’ll allow it.
Jessica Thanks, I think.