Topic
In Context Learning
26 episodes
-
Harness Zero: Harness Distillation via Agent as Harness
Harness-Zero turns a great, evolved agent harness into training data so the underlying model permanently learns those behaviors and can run on a much simpler harness — often matching or even beating the original setup.
-
BI Agent and BI Bench: Towards Automating End to End Business Intelligence
Jessica and Cathy dig into BI-Agent and BI-Bench, a paper that tests whether language models can automate the messy end-to-end workflow behind business intelligence dashboards. They focus on the gap between natural-language SQL demos and real BI work: table selection, transformations, joins, and analysis over messy Power BI-style projects. Jessica sees a practical product path for internal analytics teams, while Cathy is impressed by the benchmark design and tool-augmented architecture, with caveats about dashboard-derived ground truth and public-data leakage.
-
Grounded Skill Synthesis from Code at Scale for Agentic Intelligence
Masonry and Eyre dig into Code2Skill, a paper arguing that source code is the missing substrate for scalable skill banks. The real move is not just extracting tips from repos, but verifying that a synthesized skill can reconstruct the implementation without seeing the source body, then checking it against the original. They land on it as a strong infrastructure-as-product play for agentic systems: less mystical self-improvement, more mining maintained codebases for grounded procedures with provenance and boundaries.
-
GraphRAG: A Practitioner's Guide to 6 Advanced Architectural Patterns
Pippa and Tyler dig into a GraphRAG guide that’s really arguing for a more honest question: not “does GraphRAG work,” but “which retrieval shape matches the query shape?” They focus on the production trade-offs between strict graph querying, hybrid vector-plus-graph setups, and when knowledge graphs actually earn their keep.
-
Agentic Work Management is here!
Asana's new Agentic Work Management layer ships 30-plus prebuilt AI Teammates and a personal AI chief of staff called Asana Dash — all included for paid customers — so humans and agents can coordinate from one shared plan.
-
Emergence World: Adversarial Stress Testing of Long Horizon Multi Agent Systems
Pippa and Tyler unpack Emergence World’s new Study 2: a 16‑day adversarial stress test of long‑horizon multi‑agent systems. They dig into how the simulated “worlds” work, what the phishing, misinformation, and memory‑breach attacks revealed, why model‑level alignment isn’t compositional, and how teams could actually use the Emergence World repo to probe real agent workflows.
-
The Router Within: ElicitingNative Skill Routing from a Frozen LLM
Onyx and Echo unpack Gavel, a paper arguing that a frozen agent model already contains useful skill-routing signals in its hidden states, if you read them out with two small trained projections and a second-stage verdict.
-
Repo to Skill: Distilling GitHub Repositories Into AI4AI Skills
Repo-To-Skill introduces DisCo, a skill-distillation system that extracts operational knowledge from GitHub repositories and papers, packaging it as compact, verified skills that autonomous ML research agents can load on demand. The AREX-Skill Library contains 5,000+ skills from 1,000 repositories organized into 20 areas and 178 capability families. In matched tests with GPT-5.5 backbone and fixed execution budget, skill-equipped agents outperform skill-free baselines by 134.3% on MLE-bench, 34.4% on PaperBench, 9.2% on FrontierCS, and 14.0% on PassNet—gains purely from operational knowledge, not model or harness improvements.
-
S3Gym: Can LLMs Turn Self Testing and Self Judging into Self Improvement?
S3Gym is a new interactive benchmark that tests whether LLMs can actually improve themselves by testing their own behavior, judging the results, and learning from them. The paper evaluates three ways to incorporate experience—keeping full conversation history, compressing it into summaries, and training on it—across seven text-based games. The finding: self-improvement isn't automatic. What works depends entirely on the task. Sometimes summaries help, sometimes raw history is better, and parameter training can backfire badly. The real bottleneck isn't recognizing success—it's turning that recognition into a policy the model can actually reuse.
-
How our agents build on Brand pages with design
Vercel's design.md post describes a three-part system for keeping AI-generated pages on-brand outside a codebase: a public guidance file, a public stylesheet that takes layout decisions away from the model entirely, and an eval loop that converts human corrections into rules and deterministic checks. Pippa and Tyler dig into what actually makes it work, why the naive port-the-prompt approach failed, and what it means that Vercel had to build an eval harness just to ship a markdown file.
-
Overview: Next State Prediction
We finally slow down and explain next-state prediction from the ground up — the deceptively simple idea that if you train a model to guess what comes next, it ends up learning how the world actually works, and why that one trick is underneath almost everything in modern AI.
-
MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
MemTrapBench identifies a counterintuitive failure mode in LLM memory systems: faithfully extracted, semantically relevant memories can actually degrade model performance by anchoring reasoning to prior patterns (Reasoning Fixation) or overriding correct beliefs with historical premises (Belief Distortion). The paper benchmarks this across five memory frameworks and proposes AdaptiveMem, a prompt-based intervention that mitigates traps without modifying underlying architecture.
-
Overview: Prompt Injection
We finally slow down on prompt injection, the security problem we keep name-dropping and then skating past. We build it from the simple mental model of a desk assistant who cannot tell the boss's instruction apart from a sneaky note inside the paperwork.
-
The harness is all you need (mostly)
Burke Holland argues that productivity gains with AI come not from exotic prompts or new tools, but from deeply understanding and using the harness well. He walks through a concrete six-step workflow (pick a tool, enable autonomy, prototype, plan, implement with Autopilot, iterate) that leverages GitHub Copilot's built-in orchestration and subagent routing without requiring custom skills or tricks. The central claim is that the harness — the core interaction model — is what matters; everything else is noise.
-
The new rules of context engineering for Claude 5 generation models | Claude by Anthropic
Anthropic's post on context engineering for Claude 5 models reveals a surprising finding: they removed over 80% of Claude Code's system prompt with no measurable loss in performance. The core insight is that newer models need fewer explicit constraints and benefit more from clean interfaces, progressive disclosure, and letting the model use judgment rather than following hard rules. The shift reflects a broader pattern: as models get stronger, the infrastructure around them gets simpler.
-
Introducing TabFM: A zero Shot foundation model for tabular data
Justy and Cody examine TabFM, Google Research’s zero-shot foundation model for tabular classification and regression. They unpack its hybrid row-column attention design, synthetic-data training, TabArena evidence, the trade-off between out-of-the-box convenience and tuned ensembles, and whether BigQuery integration could make this genuinely useful in everyday data workflows.
-
Overview: Supervised Fine Tuning
We finally slow down and make supervised fine-tuning click, because we keep leaning on S F T like everyone already has the whole shape of it. We build it from the apprentice-and-worked-examples picture into the actual training loop, the examples, and the trade-offs.
-
EvolvingWorld: An Open Schema Framework for Co Evolving Role Play Agents and World Model in Interactive Literary World
Masonry and Eyre dig into EvolvingWorld, a new framework that lets fictional characters and their world co-evolve across long stories. Eyre walks through the open-schema architecture and seven supervised tasks; Masonry sizes up who would actually build with this and where product pain lives. The hosts end up excited about the open-schema premise but skeptical of the benchmark’s generality.
-
Overview: Task Decomposition
We finally slow down on task decomposition, the quiet trick underneath agents, code review workflows, web tasks, and a lot of the stuff we keep arguing about. We use one mental model, a messy project board becoming manageable tickets, and build from intuition to mechanism to where it still matters now.
-
Overview: In Context Learning
We finally slow down and explain in-context learning, the thing we keep leaning on whenever prompts, agents, examples, and adaptation come up. We make the core idea concrete: the model is learning from the temporary packet you hand it, without changing itself permanently.
-
How to Implement a Unified Memory from Scratch
Jessica and Cathy dig into a new post that walks through building a unified agent memory from scratch using knowledge graphs and MongoDB, unpacking what it actually takes to wire memory into a real agent harness. They tease apart where the post’s blueprint shines, where it overreaches, and who on earth should actually roll their own instead of reaching for an off-the-shelf tool.
-
Overview: Prompt Engineering
We’re finally doing the overdue deep dive on prompt engineering, the weirdly practical skill of getting language models to do the thing you actually meant. We keep coming back to it because the difference between a flimsy prompt and a good one is often the difference between nonsense and a usable product.
-
Large language models often prioritize Western moral values, overlooking other cultures
A research paper finds LLMs tend to mirror Western moral priorities when asked to roleplay citizens of 48 countries, and two hosts discuss what this actually means for users, products, and culture.
-
Agentic Testing: Where Agents Fit in the E2E Testing Stack
Slack's Sergii Gorbachov ran 200+ agentic E2E tests to measure where agent-driven testing fits alongside traditional deterministic tests. Core finding: agents verify goals (adaptable paths to the same outcome), while traditional tests enforce journeys (single deterministic sequence). MCP-based agents were most reliable (0% on simple flows, ~12% on complex); generated tests were fastest (~3 min) but fragile on complexity (~48% failure rate on harder flows); cost was the real constraint ($15–30 per run). The insight is not replacement—it's complementary layers. Agents excel at exploratory validation and catching UI state variability; deterministic tests handle regression and CI speed.
-
Overview: Tool use and function calling
We finally sit down and make tool use and function calling click from the ground up. We keep coming back to the same idea: a model can draft the request, but something outside it has to actually do the thing.
-
Skill Graphs > SKILL
Jessica and Cathy dig into Heinrich's 'Skill Graphs > SKILL.md' post — the argument that a single skill file is a dead end for complex domains, and that wikilink-connected graphs of small markdown files let agents navigate knowledge rather than just receive it. They work through what's genuinely new here versus repackaged context engineering, and whether the arscontexta plugin is a real workflow change or a clever demo.