Grounded Skill Synthesis from Code at Scale for Agentic Intelligence

Masonry The interesting part here is they’re basically saying the agent shouldn’t have to learn everything the hard way if GitHub already learned it for us.

Eyre Yeah. And more specifically, they’re taking a swing at the skill-bank problem from a different substrate. Not trajectories, not docs. Code.

Masonry Which honestly matters right now because the field keeps doing this thing where every agent stack wants reusable skills, but the skill source is either model exhaust or hand-written guidance. Almost a year into Exploring Next and we're still circling the same boring truth: the harness matters.

Eyre Right.

Eyre This paper’s pitch is pretty clean. Trajectory-derived skills are useful, but they’re coupled to the agent, tools, and task mix that produced them. Therefore they age badly when the stack changes. Doc-derived skills travel better, but they can drift into vibes because there’s no executable evidence underneath.

Masonry And you can feel why that gets people stuck. If you’re building a general agent, you do not have enough high-quality trajectories on day one. But if you wait for the agent to accumulate them, the product is bad during the part where you need it to be useful.

Eyre Mm-hm.

Masonry Also, tiny catch-up beat, this is very you. Every time I get excited about a memory layer, you drag me back to, okay, but what is the evidence substrate. Annoying. Usually correct.

Eyre Okay, that's fair. And this one earns it, because the mechanism is not just “summarize the repo.” They rank source units for reusable procedural content, then lift them into three record types: atomic operations, composite workflows, and recurring patterns.

Masonry So not one giant mushy skill blob. You’ve got little moves, bigger task recipes, and then the repeated pattern level. That taxonomy is actually product-friendly.

Eyre Yeah.

Eyre The part I actually like is the verifier. They generate a skill record that’s supposed to capture applicability, steps, invariants, failure cases, anti-goals, and evidence. BUT then they do source-body-blind reconstruction. The model has to recreate the implementation from the skill without seeing the source body. Then a source-aware judge compares reconstruction to original code and filters out unsupported or incomplete records.

Masonry That’s good. That’s really good. Because otherwise this is just code-to-blog-post with better branding.

Eyre Exactly.

Masonry And the scale is not tiny. They run it over nineteen thousand seven hundred sixty-nine popular, actively maintained GitHub repos and accept one million six thousand eight hundred twenty-two records into CodeSkillBank. That at least clears the “cute demo” bar.

Eyre Right, though I do want to stay calibrated on what the eval means. The headline average is an eleven point seven percent relative gain over matched baselines, fifty-seven wins out of seventy-two runs, and they say all nine SWE-bench Verified comparisons improved. That's solid. It is not the same thing as proving universal skill transfer.

Masonry Sure. But for who builds with this, I think the answer is pretty concrete. Teams doing coding agents, tool-heavy internal assistants, maybe domain copilots. You plug this in as retrieval for planning or critique, and suddenly the model has procedures with provenance instead of just “I’ve seen something like this.”

Eyre Right, right.

Eyre They also compare against trajectory-derived banks under one interface and beat Trace2Skill, ExpeL, and SkillRL-Bank on all seven shared benchmarks. Averaged, Code2Skill gets forty-nine point five versus thirty-one point zero, twenty-seven point nine, and thirty-two point eight. That’s a meaningful gap, not rounding noise.

Masonry Which connects directly to that operational-knowledge thing we kept coming back to. The missing layer is often not raw reasoning. It’s setup knowledge, boundary conditions, the annoying little invariants somebody learned while maintaining the codebase.

Eyre Yes, and this is where I think the paper is strongest. It’s one more entrant in that same fight we’ve been having about whether agent competence comes from bigger models or better externalized structure. This is very much the structure camp.

Masonry You’re sounding almost cheerful. I need to mark the date.

Eyre Stop it.

Eyre I still have concerns. Repository code is full of framework glue and project-local assumptions, and the paper knows that. Their whole challenge section is basically: how do you abstract the reusable procedure without importing accidental weirdness. I buy the reconstruction filter as a decent guardrail, but I’d still want stronger evidence on deduping near-identical patterns and on how retrieval behaves when ten similar skills compete.

Masonry Yeah, that’s my practical question too. Production is not “do you have a million skills.” Production is “does the agent pull the right one fast enough, with the right scope, and not drown in cousins.” A hundred brilliant agents, defeated by one shared folder, except the folder is now a skill index.

Eyre Okay, that's genuinely funny.

Masonry You know I’m right.

Eyre Unfortunately, yes. Also the AI-generated code result is interesting. Skills synthesized from tested A I-generated code got a ninety-three point five zero pass rate versus ninety-three point zero zero for human-written code. I would not overread that, but it does suggest the pipeline can keep feeding itself as more maintained code is machine-assisted.

Masonry That part has a real product consequence. If your repos are increasingly mixed-origin, this doesn’t collapse. It just says the verification layer matters more than authorship purity, which feels like the adult answer.

Eyre Mm-hm.

Masonry Build-next-wise, they do point to a dataset. CodeSkillBank is on Hugging Face under DeveloperSkills, so this is not purely paper theater. I didn’t see an official code release in what we had, so I wouldn’t oversell it as turnkey yet.

Eyre Yeah, same read. Research that is annoyingly close to shippable. Not a finished product, but not trapped in benchmarkland either.

Masonry Which, honestly, is my favorite genre. Okay, Eyre, let’s leave it there before you invent three more evaluators for the evaluator.