Measuring the Gap Between Human and LLM Research Ideas

Cooper —I’m still stuck on one number from this paper.

Miles Yeah?

Cooper They say humans hit synthesis methods like five percent of the time, but LLMs clock in at twenty-three to thirty-nine percent. That’s more than the entire human distribution—not just the mean, the whole thing.

Miles And the bridge-style motivations? Humans twelve percent, LLMs forty-seven to sixty-four.

Cooper Which, if you squint, sounds like a grant reviewer's wet dream.

Miles Very fair. I spend half my day explaining to people why straight-up 'we combined two things' isn't a contribution worth publishing.

Cooper Exactly—and the paper literally prints this out in Figure one. They didn’t just eyeball it; they built a two-axis taxonomy to profile every idea by its opportunity pattern and method paradigm.

Miles Right—and the axes came straight from NSF and NIH guidance docs. So it’s not just some ad-hoc rubric; it’s grounded in what funders actually value across both ML and the natural sciences.

Cooper Product people call that 'voice of the customer in the room.' Miles, meet your new favorite artifact.

Miles Fine, but the real trick is how they built the corpus.

Cooper Yes—

Miles For every human paper they reverse-engineered the closest prior works that likely inspired the core idea, then fed the same set of titles and abstracts to the models and asked for a fresh motivation and method.

Cooper So it’s not 'here’s a prompt, generate an idea about transformers'—it’s constrained literature-grounded ideation. Human idea is the one that actually shipped; LLM idea is generated from reconstructed local context.

Miles That framing neutralizes a ton of confounds. If my idea is 'build a new transformer variant,' nothing about that tells you whether my taste is broad or narrow.

Cooper And the paper’s method disambiguates that. They break each idea into motivation and method, classify it on both axes, then compare the distributions.

Miles Yep. Human distributions sit across the board: failure-mode discovery, assumption relaxation, measurement instruments, formal explanations, system building. LLMs cluster around connecting literatures or unifying methods—bridge moves and synthesis all the way down.

Cooper Which, in practice, means if I prompt an LLM for research ideas on my topic it’s going to send me the same 'combine these two domains' playbook every single time.

Miles Until the taxonomist in the loop rejects it. Speaking of which—the annotation pipeline is worth a second look. They used an LLM to do the heavy lifting, then validated it against independent human judgments.

Cooper So you’re telling me they crowdsourced the literature to a model, then used another model to grade the outputs, then sanity-checked with humans—

Miles No, the annotation pipeline—

Cooper —I know, I know. It’s still a big step up from 'give me a list of ideas and a thumbs-up from a grad student.'

Miles It’s necessary. If you want to steer idea diversity at scale, you need an evaluator that can grade research taste in-system. Otherwise you’re just shuffling the same cluster around.

Cooper So what does production look like? I can imagine dropping this taxonomy into an AI-scientist stack as a reranker—high taste-score ideas get more compute, low ones get dropped or re-prompted.

Miles That’s the cleanest use case right now. If you’re running an autonomous research shop, you could use it to filter and route generated ideas through a 'breadth check' before they fire off experiments.

Cooper And if you’re a human researcher, you could pipe your latest literature set into the same pipeline, skim the distribution over ideas, and spot blind spots in your own taste.

Miles Sure, but let’s not pretend we’ve solved the narrowness problem. The paper admits the taxonomy is a first pass and the current gap is baked into the models themselves.

Cooper So we’re stuck with LLM ideas that sound like NSF proposals written by committee. That’s our near-term reality.

Miles Exactly. Which means the best workaround might be the boring one: give the model a strong, human-curated prompt, then constrain the outputs with a taxonomist on the inside.

Cooper Or just switch to Miles mode mid-prompt—problem solved.

Miles Nice try. Anyway, the repo’s public—zi yu u c slash Taste Gap.

Cooper Which is, naturally, precisely the kind of name that makes product people groan and researchers bookmark.

Miles Yeah, well, at least the paper backs it up.

Cooper And for once, that’s not the most damning thing you could say about a research artifact.

Miles Truer words.