Ep 1119 Research Paper 5:21 w/ Pippa & Tyler

Trajectory2Gym: Decision Grounded Compositional Interaction Synthesis for Tool Use Agents

Trajectory2Gym reframes how tool-use agents learn from trajectories: successful completion doesn't guarantee useful supervision. The paper identifies two coupled problems—compositional coverage (multi-step dependencies underrepresented) and decision grounding (actions unjustified by agent-visible history)—and builds a framework that synthesizes interactions around weak spots, certifies only grounded decisions, and reweights by structural support. Results: 16.9–29.8 percentage points across Tau1, Tau2, and BFCL at all five Qwen3 scales, with a 4B model outperforming the public 30B baseline.

Embed this episode

Paste this on any site — the player is a self-contained iframe with no cookies or trackers.

<iframe src="https://sandrise.io/exploring-next/embed/1119"
  width="100%" height="180" style="max-width:640px;border:0;border-radius:12px;overflow:hidden"
  title="Exploring Next — Episode 1119 audio player"
  loading="lazy" allow="autoplay" referrerpolicy="strict-origin-when-cross-origin"></iframe>
Embed & API docs →
Script Haiku 4.5 Voice Inworld TTS 2

Transcript

Pippa Okay, so the thing that's been sitting wrong with me about trajectory supervision is that we've been filtering by success the entire time. Task completed, great, that's training data.

Tyler Right.

Pippa But this paper—Trajectory2Gym—basically says successful completion is the WRONG filter. You're keeping actions the agent can't actually justify from what it saw.

Tyler Yeah, and they have a concrete example. Retail task: user asks for two items, then confirms only one. The API accepts both requests fine. If you filter by execution success, you keep both. But the agent should only learn the one the user actually wanted.

Pippa Exactly. And the broader pattern they're naming is that 40 percent of trajectories in their pool have at most two supervised decisions. You're not even exercising multi-step dependencies.

Tyler Mm-hm.

Pippa So they introduce two coupled requirements: compositional coverage—make sure you're actually demonstrating observation-action chains across steps—and decision grounding, which means every action you supervise has to be justified by what the agent could see at that moment in the trajectory.

Tyler The mechanics are interesting because they're not just filtering harder. They're synthesizing new trajectories around the gaps. If a particular interaction pattern is underrepresented, they build tasks that exercise it in the real environment, execute them, and then only certify decisions that were actually grounded in the agent-visible history.

Pippa So it's not synthetic data that's divorced from execution.

Tyler No. They construct the task, run it live, and then look at what happened and what the agent could have known. If an error occurs—say the lookup fails—they keep that error as recovery context, because that's compositional. Later decisions depend on seeing that failure.

Pippa I'm into that. They're reframing trajectory generation as deliberately building learnable experience, not collecting whatever succeeded. And then the third stage is selection—they allocate trajectories by structural support depth and train with something called parent-normalized supervised fine-tuning.

Tyler Which is… the weight is normalized per parent trajectory rather than per step. So the semantic interaction—the whole multi-step workflow—is the unit of training weight, not individual actions.

Pippa Right. Because if you weight by step, you're back to treating trajectories as bags of decisions, and you lose the dependency structure.

Tyler Exactly.

Pippa Okay, so the results. They test on five Qwen3 scales—one point seven B all the way to thirty B—and they see 16.9 to 29.8 percentage point gains across Tau1, Tau2, and BFCL at every scale.

Tyler The wild part is the cross-scale story. The four B model post-trained on Trajectory2Gym data beats the public thirty B baseline on Tau2. Sixty-nine point seventeen versus twenty-eight point thirty-three.

Tyler So you're not just seeing 'more data helps,' you're seeing 'better-constructed data with smaller models beats much larger baselines.'

Pippa And the gains are strongest in stateful, multi-turn interaction. Which, yeah, that's the compositional stuff—where later decisions depend on earlier observations and feedback. That's exactly what agents need in production.

Tyler Right, and they show it with ablations. Synthesis without grounding teaches shortcuts. Grounding without synthesis keeps the pool where recovery and discovery stay rare. You need both.

Pippa The thing I keep coming back to is how much of agent training has been a collection problem—get more trajectories, filter by success, scale up. This says the actual problem is composition and grounding. You could have fewer trajectories if they're the right ones.

Tyler Which is a systems insight, not a capability insight. It's not 'the model is smarter now,' it's 'the supervision is learnable now.' The agent can actually see why each action was taken.

Pippa Mm-hm.

Tyler And production-wise, if you're running agents on real tasks—customer service, ecommerce, workflows where state and feedback matter—this is exactly the kind of data you'd want to construct. Not 'did the task complete,' but 'what did the agent need to learn to handle the next similar task.'

Pippa Yeah. I'd ship this. The gains are consistent, the ablations are clean, and the insight about success not being supervision is something teams should internalizing.

Tyler One thing I'd want to see is how this scales to much larger models and whether the insight holds when you're talking about inference-time scaling or tree search. This is SFT-focused, and at frontier scale some of that work might be done by search rather than learned policy.

Pippa Fair. But for the scale range they tested and for most teams actually training agents, this is a solid move.

Tyler The paper links to an open-source platform—vecna-labs/open-trajectory-gym on GitHub. It's a framework for post-training on multi-turn tool-use trajectories. Bring your own agent, model, benchmark, reward function.

Pippa So if you're building agents and you want to move beyond 'filter by success,' there's actually something to build on.

Tyler Exactly. It's not just a paper result; the infrastructure is there.

Pippa All right. This is the kind of work that's going to matter for anyone training agents at scale—especially if you care about stateful, multi-turn tasks. Worth a close read.