WorldLines: Benchmarking and Modeling Long Horizon Stateful Embodied Agents
WorldLines (HKUST team) ships a benchmark + model (ObsMem) for long-horizon home robots that must remember daily routines, unseen state changes, and fix mistakes over days. Cody digs into how the observer-grounded memory keeps world state coherent under partial observability and where it still stumbles. Justy sizes who actually builds with it and what shipping looks like.
Transcript
Justy So the robot saw Bob set the coffee timer for four A.M. on Sunday and still somehow forgot to flip it back
Cody —on a Tuesday morning two days later it finally remembered
Justy Exactly—and that’s the whole benchmark. WorldLines: long-horizon stateful embodied agents in real homes. It’s the first one that doesn’t punt on cross-day state
Cody They’re finally benchmarking ‘live with me’ instead of fetch-the-key-30-seconds
Justy Which also means if the thing keeps asking the same question because it forgot you said it yesterday—trust is gone. That’s the product failure they’re trying to fix
Cody Right, right—current long-memory benchmarks are text logs; embodied ones are instant-episode killjoys. WorldLines threads the needle: temporal-spatial reasoning, object state, embodied planning, and proactive assistance
Justy Four pillars Cody laid out. They generate full project-driven traces—dialogues, human actions, robot steps, device updates, execution feedback—then cut it at a timestamp and ask Memory QA and Embodied Task Planning
Cody Wait—are they simulating a real smart-home stack or just faking device states?
Justy They’re faking the household devices so far—simulated smart bulbs, robot vacuum, fridge—but the state changes are still modeled like the real thing, which is fine for the benchmark
Cody Feature, not a bug, until the robot actually has a real confirmation loop with the fridge
Justy Okay fine—but their Memory QA samples are brilliant. You freeze the trace at a cutoff, then ask what happened before. Robot has to reconstruct Bob’s Sunday coffee mistake from interleaved actions and device logs
Cody And the Embodied Task Planning samples expect the robot to use that remembered state to decide what to do next—so if Bob asks for coffee at seven A.M., the robot shouldn’t suggest ‘brew now’ when it already knows Bob’s routine is eight thirty
Justy That’s the insight—agents that only retrieve text fall apart on stateful tasks because they can’t tell ‘I saw it’ from ‘I was told it’ from ‘I inferred it.’ ObsMem’s whole move is observer-grounded memory
Cody Observer-grounded meaning it keeps the raw observation buffer around, not just a retrieval index
Justy Three layers—evidence, state trail, and belief record. Evidence is the video clip or sensor log; state trail is the structured delta map of what changed when; and belief record is the robot’s running best guess about the world given what it’s sure of
Cody Uh-huh—and the visibility-aware twist is key. If the robot didn’t see Bob move the fruit from counter to fridge, the state trail literally says ‘unknown’ instead of hallucinating a location
Justy Which solves the classic ‘where did I put my keys’ disappearing act. ObsMem doesn’t let the robot guess—it forces it to mark the state as uncertain until it gets a direct observation or an action receipt
Cody Nice on paper. Results show ObsMem outperforms six baselines on Memory QA and Embodied Planning, but—surprise—the ceiling is still 62% on the hardest cross-day tasks
Justy So robots are wrong a third of the time on multi-day plans? That’s not lights-out autonomy
Cody The paper calls it ‘significant challenges remain in partial observability and overwritten world states’—which is polite for ‘people still have to babysit the fridge sensor’
Justy Okay so who actually builds with this. WorldLines ships both the benchmark and the model skeleton, so HomeRobot Inc. can fork and extend the pipeline
Cody Real talk—they’ll need a closed-loop confirmation protocol so the robot can ask ‘are you sure the fruit is in the fridge?’ without tanking latency
Justy and Cody’s already complaining that asking the user is cheating
Cody It’s stealing the robot’s soul—next thing you know the robot’s a glorified Alexa pinging for every state change
Justy Not if the confirmation is under 500 ms and the world state sync protocol keeps the phone, vacuum, and fridge all agreeing. Cody’s only half-right
Cody I’m half-right? Fine, half-right. But the paper barely touches state consensus—it assumes the observer has one clean view, not a split-brain smart-home net
Justy They’re shipping a reference architecture, which is still huge. ObsMem’s the closest thing we’ve got to a robot that remembers like a human instead of retrieving like a chatbot
Cody Agreed—it’s the step we needed. But until the architecture includes repair mechanisms when world state conflicts—like ‘the fridge says empty but I just dropped in fruit’—it’s still lab work
Justy So the honest take: landmark benchmark, solid first architecture, not yet a shippable brain for your home robot
Cody Exactly—and the next milestone isn’t bigger memory, it’s robots that act on remembered state without bugging you every time
Justy Which means we’re still years away from the ‘bring me coffee’ robot that doesn’t also bring you last Tuesday’s forgotten breakfast