Overview: Durable Execution

Justy Okay, we have to do this one properly. Durable execution keeps coming up and every time we mention it, I can feel us hand-waving past the part that actually matters.

Cody Yeah. And then we both act surprised when the explanation turns into, uh, a tiny swamp. Which is very on brand for this show.

Justy It is. Also, I think people hear the phrase and assume it’s some abstract platform thing, when it’s really about a workflow not forgetting where it was.

Cody Right. Think of it like a book with bookmarks. You do a step, you leave a mark, and if the lights go out, you don’t start from page one and pretend nothing happened.

Justy So the boring version is: you’ve got a multi-step job, something crashes halfway through, and the system needs to know what already finished. That’s the whole shape of it.

Cody Exactly. And that shape matters because the naive version is awful. If you just retry blindly, you can fetch the same thing again, save the same thing again, or send the same notification twice.

Justy Which is where I always want to say, okay, that sounds like a bug, but in distributed systems it’s basically Tuesday.

Cody Yeah, and the reason this concept exists is because the world is not a neat little single-process script. Machines restart, networks hiccup, services time out, and your nice linear flow gets interrupted.

Justy Before we go deeper, we should name the prerequisite stuff we keep smuggling in. We did a whole Overview on state serialization, episode seven hundred fifty eight, and the quick gloss is just: you save the state so you can load it later.

Cody Mm-hm.

Justy And we did retry loops and error recovery in episode seven hundred thirty two. That one’s the idea that if something fails, you try again in a controlled way instead of panicking or giving up.

Cody Right, and append-only logging, episode seven hundred fifty nine, is the other piece we keep circling. That’s just keeping a record by only adding new entries, never quietly rewriting the past.

Justy Okay, so with those in place, durable execution is like the workflow version of all three. It remembers its progress, it can resume after failure, and it keeps a trail of what happened.

Cody And the key thing is that it’s not just a log for humans. The system itself reads that record and decides, ‘I already did this step, so I’m not doing it again.’

Justy That’s the part that makes it feel less like magic and more like bookkeeping, which I mean as a compliment. Bookkeeping is underrated when the machine is trying to do something expensive and long-running.

Cody That is such an Exploring Next line. But yes, bookkeeping is exactly the right annoyance. The workflow becomes legible enough that you can stop and start it without losing the plot.

Justy So walk me through the mechanics like I’m the one who got paged because the process died halfway through.

Cody The system wraps the workflow and watches each step. Before a step runs, it checks whether that step already has a recorded result. If it does, it reuses it. If it doesn’t, it runs the step, stores the outcome, and moves on.

Justy Mm-hm.

Cody If the process crashes after step three, the next invocation looks at the stored progress, sees that steps one through three are done, and resumes at step four. That’s the durable part: progress survives the crash.

Justy And that means the workflow’s state is not living only in RAM, right? It’s in something persistent, like a database or a service that outlives the process.

Cody Yep. Persistent storage is the spine here. Without that, you’re just crossing your fingers and calling it architecture.

Justy I knew you’d say something mean about architecture.

Cody I contain multitudes. Also, this is where idempotency shows up, which is just a fancy way of saying a step is safe to run more than once because it gives the same result or doesn’t double-count side effects.

Justy So if a step can’t safely be repeated, the system has to store its output and reuse it instead of rerunning it.

Cody Yep. Anything where failure halfway through is expensive, visible, or annoying to clean up.

Justy And this is where the book-with-bookmarks picture stays useful. You’re not rereading the whole book every time. You’re opening to the last mark and continuing from there.

Cody Nice, and if we want the more technical version, the system is effectively checkpointing workflow state and replaying enough of the path to know what needs to happen next. That replay has to be deterministic enough that the record still lines up with reality.

Justy That word, deterministic, is doing work there. It means if you run the same logic again with the same inputs, you get the same path, so the checkpoints still make sense.

Cody Mm-hm. If the workflow takes different branches every time for the same input, the stored record becomes hard to trust. Then your bookmark points to a page that no longer exists.

Justy Okay, I like that. And it also explains why this is such a big deal for debugging. If you know which steps happened, you know where the failure actually was instead of staring at a dead process and guessing.

Cody Exactly. The visibility is part of the win. You get a trail of completed steps, failed steps, retries, and timestamps, which makes the whole thing much easier to reason about.

Justy That’s very close to the receipts thing we keep obsessing over. The record is not decorative. It’s what lets you inspect the path, not just the final mess.

Cody Yeah, and that’s why I’m suspicious of any system that says it’s durable but can’t tell you what it already did. If the state is hidden, you don’t really have recoverability, you have vibes.

Justy Vibes are not a recovery plan.

Cody No. They are, at best, a way to make a bad incident feel more philosophical.

Justy So where does this sit now? Is durable execution still the main way people build these workflows, or is it one of those ideas the field half-absorbed and renamed?

Cody Still very live. The basic pattern is absolutely current, and the tools around it have gotten more operationally mature. Temporal’s worker controller going generally available is a good example of the boring part getting better, which is usually the part that matters.

Justy Right, so the concept didn’t get replaced. It got buried inside better tooling and better ops around the same need.

Cody Yeah. Azure Durable Functions, AWS Step Functions, Temporal, Restate, Cloudflare Workflows — they’re all different takes on the same underlying promise: keep the workflow’s progress somewhere persistent so it can survive interruption.

Justy And that lines up with the stuff we’ve looked at before. The orchestration piece, the receipts piece, the ‘why every decision needs a receipt’ piece — same family, just different angle.

Cody Exactly. The field keeps rediscovering that once work gets long-lived and multi-step, the runtime has to remember more than a single request. That’s not a fad. That’s just the shape of the problem.

Justy And the catch is still the catch: durability costs something. You pay in extra storage, extra coordination, and some design discipline because you can’t just fling side effects around and hope.

Cody Right. There’s latency in writing state, there’s complexity in making steps resumable, and there’s a real boundary between tasks where durability is essential and tasks where it’s just overhead.

Justy Okay, that’s the trade. If the work matters enough that replaying it would hurt, durability starts looking cheap very fast.

Cody Yeah, and if the work is just a quick stateless call, then wrapping it in all this machinery is probably just self-inflicted ceremony.

Justy Which is why I keep coming back to the same thought: this is not about making code fancier. It’s about making interruption survivable.

Cody That’s the one thing to keep. Durable execution means the workflow can lose the machine and keep its place.

Justy Okay, good. I feel like we finally did the bookmarks thing justice, which is honestly a miracle for episode seven sixty.