An agent that can wait eight days

TL;DR

A durable invocation is pinned to the deployment that started it. An approval that waits 8 days keeps that deployment running for 8 days.
We keep the old behaviour for 5 minutes. After that the wait moves into a virtual object, and the turn waits as state instead of as a suspended invocation.
When the person answers, the tool runs on the latest deployment and the same turn resumes in a new invocation. Users see no difference.

The cost of waiting

Our agent runtime is built on durable execution. Every step of a turn is journaled, so a crash or a deploy does not lose work: the invocation replays its journal and carries on. For tools that wait on a person (an approval, a choice of connection, an answer to a question) that is exactly what you want. The agent can ask on Friday and continue on Monday.

Then we looked at our cluster after a normal week of releases. Fourteen versions of the tool executor were still running, two pods each. One killed invocation’s journal ended in a sleep due to wake in 7 days and 22 hours: the approval timeout. Nothing was broken. People simply had not clicked yet.

14old executor versions still running

8 daysmaximum wait for a person

5 minmaximum pinned time after the change

Fig. 1Press play, scrub or hover. Releases over eight days. Before: every version lives as long as its longest open wait. After: no invocation waits longer than five minutes, so old versions drain shortly after each release.

Why invocations pin

Durable execution engines like Restate replay a journal against code. For replay to be correct, the code must be the code that wrote the journal. So an invocation is pinned to the deployment that started it, and a suspended invocation keeps its deployment alive. The operator scales an old version to zero only after its last invocation finishes.

Two pins, not one The agent’s turn (doTurn) awaited the executor as a request-response call, so a waiting tool pinned both services.

Our human waits lived in one middleware function. It created an awakeable, wrote a waiting row and a card, and waited with an 8-day timeout. The turn awaited the executor, so it stayed suspended for the same time. Two invocations, two pinned deployments, per open card.

chat-tool-executor/middleware.ts (before, simplified)ts

const { id, promise } = ctx.awakeable<Answer>();
await writeSessionAwakeable(id, toolCallId);
await emitCard("tool-awaiting", request);

// Suspends this invocation, and the caller's, for up to 8 days.
const answer = await promise.orTimeout(days(8));

Two phases

The constraint was strict: the turn must block until the tool result exists. No “continue without the tool”. And the approval logic had to stay in the executor; the agent runtime should only learn that a result may arrive later.

Most answers come quickly, so we kept today’s behaviour for the first five minutes, on the same code path. After five minutes, the executor hands the wait to a virtual object and returns a special result, result_pending. The turn records that it is waiting, as state on the session’s controller, and ends its invocation. Step through the hand-off below.

Fig. 2The hand-off after five minutes, step by step. Grey lifelines are services whose invocation has ended. From step 3 to step 4 nothing is pinned, however long the person takes.

The interesting property is in step 4. When the person answers, the virtual object re-runs the tool on the latest deployment, not the one that asked. That also makes it stricter than before: the re-run re-resolves identity and the tool catalog, so a user who lost access during the wait is rejected.

Why not signals? A signal targets one specific invocation. Once the turn waits as state, its invocation has ended, so there is nothing to signal and the value would be lost. The executor reports with a one-way call to a handler on the controller, which is deliverable whether or not an invocation is running.

Fig. 3A synchronous tool wait: in-call for up to five minutes, then as state for up to eight days. Three exits: an answer resumes the same turn, the timer expires it, or a new message ends it like a steer.

The busy rule

Moving the wait out of an invocation removed an implicit lock. Before, “a turn is running” meant “an invocation is active”. Now a turn can be in progress with no invocation at all. So the controller treats activeTurnInvocationId || waitingTurn as busy everywhere it decides whether a turn is running: new messages queue, steers apply to the waiting turn, and background results wait their turn in the FIFO.

The one visible change is deliberate. A new message to a session that waits for answers ends the waiting turn, like a steer, instead of queueing behind a card the person might never answer.

The same machinery made background calls almost free. A tool or sub-agent can be sent to the background at any point, even after the hand-off: the controller flips the waiting entry and resumes the turn with a placeholder. The result arrives later through the same toolFinished path and becomes a new message. The design side of that is in Agent action patterns.

What we rejected

Alternative	Why not
Re-pin suspended invocations to the latest deployment after each release	Makes every release a journal-compatibility contract. Any mismatch fails a user’s turn.
A tiny, rarely released router service holds the wait	Moves the pinning instead of removing it. The agent runtime still pins.
End the turn as “waiting” and start a new turn on the answer	Changes turn semantics, the event stream and the transcript. Splits one reply across two turns.
Reuse the sub-agent signal path	Either keeps the parent suspended (the same pinning) or continues without the tool.

Durability below the turn

Waiting is one half of durability. The other half is making sure that whatever continues the work is allowed to. For our coding agent, every thread has one owner host that is the only writer of its log, and every write is fenced by the owner’s epoch. A host that pauses and wakes up after the thread moved on cannot write.

Fig. 4Epoch fencing. Reassigning a thread bumps its epoch inside the same locked row that every append checks, so a stale owner’s write fails instead of forking history.

Fencing protects the database, not the world. It cannot cancel a request an old host already sent, and a shell command’s side effects are not transactional. So recovery is conservative: the new owner continues from committed history under the original turn, and holds the turn when a durable tool intent has no known result. A provider error does not prove an effect did not happen.

Trade-off Conservative suspension sometimes asks a person to confirm work that may already have happened. We prefer one extra question to one duplicated email.

Lessons

Durable is not free. Every suspended invocation is a deployment you cannot retire. Count them.
Keep the fast path unchanged. Most answers arrive in minutes. The hand-off only exists for the long tail, so the common case is still the code we already trusted.
State outlives processes. A turn that waits is data, not a sleeping function. Once we modelled it that way, background calls and steering became small additions.
Be honest about effects. Exactly-once inside our system is achievable. Exactly-once in someone else’s inbox needs their idempotency contract.