Six writes to one: what happened to our first token

Where the time went

Time to first token is the latency people feel. The model’s own time to first token is mostly out of our hands, so we care about everything before it: from the moment our server receives the message to the moment the request leaves for the model provider.

For our coding agent, that path ran through three services and Postgres. In a development build we measured 86 ms to turn/started and 133 ms to the first model request. Not terrible. But the trace showed something worse than the number: the path was doing a lot of coordination work that no one had asked for.

Anatomy of a send

One warm user message went like this. The engine authorized the request (two reads) and enqueued it (a transaction plus a NOTIFY). It authorized again, extended a grant, and called the harness host’s Wake. The host claimed its mail (three statements), claimed the queue item (another NOTIFY), started the turn, recorded the input (NOTIFY) and recorded the turn start (NOTIFY). Only then did it publish turn/started.

Every notifying write made every engine re-read the thread and push an update to every viewer. Per send, that was six info updates and three queue re-lists in each browser. Toggle between the two runs below.

One owner, one log

Removing hops one at a time would have helped a little, and kept the shape that caused them: three parties writing the same thread’s state and using Postgres notifications to tell each other. We changed the shape instead.

Every thread gets one owner host, and that host is the only writer of the thread’s durable state. The state is an ordered log in Postgres. Its projections (queue, status, turns, unread counts) commit in the same transaction as each entry. The engine becomes a stateless edge: it authorizes once, looks up the owner, routes the command and fans the owner’s stream out to viewers.

A single writer is only safe if a stale writer cannot write. Each Append runs inside a transaction that locks the thread’s control row and checks a fence: the owner’s incarnation and epoch. Assigning a thread to a new host bumps the epoch, so an old owner that wakes up after a pause fails its next write instead of corrupting the log.

SELECT owner_incarnation, owner_epoch
  FROM harness_thread_control
 WHERE thread_id = $1 FOR UPDATE;

-- assigned: must match (incarnation, epoch) + live heartbeat
-- unassigned: incarnation IS NULL AND epoch = expected
INSERT INTO harness_thread_log (...) VALUES (...);
-- projections: queue, status, turns, unread (same transaction)
COMMIT;

Measuring it

We measured the final build with live traces: seven turns on a long existing conversation, one cold, four warm short answers and two warm turns with three sequential shell commands each. That gives local medians, not production percentiles, and the “before” column is an earlier observation rather than a controlled A/B test. With those caveats, the database side moved clearly.

The latency did not follow. Socket receipt to the gateway request was about 100 ms before and a median of 117 ms after, within noise of each other. The table shows why: the single append is cheap, but the work around it is not.

What remains is cold router construction, repeated environment reads and context capture on every step. None of these are coordination; they are just work. Removing the coordination made them visible.

The gateway hop

The last stop before the provider is our model gateway. Every model call that leaves Langdock passes through one Go service that embeds the Bifrost library for provider protocols. Around it we add caller identity, tenant authorization served from caches, a fail-closed capability policy and admission control.

Admission is where latency and correctness meet. A spend check that reads counters updated after usage lets concurrent requests all pass. The gateway reserves capacity atomically in one Redis round trip instead, and falls back to a bounded local limiter if Redis is down. Billing moves off the hot path: the gateway emits usage evidence and a separate accounting service settles it later.

What we keep

• One writer per thread. Fewer moving parts, fewer transactions, and a clear answer to “who may change this”.
• State in the notification. Viewers get at most two control notifications per send, each carrying the state it announces. No re-reads.
• Connection reuse. Reusing HTTP transports avoided repeated root-store and connection setup, a measured win.
• State the workload. Every number in this post comes with its build, topology and sample size. A latency claim without them is a guess.

Next up are release-build measurements in our production topology, and Postgres commit load under concurrency. If you enjoy this kind of trace-reading, we will publish those numbers here too.