Two brains, one voice: the dual-model call architecture

A ten-second pause in chat usually reads as "working on it". On a phone call, three seconds of silence makes the caller say "hello?". The model reading a thread or calling a tool needs those seconds and often many more, so one model cannot both hold the conversation and do the work well.

We therefore run two models. A fast brain sits beside the audio pipeline and uses a small model with low reasoning effort. A slow brain has the larger model, the tools, and the conversation history. It runs in the parent process and talks to the fast brain over a Unix socket. We already kept conversation separate from work, as described in the conversation layer post. Voice applies the same split under a much tighter time limit.

Dual-brain voice architecture: the caller connects through a LiveKit room to the fast brain process (STT, VAD, TTS, fast LLM), which exchanges utterance events and FastBrainNotifications with the slow brain ConversationManager over an IPC socket. — The two processes and the socket between them.

The design has one test: the caller cannot tell which model produced a line.

One structured call per turn

When the caller finishes speaking, the fast brain makes one structured model call. It returns a classification and, when needed, a short line to speak. Many turns need no reply, so deciding whether to speak is as important as choosing the words.

silence, for when the caller said "ok" or "mm-hm". Say nothing, and don't wake the slow brain either. Half of a natural phone call is acknowledgement noises, and a system that replies to every "yeah" is unbearable. This is also where most of the cost saving comes from, since bare acks never touch the big model.
smalltalk, for "how's your day going?" The fast brain answers alone. The slow brain still sees the exchange afterwards but almost always decides to stay quiet rather than pile a second answer onto a pleasantry.
defer, for anything needing data, tools, or actual thought. The fast brain speaks one short contextual line as a lead-in, capped at 160 characters, along the lines of "let me check that thread now", and hands the turn down. The slow brain composes the real answer.
continuation and hang_up, for interruptions and call endings, which get their own sections below.

One round-trip on a small model already consumes most of the latency budget, so the fast brain is not an agent with tools. Its prompt says that a slower model will answer anything substantive moments later. Its job is to hold the floor without committing to an answer that may be wrong.

Swimlane timeline of one engaged turn: the caller asks a question, the fast brain classifies it as defer in one structured LLM call and speaks a contextual filler, the slow brain reads the email thread and composes the answer, and the fast brain speaks it verbatim. — One engaged turn, both brains, no dead air.

The slow brain writes every real answer

The substantive reply always comes from the ConversationManager, and the fast brain speaks it verbatim. We do not ask the fast brain to make it chattier. The line the slow brain reasoned about and committed to memory is therefore the line the caller hears. A rewriting step would give the two models another place to disagree.

The state model is symmetric and reads a bit strangely at first. The slow brain's prompt describes the voice agent as its own mouth, so the agent's spoken lines appear in its history as if the CM had said them itself, and it's told the agent emits a brief filler on each turn to cover latency but never composes substantive replies. Meanwhile the fast brain gets hydrated at call start with the last fifty messages of history on that channel, so a caller who says "about what you emailed me yesterday" isn't met by an amnesiac. It's an odd setup but it means both sides behave as though they're one speaker.

The race between the filler and the answer

Once a turn is deferred, the filler and the real answer are both in flight. Usually the filler plays first. Sometimes the slow brain finishes first, which can produce a stale lead-in after the answer: "Thursday at 2pm is confirmed. Let me check that thread now..."

A turn counter prevents that. Every filler remembers which turn it belongs to. If the slow brain has already spoken on that turn, the queued filler is discarded. The slow brain is also told to continue from a filler rather than repeat it. When the caller asks two questions while it is still working, the fast brain receives a note not to use the same deferral twice.

We deliberately didn't build a reconciliation model, meaning some third LLM that merges the two brains' output into one smooth stream. Every coordination problem along this path gets solved with counters, queue clearing, and prompt rules instead, because putting a model in the hot path would just reintroduce the latency the split exists to remove.

Filling silence, reluctantly

Long silences happen, either because the slow brain is three tools deep into something, or the caller is thinking, or both. We fill some of them, though I'm fairly opinionated that a bad unprompted line is worse than silence, since nothing marks a bot faster than chirpy filler on a timer.

Every utterance resets the silence timer. After five quiet seconds, with nobody speaking and nothing queued, the slow model decides whether to say anything. It sees the transcript, the status of running work, and relevant screenshots. It may return another delay instead of a line. If the caller speaks before a queued line plays, we discard it.

Using the expensive model to decide whether to say "still on it, the export is about half done" looks extravagant, but generating the words was never the expensive bit. Working out whether anything is worth saying at all takes a judgment call over the whole call state, and that's what the slow brain is there for.

Hanging up without being weird

Ending a call requires two decisions. The slow brain judges whether the task is complete and whether the caller is finished or merely distracted. The fast brain chooses the sub-second moment to hang up. Giving both decisions to the fast brain cuts callers off; giving both to the slow brain makes the goodbye arrive too late.

So we split it with a gate. By default the fast brain can't end a call at all, so "ok, bye then" gets a normal reply rather than a dropped line. When the slow brain judges the call is genuinely done it arms the gate with a reason, and only then does hang_up appear among the fast brain's classifications. A goodbye-shaped turn now earns a brief warm closing line and then the hang-up, and twelve seconds of dead air earns a fixed farewell instead of more filler. Proactive silence-filling switches off once the gate is armed, because chattering at someone you're trying to let go of is its own kind of rude.

Three-stage hang-up gate: gate closed by default (fast brain cannot end the call), the slow brain arms it with allow_hang_up, and once armed a goodbye-shaped turn or 12 seconds of dead air leads to a farewell with a 1-second grace window that any speech aborts. — The slow brain decides whether the call is finished, and the fast brain picks the moment.

Even then there's an escape hatch, because after the farewell is spoken there's a one-second grace window where any speech from the caller aborts the close. "Oh wait, one more thing..." works on our calls, because it works on human calls. The actual teardown, meaning the Twilio hangup and the LiveKit room deletion, is owned by the ConversationManager, so a call always ends through the same path regardless of which brain started it.

Getting interrupted gracefully

Callers talk over the assistant constantly and that's fine, so barge-in stops the TTS immediately and we keep only the words that were actually spoken aloud in the transcript. We do that because the transcript is shared ground truth for both brains, so it needs to record what the caller heard rather than what the system meant to say.

The unspoken remainder isn't thrown away though. It gets stashed, and on the caller's next turn the fast brain decides what the interruption meant. "Sorry, go on" resumes the stashed line verbatim with no slow brain involved. A new question that changes the ask drops the remainder and forwards the interruption down as a voice interrupt for the slow brain to handle properly. If the caller barged in with no actual words, like a cough or a false start, the line just resumes without any model call at all. One asymmetry there is that an interrupted filler is never resumed, on the grounds that if you cut off "let me check that thread now...", nothing of value was lost.

The boring failure modes

If the slow brain fails mid-turn, the fast brain speaks a fixed apology and cancels any pending proactive line. Otherwise an error could be followed by a cheerful update that is no longer true. On outbound calls, we hold the opener until the callee says something or three seconds pass, so the assistant does not talk over their "hello?". Every utterance then follows the same message pipeline as chat, which lets a later conversation pick up where the call ended.

Where the split leaves us

Silence is bad on a live call, but a fast wrong answer is worse. The fast brain therefore classifies the turn and keeps the conversation moving without making substantive claims. Everything with consequences goes to the slower model. Turn counters and a hang-up gate keep the two paths consistent. I still expect this part of the system to change more than most, because timing failures only become obvious in real conversations.