Voice AI engineering: latency, interruptions, and turn-taking
A voice agent that pauses like a person is an engineering problem, not a prompt. I build the pipeline so it feels usable on a real call.
Why most voice demos die
Text chat can wait a second. Voice cannot. If the model thinks for too long, if TTS starts late, or if the agent cannot hear a “wait, stop,” users hang up. I treat those as first-class bugs.
I write about this in the production voice AI guide on this site. The work is the same on a client project: stream partial transcripts, keep a tight agent graph, and never hide the tool calls from you.
What you walk away with
A repo, a deployed endpoint, and a recording of a call that does not sound like a hold menu. If you also need the web app around it, that is the full-stack service - same engineer, one thread.
What you get
- End-to-end voice loop: capture, transcribe, decide, speak
- Barge-in and interruption handling so users can cut in
- Turn-taking that does not talk over the caller
- Tool-calling agents behind the voice, with a visible fallback
- Latency budgets you can measure, not “it felt fast”
How we work
01
Scope
We pick one call type (support, booking, internal) and a latency budget. No kitchen-sink persona.
02
Build
I wire STT, the agent, TTS, and a debug timeline so you can see every delay.
03
Tune
Interruptions, silence, and failure audio. Then we put it on a real number.
