Digital MacaroniVoice agent architecture
How AI voice agents are built
Every voice agent has to hear, think, call tools and speak. How those parts fit together decides how fast it replies, how well it handles interruptions, how accurately it hears details and who can maintain it. There are three main ways to build one, and we run a live demo of each.
By Leif Johansen, founder of Digital Macaroni · Updated 5 October 2026
The parts of a voice agent
- Phone line
- A carrier such as Twilio or Telnyx answers the number and streams the call audio in real time.
- Hear
- Turns the caller’s voice into something the agent can work with: a transcript, or audio a model understands directly.
- Think
- A language model works out what the caller wants, decides what to do next and calls tools.
- Speak
- Turns the reply into a voice, ideally while the reply is still being written.
- Tools
- The business’s own systems the agent can use: schedules, catalogues, bookings, orders, messages.
The three architectures below differ in one place: how hearing, thinking and speaking are put together.
01 · Speech-to-speech
Single bidirectional model
A single model takes the caller’s audio in and sends audio back out, reasoning and calling tools in between, all in one live session. Nothing is handed from one model to another, which is why it replies fastest and handles interruptions most naturally.
One live session, audio in and out at once.
Good at
- Lowest delay
- Natural interruptions
- Hears the voice, not a transcript
- Tool calls during the conversation, sync or async
Trade-offs
- Voices come from the model’s built-in set rather than a voice designed from scratch.
- Less control over exact wording than when a separate text model writes every reply.
- Hearing, reasoning and speech all come from one provider.
Our buildGoogle Gemini
Google ADK
Twilio
Google Cloud
Resend
02 · Voice platform
Managed voice platform
A voice platform runs the real-time conversation: it transcribes the caller, runs the reasoning model you choose and speaks the reply in one of its voices. When the agent needs the business’s data, the platform calls back to your own server, so the data and the records stay with you.
ElevenLabs AgentsScribe RealtimeGemini 3.7 FlashEleven v4 TurboGood at
- The real-time loop is handled
- Pick the brain
- Tools run on our side
- Easy to own and hand over
Trade-offs
- The platform charges its own per-minute fee on top of the call.
- You build within the platform’s features and limits.
- One more service in the path of every call.
Our buildElevenLabsGoogle GeminiTwilioGoogle CloudResend
03 · STT → LLM → TTS
Cascade: hear, think, speak
Three separate models work in a row: speech-to-text writes down what the caller said, a language model decides what to do and calls tools, and text-to-speech says the reply. Your own code decides when each turn starts and ends. It’s a little slower, but each part can be chosen for its job and swapped on its own.
AssemblyAI Universal-3.6 ProGemini 3.5 Flash-LiteGemini 3.8 Flash-Lite TTSGood at
- The best ear for phone audio
- Every part chosen for its job
- Turn-taking we control
- Exact where it matters
Trade-offs
- A slightly longer pause before each reply than a single bidirectional model.
- More moving parts: three models from two providers, plus our own turn-taking.
- More engineering to build and to maintain.
Our buildAssemblyAIGoogle Gemini
Duffel
TelnyxGoogle CloudResend
Side by side
| Compare | Single bidirectional model | Managed voice platform | Cascade: hear, think, speak |
|---|---|---|---|
| How it works | One model hears, thinks and speaks in one live stream | A voice platform runs the conversation; tools stay on our server | Separate models hear, think and speak, one after another |
| Reply speed | Fastest | Fast | A little slower |
| Interruptions | Most natural | Handled by the platform | Handled by our own code |
| Names, numbers and codes | Good | Good | Best in our tests |
| Voice | The model’s built-in voices | A large library of premium voices | Any voice, including one designed for you |
| Control over wording | Less | More | Most |
| Extra platform fee | No | Yes | No |
| Easiest to hand over | Medium | Easiest | Needs more engineering |
How to choose
- Single bidirectional model
- Your callers interrupt, change plans and want it to feel like talking to a person.
- Managed voice platform
- You want the agent in your own account, simple to look after, with a premium voice.
- Cascade: hear, think, speak
- Every call is full of names, numbers or codes that have to be exactly right.
Questions about voice agent architecture
What is a voice agent architecture?
It’s how the parts of a voice agent fit together: what hears the caller, what decides what to do and calls the business’s tools, and what speaks. Those choices decide how quickly it replies, how well it handles interruptions, how accurately it hears details, what each call costs and who can maintain it.
What’s the difference between speech-to-speech and a cascade?
A speech-to-speech model takes the caller’s audio in and produces audio out in one step, so it replies quickly and handles interruptions naturally. A cascade splits the job: speech-to-text writes down what the caller said, a language model decides what to do, and text-to-speech says the answer. It’s a little slower, but each part can be chosen and tuned for its job.
Which architecture is fastest?
A single speech-to-speech model, because nothing is handed between separate steps. A managed voice platform comes close. A cascade adds a short pause before each reply, which we keep small by starting the reply early and preparing common lines in advance.
Which is most accurate with names, numbers and codes?
In our own tests on noisy phone audio, a cascade with a transcriber built for phone calls caught 97% of key details, against 70% for the first transcriber we tried. That’s why our booking demo uses a cascade.
Do I need a voice platform like ElevenLabs?
Not always. A platform is the quickest way to a natural voice and the easiest setup to hand over, but it adds its own per-minute fee. Speech-to-speech and cascade builds run without one.
Can the architecture change later?
Yes. In our builds the business’s tools and data are kept separate from the engine that hears and speaks. All three of our demos use the same kind of tool layer, so the engine can be swapped as better models arrive without rebuilding the agent.
Does the architecture affect my phone number or carrier?
No. Any of them works with carriers such as Twilio and Telnyx, and with your existing business number through call forwarding.
Not sure which one fits your calls?
Tell us how your phone line works. We’ll recommend an architecture and explain why.