GuideAdvanced
Voice and Realtime Agents
A chat waits for you to finish typing. A phone call waits for nobody: audio arrives in 20 millisecond pieces, people pause in the middle of a sentence, cut the agent off once they understood, cough, say "uh huh" while the agent talks, and hang up after one second too many of silence. This guide teaches you to build what makes a voice agent work in that world: **bidirectional streaming** of audio and events, **turn detection**, **interruptions** (barge-in) with the conversation trimmed to what the user actually heard, **end-to-end latency** measured stage by stage, **tools** that run without leaving dead air, and **telephony**, with its 8 kHz μ-law audio and its media stream bridge. You learn it by building the phone line of **Cardo Vet**, a fictional veterinary clinic: an agent that answers, understands you want an appointment for your dog, checks the calendar, offers times, stops talking when you cut in, confirms your pet's name by reading it back, and books the visit. You write the client that sends the microphone over WebSocket and plays the agent's voice, the policy that decides when an interruption is real and when it is an "uh huh", the bridge that connects a phone call to the model, and a suite of scripted calls that measures whether the agent is ready for production. All the code runs on your laptop with no API key and no accounts: a mock server speaks a subset of the event protocol of OpenAI's Realtime API (the same event names, cited from its reference) over a real WebSocket, and a scripted "caller" talks with one tone per letter that the server can read back. Latency is **measured** with a real clock on that simulator, not made up. Every piece is connected to its counterpart in the Realtime API, Gemini's Live API, a phone carrier's media streams and the open source voice frameworks, cited from their documentation.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Free
- access
Outcomes
What you'll be able to do
- Choose between a speech-to-speech architecture, an STT, LLM and TTS chain, or a full duplex model that delegates, by latency, control and cost
- Explain the digital audio that travels in a voice session (PCM16, sample rate, frames, base64, dBFS) and calculate how much it weighs
- Write an event client that sends and receives at the same time over WebSocket, and choose between WebSocket, WebRTC and SIP with ephemeral keys
- Configure and measure turn detection: threshold, prefix padding, silence, semantic VAD, noise, push-to-talk and idle timeout
- Implement full barge-in: stop playback, trim the conversation to what was heard, cancel the response, and ignore the "uh huh" and the echo
- Measure end-to-end latency per stage, report it as p50 and p95, and decide which lever moves each millisecond
- Use tools in a voice conversation without dead air, with preamble phrases, slow tools and spoken confirmation
- Connect a phone call to the model with a media stream bridge: μ-law, resampling, `clear` and marks, keypad and transfers
- Evaluate a voice agent with scripted callers and take it to production with a release gate
Before you start
What you need to bring
It's for you if...
- Engineers who have already called an LLM from code and want to build an agent that speaks and listens in real time
- Teams about to put an agent on the phone, or a "talk to us" button in their product
- Tech leads who have to choose between a speech-to-speech API, a chain of services or a voice framework
Requirements and materials
- Intermediate Python: functions, `dataclass`, dictionaries, JSON and basic `asyncio` (`async def`, `await`, `asyncio.create_task`). The guide explains the rest
- Having seen an LLM call with tools helps; the guide reviews the whole loop inside voice
- Python 3.10 or newer. Nothing else: no API keys, no accounts, no packages to install
- This is an advanced, optional guide. If you want to see vision, documents and batch audio first (transcribing a file, generating speech from text), the AI Multimodal Guide covers them; this guide does not need it
Content
The syllabus, module by module
Open any of them to see its lessons.
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!