Deepgram

Everyone has heard the moment a voice agent fails. You finish speaking, nothing happens, you start to repeat yourself, and the system talks over you. The technology was not wrong — it was late.

That single property, latency, decides whether a voice interface feels like a conversation or like an automated phone menu. And it is where the conventional architecture falls down: chain speech-to-text, then a language model, then text-to-speech, and every hop adds delay that accumulates into exactly that pause.

Deepgram's product decisions all point at that problem. It is worth understanding why, because it explains what the company is good at and what it does not tell you.

What Does Deepgram Provide?

• Two Flux models: Flux STT for speech-to-text, described as conversation-aware, and Flux TTS for synthesis, described as thinking before it speaks.
• A Voice Agent API combining speech-to-text, text-to-speech and LLM orchestration in one interface — the reason latency stays low.
• 80ms text-to-speech latency, which is the number that decides whether a voice agent feels conversational or broken.
• Real time or batch, cloud or self-hosted — self-hosting is presented as a normal deployment option, not an enterprise negotiation.
• Infrastructure customers including Twilio, Cloudflare, IBM, Sierra and Cresta, which are companies that evaluate voice APIs seriously.

Why Does One API Beat Three Services?

Deepgram's Voice Agent API combines speech-to-text, text-to-speech and LLM orchestration into a single API, stated as reducing complexity, latency and cost.

The latency argument is the substantive one. In a chained architecture, audio travels to a transcription service, the text travels to a model, the response travels to a synthesis service, and the audio comes back. Each leg has network time, queuing and processing, and they add rather than overlap.

Human conversational turn-taking has a narrow tolerance. Past roughly a second of silence people assume something has gone wrong and start speaking again, which produces the collision everyone recognises. Removing hops is the only structural way to buy that headroom back — you cannot optimise your way out of three round trips.

The complexity argument matters too, and is less discussed. Three services means three sets of credentials, three failure modes, three billing relationships and three places for a version change to break your agent at two in the morning.

The trade is control, and it is real. A chained pipeline lets you swap in a different language model, tune each stage independently and inspect what happened at each step. An integrated API is faster and gives you fewer places to intervene when something is subtly wrong.

The alternative is assembling the same stack from parts: transcription from one vendor, synthesis from ElevenLabs, orchestration of your own. That is entirely workable, and it leaves you owning three latency budgets instead of one.

You cannot optimise your way out of three network round trips. Removing a hop is the only thing that actually buys latency back.

What Conversation-Aware Actually Means

Deepgram describes Flux STT as conversation-aware and Flux TTS as the first voice model that thinks before it speaks. Both phrases are doing real work, though neither is defined on the site.

For transcription, conversation-awareness most plausibly refers to turn detection — knowing when a speaker has actually finished rather than merely paused. That is the hardest part of real-time speech and the source of most agent interruptions, because a system that treats every pause as a turn will cut people off mid-sentence.

For synthesis, thinking before it speaks suggests the model plans an utterance's prosody before generating audio, rather than producing it word by word. That would explain more natural emphasis and intonation, which is what separates a voice that sounds human from one that sounds like a reader.

Both are plausible and neither is documented in detail on the pages we read. Test them directly: interrupt the agent mid-sentence, pause mid-thought, and see whether it waits or jumps in. Turn handling is where voice agents actually fail, and it is easy to evaluate in five minutes.

Self-Hosting Changes Who Can Buy It

Deepgram states that its models run in real time or batch, in the cloud or self-hosted, with dedicated self-hosting documentation.

That single option determines whether the platform is usable for a whole category of buyer. Voice data is unusually sensitive: a recorded call contains a person's voice — which is biometric identity — plus whatever they said, which in healthcare, finance and legal contexts is regulated separately.

For organisations under strict data residency rules or handling protected health information, sending audio to a third-party cloud is frequently prohibited outright, regardless of the vendor's certifications. On-premise deployment removes that objection rather than arguing with it.

What makes Deepgram's position notable is that self-hosting is presented as an ordinary deployment mode rather than an enterprise concession extracted in negotiation. Many competitors treat it as a bespoke arrangement, which in practice means slow and expensive.

The questions to settle before relying on it are practical: what hardware the models need, whether self-hosted versions track cloud releases, and how licensing works. Self-hosted AI infrastructure has real operational cost, and it is worth knowing that cost before choosing it over a compliance conversation you might have won.

Deployment terms are also the axis on which its closest competitors differ. AssemblyAI and Speechmatics each make a different case here, and which matters more is a procurement question rather than a technical one.

• Establish the hardware requirement before assuming self-hosting is cheaper than cloud.
• Ask whether self-hosted models receive the same updates as the cloud versions, and how quickly.
• Confirm licensing terms for on-premise deployment separately from cloud pricing.
• For regulated audio, get data residency and retention terms in writing either way.
• Test the same audio through both deployments if you may switch later.

Reading The Customer List Properly

Deepgram names Twilio, Cloudflare, Sierra, IBM, Daily, Cresta, Granola, VAPI, Decagon, Kore.ai, NICE-Cognigy and Coval.

This list is worth more scrutiny than a typical logo wall, because of what these companies are. Twilio, Daily and VAPI are communications infrastructure providers. Sierra, Decagon, Cresta and Kore.ai build conversational AI products. These are not end users who bought a subscription; they are companies whose own products depend on the voice layer working.

That kind of customer evaluates differently. An infrastructure provider integrating a speech API tests latency under load, failure behaviour and accuracy on real traffic, because their own reliability inherits from it. Passing that scrutiny repeatedly is stronger evidence than any published benchmark.

It also tells you where Deepgram sits in the market. This is a supplier to the people building voice products, which fits the infrastructure-first positioning and explains why the marketing pages assume an engineering audience.

The corollary for a smaller buyer is that you are buying something built to satisfy demanding integrators, which is generally good — and that the documentation assumes you are one.

If your need is transcription of recorded audio rather than live agents, a service like Rev solves that problem more directly and at far less architectural cost.

Testing A Voice Agent Before You Build One

Voice infrastructure is unusually easy to evaluate badly, because a demo in a quiet room with clear speech tells you almost nothing about production.

The bar is step three. Turn detection is where voice agents fail, and it fails on real speech rather than on scripted prompts.

1. Build the smallest possible agent through the Voice Agent API rather than testing transcription alone — the integrated path is what you would actually ship.
2. Measure round-trip latency from end of speech to start of response, over a normal connection rather than a fast one. That number is your product experience.
3. Interrupt it mid-sentence, then pause mid-thought without finishing. A system that talks over you or cuts you off has failed the test that matters.
4. Feed it your real audio conditions: background noise, an accent your users have, a caller on a hands-free phone in a car.
5. If you need self-hosting, deploy it and compare output against the cloud version on identical audio before committing to either.
6. Ask directly for language coverage, accuracy figures and compliance documentation, since none appear on the marketing pages.

What Deepgram Charges, And What It Omits

Pricing is not disclosed on the pages we read. In this category billing is normally by audio duration for transcription and by characters or duration for synthesis, with self-hosting licensed separately.

That structure means cost scales with usage rather than seats, which suits an infrastructure product — a small team processing heavy volume pays more than a large team processing little, which is the correct shape.

More notable is what else is missing from the marketing pages. No language count appears anywhere we read, which is the first specification most buyers need. No word error rate or accuracy figure is published. No compliance certifications — SOC 2, HIPAA, GDPR — are stated where procurement would look for them, despite self-hosting being offered specifically for regulated buyers.

None of that means the answers are unfavourable; infrastructure companies frequently keep these in documentation or supply them on request. It does mean a buyer cannot complete an evaluation from the website, which is a real friction for a product otherwise aimed at developers who prefer to self-serve.

We do not publish amounts. Request the language list, accuracy data and compliance pack together in your first contact, since you will need all three.

Voice Agent Terms That Decide A Build

Five terms that determine whether a voice product feels usable.

Turn detection

Knowing when a speaker has genuinely finished rather than merely paused. The hardest part of real-time voice, and the source of nearly every interruption users complain about.

Time to first byte

How long after the user stops speaking before audio starts coming back. Past roughly a second, people assume the system has failed and start talking again.

Barge-in

A user interrupting the agent mid-sentence and the agent stopping to listen. Essential for anything conversational, and frequently missing from cheaper stacks.

Streaming versus batch

Streaming returns partial results as audio arrives and optimises for latency; batch sees the whole file and optimises for accuracy. Different services with different results.

Self-hosting

Running the models on your own infrastructure. Often the only way to use voice AI where data residency rules or health information regulations apply.

Deepgram: Models, Latency And Deployment