AssemblyAI

There is a category error that costs teams months, and it is worth naming before anything else: a transcription product and a speech API are not the same purchase.

A transcription product such as Otter.ai gives you an interface, a file list and an export button. A speech API gives you an endpoint, and everything a user would touch is yours to build. Buying the second when you wanted the first means paying to develop a product you could have subscribed to.

AssemblyAI is unambiguously the second. It describes itself as voice AI infrastructure for developers, and every part of the platform assumes there is an engineering team on the other side. This review covers what the models do, why the understanding layer matters more than the transcript, and the specifications you will have to dig for.

What Does AssemblyAI Actually Sell?

• Speech-to-text as infrastructure, not a transcription app — production APIs you build a product on, with no end-user interface.
• Two speech models with different jobs: Universal-3.5 Pro is the highest-accuracy pre-recorded model and the recommended default; Universal-2 covers broader languages and legacy features.
• Real-time and pre-recorded are separate paths, and a Voice Agent API runs speech-to-speech agents over a single WebSocket.
• Speech Understanding is the second product — structured insight from transcripts rather than the transcript itself.
• Guardrails are first-class: PII redaction, profanity filtering and content moderation ship as platform features rather than as something you build.

Universal-3.5 Pro And What Changed

AssemblyAI documents two pre-recorded models with an explicit division of labour. Universal-3.5 Pro is described as the highest-accuracy model and the recommended default. Universal-2 is positioned for broad language coverage and legacy feature support.

That split is more useful than a single flagship, and the honesty is worth crediting. Speech recognition accuracy and language breadth pull against each other — the best English model is rarely the best Vietnamese one, because the training data and the acoustic modelling differ.

For a buyer the implication is concrete. If your audio is English-language customer calls, take the Pro model and expect the better result. If your product serves a long tail of languages, check Universal-2's coverage against your actual language mix before committing, because a model that is excellent in twelve languages is useless if your users speak the thirteenth.

Around the models sit the features that make transcripts usable: speaker labels for diarisation, word-level timestamps, language detection, custom spelling, key terms, and medical terminology support. Custom spelling matters more than it sounds — every product has proper nouns, product names and jargon that a general model will render phonetically and wrongly, forever, until you tell it otherwise.

The best English model is rarely the best model for your thirteenth language. A vendor offering two and telling you which is which is being straight with you.

Can You Trust A Transcript You Did Not Check?

No speech system is perfect, and the way they fail matters more than the rate at which they fail.

Recognition errors are not random. They cluster on exactly the words that carry meaning — names, numbers, technical terms, and anything spoken quickly or over background noise. A transcript can be 95 per cent accurate overall and wrong on the account number, which is the only part anyone needed.

This has a direct design consequence. If a transcript drives an automated action — routing a case, updating a record, triggering a payment — build a confidence threshold and a human path for anything below it. Word-level timestamps exist partly so a person can jump to the audio and check rather than reading the whole thing.

It also means you should measure on your own audio rather than trusting any benchmark. The same caution applies to Speechmatics and every rival: vendor accuracy figures are measured on clean corpora; your audio has hold music, crosstalk, accents and a caller in a car. Transcribe fifty of your own recordings, count the errors that would actually matter, and that number is the only one worth planning around.

Why Speech Understanding Is The Harder Half

AssemblyAI sells a Speech Understanding API for structured insight from transcripts, separately from the transcription itself. That separation reflects where the real work sits.

A transcript is not an answer. Nobody wants ten thousand words of a support call; they want to know what the customer asked for, whether it was resolved, how the caller sounded, and whether anything needs escalating. Getting from one to the other used to mean building a second pipeline with its own models and its own maintenance burden.

Buying that layer rather than building it is usually the right call for a product team, for an unglamorous reason: the second pipeline is the one that rots. Transcription is stable; summarisation, classification and sentiment analysis have to be revised every time the underlying models improve, and that is a permanent cost on your roadmap rather than a one-off.

Pairing it with a voice generator such as ElevenLabs completes an agent, and the LLM Gateway points at the same idea — a route to language models over the same platform, so the understanding step does not require separate vendor management.

The judgement to apply is where the derived output is trusted. Summaries are fine as an aid to a human reading the transcript. Sentiment scores driving an automated escalation decision deserve considerably more scepticism, because inferring emotional state from words is unreliable in exactly the cases — sarcasm, restraint, cultural difference — where it would matter.

• Use transcription output for search, records and evidence — it is checkable against the audio.
• Use derived summaries to help a person read faster, not to replace their reading.
• Treat sentiment as weak signal, and never as the sole trigger for an automated action.
• Apply PII redaction at ingest if you will store transcripts, not as a later cleanup step.
• Keep the audio, since every derived output can be regenerated but the recording cannot.

Voice Agents Over A Single WebSocket

The Voice Agent API runs speech-to-speech agents over a single WebSocket, and the architectural detail in that sentence is the whole point.

The conventional way to build a voice agent chains three services: speech-to-text, then a language model, then text-to-speech, each with its own connection and its own latency. The delays add, and the result is the stilted pause everyone recognises from automated phone systems — the thing that makes callers press zero.

Collapsing that into one connection is a latency argument rather than a feature. Conversational turn-taking has a fairly narrow tolerance; past roughly a second of silence, people start talking over the system or assume it has failed. Every hop you remove buys headroom.

The trade is control. A chained pipeline lets you swap the language model, tune the voice, and inspect each stage independently. A single integrated path is faster and gives you fewer places to intervene.

The right choice follows from what the agent does. For a scripted, high-volume interaction where speed is the experience — checking an order, booking a slot — integrated latency wins. For an agent that needs careful reasoning, tool use and auditable steps, the chained approach remains easier to build and debug.

Benchmarking It On Your Own Audio

Every speech vendor publishes accuracy measured on clean corpora, and your audio is not clean. The only benchmark that predicts what you will get is one you run yourself, and it takes an afternoon.

The bar is step four. An error rate on words that carry meaning is the number your product actually depends on, and it is very different from an overall rate.

1. Collect fifty real recordings that span your actual conditions — background noise, accents, crosstalk, a caller on a bad line. Not your cleanest samples.
2. Load your domain vocabulary into custom spelling and key terms before you start. Skipping this is the most common reason a first test disappoints.
3. Transcribe them all on Universal-3.5 Pro, and repeat on Universal-2 if any of your languages sit outside the major ones.
4. Count errors only on words that would change an outcome — names, numbers, product terms, dates. An overall accuracy figure hides exactly the failures that matter.
5. Test diarisation separately on your hardest recordings, since speaker attribution degrades on crosstalk long before transcription does.
6. If you are building an agent, measure round-trip latency on the streaming path rather than the pre-recorded one; they are different services with different characteristics.

How AssemblyAI Bills For Audio

Speech APIs in this category are metered by audio duration rather than by seat, and the practical unit is hours or minutes of audio processed.

That model has a property worth designing around: cost scales with volume, not with team size. A three-person startup processing a large volume of calls pays more than a hundred-person company processing few, which is unusual and generally fair.

The variables that move a bill are duration, whether you use the real-time path, and how many understanding features you layer on. Streaming is typically priced differently from pre-recorded because it is a different service, and each derived output is additional work.

The engineering lever most teams miss is not sending audio you do not need. Silence trimming, voice activity detection, and simply not transcribing hold music are the difference between a predictable bill and a surprising one at scale.

We do not publish amounts, since rates change. Model your cost on real audio duration from a representative week, and check whether committed-volume terms exist before you scale — in this category they usually do, and they are worth asking about.

AssemblyAI: The Developer Specification

Speech API Terms That Decide A Build

Five terms that determine whether a speech platform fits what you are building.

Word error rate (WER)

The standard accuracy measure — insertions, deletions and substitutions against a reference transcript. Useful for comparing models on the same audio, and misleading across different audio, because clean corpora flatter everyone.

Diarisation

Working out who spoke when, and labelling the transcript accordingly. Essential for calls and meetings, and the feature most likely to degrade with crosstalk or similar-sounding voices.

Streaming versus pre-recorded

Two different services. Streaming returns partial results as audio arrives and optimises for latency; pre-recorded sees the whole file and optimises for accuracy. Expect different models, different prices and different results.

PII redaction

Automatically removing personal data — names, card numbers, addresses — from a transcript. Apply it at ingest if you store transcripts, because retro-fitting redaction over stored data is far harder.

Custom spelling and key terms

Telling the model your proper nouns, product names and jargon in advance. The cheapest accuracy improvement available, and the one teams most often skip.