You speak into your phone or laptop. Within a second, your words appear on screen as text. The experience feels almost magical, and for clean audio with one speaker and no background noise, the accuracy is often genuinely impressive. But add a second person talking, a noisy room, or a technical term the model has never encountered, and errors start appearing.
Understanding how AI transcription actually works makes the errors less surprising and the good results more appreciable. It also tells you exactly what you can do to get better results from any tool you use.
The Three Stages That Convert Speech to Text
Modern AI transcription is not a single process. It is three distinct stages working in sequence, each one building on what the previous one produced.
Stage 1: Audio Processing
Before any language understanding happens, the raw audio signal needs to be converted into a format the AI can analyse. The system converts incoming audio into a spectrogram, a visual representation that maps sound frequencies against time. Different sounds produce different patterns in this representation.
This stage also handles noise reduction and normalisation. Background noise, inconsistent volume, and room echo are partially removed here before the audio ever reaches the language model. How well this stage works depends heavily on the original recording quality. A microphone placed close to the speaker produces a spectrogram with clean, distinct patterns. A laptop microphone picking up keyboard sounds, fan noise, and a distant speaker produces a messier one that the subsequent stages have to make sense of.
Stage 2: Acoustic Modelling
The acoustic model analyses the spectrogram and identifies phonemes, the basic units of sound that form words. This is the part trained on millions of hours of audio matched to text. It learns what different sounds look like in spectrogram form and maps them to the most likely phonetic sequences.
Modern acoustic models, including those based on OpenAI's Whisper architecture, use transformer neural networks trained on vast amounts of audio data. They have become very good at recognising phonemes even when they overlap, are partially obscured by noise, or are pronounced in unfamiliar ways. The acoustic model does not produce words directly. It produces probabilities: sequences of sounds that are likely to have been said, ranked by confidence.
Stage 3: Language Modelling
The language model takes the acoustic model's output and determines which actual words were most likely spoken, based on the context of surrounding words. This is why AI transcription correctly handles homophones. When the acoustic model hears a sound that could be "their," "there," or "they're," the language model resolves the ambiguity by looking at what words surround it. "Their meeting" makes linguistic sense. "There meeting" does not. The language model picks the right one without any explicit rule about grammar.
This context-awareness is what makes modern AI transcription far more accurate than earlier phonetic matching systems. It does not just match sounds to words in isolation. It understands that certain word sequences are far more probable than others.
For real-time transcription, these stages happen in rapid succession on audio chunks of a second or less. The transcript appears almost immediately after you speak because the model does not wait for a complete sentence before beginning to process.
Why Errors Happen
Given how sophisticated this pipeline is, the errors it produces follow predictable patterns. Understanding them explains most of the mistakes you see.
Audio Quality Is the Biggest Factor
The acoustic model can only work with what the microphone gives it. If a human would struggle to hear what was said clearly, the AI will too, and usually perform worse because it lacks the contextual knowledge humans bring to difficult listening situations.
In tests across major transcription services, accuracy on clean single-speaker audio consistently reaches 95 to 99 percent. On real-world audio with background noise, multiple speakers, or inconsistent recording conditions, accuracy typically drops to 85 to 90 percent and can go lower in challenging environments. The difference between providers on the same audio is usually only a few percentage points. The difference caused by audio quality can be fifteen percentage points or more.
This is the most actionable insight about AI transcription. Improving the recording matters more than switching providers.
Speaker Overlap Confuses the System
When two people talk at the same time, the acoustic model receives two overlapping phoneme streams. It is built to decode one coherent stream into words. Crosstalk produces a spectrogram pattern that does not cleanly correspond to either speaker's words. The model typically captures the louder voice and drops the quieter one, producing a transcript that silently omits whatever the quieter speaker said.
This is why meeting transcription is consistently harder than recording a single person speaking. Group conversations with frequent interruptions and overlapping commentary produce significantly more errors than solo recordings of equivalent length.
Unfamiliar Vocabulary Has No Statistical Support
The language model predicts likely word sequences based on patterns in its training data. A word that appears rarely or not at all in training data has low statistical support. When the acoustic model produces sounds that could correspond to a rare technical term, the language model often substitutes a more common-sounding word instead.
Medical terminology, legal language, scientific nomenclature, company-specific product names, and proper nouns are all vulnerable to this. A drug name might be replaced with a phonetically similar common word. A person's unusual name might become something recognisable. The model is not making a random error. It is making a statistically informed guess that happens to be wrong because the right answer was underrepresented in training.
Some transcription services allow custom vocabularies that boost the probability of specific terms. This directly addresses the statistical support problem and improves accuracy significantly for specialised content.
Accents Are a Training Data Problem
AI transcription models are trained on audio data that reflects the demographics and geographies of whoever contributed to the training set. Accents, dialects, and speech rhythms that are well-represented in training data transcribe more accurately. Those that are underrepresented produce more errors because the acoustic model has seen fewer examples of how those sounds map to words.
Accuracy for standard American and British English consistently leads at 98 percent or higher. Other languages and regional accents vary, sometimes significantly. This is a known limitation being actively addressed through more diverse training data, but it has not been fully resolved.
Real-Time Systems Face an Additional Constraint
Standard transcription processes the entire audio file before producing output, giving the language model access to complete sentences and full context. Real-time transcription must produce output within a second or less of each spoken phrase, which means the language model has access only to recent context rather than the complete utterance.
This occasionally produces errors that a non-real-time system would catch. A word in the middle of a sentence might be misidentified because the disambiguating context comes later in the sentence and is not yet available when the real-time model commits to its output. Some real-time systems partially address this by retroactively revising earlier words when later context clarifies them, which is why you sometimes see a word in a live transcript change a moment after it first appears.
What You Can Do to Get Better Results
Most transcription errors are addressable at the recording stage rather than the processing stage.
Use a close-proximity microphone. A headset, lapel mic, or quality desktop microphone positioned close to the speaker produces dramatically cleaner audio than a built-in laptop or phone microphone picking up room acoustics from a distance. This single change produces the largest accuracy improvement available to most users.
Minimise background noise. The acoustic model partially filters noise, but it cannot eliminate it. Recording in a quieter environment, closing windows, turning off fans, and muting microphones when not speaking all improve the signal-to-noise ratio that the model works with.
Speak at a moderate pace with clear articulation. Very fast speech compresses phonemes and produces a spectrogram with less distinct patterns. A slightly slower and more deliberate pace gives the acoustic model more distinct signal to work with.
Add custom vocabulary for specialised content. If you regularly transcribe content with industry-specific terms, proper nouns, or unusual vocabulary, any transcription service that supports custom vocabulary or terminology lists is worth using. Boosting the probability of the specific terms you need is the most direct fix for vocabulary-related errors.
Review sections that matter most. At 95 to 98 percent accuracy on clear audio, a thousand-word transcript contains roughly twenty to fifty errors. A quick scan for the critical sections, names, numbers, technical terms, and action items, catches most of what matters without requiring a word-by-word review.
Frequently Asked Questions
Why does AI transcription sometimes change a word I already saw in the transcript?
Real-time systems commit to words as they are spoken but may revise them when subsequent context changes what the language model believes was most likely said. This retroactive correction is a feature rather than a bug. It means the final transcript is more accurate than the initial output, even though the live experience can look slightly unstable as words update.
Is transcription accuracy affected by speaking speed?
Yes. Very fast speech compresses phonemes and makes the acoustic model's job harder. Deliberate pauses between sentences also help the system correctly segment speech into distinct utterances rather than running words together. Speaking at a natural but unhurried pace consistently produces better results than speaking very quickly or very slowly.
Why does AI transcription struggle with names and proper nouns?
Names have low statistical frequency in training data compared to common words. When the acoustic model produces sounds corresponding to an unusual name, the language model lacks statistical support for that specific word and substitutes something it considers more probable. Providing a custom vocabulary list with names you expect to appear is the most effective fix for this specific problem.
Discussion (0)
Be the first to comment.