What is OpenAI Whisper?
OpenAI's open-source speech recognition model. It turns audio in many languages into text, or translates it into English, and runs on your own machine.
beginner · Also called OpenAI Whisper
Whisper is the speech-to-text model OpenAI released openly in 2022. Hand it an audio file in almost any common language and it returns the transcript, with timestamps. It can also translate the speech straight into English text. Because the weights are free under the MIT licence, it runs on your own hardware, and it sits underneath a great many transcription apps.
Why it was different
Earlier speech systems were mostly trained on carefully transcribed recordings, which are expensive, so there were never very many. OpenAI took the opposite route. It collected 680,000 hours of audio from the internet together with the transcripts that came with it: messy, imperfect, but enormous. The result was not the very best on any one tidy benchmark, but it held up far better on real recordings with accents, background noise and technical words, without any extra training for each new kind of audio.
Under the hood it is a TransformerTransformerThe design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once.: the audio is turned into a spectrogram (a picture of sound frequencies over time), an encoder reads it, and a decoder writes the text one TokenTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. at a time, much like a language model.
The sizes
| Size | Parameters | Video memory, about |
|---|---|---|
| tiny | 39M | 1 GB |
| base | 74M | 1 GB |
| small | 244M | 2 GB |
| medium | 769M | 5 GB |
| large | 1,550M | 10 GB |
| turbo | 809M | 6 GB |
The larger models are more accurate and slower. Turbo is a trimmed version of large that runs about 8 times faster with a small loss in accuracy. For English-only audio, the .en versions of the smaller sizes are usually more accurate than their multilingual twins.
Running it
pip install -U openai-whisper
whisper meeting.mp3 --model turbo
That writes the transcript next to the file, in plain text and subtitle formats. Faster community versions, such as faster-whisper and whisper.cpp, run the same weights with less memory, including on laptops without a GPU.
Where it goes wrong
Whisper writes what it hears as fluent text, and like other generators it can produce words that were never said. For anything where a wrong word matters, such as medical notes, legal records or quotes, check the transcript against the audio. It also has no idea who is speaking; telling speakers apart needs a separate step called diarisation.
Sources