# What is OpenAI Whisper?

OpenAI's open-source speech recognition model. It turns audio in many languages into text, or translates it into English, and runs on your own machine.

- Page: https://tecyfy.com/concepts/whisper
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: beginner
- Also called: OpenAI Whisper

Whisper is the speech-to-text model OpenAI released openly in 2022. Hand it an audio file in almost any common language and it returns the transcript, with timestamps. It can also translate the speech straight into English text. Because the weights are free under the MIT licence, it runs on your own hardware, and it sits underneath a great many transcription apps.

## Why it was different

Earlier speech systems were mostly trained on carefully transcribed recordings, which are expensive, so there were never very many. OpenAI took the opposite route. It collected 680,000 hours of audio from the internet together with the transcripts that came with it: messy, imperfect, but enormous. The result was not the very best on any one tidy benchmark, but it held up far better on real recordings with accents, background noise and technical words, without any extra training for each new kind of audio.

Under the hood it is a [Transformer](https://tecyfy.com/concepts/transformer): the audio is turned into a spectrogram (a picture of sound frequencies over time), an encoder reads it, and a decoder writes the text one [Token](https://tecyfy.com/concepts/token) at a time, much like a language model.

## The sizes

| Size | Parameters | Video memory, about |
|---|---|---|
| tiny | 39M | 1 GB |
| base | 74M | 1 GB |
| small | 244M | 2 GB |
| medium | 769M | 5 GB |
| large | 1,550M | 10 GB |
| turbo | 809M | 6 GB |

The larger models are more accurate and slower. Turbo is a trimmed version of large that runs about 8 times faster with a small loss in accuracy. For English-only audio, the `.en` versions of the smaller sizes are usually more accurate than their multilingual twins.

## Running it

```bash
pip install -U openai-whisper
whisper meeting.mp3 --model turbo
```

That writes the transcript next to the file, in plain text and subtitle formats. Faster community versions, such as faster-whisper and whisper.cpp, run the same weights with less memory, including on laptops without a GPU.

## Where it goes wrong

Whisper writes what it hears as fluent text, and like other generators it can produce words that were never said. For anything where a wrong word matters, such as medical notes, legal records or quotes, check the transcript against the audio. It also has no idea who is speaking; telling speakers apart needs a separate step called diarisation.

## Related

- [Speech‑to‑Text](https://tecyfy.com/concepts#speechtotext): The task or model that converts spoken language audio into written text.
- [Transformer](https://tecyfy.com/concepts/transformer): The design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once.
- [Multimodal Model](https://tecyfy.com/concepts/multimodal-model): A model that works with more than one kind of input or output, such as text and images, or text, sound and video, instead of text alone.
- [Zero‑shot Learning](https://tecyfy.com/concepts#zero-shot-learning): Solving a task without any task‑specific examples

## Sources

- [Robust speech recognition via large-scale weak supervision](https://arxiv.org/abs/2212.04356)
- [Whisper](https://github.com/openai/whisper)