What is a vision-language model?

Paste a screenshot of an error into a chat assistant and ask what went wrong. The answer comes from a vision-language model (VLM): a language model that has been given a way to see. It reads the pixels, turns them into something the language model understands, and replies in words.

How a picture gets into a language model

Most open vision-language models are built from 3 parts:

1. An image encoder cuts the picture into small square patches and turns each into a vector. It has usually been pre-trained on pairs of images and captions.
2. A projector, a small network, maps those vectors into the same space as the language model's word embeddingsEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them..
3. The language model then reads the image patches as if they were extra tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. at the start of the prompt, followed by your question, and writes the answer.

LLaVA, from 2023, is the well-known example: a CLIP image encoder, a projector, and an open language model, trained on conversations about images. Other models are trained on images and text together from the start, but the idea, pictures become tokens that sit beside the words, is the same.

Where the image understanding comes from

The key step came in 2021 with CLIP from OpenAI. It learned from 400 million pictures with their captions from the internet, by learning which caption belongs with which image. That gave image vectors that line up with meanings in words, which is exactly what a language model needs in order to talk about a picture.

What they are good at

• Reading text in images: screenshots, receipts, signs, scanned pages.
• Charts, tables and diagrams, including turning them into data.
• Describing photos and answering questions about them.
• Looking at a web page or app screen and deciding what to click, which is how many computer-use agents work.

Where they go wrong

They can describe things that are not in the picture, a visual kind of HallucinationHallucinationWhen a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist.. They are often weak at counting, at exact positions, and at fine detail in small or low-resolution images. And an image costs tokens: a large picture can cost as much as a page or more of text, so resize images to what the task needs before you send them.

When to use one

Use a VLM when the information lives in pixels: a scanned form, a chart, a photo of a damaged part. If the same information exists as text (the PDF's text layer, the table as CSV, the error message itself), send the text instead. It is cheaper and usually more accurate.