# What is a vision-language model?

A model that takes images as well as text and answers in text, so it can describe a photo, read a chart or a screenshot, or answer questions about a document.

- Page: https://tecyfy.com/concepts/visionlanguage-model
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: beginner
- Also called: VLM, Vision Language Model

Paste a screenshot of an error into a chat assistant and ask what went wrong. The answer comes from a vision-language model (VLM): a language model that has been given a way to see. It reads the pixels, turns them into something the language model understands, and replies in words.

## How a picture gets into a language model

Most open vision-language models are built from 3 parts:

1. **An image encoder** cuts the picture into small square patches and turns each into a vector. It has usually been pre-trained on pairs of images and captions.
2. **A projector**, a small network, maps those vectors into the same space as the language model's word [embeddings](https://tecyfy.com/concepts/embedding).
3. **The language model** then reads the image patches as if they were extra [tokens](https://tecyfy.com/concepts/token) at the start of the prompt, followed by your question, and writes the answer.

LLaVA, from 2023, is the well-known example: a CLIP image encoder, a projector, and an open language model, trained on conversations about images. Other models are trained on images and text together from the start, but the idea, pictures become tokens that sit beside the words, is the same.

## Where the image understanding comes from

The key step came in 2021 with CLIP from OpenAI. It learned from 400 million pictures with their captions from the internet, by learning which caption belongs with which image. That gave image vectors that line up with meanings in words, which is exactly what a language model needs in order to talk about a picture.

## What they are good at

- Reading text in images: screenshots, receipts, signs, scanned pages.
- Charts, tables and diagrams, including turning them into data.
- Describing photos and answering questions about them.
- Looking at a web page or app screen and deciding what to click, which is how many computer-use agents work.

## Where they go wrong

They can describe things that are not in the picture, a visual kind of [Hallucination](https://tecyfy.com/concepts/hallucination). They are often weak at counting, at exact positions, and at fine detail in small or low-resolution images. And an image costs tokens: a large picture can cost as much as a page or more of text, so resize images to what the task needs before you send them.

## When to use one

Use a VLM when the information lives in pixels: a scanned form, a chart, a photo of a damaged part. If the same information exists as text (the PDF's text layer, the table as CSV, the error message itself), send the text instead. It is cheaper and usually more accurate.

## Related

- [Multimodal Model](https://tecyfy.com/concepts/multimodal-model): A model that works with more than one kind of input or output, such as text and images, or text, sound and video, instead of text alone.
- [Large Language Model](https://tecyfy.com/concepts/large-language-model): A model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions.
- [Embedding](https://tecyfy.com/concepts/embedding): A list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them.
- [Transformer](https://tecyfy.com/concepts/transformer): The design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once.
- [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.

## Sources

- [Learning transferable visual models from natural language supervision](https://arxiv.org/abs/2103.00020)
- [Visual instruction tuning](https://arxiv.org/abs/2304.08485)
- [Vision language models explained](https://huggingface.co/blog/vlms)