# What is a multimodal AI model? A model that works with more than one kind of input or output, such as text and images, or text, sound and video, instead of text alone. - Page: https://tecyfy.com/concepts/multimodal-model - Subject: How AI works (https://tecyfy.com/concepts) - Level: beginner - Also called: Multi-Modal Model, Multimodal AI - Last checked: 2026-10-14 Early chatbots only read and wrote text. A multimodal model handles more than one kind of material, or "mode": text plus images, and often sound, video and documents too. You use one when you photograph a menu and ask what to order, or drop a chart into a chat and ask what it shows. ## In and out are different questions "Multimodal" can mean 2 separate things, and it pays to ask which. **What it can take in.** Claude and Gemini both read images, and Gemini also takes audio. Google says its Gemini models were built to be multimodal from the start, and can caption a picture, answer questions about it, and mark where objects are. **What it can put out.** Reading an image is not the same as making one. Claude is a clear example: Anthropic says it understands images but cannot generate or edit them. Making pictures is a different job, often done by a different kind of model, such as a [diffusion model](https://tecyfy.com/concepts#diffusion-model). ## How a model "sees" A picture is turned into [tokens](https://tecyfy.com/concepts/token), just as text is. Claude cuts an image into small squares of 28 by 28 pixels and treats each square as a token, so a 1,000 by 1,000 pixel photo costs 1,296 tokens. For Gemini, a minute of audio is 1,920 tokens. 2 things follow. Pictures and recordings fill the [context window](https://tecyfy.com/concepts/context-window) and cost money like text does. And a large image is usually shrunk first, so tiny print in a big screenshot may become unreadable to the model. Crop to what matters. ## Where it goes wrong Anthropic's own list of Claude's limits is a good checklist for any model that reads images. Counts of many small objects are approximate. Positions and layouts are rough. Blurry, rotated or very small images cause mistakes. And it cannot tell you whether a picture was made by AI, so do not ask it to judge. A model can also describe an image confidently and wrongly, the visual version of a [Hallucination](https://tecyfy.com/concepts/hallucination). ## Using it well Give it a clear, well-lit, upright picture. Put the image before your question. Ask for what you can check: "read the numbers in this table" beats "what does this mean?". And for anything important, from a medical image to a contract photo, treat the reading as a first draft. ## Related - [Large Language Model](https://tecyfy.com/concepts/large-language-model): A model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions. - [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. - [Vision‑Language Model](https://tecyfy.com/concepts#visionlanguage-model): A multi‑modal model jointly trained on image and text data to link visual and textual understanding. - [Speech‑to‑Text](https://tecyfy.com/concepts#speechtotext): The task or model that converts spoken language audio into written text. - [Diffusion Model](https://tecyfy.com/concepts#diffusion-model): A generative model that iteratively denoises data starting from random noise - [Context Window](https://tecyfy.com/concepts/context-window): Everything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out. ## Sources - [Image understanding](https://ai.google.dev/gemini-api/docs/image-understanding), Google, Gemini API documentation, 2026-10-14. Gemini models are built to be multimodal from the ground up and can handle image tasks without specialised machine learning training. Supported tasks include image captioning, classification, visual question answering, object detection with bounding boxes and segmentation. - [Audio understanding](https://ai.google.dev/gemini-api/docs/audio), Google, Gemini API documentation, 2026-10-14. Gemini can analyse and understand audio input and generate text responses, including describing, summarising or answering questions about audio, and transcription and translation. Audio counts as 32 tokens per second, so 1 minute is 1,920 tokens. - [Vision](https://platform.claude.com/docs/en/build-with-claude/vision), Anthropic, Claude API documentation, 2026-10-14. Claude can understand and analyse images and accepts JPEG, PNG, GIF and WebP. It is an image understanding model only and cannot generate or edit images. Claude views images in 28 by 28 pixel patches, each a visual token. A 1,000 by 1,000 pixel image costs 1,296 tokens. Limitations include approximate counting and spatial reasoning, mistakes on low-quality, rotated or very small images, and that it cannot tell whether an image is AI-generated. It will not name people in images. It works best when images come before the text. A resized image can make text less legible. It is not designed to interpret complex diagnostic scans, and its readings should be reviewed, especially for high-stakes uses. - [Understand and count tokens](https://ai.google.dev/gemini-api/docs/tokens), Google, Gemini API documentation, 2026-09-25. Pictures and sound are turned into tokens too. For Gemini, an image of up to 384 pixels on each side counts as 258 tokens.