A multimodal model is an AI model that accepts or produces more than one type of data, such as text, images, audio, video or documents, within a single model or API. Most current large language models from major providers accept images as well as text, and a growing set of models also take audio and video or generate speech, images and video. This page explains what "multimodal" means in practice, how input and output modalities differ, how providers count images, audio and video as tokens, and what each provider documents. It is for developers, product teams and buyers comparing what models can actually take in and give back.
Last checked: 2026-11-15. Providers are listed alphabetically; order is not a ranking.
This is an independent reference page from NHI Mgmt Group based on public provider documentation; it is not sponsored or endorsed by any provider.
Key takeaways
- Input and output are separate questions. Anthropic and OpenAI document their latest general models as text and image in, text out; generating images, speech or video is usually done by separate, specialised models.
- "Omni" models handle the widest mix. Alibaba lists Qwen3.8-Omni-Flash for audio, video and visual understanding, and Google lists text, image, audio, video and PDF for Gemini 4 Argon.
- Media is billed as tokens. Anthropic counts 28x28 pixel patches, Google charges 258 tokens per small image or 768x768 tile and about 100 tokens per second of default-resolution video, and OpenAI uses patch or tile sizing that varies by model.
- Limits differ widely. Documented maximums range from 600 images per request (Anthropic API, most models) to 1,500 (OpenAI) and 3,600 (Google).
- Stated limitations matter. Providers document weaknesses such as small or rotated images, approximate counting and spatial outputs, and no reliable detection of AI-generated images.
What does multimodal mean?
A modality is a kind of data: text, image, audio, video, or structured documents such as PDFs. A model is multimodal when it can work with more than one of them. Providers describe the direction separately:
- Multimodal input (understanding): the model reads an image, recording, video or document and answers in text. Examples: describing a chart, transcribing speech, summarising a video.
- Multimodal output (generation): the model produces an image, speech, music or video. Examples: image generation and editing, text-to-speech, video generation.
- Any-to-any or "omni": one model takes several input types and can answer in more than one, such as speech-to-speech conversation.
- Native vs pipeline: some products chain separate models (for example speech-to-text, then a text model, then text-to-speech). Open-weight cards sometimes describe the architecture: the Qwen3.8-27B card calls it a "Causal Language Model with Vision Encoder".
| Task type | Input | Output | Example model IDs from official docs |
|---|---|---|---|
| Vision (image understanding) | Text, image | Text | claude-opus-5-5, command-a-vision-07-2025, gpt-5.5, mistral-large-4 |
| Video understanding | Text, video (frames and audio) | Text | gemini-4-argon, Qwen/Qwen3.8-27B |
| Speech to text | Audio | Text | cohere-transcribe-03-2026, gemini-3.5-transcribe, gpt-transcribe, Voxtral Transcribe 3 |
| Text to speech | Text | Audio | gemini-3.8-flash-tts, Voxtral TTS 3 |
| Realtime speech | Audio, text | Audio, text | gemini-3.8-live, gpt-realtime-2.1, qwen3.8-omni-flash-realtime, Grok Voice API |
| Image generation and editing | Text, image | Image | gemini-3.1-flash-image, gpt-image-2.6-sunburst, grok-imagine-image-2.0 |
| Video generation | Text, image | Video | gemini-omni-1.1-flash, grok-imagine-video-1.5 |
| Document OCR | Document | Text | Mistral OCR 4.1 |
| Multimodal embeddings | Text, image, mixed (PDF) | Vector | embed-v5.0-pro |
As of 2026-11-15. Input columns describe the task type; check each model page for the exact modalities it accepts.
Which multimodal models do the main providers offer?
Identical fields for every provider. Where the documentation fetched does not state a point, the cell says so.
| Provider | General model input (as documented) | General model output | Audio and speech models | Image or video generation | Media token method |
|---|---|---|---|---|---|
| Alibaba Cloud (Qwen) | qwen3.8-omni-flash: listed for audio, video and visual understanding; Qwen3.8-27B (open weights): text, image, video | Text; realtime Omni models listed for speech-to-speech | qwen3.8-omni-flash-realtime, qwen3.5-omni-plus | Image: qwen-image-3.0-pro, wan2.7-image-pro; video: wan3.0-video, HappyHorse 1.1 models | Not stated by Alibaba Cloud on the models page fetched |
| Anthropic | Text and image (all current models) | Text | Not stated by Anthropic in the models overview | Not stated by Anthropic in the models overview | 28x28 pixel patches |
| Cohere | command-a-plus-05-2026 and command-a-vision-07-2025: text and images | Text | cohere-transcribe-03-2026, cohere-transcribe-arabic-07-2026 (audio in) | Not stated by Cohere on the models page | Not stated by Cohere on the models page |
| DeepSeek | deepseek-flash (V4.1 Flash): native multimodal support; image files accepted as file input (13 November 2026) | Text | Not stated by DeepSeek in the change log | Not stated by DeepSeek in the change log | Not stated by DeepSeek in the change log |
| Google (Gemini) | gemini-4-argon: text, image, audio, video, PDF | Text; Live models for voice | gemini-3.8-live, gemini-3.8-flash-tts, gemini-3.5-transcribe | Nano Banana image models; gemini-omni-1.1-flash video generation and editing | 258 tokens per image or 768x768 tile; 32 tokens per second of audio |
| Mistral AI | mistral-large-4: text and image (open weights); Ministral 3: text and vision | Text (with reasoning) | Voxtral Transcribe 3 family, Voxtral TTS 3 | Not stated by Mistral AI on the models overview | Not stated by Mistral AI on the pages fetched |
| OpenAI | Text and image (all latest models); gpt-5.5: audio and video not supported | Text | gpt-realtime-2.1, gpt-live-1, gpt-transcribe | gpt-image-2.6-sunburst and other GPT Image models | Patch or tile sizing, varies by model |
| xAI | grok-4.7: modalities not stated by xAI on the models page | Text | Grok Voice API: speech-to-speech, text-to-speech, speech-to-text | grok-imagine-image-2.0, grok-imagine-video-1.5 | Not stated by xAI on the models page |
As of 2026-11-15. Sources: provider model pages listed under Sources.
How are images, audio and video counted as tokens?
Media inputs consume context window space and are billed at the model's input token rate. The three providers that publish a method in the pages fetched describe it as follows.
| Provider | Images | Audio and video | Worked example from the docs |
|---|---|---|---|
| Anthropic | ⌈width / 28⌉ × ⌈height / 28⌉ visual tokens; images downscaled to 1,568 px long edge (1,568 tokens max), or 2,576 px (4,784 tokens max) on Claude 4.7 and later | Not stated by Anthropic in the vision guide | 1000x1000 px image: 1,296 tokens on both tiers |
| 258 tokens if both sides are 384 px or less; otherwise 768x768 tiles at 258 tokens each | Audio 32 tokens per second; video frames sampled at 1 FPS, 66 tokens per frame at low resolution or 258 otherwise; about 100 tokens per second at default resolution, about 300 at high | 960x540 image: 6 tiles | |
| OpenAI | Detail levels low, high, original, auto; patch-based or tile-based sizing depending on model; up to 30,000 patches per image after resizing | gpt-5.5: audio and video input not supported | gpt-5.5 high: up to 2,500 patches, 2,048 px maximum dimension |
As of 2026-11-15. Google notes these figures apply to static processing; agentic processing uses different accounting. Cost per token is covered in Model Pricing Explained, and context limits in Context Windows Explained.
What are the documented limits on multimodal input?
| Provider | Image formats | Max images per request | Size limits |
|---|---|---|---|
| Anthropic | JPEG, PNG, GIF (first frame), WebP | 600 (API); 100 for 200K-context models; 20 per message on claude.ai | 8000x8000 px; 10 MB per image on the Claude API (5 MB on Amazon Bedrock); 128 MB request; 3,000 px cap when over 20 images |
| PNG, JPEG, WebP, HEIC, HEIF | 3,600 image files | 20 MB total inline request; File API for larger; video up to 3 hours at low resolution in a 1M-token context | |
| OpenAI | PNG, JPEG, WebP, non-animated GIF | 1,500 | 512 MB total payload per request; images over 30,000 patches rejected |
As of 2026-11-15. Alibaba Cloud, Cohere, DeepSeek, Mistral AI and xAI limits were not stated on the pages fetched for this page; check each provider's own vision or file input documentation.
What can multimodal models not do reliably?
Anthropic's vision guide lists limitations that apply in some form to vision models generally, and are useful as a test checklist for any provider:
- Image quality: mistakes are more likely with low-quality, rotated or very small images under 200 pixels.
- Spatial outputs: coordinate and localisation outputs are approximate.
- Counting: counts of many small objects may not be precise.
- Synthetic media: the model cannot determine whether an image is AI-generated.
- People: Anthropic states that Claude does not name people in images.
- Medical imaging: not designed to interpret diagnostic scans such as CTs or MRIs.
Other providers' limitation statements were not read for this page; test each model against your own data.
What should security and data teams check?
- Hidden instructions in media: images, audio and documents can carry text a model will follow. See the Multimodal Prompt Injection Guide and AML.T0129 Triggers in Multimodal Inputs.
- Sensitive data in uploads: screenshots, scans and recordings often contain personal data or secrets that text filters miss. See the LLM Firewalls vs DLP for GenAI.
- Retention of uploaded files: file APIs store media for reuse; confirm each provider's retention terms and delete files after use.
- Output handling: generated images, speech and video need the same review as text output, including provenance labelling where required.
How to choose a multimodal model: factual criteria
- List the modalities you need in each direction (for example image in, text out; audio in, audio out).
- Check the model page, not the family: modalities differ by model, as
gpt-5.5(no audio input) andgpt-realtime-2.1show. - Estimate media tokens with the provider's published method and your typical image size or video length.
- Compare documented limits on image count, file size and video length against your workload.
- Check deployment needs: open-weight options with image input include Mistral Large 4 and Qwen3.8-27B (Apache-2.0). See Running LLMs Locally Explained.
- Test on your own data, including low-quality and rotated images, before production.
Practitioner checklist
- Record input and output modalities for every model ID in use.
- Resize images before upload to control token cost and stay within dimension caps.
- Use file APIs for large or repeated media, and delete stored files when no longer needed.
- Scan uploaded media for personal data and secrets before sending it to a provider.
- Treat text found inside images, audio and documents as untrusted input.
- Benchmark vision accuracy on representative, imperfect samples.
- Recheck model pages quarterly: modalities and limits change between releases.
Standards and references
- Alibaba Cloud: Model Studio models and Qwen: Qwen3.8-27B model card (accessed 2026-11-15)
- Anthropic: Vision and Models overview (accessed 2026-11-15)
- Cohere: Models overview (accessed 2026-11-15)
- DeepSeek: API change log (accessed 2026-11-15)
- Google: Gemini models, Image understanding and Video understanding (accessed 2026-11-15)
- Mistral AI: Models overview and Mistral Large 4 model card (accessed 2026-11-15)
- OpenAI: Images and vision, Models and GPT-5.5 (accessed 2026-11-15)
- xAI: Models (accessed 2026-11-15)
Related NHI Mgmt Group resources: AI Platforms Guide · Multimodal Prompt Injection Guide · LLM Naming and Versioning Explained · Model Cards and System Cards Explained