A multimodal model accepts more than text. Images, audio, documents and video are encoded into the same representation space as language, so a single model can read a chart, transcribe a meeting, describe a photograph or hold a spoken conversation without handing off to separate services.
How it works
- Encoders per modality. A vision encoder turns image patches into tokens; an audio encoder does the same for spectrogram frames.
- Projection. Those tokens are mapped into the language model's embedding space by a small adapter, so the transformer treats them like words.
- Joint training. Contrastive pre-training (as in CLIP) aligns images with captions; instruction tuning on interleaved data teaches the model to answer questions about what it sees.
- Generation. Output may be text, or the model may call an image or speech model — and native any-to-any models increasingly emit audio and images directly.
What it unlocks
- Document understanding — reading scanned invoices, forms and slides without a separate OCR pipeline.
- Visual question answering — interpreting screenshots, dashboards, diagrams and photographs.
- Speech interfaces — low-latency voice conversation with interruption handling, instead of transcribe-then-reply chains.
- Computer use — an agent that looks at a screenshot and decides where to click.
- Accessibility — automatic alt text, described video and live captioning.
Practical notes
- Images are not free: a high-resolution page can cost more tokens than several paragraphs of text.
- Resolution and cropping drive accuracy — small text in a downscaled screenshot is the usual cause of misreads.
- Ask for structured output (JSON with fields you name) when extracting data, and validate it before use.