Inside the Model
Images and audio become tokens too
Scroll
Images and audio become tokens too
A multimodal model accepts more than text. For images, a vision encoder cuts the picture into small square patches, turns each patch into a vector, and maps those vectors into the same space as text tokens. The language model then reads one sequence in which some tokens came from pixels and some from words.
That has a cost. An image is billed and budgeted in tokens like everything else, and one high-resolution image commonly takes from several hundred to a few thousand, depending on the provider and how the image is resized. Audio is handled one of two ways: transcribed to text first, or encoded directly into audio tokens by a model built for it. The second keeps tone and timing that a transcript loses.
It also widens the attack surface. Text inside an image is still text the model reads, so instructions hidden in a screenshot, a slide or a photo of a sign can act as a prompt injection, and a human reviewer can easily miss them. Models also misread small print and invent details in busy images. Check facts taken from images the way you would check any other model output, and treat images from untrusted sources as untrusted input.
Input