What does an AI model actually see in your screenshot?

A 1320×2868 iPhone screenshot is 3.8 million pixels, and no language model ever sees them. Before the first layer runs, every major API shrinks the image to a fixed budget, cuts it into a grid of squares, and treats each square as a token — and the three big providers draw that grid differently enough that the same file is legible to one and a smudge to another. Here's what actually reaches the model, what it costs, and how to send less without losing the answer.

The picture becomes a grid

A language model can't take pixels. Somewhere before inference, the image is resized to fit a budget and split into square cells, and each cell becomes a token that is priced like a word of input. The three big APIs all do this; they just disagree about the cell.

Gemini's documented rule is the simplest: an image no bigger than 384 pixels on either side costs a flat 258 tokens, and anything larger is tiled into 768×768-pixel tiles at 258 tokens each. OpenAI's GPT-4o-class models work in two steps at high detail: scale the image to fit inside 2048×2048, then scale again until the shortest side is 768, then count 512-pixel tiles at 170 tokens each plus 85 for the image itself. (Low detail skips the tiling and charges 85 tokens regardless of size.) Their newer GPT-5 and o-series models count 32-pixel patches instead, with a per-model multiplier and a high-detail cap of 2,500 patches, shrinking the image when it would exceed that. Claude uses 28-pixel patches — ⌈width/28⌉ × ⌈height/28⌉ visual tokens — and downscales anything whose long edge exceeds 1,568 pixels on the standard tier, or 2,576 on its newer high-resolution models.

None of that is stable enough to memorize. The Claude formula we had in our notes was out of date when we checked it this week, and OpenAI has two incompatible schemes running side by side. An assumption about image cost is the same kind of dependency as a hardcoded model name: check it against the provider's page, and expect it to move.

What one screenshot costs

Run the current rules against that 1320×2868 screenshot. Claude's standard tier scales it to 722×1568, which is 26 by 56 patches: 1,456 tokens. GPT-4o at high detail fits it to 943×2048, then shrinks the short side to 768, giving 768×1668; that's two tiles across and four down, so 85 + 8 × 170 = 1,445 tokens. Gemini doesn't publish its tile placement precisely enough to give an exact count, but a 2,868-pixel-tall image is several 768-pixel tiles at 258 each, so a couple of thousand tokens is the right order.

Call it 1,500 tokens for one screenshot, which is roughly the length of this article. A six-frame store set at full resolution is a 9,000-token prompt before you've written a word of instructions, and every retry, every self-review pass and every "try again" tap resends it. This is where an AI feature's bill actually comes from — not the fifty-word prompt you spent a week tuning.

What survives the shrink

The cost is the smaller problem. The bigger one is that the model reads the resized copy, and resizing has a floor below which text stops being text. iOS body copy is 17 points, which is 51 pixels on a 3× screen. Claude's 1,568-pixel ceiling scales that screenshot by 0.55, so body text lands at about 28 pixels: comfortably readable. An 11-point caption starts at 33 pixels and ends at 18, which is borderline. A one-pixel divider becomes half a pixel, which is to say a faint smear or nothing.

Now do the same arithmetic on your own downscale. If you shrink the screenshot to 512 pixels on the long side before sending — a common choice, and the one we make for most calls — the scale factor is 0.18. Body text is nine pixels tall. It's gone. A 120-pixel marketing headline survives at 21 pixels, colours and layout survive perfectly, icons survive as shapes. OpenAI and Anthropic both say this in their own docs: if the text matters, make it bigger in the image, and don't crop away the context just to do it. The practical version is that resolution is a decision about which question you're asking, not a quality setting.

Send the size the question needs

ShotCanvas sends screenshots to Gemini for several different jobs, and each gets its own size. "Which of these frames is the strongest?" is judged at 768 pixels, because the model is grading a composition, not reading it. "Pick a palette, a font and a layout for this app" gets 512. When the whole set goes along as context, up to six frames, each is cut to 320, which sits under Gemini's 384-pixel flat rate: six screenshots for about 1,550 tokens, less than a long email. The self-review pass that critiques a rendered design looks at a 480-pixel copy, because contrast and cohesion are visible at that size and small type isn't the question.

Two details matter more than the numbers. The downscaled copies are re-encoded as PNG, not JPEG: a UI screenshot is flat colour and hard edges, PNG handles that well, and Anthropic's docs warn specifically that heavy lossy compression makes text harder for the model to read — a second compression pass on an already-compressed image is the worst case. And the copies are memoized against the source bytes, so the design call, the caption call and the review pass reuse one resize instead of decoding a four-megapixel PNG three times on a phone.

When the job really is reading — a button label, a price, an error string — don't send a bigger frame. Crop to the element. A 600×200 crop of a toolbar costs a few hundred tokens at most on any provider and puts the text at native size in front of the model, which a 1,500-token full frame can't do. Keep the whole frame for questions that need the context; cut it down for questions that need the letters.

Ask for what a grid can answer

The same grid explains the documented weak spots. OpenAI and Anthropic both say their models are approximate at precise localisation and at counting many small objects, and all three providers tell you to send images the right way up. Claude's docs add that it receives no image metadata at all, so an EXIF rotation flag is invisible; bake the rotation into the pixels before sending. Then design the ask around the limits. When ShotCanvas needs the most impressive region of a screenshot, it asks for the centre as fractions of the frame between 0 and 1 and a width between 0.1 and 0.6 of the image, never a pixel box. The answer is coarse by construction, which is fine, because the template only needs "the chart, upper middle", and a coarse answer can't be wrong by a decimal place.

Even prompt order is provider-specific. Gemini's docs say to put the text before the image for single-image requests; Claude's say images work best before the text. Neither is wrong; they describe different models. It's one more reason to keep image handling in a single place per provider rather than scattered across features.

Where this shows up in ShotCanvas

Every AI call in the editor — palette, layout, headline, focal region, set review — runs through one resize path with a per-job size, so a regenerate tap is measured in hundreds of image tokens, not thousands. You don't have to think about any of this to use it. But if you're wiring a vision model into your own app, the numbers above are the ones to start from.