One sentence, thirteen languages, one token budget.

Every price list for every model is quoted per million tokens, and every developer silently converts that into "per word" using English as the exchange rate. Then you localise, and the arithmetic quietly stops working. I put the same sentence through a tokenizer in thirteen languages to see how far off it gets.

A token is not a word, and it isn't a character either

Models don't read text. They read a list of integers, produced by a tokenizer that chops the string into pieces it has seen before. The pieces are learned by frequency: common English fragments like ing, tion and the get their own single ID, so ordinary English prose lands around four to five characters per token.

That ratio is not a property of language in general. It's a property of what the tokenizer's training corpus happened to contain, and those corpora are overwhelmingly English. Everything else pays for the mismatch — one token per syllable, sometimes one token per character, occasionally several tokens for a single character that needed more than one byte of UTF-8.

The measurement

Here is a plain product sentence — the kind that ends up in an App Store description — run through OpenAI's o200k_base encoding, the tokenizer behind the current GPT-4o and GPT-5 generation. Eighteen words, 112 characters, 20 tokens in English.

I re-ran the whole thing on a completely different sentence — a subscription renewal disclosure — to check that one example wasn't carrying the result. The ratios held: German 1.44×, French 1.48×, Japanese 1.40×, Thai 1.84×, Chinese 1.16×. Russian was the only one that moved much (1.50× on the first sentence, 1.20× on the second), which is what you'd expect from a language whose common inflected forms are hit-or-miss in the vocabulary.

Two things in that table are worth sitting with. The first is that character count tells you almost nothing. Japanese says the same thing in 60 characters — barely half of English — and still costs 55% more. The second is that Chinese is essentially free. One character carries a whole morpheme, and the tokenizer has enough Chinese in its vocabulary to map most characters to one token, so density and coverage cancel out.

The tokenizer sets the price, not the model

The part that surprised me most is how much of this is a versioning artefact. Here are the same thirteen strings under cl100k_base — the older encoding from the GPT-3.5 and GPT-4 era — next to the current one.

English didn't move by a single token. Hindi got 69% cheaper. That is the whole story of what a bigger vocabulary buys: under the old encoding, Devanagari was being shredded into near-per-byte fragments — 95 tokens for 106 characters — and the fix had nothing to do with the model's reasoning, its context window, or its price per million. Somebody added more of the world's scripts to the vocabulary.

The practical consequence: token counts are not portable across model families. Gemini, Claude and the Llama family all use different tokenizers with different vocabularies. A budget you calibrated against one provider's encoding can be 30% wrong on another for exactly the same text, and wrong in a direction that depends on which languages you ship. If the number matters, count it with the tokenizer that will actually process the request, or call the provider's token-counting endpoint.

Where this actually bites: truncation, not cost

The money is usually the boring half. A store listing is a few hundred tokens; even at 1.8× and thirty locales you're spending pennies. The failure that hurts is max_output_tokens.

Here's the shape of it. You build a translation call, test it in English and German, watch the output land comfortably inside a 512-token cap, and ship. Thai, Japanese and Hindi go out in the same batch with the same cap — and the response stops mid-sentence. Not an error. Not a 4xx. A finish_reason of length and a truncated string that flows straight into your UI, or worse, straight into a store listing. If you're asking for JSON back, truncation is even nastier: the object never closes, your parser throws, and the stack trace points at your parsing code rather than at the budget you set three files away.

So: size the cap for your worst language, not your test language. English word count × 1.4 is a reasonable floor for European languages; × 2 covers Thai, Japanese and Korean with headroom. And always check the finish reason before you trust the payload — a truncated response is a successful HTTP call.

Four things worth doing

• Batch per locale, not per string. One request that translates twelve headlines into Japanese amortises the system prompt across all twelve. Twelve requests pay for it twelve times, and the prompt is usually the bigger half of a short-string call.
• Cache on the source text, not the rendered result. Key on locale + source string. Editing one headline in a set of twelve should cost one re-translation, not twelve.
• Cap concurrency. Fanning thirty locales out at once is the fastest way to discover your rate limit. Four to six in flight keeps throughput high and 429s rare.
• Don't translate your character limits. A 30-character App Store subtitle is still 30 characters in German, where the same phrase is reliably longer. Ask for a constrained rewrite, not a translation — and verify the length yourself, because models can't count characters in a script they see as tokens.

What we do about it

ShotCanvas translates screenshot headlines into every locale you export to, and all four of those rules are in the implementation: one batched call per locale rather than per string, an in-memory cache keyed on locale plus source text, concurrency capped at four locales at a time, and headline-length validation on the way back rather than trust in the model's character counting. None of it is clever. It's just what falls out of accepting that a token budget calibrated in English is calibrated for one of the languages you ship to.