Same prompt, different answer: how sampling works.

Run the exact same prompt twice and you'll get two different answers. Nothing changed — not the model, not the context, not your code. That's not a bug, and it isn't the model "thinking differently today": the last step of every token an LLM produces is a weighted dice roll, and the dials you're given just reshape the dice. Here's what's actually being rolled.

The model never picks a word

Strip away the chat interface and a language model does one thing: given the text so far, it assigns a score to every token in its vocabulary — tens of thousands of them — for how well each would continue the text. A function called softmax turns those scores into probabilities that sum to 100%. After "The capital of France is", the distribution is a spike: "Paris" holds nearly all of the mass. After "My favorite city is", it's a broad hill — dozens of cities with real probability, hundreds more scattered through the tail.

Generation is that step in a loop: score everything, draw one token from the distribution, append it, repeat until done. The draw is where the randomness lives. And it isn't noise bolted on for fun — always taking the single most likely token (greedy decoding) is known to produce flat, repetitive text that circles back on itself. Human writing isn't a chain of most-probable words, so text forced down the most-probable path stops reading like writing. The dice are load-bearing.

What temperature actually does

Temperature is a number that divides the scores just before they become probabilities. Below 1, gaps between candidates get amplified: likely tokens become more likely, the tail collapses. Above 1, the distribution flattens and the tail gets real chances. At 0, it stops being sampling at all — take the top token, every time.

Make it concrete. Suppose at some step the candidates for the next word are "fast" at 60%, "quick" at 30%, "clean" at 10%. Cool the distribution to temperature 0.5 and the odds become roughly 78 / 20 / 2 — "clean" is nearly gone. Heat it to 1.5 and you get roughly 52 / 33 / 16 — the underdog is suddenly one roll in six. That's all temperature is. It doesn't inject creativity and it doesn't make the model smarter or more careful; it decides how much of the tail is in play. The major APIs default to 1.0 — the distribution exactly as the model produced it.

Top-p: amputating the tail

There's a problem with the tail. Even at modest temperatures, thousands of tokens each hold a sliver of probability, and a response is hundreds of rolls in a row — eventually a genuinely weird draw lands. Top-p (nucleus sampling) fixes this: sort the candidates, keep the smallest set whose probabilities add up to p, discard everything else, re-normalize, then roll. At p = 0.9, the plausible head of the distribution stays in play and the long tail is simply cut off. Its blunter cousin top-k keeps a fixed number of candidates instead.

The practical advice, straight from the API docs: tune temperature or top-p, not both. They reshape the same distribution in overlapping ways, and stacking them makes behavior hard to reason about.

Why temperature 0 still surprises you

Set temperature to 0 and you'd expect a lookup table: same input, same output, forever. In practice, hosted APIs don't promise that, and sooner or later every developer notices two identical temperature-0 requests coming back different. The cause sits below the model. Floating-point addition is non-associative — (a + b) + c and a + (b + c) can differ in the last bits — and a GPU sums thousands of numbers in whatever order lets it run fastest. Your request is also batched with strangers' requests, and the batch's size and shape change which kernels run and in what order those sums happen.

None of that would matter, except that sometimes the top two tokens sit within rounding distance of each other — and then the "always take the top token" rule flips on a bit of arithmetic noise. One flipped token is enough. Everything after it is conditioned on it, so a last-bit wobble in step twelve becomes a visibly different paragraph by step two hundred. Some APIs expose a seed parameter for reproducibility, and the documentation calls it best-effort — for exactly this reason.

Using the dice instead of fighting them

Once you see the roll, a few engineering decisions make themselves.

When you want variety, sampling is the feature. A "regenerate" button works because each press is a fresh draw from the same distribution. The first output isn't "the answer" — it's one sample. Products that generate copy, names, or designs should lean into that: re-rolling is the cheapest form of exploration you'll ever ship.

When you want reliability, validate — don't trust the dial. Lowering temperature makes malformed output rarer, not impossible. If your code needs JSON, a length limit, or a value from a fixed set, check it in code and retry on failure. Retries work because of sampling: a second draw rarely repeats the first draw's mistake. That's the whole ladder in our structured-output post.

Don't try to prompt the randomness away. "Always respond with exactly the same answer" can sharpen the distribution, but the roll still happens after the prompt has done its work. A prompt can move the odds; it cannot fix the outcome.

Match the dial to the task. Classification, extraction, and anything you'll parse: temperature at or near 0. Headlines, descriptions, naming, anything a human will pick from: leave headroom, generate several, choose.

Test properties, not strings. An exact-match test on model output is flaky by design. Assert what must be true — it parses, it's under the limit, it mentions the product — and let the wording float.

Where we landed

ShotCanvas generates App Store headlines and metadata with AI, and both halves of this post are load-bearing in it. Generation keeps temperature headroom on purpose — the regenerate button is an honest re-roll, which is exactly what you want when you're hunting for the line that fits your app. But nothing the model says is trusted: subtitle and keyword limits are enforced in code, and an over-limit draw is thrown away and rolled again. The dice write the copy; the code keeps the promises.