Why your AI keeps making things up.

It invents an npm package that's never existed, cites a paper with the wrong author, or names an Apple API method that isn't in the docs — and states every one of them in the exact same confident tone it uses for things that are actually true. That's not a knowledge gap. OpenAI's own researchers published the mechanism in 2025, and it's not what most people assume.

The common explanation is incomplete

The usual story is "bad training data" or "not enough scale yet" — implying hallucination is a knowledge problem that better data or a bigger model eventually fixes. That's part of the picture, but it doesn't explain why hallucination survives in models trained on enormous, largely accurate datasets, and it doesn't explain the specific shape of the failure: not random noise, but confident, fluent, specific wrongness.

The real cause: guessing is what gets rewarded

In September 2025, OpenAI researchers Adam Tauman Kalai and Ofir Nachum published "Why Language Models Hallucinate", reframing the problem from a data-quality issue into an incentive problem — and the argument holds even for a model trained on perfectly accurate data. Pretraining optimizes next-token prediction: given everything so far, predict the next token. That objective works beautifully for patterns the training data actually contains. It has nothing to work with for arbitrary, low-frequency facts that don't follow a pattern — a specific person's birthday, an obscure citation, the exact version number a package shipped a feature in. There's no statistical signal to learn from, so the model is structurally forced to guess.

That alone would just mean occasional wrong guesses. The part that makes it worse is what happens after pretraining, during the evaluation and fine-tuning that shapes how a model actually behaves.

Your model is graded like a multiple-choice test

Most benchmarks and leaderboards score models on accuracy alone: right answer, one point; anything else, zero. Under that scheme, "I don't know" and a confidently wrong guess get scored identically — both zero. But a guess has a nonzero chance of landing on the correct answer, while abstaining guarantees zero every time. The paper's comparison is exactly this: it's a student facing a multiple-choice exam with no penalty for a wrong answer. The rational strategy is to always guess, never leave a blank. Models trained and selected against accuracy-only benchmarks learn the identical strategy — bluff with confidence, because bluffing scores better than honesty, on average, under that grading rule.

This isn't a one-paper finding, either. A 2026 paper in Nature reached a matching conclusion studying evaluation regimes more broadly: grading large language models on raw accuracy structurally incentivizes hallucination, because it never rewards calibrated uncertainty. Two independent lines of research landing on the same mechanism is a strong signal it's not an artifact of one paper's framing — it's how the current generation of training and evaluation actually works.

What actually helps, if you're shipping AI features

Ground factual claims instead of trusting recall. The failure mode above is specific to answers pulled from the model's parametric memory. Retrieval — handing the model the actual source text and asking it to work from that, rather than asking it to remember — sidesteps the guessing problem for anything you can look up. If your feature states a fact, prefer "here's what the source says" over "here's what the model recalls."

Design your own evals to reward "I don't know." If you're testing prompts or fine-tuning anything, don't grade purely on whether the output matches an expected answer. Include cases with no correct answer available and check whether the model hedges or invents one — that's the exact test the OpenAI paper describes, and it's cheap to add to an existing eval suite.

Keep factual and creative asks in separate calls. A prompt that mixes "write something persuasive" with "and get the specific facts right" pulls the model in two directions at once. Split them: one pass for the facts you can verify or retrieve, a separate pass for the tone and copy built on top of those facts.

Verify at the boundary, always. Never surface raw model output as a fact in a user-facing product without a check — a real citation, a real package name, a real API signature. Treat the model as a fast drafting engine, not a source of truth, for anything where being wrong is expensive.

It won't fix itself next model generation

This is the part worth sitting with: because the cause is structural — how these models are pretrained, and how they're graded afterward — a bigger, newer model doesn't make it go away. It can make hallucination rarer on well-represented facts while leaving the underlying incentive to bluff on the rare, unrepresented ones fully intact. Until evaluation broadly rewards calibrated uncertainty instead of just accuracy, "sounds confident" and "is correct" will keep being two different things you have to check separately — which is exactly why every Gemini call in ShotCanvas surfaces the raw model output (and the raw error, when there is one) instead of quietly presenting it as settled fact.