WHY AI MAKES THINGS UP
Not a glitch. Not a glitch. Explained plainly, with sources named and dated.
Models are trained and then scored in a way that gives full credit for a lucky guess and zero credit for saying "I don't know". Under that scoring, guessing is always the better strategy. So they guess — fluently, confidently, and in the correct format.Kalai, Nachum, Vempala & Zhang, "Why Language Models Hallucinate", OpenAI, September 2025 (arXiv 2509.04664).
The arithmetic that causes it
Almost every benchmark used to rank AI models scores answers as right or wrong, with nothing in between. Consider a model that is 60% confident in an answer:
That is the whole problem. Under binary scoring, an honest abstention scores worse than a coin-flip guess — so a system optimised against those scoreboards learns to answer everything. The researchers' comparison is exact: models behave like students who have worked out that leaving a question blank guarantees zero, while a plausible answer might score.
Four mechanisms, stacked
Prediction has no concept of "unknown"
A model produces the most probable continuation of the text so far. There is no separate step where it checks whether it knows the answer — the machinery that generates a true fact and the machinery that generates a plausible-looking false one are the same machinery.Mechanism explained further in the six layers.
Rare facts have a floor
Facts that appear only once in the training data — a specific person's birthday, an obscure dissertation title — cannot be reliably recalled. The research finds that the hallucination rate for such facts is bounded below by how common those one-off facts are in the corpus, and estimates that a substantial share of biographical facts fall into this category.
This is why models are strong on well-covered subjects and unreliable in exactly the specific, verifiable details people most want. The paper's own example: asked for one author's dissertation title, a widely used chatbot gave three different answers across attempts, none correct. Asked for his birthday, three different dates, all wrong.
Generating is harder than recognising
A model may be able to judge whether a statement is true more reliably than it can produce a true statement unprompted. The paper formalises this: the error rate when generating is bounded at roughly twice the error rate when classifying.Kalai et al., OpenAI, September 2025 — stated as a formal bound in the paper, not an empirical measurement.
The practical consequence is directly useful: asking a model to check a claim is more reliable than asking it to supply one.
Partial familiarity triggers confidence
Interpretability work suggests that when a model half-recognises a name or topic, internal features associated with "this is something I know" activate — and the answer proceeds as though the knowledge is there. Which is why the most confident errors cluster around things the model almost knows, not things it has never encountered.
The same research indicates refusal can be trained as a learned behaviour rather than bolted on as a prompt instruction — meaning this is tractable, not fundamental.
What actually reduces it
- Change the scoring. The paper's central recommendation is borrowed from ordinary exams: penalise confident wrong answers more than abstentions, or give partial credit for well-expressed uncertainty. Unglamorous, and it targets the cause rather than the symptom.
- Ask it to check rather than to supply. Follows directly from mechanism 03, and costs nothing.
- Give it the source material. Retrieval reduces the problem substantially — though studies have found it is not a complete fix, particularly in specialist domains.
- Ask for uncertainty explicitly. "Say if you are not sure" works more often than people expect, because the default behaviour is a scoring artefact rather than an inability.
It suggests a perceptual glitch — something going wrong inside. What is actually happening is a system producing its best guess under rules that reward guessing. Calling it hallucination implies the fix is technical and mysterious. Calling it what it is — optimised bluffing — points at the fix, which is changing what gets rewarded.
This is the clearest example on the site of a problem that looks like a mystery and is actually an incentive. It is genuinely reducible, it is being reduced, and it will not reach zero with current methods. Which leaves the practical position unchanged from every other page here: verify anything checkable and consequential, and treat fluency as telling you nothing whatsoever about accuracy. A model has no way to signal the difference between recall and invention, because internally there is no difference.