Have you ever received an AI answer that sounds so certain that you almost want to tattoo it on your arm, only to later discover that it was completely made up? This AI phenomenon is called hallucinations. OpenAI now believes it understands why this happens and the discussion is ongoing on how to make AI admit I don’t know rather than serve pure fantasies.
Table of content
The reward system is a major cause of hallucinations
Researchers at OpenAI argue that the reward system itself is a major culprit. When models are trained, they get higher scores for attempting to provide an answer than for saying they don’t know. It’s a bit like a school test where you always get extra points for guessing, even if the answer is wrong. The result is a model that prefers to make things up rather than leave the question unanswered. This makes the model good at performing well on tests but less good at delivering the truth.
An important question arises here. Should we train AI to dare to acknowledge uncertainty? If the model actually said I don’t know instead of throwing out a false claim, many problems would be solved. At the same time, users might feel disappointed when the machine refuses to give an answer. We humans seem to prefer a bad answer over no answer at all, and that influences how the models are built.
AI is not trained on absolute truths
Language models work by predicting the next word in a sentence based on previous words. This mechanism makes them incredibly skilled at mimicking human language. But if there isn’t enough factual information in the training data, the model simply fills in the gaps with something that seems plausible. It is not aiming for the truth but for the most likely next word.
That is why AI can sound so convincing even when it is completely wrong. A model that states Columbus sailed to Australia in 1492 may come across as confident, but it is only based on the word combination sounding statistically credible. There is therefore no inherent morality or fact-checking, only mathematics.
Why the problem is not an error but a feature
The interesting thing is that researchers argue hallucinations are not mistakes in the classical sense. They are rather a function of how the models are trained. Since evaluations and tests often penalize uncertainty, the models are rewarded when they guess correctly, and this creates a built-in culture of confident but sometimes incorrect answers.
Solving this is not as simple as just feeding in more data. We need to rethink how we evaluate models. Maybe we should reward honesty more than speed. Imagine an AI that gets the highest score when it admits it doesn’t know, instead of always trying to sound smart. This would fundamentally change how models are developed and how we humans use them.
Do we want AI to hallucinate?
Some researchers argue that hallucinations are not just an unfortunate side effect but an unavoidable part of the technology. Since models are based on probabilities, they will always produce something that is sometimes wrong. The more complex the tasks, the greater the risk that the model fills in with guesses.
This leads to a philosophical question. Do we even want an AI that never hallucinates? To achieve that requires an extremely cautious model that might never dare to take risks. But risks are also what sometimes make AI surprise us with creative and useful ideas. Eliminating hallucinations entirely could therefore also mean losing something valuable.
The balance between fact and imagination
There are several methods researchers are experimenting with to reduce hallucinations. One method is connecting the model to external databases and search engines. In this way, it can fact-check its answers in real time before responding. This is often called retrieval augmented generation and is already used in some commercial solutions such as Google, which has one of the world’s largest real-time datasets.
Another method is to teach the models to express uncertainty. Instead of saying Paris is the capital of Mars, the model can respond I am uncertain but I believe you mean Paris is the capital of France. This makes the answers more nuanced and reduces the risk of people taking incorrect answers as absolute truths.
Finally, researchers are experimenting with changing the very reward systems during training. By rewarding correctness over blind confidence, the models can be trained to prioritize truth even if it means sometimes acknowledging a lack of knowledge.
But all these solutions have a downside. If we make AI too strict and factual, we risk stifling its creative side. Perhaps it is the blend of facts and imagination that makes the technology so exciting. The question therefore becomes not only how we reduce hallucinations but also how much imagination we actually want to keep.
