Why does AI sometimes make things up?

The short answer

Why does AI sometimes make things up?

AI makes things up because of how a language model is built: it predicts, word by word, what is likely to come next. Most of the time that gives a good answer. Where the model does not have the answer, it still produces something that looks like one.

The tricky part is the tone: an invented fact sits in the same tidy sentences as a correct one, without any caveat. If you do not know the subject, you cannot see the difference. The technical term is hallucinating; in plain words, it is making things up.

The occasion

What does such an invention look like?

An invention looks like an ordinary answer. It is a date, a source or a fact that sounds precise and does not exist. You cannot tell it by its form, only by checking.

Researchers from OpenAI and Georgia Tech gave a simple example in September 2025. They asked a language model for the birthday of one of them, telling it to answer only if it really knew. The model gave a different date three times, and all three were wrong.

The same thing happens in court, with consequences. According to the law firm Stibbe, the Noord-Nederland district court received such a document in November 2025. The claim probably rested on rulings that did not exist or had nothing to do with the case, most likely supplied by AI and not checked.

A month later the Oost-Brabant district court ruled that a man who had drafted his summons with AI should have checked the result.

The cause

Why does a language model guess instead of saying it does not know?

A language model guesses because of how it is trained and tested: a guess sometimes scores points, ‘I don’t know’ never does. Kalai and his colleagues compare it to a student who would rather guess at a hard exam question than leave it blank.

Most tests that models are scored on award a point for a correct answer and nothing for a blank one. A model that guesses when in doubt then scores higher than a model that honestly says it does not know. So the makers teach their models to bluff, without anyone meaning to.

With a loose fact that follows no pattern, such as someone’s birthday, a guess also cannot be told apart from knowledge. According to the researchers, the guessing will decrease once the tests are scored differently. That is for the makers of the models to do; meanwhile, you can make sure a guess does not go out unseen.

My reading

Where does it go wrong in practice?

It goes wrong where the reader does not know the trade and so cannot weigh the answer. The less you know about a subject, the more convincing a confident but invented answer sounds.

In my book I use marketing as an example. Whoever makes a message with AI themselves thinks the work is done with one question, and the result looks quite nice.

Good marketing rests on craft: audience, position, message, rhythm. On the map of gain in the book, work with few steps and little craft sits bottom left, and I write about it: “If marketing really sat bottom left, his home-made version would have been good. It seldom is.” If you do not know the trade, you cannot see what is missing or what was made up.

He forgets nothing and never leaves, but he never notices himself when he is wrong. That is why the person signs.Paul Hoffman

That is how I look at every digital employee: he is fast and forgets nothing, but he has no judgment of his own. A person with a name signs for what goes out the door, from the first day you use AI.

What helps

How do you make AI make up less?

AI makes up less when you give it something to lean on and allow it to say it does not know. That does not remove the problem, so the checking stays.

• Give it your own documents: have it answer from your price list, your terms or your earlier quotes, not from whatever it picked up somewhere.
• Ask for an honest ‘I don’t know’: put in your instruction that it may say something is not in there, and that you would rather hear that than a guess.
• Have it say where it got something: a reference to the page or the passage turns checking into a matter of minutes.
• Look where it matters: not everything word for word, but where a mistake costs money, customers or your name.

I describe that last point in my book through a senior who reviews the work of digital employees. She does not read everything, but looks in a targeted way: “The assumption at the bottom that the rest rests on, the figure that comes out too neatly, the sentence that sounds too sure.”

At first you check everything. As trust grows, that supervision withdraws: “First everything, then only the doubtful cases, later only the unusual.”

In practice

What does this mean for your business?

For your business it means that a name stands under everything AI makes that goes out the door. Arrange it this week for one piece of work, for example the quotes or the newsletter.

Choose who signs. Write down for that person where this work usually goes wrong: which figure always has to add up, which assumption you always trace, which claim may never go out without a source. Agree that AI may say it does not know.

A person who checks what comes out already belongs to the first level of AI in a business. Read in the article on the five levels of AI what else belongs to that level, and take the short check there.