Why LLMs Hallucinate

A customer asked an airline's chatbot about bereavement fares. The bot described a refund policy that didn't exist. The customer relied on it, the airline refused to honour it, and a tribunal sided with the customer: the company was responsible for what its own chatbot said. A year earlier, two New York lawyers had filed a brief full of court cases that a chatbot had made up, complete with realistic names and citations.

Nothing was broken in either case. The models did what they always do. They wrote text that fit the shape of the question, and whether it was true was never part of the job.

Hallucination isn't a glitch

A hallucination is an answer that sounds right and isn't: a method that doesn't exist in the library, a citation nobody wrote, a date stated with total calm. The word suggests the model is seeing things. It isn't. It is predicting a likely next token, which is its whole job, in a spot where the likely token happens to be wrong.

If that sentence is new to you, start with LLMs explained simply. Everything below follows from it.

Why the model makes things up

Five things account for most of what you'll see in production.

• Plausible beats true. The model learned to continue text the way people write it. "The capital of Australia is Sydney" is a very natural sentence, because people get it wrong constantly.
• Rare facts are weakly learned. Common knowledge shows up thousands of times in training data. The signature of an obscure library method, or your company's refund policy, might show up a handful of times or never. The model fills the gap with something shaped like an answer.
• There is no built-in "not sure". The model always emits a token. Uncertainty exists in the probabilities, but the sentence that comes out sounds equally confident either way. Many training and evaluation setups also reward a confident guess over an honest "I don't know", so guessing gets reinforced.
• Its knowledge is frozen and its context is limited. It knows nothing after its training cutoff and nothing you didn't put in the prompt. Stuff in too much, or contradictory material, and it can miss or blend what you gave it.
• Your question can lead it. Ask "why does retry_forever() leak connections?" and the model will often accept that the method exists and start explaining.

Watch it happen

This toy sampler uses made-up probabilities, not a real model. What matters is that it hands you a fluent answer whether or not the distribution behind it is sure of itself.

import math, random
                      
                      # Toy next-token distribution after "The capital of Australia is" (illustrative numbers)
                      probs = {"Canberra": 0.55, "Sydney": 0.35, "Melbourne": 0.10}
                      
                      def sample(p, temperature=1.0):
                          weights = [v ** (1 / temperature) for v in p.values()]
                          return random.choices(list(p), weights=weights)[0]
                      
                      entropy = -sum(v * math.log2(v) for v in probs.values())
                      print([sample(probs) for _ in range(5)])   # e.g. ['Canberra', 'Sydney', 'Sydney', ...]
                      print(f"entropy: {entropy:.2f} bits")      # high entropy = the model is torn, yet the text sounds sure
                      

Two things worth noticing. The answer changes from run to run, so wrong answers show up intermittently, which is exactly why they slip through testing. And turning the temperature down only makes the model pick its top guess more often. If the top guess is wrong, you now get the same wrong answer every time.

What actually helps

No single fix removes the problem. Layered together, they make wrong answers rarer, easier to spot and cheaper to deal with.

flowchart LR
                          Q[Question] --> R[Retrieve sources]
                          R --> M[LLM answers with a quote and source id]
                          M --> V{Quote found in source?}
                          V -->|yes| A[Return answer]
                          V -->|no| F[Say 'can't find it' or escalate]
                      

Ground the answer

Put the relevant text in the prompt, from retrieval or just a database lookup, and tell the model to answer only from it. That changes the task from "recall this from memory" to "read this and summarise it", which models are much better at. They can still misread or ignore what you gave them, so this isn't the end of the story.

Make the model show its evidence, then check it yourself

Ask for the answer, the exact supporting quote and a source id in a structured format. Then verify the quote in code:

SOURCES = {
                          "S1": "Refunds are available within 30 days of purchase.",
                          "S2": "Digital goods are non-refundable once downloaded.",
                      }
                      FALLBACK = "I can't find that in our policy documents."
                      
                      def grounded(answer: dict) -> str:
                          """Accept the model's answer only if its quote really appears in the cited source."""
                          source_text = SOURCES.get(answer.get("source"), "")
                          if answer.get("quote") and answer["quote"] in source_text:
                              return answer["text"]
                          return FALLBACK
                      
                      good = {"text": "You can get a refund within 30 days.", "source": "S1",
                              "quote": "Refunds are available within 30 days of purchase."}
                      made_up = {"text": "Refunds are available for 90 days.", "source": "S1",
                                 "quote": "Refunds are available for 90 days."}
                      
                      print(grounded(good))     # You can get a refund within 30 days.
                      print(grounded(made_up))  # I can't find that in our policy documents.
                      

That check is a plain string comparison, not AI, and that's deliberate. The verifier shouldn't share the generator's weakness. It catches invented quotes. It does not catch wrong reasoning built on real quotes, which is why high-stakes answers still need a human.

Narrow what it can say

Use a schema, and enums where the answer comes from a known set. Let tools supply facts: a calculator for arithmetic, a database for records, an API for live data. Check anything the model writes that your system will act on, such as identifiers, SQL or code, before it runs.

Allow "I don't know", and test that it uses it

Say explicitly that "not in the provided documents" is an acceptable answer. Then try questions you know your sources can't answer. A model that has never been shown that exit will usually answer anyway.

The solution architect's view

You are not going to find a model that doesn't hallucinate, so I'd stop treating it as a vendor defect to be fixed in the next release. It is a property of the component, like network latency or eventual consistency, and the architecture has to absorb it. The model choice matters far less than the decisions below.

Start with the cost of being wrong

The right controls depend on what an invented answer costs you. Classify each use case before you design anything.

Draw a trust boundary around the model

Treat model output like input from the open internet: untrusted until something deterministic has checked it. Schema validation, allow-lists, quote checks and business rules all live outside the model, in code you can test. If the model drives tools, it proposes and your code decides, with permissions scoped to the action, not to the model.

Design "I don't know" as a real path

Abstaining and escalating need a UX, routing rules and someone who owns the queue. If the only path you built is "answer", that is what the system will do, including when it shouldn't.

Decide who is accountable

Your customer doesn't care that the model said it. In the airline case, the company owned the answer. Read what your vendor contract actually covers, and don't assume it covers what your product tells users. Name an owner for answer quality the same way you name one for uptime.

Turn it into acceptance criteria

Write hallucination into your non-functional requirements instead of hoping for the best.

• Set a ceiling for invented answers on an evaluation set that includes unanswerable questions, and a floor for correct abstentions.
• Budget latency and cost for the verification step; it isn't free.
• Log the sources used and the verifier's verdict for every answer.
• Watch the abstention rate. A sudden jump usually means retrieval broke or the content has a gap, so it is an early warning, not just a quality metric.
• Make the evaluation set a release gate for any change to the model, prompt or retrieval.

Questions for the design review

1. What does a confidently wrong answer cost us here, and who pays?
2. Where is the trust boundary, and what checks it?
3. What happens when the system doesn't know, and who picks it up?
4. How will we find out the rate has gone up before a customer does?
5. Who is accountable for the answers, and does the contract agree?