LLM Hallucinations in Production: Detection and Mitigation Strategies That Actually Work
Confident and wrong is the model working as designed, not a bug. Here's how we detect and contain hallucination in production LLM systems.
Hallucination Isn't a Bug to Patch: It's a Property to Manage
Large language models generate the statistically plausible continuation of a prompt, not a database lookup against verified facts. Confident, fluent, and wrong is the model working exactly as designed when it lacks grounding. Treating hallucination as an occasional glitch that better prompting will eventually eliminate leads to production systems that fail in exactly the situations where confidence matters most.
Grounding Reduces Hallucination: It Doesn't Eliminate It
Retrieval-Augmented Generation meaningfully reduces hallucination by giving the model source material to draw from instead of relying purely on parametric memory, but it doesn't guarantee the model actually uses that source material correctly. We've seen RAG systems still hallucinate by misattributing a claim to a retrieved document, subtly misquoting a number, or blending two retrieved passages into a plausible-sounding but incorrect synthesis.
Detection Techniques That Work in Production
Citation-checking (verifying that a generated claim's cited source actually contains that claim), consistency sampling (generating the same response multiple times and flagging disagreement as a hallucination signal), and structured output constraints (forcing the model to extract rather than freely generate, wherever the task allows it) are the three detection approaches that have actually caught real hallucinations in our production systems, not just theoretical ones.
Human-in-the-Loop Checkpoints for High-Stakes Output
For any output that informs a financial decision, a legal claim, or a customer-facing factual statement, we route the model's output through a human-review checkpoint before it goes live, not because the model is usually wrong, but because the cost of an undetected hallucination in those categories is asymmetrically high compared to the cost of a brief review delay.
Building Hallucination Monitoring Into the Product, Not Just the Prompt
The systems we consider production-ready log every generation alongside its retrieved sources and confidence signals, track a hallucination-rate metric over time (via spot-checking and user-reported corrections), and alert when that rate drifts upward, treating it as an operational metric to monitor, the same way you'd monitor API error rates or latency.
Four Patterns to Look For, With Short Examples
These examples are made up to show the patterns. Real cases look very similar. An invented source. A customer asks about the refund rules. The assistant quotes clause 7.2 of the policy, but the policy has no clause 7.2. The model filled a gap with something that looks right. A wrong number. A summary of a report says sales grew 14 percent. The report says 4 percent. The sentence reads well, so a quick check does not catch it. A blended answer. The system retrieves an old price list and a new one, then writes one answer that matches neither. Both sources were real. The mix was not. An answer to a question the documents cannot answer. Someone asks for a delivery date and the documents do not contain one. A safe system says it does not know. A weak one makes up a date.
Measure It With a Test Set You Build Yourself
You cannot manage a rate you do not measure. Start with 50 to 200 real questions that your users ask, taken from support tickets, search logs or sales calls. For each question, write the correct answer and mark the exact passage in your documents that supports it. Run the system on this set and have a reviewer mark each answer as supported, partly supported or unsupported. The share of unsupported answers is your hallucination rate. Run the same set after every change to the prompt, the model or the documents, so you can see whether a change helped. This step is often skipped because it is slow, careful work. Someone has to write the correct answers and check them against the sources. That is data labeling, and the quality of the labels decides how much you can trust the number.
Changes That Help, and What Each One Cannot Do
Tell the model to answer only from the text you give it, and to say when it does not know. This reduces invented answers but does not stop misquotes. Ask for a short quote and a source name with every claim. This makes errors easier to find, because a reviewer can check the quote. The model can still quote a passage wrongly. Lower the temperature setting. This makes answers more repeatable, but a repeatable wrong answer is still wrong. Improve retrieval. Many wrong answers start with the wrong passage being found. Better chunking and search settings often help more than changing the prompt. None of these removes the risk. Used together with a test set and a human check for important answers, they bring the rate down to a level you can plan around.
Frequently asked questions
What is an LLM hallucination?
A hallucination is an answer that sounds confident but is not supported by facts or by the documents the system was given. It can be an invented fact, a wrong number, a made up source or a mix of two real sources.
Can hallucinations be removed completely?
No. A language model writes the most likely next words, so it can produce wrong text that reads well. You can reduce hallucinations a lot and catch most of the rest, but you should plan for some to get through.
Does RAG stop hallucinations?
RAG reduces them because the model gets source text to work from, but it does not stop them. The system can retrieve the wrong passage, the model can misquote the right one, or it can blend two passages into a new claim.
How do I measure my hallucination rate?
Build a test set of real questions with correct answers and the passages that support them. Have a reviewer mark each answer from your system as supported or unsupported. The share of unsupported answers is your rate. Repeat after every change.
How many test questions do I need?
Fifty to two hundred real questions is a practical start. Cover your main topics, and include a few questions your documents cannot answer, to see whether the system says it does not know.
Which answers need a human check?
Anything that affects money, legal rights, health or safety, and any factual statement shown to customers. If a wrong answer is hard to undo, review it before it goes out.
Can you help us build the test set?
Yes. Our AI data team can write and label evaluation sets and check model answers against your sources, under NDA. Tell us what you are building and we will suggest a small pilot first.
Need clean evaluation data for your AI system?
Tell us what you are building. We will suggest a small pilot batch first, and we work under NDA.
Need Expert Technology Implementation?
Our team builds websites, custom software and blockchain systems, tests web apps and APIs, and runs hosting, SEO and AI data work. Tell us what you are planning.
Related service: AI Data Annotation
Related Technical Insights
Deep dives into cloud SaaS, AI data, and cybersecurity.