Blog

LLM hallucinations: why models make mistakes and how to reduce them

Learn why LLM hallucinations happen, see real examples from law and research, and follow a seven-step process to measure and reduce them in production.

By Appen · Expert insight
September 2025 · Updated October 2026

When Stanford researchers tested commercial legal research tools marketed as hallucination-free, the tools still gave incorrect or misgrounded answers between 17% and 33% of the time (Magesh et al., 2024). Large language model (LLM) hallucinations are outputs that sound fluent and confident but are factually wrong, unsupported by their source, or invented outright. This article explains what LLM hallucinations are, why training and evaluation incentives produce them, and a seven-step process for measuring and reducing them in production.

What are LLM hallucinations?

LLM hallucinations are responses in which a large language model states false, fabricated, or unsupported information with apparent confidence. They include invented citations, wrong facts, and claims that contradict a supplied source document. Hallucinations arise from how models are trained and scored, not from rare bugs, so production models produce them at a measurable rate.

Huang et al. (2025) group them into factuality and faithfulness errors:

  • Factual fabrication: The model invents an entity, a statistic, or a citation that doesn't exist.
  • Factual inconsistency: The model misstates a real fact, such as a wrong date, author, or dosage.
  • Instruction inconsistency: The output ignores or departs from the user's instructions.
  • Context inconsistency: The output contradicts the document or retrieved passage it was given.
  • Logical inconsistency: An intermediate reasoning step is wrong, and the final answer inherits the error.

The National Institute of Standards and Technology (NIST) calls this risk "confabulation" in its generative artificial intelligence (AI) profile (NIST, 2024). String matching rarely catches either type of error, so evaluation programs pair automated checks with human in the loop review, where trained reviewers check each claim against its source.

LLM hallucination examples and failure modes

Hallucinations do the most damage where people trust outputs without checking them.

  • Legal filings: Attorneys in Mata v. Avianca cited six nonexistent cases generated by ChatGPT and were fined $5,000 (Merken, 2023). In 2025, counsel for Anthropic apologized for a filing that contained an inaccurate Claude-generated citation (Zeff, 2025).
  • Research lookups: Asked for the title of co-author Adam Kalai's PhD dissertation, several widely used models gave different, confident, incorrect answers (Kalai et al., 2025).
  • Simple counting: Asked how many Ds are in "DEEPSEEK," models answered "2," "3," or as high as "7." The correct answer is 1 (Kalai et al., 2025).
  • Customer policy: A Canadian tribunal held Air Canada liable after its chatbot told a customer he could claim a bereavement fare refund after travel, which the airline's actual policy did not allow (Moffatt v. Air Canada, 2024).

Many of these failures can be found before release through adversarial testing. See our guide to large language model red teaming.

Why do language models hallucinate?

Kalai et al. (2025) argue that hallucinations are a predictable result of how models are trained and graded:

  • Pretraining sets a floor: The pretraining objective can't reliably learn arbitrary facts that rarely appear in training data, so some error remains even when the data is clean.
  • Binary grading rewards guessing: Most evaluations give 1 point for a correct answer and 0 for both a wrong answer and "I don't know," so a guess always has a higher expected score than abstaining.
  • Leaderboards reinforce the incentive: Accuracy-only benchmarks such as GPQA, MMLU-Pro, and the software engineering benchmark SWE-bench let a model that bluffs outrank a better-calibrated one.

Worked example. Two models answer 100 closed-book questions, and both know the answer to 70. On the other 30, Model A guesses (20% of its guesses happen to be right), while Model B says "I don't know." Kalai et al. propose stating a confidence target t in the prompt, where each wrong answer costs t/(1−t) points. At t = 0.75, each error costs 3 points.

Binary vs confidence-target scoring for the same two models

Dimension Model A (guesses) Model B (abstains)
Correct answers 76 70
Wrong answers (error rate) 24 (24%) 0 (0%)
Abstentions 0 30
Score under binary grading 76 70
Score under confidence-target grading (t = 0.75) 4 70

Under binary grading, the bluffing model wins by 6 points. Under confidence-target grading, it loses by 66. Reporting error and abstention rates alongside accuracy makes this trade-off visible. For how this is measured in practice, see AI hallucination benchmarking.

How to reduce LLM hallucinations

Each method below targets a different cause, so production systems layer them. Retrieval-augmented generation (RAG) grounds answers in documents fetched at query time (Lewis et al., 2020). Uncertainty detection flags answers that change across repeated samples (Farquhar et al., 2024; Manakul et al., 2023).

LLM hallucination detection and mitigation methods compared

Dimension RAG Confidence-target scoring Uncertainty detection Expert human review
Cause addressed Missing or outdated knowledge Incentive to guess Confabulation at inference Subtle, domain-specific errors
Scalability High High (evaluation change only) Medium (several samples per prompt) Lower (cost grows with volume)
Auditability Medium (retrieved passages can be logged) High (error and abstention reported separately) Medium High (reviewer reasoning recorded)
Main residual risk Retrieval misses; citing sources that don’t support the claim Too many abstentions reduce usefulness Misses errors the model repeats consistently Reviewers disagree without calibrated guidelines
  1. Define a hallucination taxonomy: Write labeling guidelines that use the five categories above and three severity tiers: critical (could cause legal, safety, or financial harm), major (materially wrong), and minor (imprecise but not misleading). Pilot with at least three annotators. Proceed only when Krippendorff's alpha is at least 0.80, or 0.667 for tentative conclusions (Krippendorff, 2004).
  2. Build a domain golden set: Collect at least 300 prompts with expert-verified reference answers. Include long-tail facts and questions the model should decline.
  3. Score accuracy, error, and abstention separately: Add a confidence target to evaluation prompts. Fail any release where the error rate rises, even if accuracy improves.
  4. Ground answers in retrieved sources: Split each response into atomic claims and check each one against the evidence (Min et al., 2023). Set a claim-support threshold for each domain, for example 95% for regulated content.
  5. Flag high-uncertainty outputs: Run semantic entropy or SelfCheckGPT on sampled responses and send the most uncertain 10% to human review.
  6. Gate releases with expert review: Require zero critical hallucinations on the golden set before deployment, and have domain experts settle disagreements.
  7. Monitor production and retrain: Each week, label 1% to 2% of live traffic using the same guidelines. Feed corrections into reinforcement learning from human feedback (RLHF), or into reinforcement learning with verifiable rewards (RLVR) where answers can be checked automatically.

How Appen supports LLM hallucination evaluation

Your research and engineering teams own the model and the release decisions. Appen provides the human data and quality controls that the seven steps above depend on.

  • Golden sets and test questions: Domain experts review outputs against verified ground truth in medicine, law, science, and finance, and test questions are blended into live tasks. When a contributor falls below the accuracy threshold, the job manager is notified and can remove them and their work from the job.
  • AI Data Platform (ADAP): Claim-level text annotation runs in ADAP with multi-stage review, adjudication steps, and inter-annotator agreement tracked through test questions.
  • Domain experts and native-language reviewers: Appen's network of 1M+ vetted contributors across 170+ countries and 235+ languages includes verified PhDs, MDs, and JDs, so faithfulness checks match your production domain and languages.
  • Off-the-shelf datasets: Pre-collected, pre-licensed datasets add long-tail and multilingual coverage to golden sets when building from scratch would slow evaluation down.

This approach builds on language data work going back to Appen's founding in 1996. It shaped Appen's multilingual LLM translation research, which tested 20+ languages for cultural nuance rather than literal accuracy, and its LLM evaluation rubrics. Talk to an Appen evaluation specialist to scope your hallucination test program.

Frequently asked questions about LLM hallucinations

Can hallucinations in LLMs be eliminated entirely?

Not for base models answering open-ended factual questions. Kalai et al. (2025) show that if 20% of facts of one type, such as birthdays, appear only once in pretraining data, a base model should be expected to hallucinate on at least 20% of them. Post-training can lower the observed rate, and a model that says "I don't know" when unsure avoids these errors, but binary grading scores that answer the same as a wrong one.

Does retrieval-augmented generation stop hallucinations?

No. It reduces them. The legal tools that Magesh et al. (2024) tested were built on RAG and still failed, often by citing a real source that didn't support the claim. Retrieval coverage, how documents are split into chunks, and whether the model actually uses what it retrieves each need their own evaluation.

Do reasoning models hallucinate less?

Not consistently. In OpenAI's system card, o3 hallucinated on 33% of PersonQA questions and o4-mini on 48%, compared with 16% for the earlier o1 model (OpenAI, 2025). Models that make more claims per answer produce more correct and more incorrect statements, so judge reasoning models by error rate, not accuracy alone.

How do regulations treat LLM hallucinations?

The European Union (EU) AI Act requires high-risk AI systems to reach an appropriate level of accuracy and to state their accuracy metrics in their instructions for use (European Union, 2024, Article 15). NIST's generative AI profile lists confabulation among 12 risks and recommends documented evaluation. Teams in regulated sectors should keep auditable records of their hallucination tests.

Related reading: rubric-based LLM evaluation.

References

Tags:
Hallucinations
Model Evaluation
Human Feedback
Talk to our team