Confession: I shipped a feature last spring that confidently told a user our refund window was 60 days. It’s 14. The model made up the number, wrote it in a clean, polite sentence, and I didn’t catch it until the user did. That’s the thing about LLM hallucination examples that nobody warns you about. They don’t look like errors. They look like answers.<\/p>\n
I’ve spent a chunk of this year building retrieval-augmented chatbots for small teams, and I’ve collected a small museum of these failures. Some are funny. A few cost me a weekend. What I want to do here is show you actual examples, explain why they happen even when you “did everything right,” and walk through the cheap checks I now run before anything reaches a user.<\/p>\n
Let me start with specifics, because abstract talk about “factual reliability” never helped me debug anything.<\/p>\n
The made-up policy number. The refund example above. The docs said nothing about 60 days. The model filled a gap with a plausible figure because plausible is what it optimizes for.<\/p>\n
The phantom citation. I asked a support bot to answer “from the docs only,” and it produced a tidy answer with a reference to The confident API method. This one stung because I trusted it. A coding assistant told me to call The half-right merge. The worst kind. The model pulls two real facts from context and stitches them into one sentence that’s wrong. “Your plan includes 50GB of storage and unlimited seats” when the plan has 50GB and five seats. Both numbers came from the docs. The combination is fiction.<\/p>\n If you want a longer tour of which models I run locally and how often they do this, I wrote up my setup in my notes on running local LLMs in 2026<\/a>. The short version: every model I’ve tried does this. Bigger ones just do it more fluently.<\/p>\n The intuitive fix is retrieval. Give the model the right documents, tell it to answer only from them, and the made-up answers should stop. I believed this for an embarrassingly long time.<\/p>\n They don’t stop. They drop. Big difference.<\/p>\n A token predictor doesn’t “know” where its knowledge ends. When the retrieved chunks don’t contain the answer, the model doesn’t go quiet. It blends what it retrieved with what it half-remembers from training and gives you a sentence that sounds grounded. The retrieval step lowered the odds. It didn’t install a brake.<\/p>\n There’s research that backs up the uncomfortable part of this. A 2026 paper on Evidence Graph Consistency in retrieval-augmented generation<\/a> evaluated hallucination detectors across six models on thousands of RAG responses and found something I didn’t expect: the structural signal that flags a hallucination in one model family points the opposite way in another. Their conclusion was blunt. Embedding-based graph consistency “cannot serve as a model-independent hallucination detection signal.” Translation for the rest of us: a check that works on your Llama setup might quietly fail when you swap in a different model. There’s no universal smoke detector you can bolt on once and forget.<\/p>\n So retrieval helps, and it’s still worth doing well. It just isn’t the finish line I wanted it to be.<\/p>\n After the refund incident I stopped trusting the model’s prose and started trusting its receipts. The rule I settled on: every factual claim has to point at a retrieved chunk, and I verify the pointer in code, not in the prompt.<\/p>\n The trick is to make the model emit structured output with explicit source IDs, then check that those IDs were actually in the context you sent. Here’s the shape of it:<\/p>\n That last block is doing the real work. If the model cites The half-right merges and misread numbers slip past citation checks. For those I lean on a different idea: ask the model more than once and watch whether it agrees with itself.<\/p>\n A hallucinated fact tends to wobble. Confident, real answers tend to repeat. So I sample a few responses at a non-zero temperature and compare. If the answers diverge on the specific claim, I flag it for a human or fall back to “I’m not sure.”<\/p>\n It’s crude. It costs you n times the tokens, so I only run it on high-stakes paths like billing or anything that quotes a number. But it works often enough to earn its keep. A 2026 paper called BEACON<\/a> formalizes this instinct far better than my five-sample hack: it pulls a 31-dimensional feature vector from multi-pass generation, including paraphrase stability and chain-of-thought consistency, and gets around 0.81 AUROC at flagging hallucinations from black-box model outputs alone. The takeaway I drew from it is that consistency across samples is a genuine signal, not just a folk remedy. You can build something serious on it.<\/p>\n There’s also promising work on structure. One group cut hallucinations on a hard Wikipedia question-answering benchmark by giving the retriever a lightweight graph of the evidence<\/a> instead of a flat pile of chunks, which improved factual precision and recall. I haven’t shipped that in production yet, but it’s next on my list to try.<\/p>\n Here’s the pipeline I run, stripped down. None of it is fancy and that’s the point.<\/p>\n First, retrieve well. Better chunks mean fewer gaps for the model to paper over. Most of my early “hallucinations” were really retrieval misses wearing a costume.<\/p>\n Second, force structured answers with citations, and verify the citation IDs in code. Reject anything that points at sources you never sent.<\/p>\n Third, run self-consistency on the money paths only. Numbers, dates, policy claims. If the samples disagree, I’d rather say “I don’t know” than guess on a user’s behalf.<\/p>\n Fourth, log every “I don’t have that in the docs” response. Those are gold. They tell me exactly which questions my retrieval is failing, which is usually a content gap I can fix by writing one more doc.<\/p>\n And fifth, never let one model grade its own homework. I learned this the hard way and wrote about it separately in why I stopped letting one model fix itself<\/a>. A second model, or a plain code assertion, catches things the first one is structurally blind to.<\/p>\n Pick your highest-stakes endpoint, the one that quotes a number or a policy. Add the citation-verification block above. That’s it. Maybe fifteen lines. You’ll either prove your model is well-behaved on that path, or you’ll catch it inventing a source ID, and both outcomes are worth the afternoon.<\/p>\ndocs\/billing\/proration.md<\/code>. That file doesn’t exist. It invented a filename that matched the shape of our repo.<\/p>\nclient.embeddings.batch_create()<\/code>. Looks real. Reads real. The method has never existed. I lost an hour before I checked the actual SDK.<\/p>\nWhy do LLMs hallucinate even with retrieval?<\/h2>\n
The cheapest check that actually caught things: citation grounding<\/h2>\n
import json\n\ndef answer_with_citations(client, question, chunks):\n # chunks: list of {"id": "doc_3", "text": "..."}\n context = "\\n\\n".join(f"[{c['id']}] {c['text']}" for c in chunks)\n valid_ids = {c["id"] for c in chunks}\n\n prompt = (\n "Answer using ONLY the sources below. "\n "Return JSON: {\\"answer\\": str, \\"citations\\": [source_id, ...]}. "\n "If the sources don't contain the answer, set answer to "\n "\\"I don't have that in the docs.\\" and citations to [].\\n\\n"\n f"{context}\\n\\nQuestion: {question}"\n )\n\n raw = client.complete(prompt) # your model call\n result = json.loads(raw)\n\n # The part that matters: verify the receipts in code.\n cited = set(result["citations"])\n if not cited.issubset(valid_ids):\n bogus = cited - valid_ids\n raise HallucinationError(f"model cited unknown sources: {bogus}")\n\n return result\n<\/code><\/pre>\ndoc_9<\/code> and you only sent doc_1<\/code> through doc_5<\/code>, you know it’s improvising before the user sees a word. This single check would have caught my phantom-citation bug. It does nothing for the made-up refund number, though, because the model could cite a real chunk and still misread it. Which is why I don’t rely on it alone.<\/p>\nDetecting the subtle ones with self-consistency<\/h2>\n
from collections import Counter\n\ndef self_consistency_flag(client, prompt, n=5):\n answers = [client.complete(prompt, temperature=0.7) for _ in range(n)]\n norm = [a.strip().lower() for a in answers]\n top, count = Counter(norm).most_common(1)[0]\n agreement = count \/ n\n return {"answer": answers[0], "agreement": agreement,\n "trustworthy": agreement >= 0.6}\n<\/code><\/pre>\nWhat I actually do now, in order<\/h2>\n
Try one thing this week<\/h2>\n