Short version for the impatient: the line I bolted onto my RAG prompt to stop hallucinations, “If the answer isn’t in the context, say you don’t know,” was quietly costing me correct answers. If you want to know why, read on.<\/p>\n
I found this out the annoying way. I had a support bot pulling from about 40 pages of product docs stuffed into a single long context window. It worked fine on my test questions. Then a teammate asked it something whose answer was plainly sitting in paragraph three of the docs, and the bot said “I couldn’t find that in the provided material.” The fact was right there. I’d read it myself thirty seconds earlier. I spent the next hour convinced my retrieval was broken. It wasn’t. My prompt was.<\/p>\n
A paper that landed on arXiv this month put a name to what I’d been fighting, so I stopped guessing and started measuring. Here’s what I learned, and what I actually changed.<\/p>\n
The classic way people check whether a model can use a big context is the needle-in-a-haystack test: bury one specific fact somewhere in a huge blob of text, then ask about it. If the model finds it, the context window “works.” Vendors love quoting near-perfect scores on this.<\/p>\n
The problem is that a single clean needle in an otherwise irrelevant haystack is not what your production data looks like. Real corpora have facts scattered across many documents, near-duplicates, half-relevant passages, and contradictions. The recent paper “Not All Needles Are Found<\/a>” builds an extended version of the test that varies where facts sit and how they’re distributed, then runs it across Gemini 2.5 Flash, ChatGPT-5-mini, Claude 4.5 Haiku, and DeepSeek. Their headline finding is that reliability swings a lot depending on those conditions, even when the fact is definitely present.<\/p>\n They separate three things most benchmarks smash together: pulling a fact out verbatim, reasoning over it, and the risk of making something up. Once you split those apart, the failure modes get much easier to see. And two of them are worth your attention.<\/p>\n Here’s the one that got me. The authors show that anti-hallucination prompts, the “don’t make it up, only answer from the provided text” instructions we all paste in, can push a model into refusing to answer questions whose answers are actually present. They call it a Safety Tax. You told the model to be careful, so it got careful about everything, including the stuff it should have confidently returned.<\/p>\n That matched my bug exactly. My prompt looked like this:<\/p>\n Read that as a nervous intern would. “Explicitly stated” is doing a lot of work. If the answer requires stitching two sentences together, or the phrasing in the docs doesn’t match the phrasing in the question, a cautious model decides it wasn’t “explicit” enough and bails. I’d optimized for never being wrong and accidentally optimized for often being useless.<\/p>\n The fix isn’t to throw caution out. It’s to stop conflating “I’m not certain this is stated word-for-word” with “this isn’t here.” Those are different claims, and the model will happily treat them as the same one if you let it.<\/p>\n The second failure mode they document is what happens when the evidence you need is spread out instead of sitting in one tidy chunk. Performance drops when facts are dispersed across the context, even though nothing was removed. The model has all the pieces; it just struggles to pull them together and prioritize the relevant ones.<\/p>\n If that sounds familiar, it should. Back in 2023 the “Lost in the Middle<\/a>” work showed models use information at the start and end of a long context far better than information buried in the middle. The new results say the position problem never fully went away, it just got more subtle. Bigger windows didn’t make it disappear. They mostly made us stop checking.<\/p>\n This is the part that changed how I think about retrieval. I used to treat a giant context window as a reason to be lazy: dump everything in, let the model sort it out. What the data says is that dumping everything in actively works against you when the answer depends on scattered evidence. Fewer, tighter, better-ordered chunks beat a wall of maybe-relevant text.<\/p>\n Three concrete changes, in the order they helped most.<\/p>\n First, I stopped forcing a binary. Instead of “answer or refuse,” I gave the model room to answer with a confidence signal. Something closer to:<\/p>\n That last line matters more than it looks. It flips the default. The refusal is now the exceptional case, not the safe default the model reaches for under pressure.<\/p>\n Second, I stopped stuffing the window. My retrieval used to return the top 20 chunks because I had the tokens to spare. I cut it to the top 6 and reranked them so the strongest evidence sits at the top, where the model actually reads carefully. My accuracy went up and my token bill went down, which is a rare combination. If you’re doing retrieval at all, a reranking pass is the highest-leverage thing you can add, and I go deeper on the storage side of this in my pgvector write-up<\/a>.<\/p>\n Third, I started measuring refusals as their own metric. Not just “was the answer right,” but “did the model refuse when it shouldn’t have.” That number was invisible to me before because a wrong refusal looks polite. It doesn’t throw an error. It just quietly fails to help someone. If you only track accuracy on answered questions, the Safety Tax hides from you completely. I learned that lesson the hard way while building out evals, which I wrote about in what I trust after LLM judges failed me<\/a>.<\/p>\n I don’t want to oversell the fix. There are cases where you genuinely want the model to refuse rather than hedge. Anything touching medical dosing, legal citations, financial figures, or account actions should fail closed, not open. A hedged guess about someone’s medication is worse than a refusal every time.<\/p>\n So the real answer is that the “don’t make it up” instruction isn’t wrong, it’s just unconditional in most people’s prompts. You want it loud in the high-stakes paths and quiet in the low-stakes ones. Treating every question as if a wrong answer could get someone hurt is how you end up with a bot that refuses to tell you its own opening hours. The vendors’ own guidance on context windows<\/a> is decent on the mechanics but light on this tradeoff, so you mostly have to tune it yourself.<\/p>\nThe “Safety Tax” nobody warns you about<\/h2>\n
Answer ONLY using the context below. If the answer is not\nexplicitly stated in the context, respond with:\n"I couldn't find that in the provided material."\n\nDo not use outside knowledge. Do not guess. Do not make\nanything up.\n<\/code><\/pre>\nDistributional collapse, or why dispersed evidence hurts<\/h2>\n
What I changed in my prompts and my retrieval<\/h2>\n
Answer the question using the context below.\n\n- If the context directly supports an answer, give it.\n- If the context partially supports an answer, give your\n best answer and note which part is uncertain.\n- Only say "not covered" if nothing in the context is\n relevant at all.\n\nPrefer a useful, hedged answer over a refusal.\n<\/code><\/pre>\nWhen a strict prompt is still the right call<\/h2>\n