I spent an afternoon last week making a prompt worse, and I didn’t notice until the eval numbers came back. My support-ticket classifier kept mislabeling refund requests as billing questions, so I did the obvious thing. I added more examples. Three became eight. Accuracy dropped four points. I stared at the diff for a bit, decided the model just needed a couple more, and added two more out of pure stubbornness. It dropped again.<\/p>\n
That’s when I went looking for whether this was a known problem or just me having a bad day. It’s a known problem. It even has a slightly annoying name: over-prompting. And once I understood what was happening, I stopped treating “add more examples” as a free lunch. Here’s what I’ve learned since, including the part where the research disagrees with the advice most of us have been repeating.<\/p>\n
Few-shot prompting means you show the model a handful of input-output examples before asking it to handle a new case. Zero-shot is when you just ask. One-shot is one example. Few-shot is a few. The idea got popular with the GPT-3 paper<\/a>, which was literally titled “Language Models are Few-Shot Learners,” and the headline finding was that you could steer a model’s behavior with examples in the prompt instead of fine-tuning it.<\/p>\n In practice it looks like this:<\/p>\n The model reads the pattern and fills in the last label. When it works, it feels like magic. You didn’t train anything. You just showed it what you wanted. For a lot of tasks, especially formatting and classification, a few good examples beat a paragraph of instructions.<\/p>\n So the folk wisdom became: if the output is wrong, add more examples. More is more. That’s the part that turned out to be shaky.<\/p>\n Over-prompting is what happens when extra examples stop helping and start hurting. My classifier is a clean case. The first three examples pinned down the format and the boundary between “billing” and “refund.” Examples four through ten mostly added noise. A few of them were edge cases I’d hand-picked because they were tricky, which meant I was teaching the model that the tricky reading was the normal one.<\/p>\n There’s a paper from this year, The Few-shot Dilemma: Over-prompting Large Language Models<\/a>, that puts numbers on this. The authors test the standard assumption that more relevant few-shot examples always help. Their result, in their words, “contradicts the prior empirical conclusion that more relevant few-shot examples universally benefit LLMs.” Piling on domain-specific examples degraded performance for some of the models they tested. Not all. That’s the part that makes this genuinely annoying to reason about.<\/p>\n I wrote about a related failure a while back, where a longer prompt meant to reduce hallucinations made the output worse instead<\/a>. Same shape of problem. You add something that should obviously help, the model gets more to chew on, and quality goes down. Prompts aren’t additive the way we keep pretending they are.<\/p>\n The paper is worth reading properly rather than taking my summary for it, but here’s the gist. They set up a framework with three ways of picking which examples to include: random sampling, semantic embedding similarity, and TF-IDF vectors. Then they ran it across a spread of models, including GPT-4o, GPT-3.5-turbo, DeepSeek-V3, Gemma-3, several LLaMA versions, and Mistral.<\/p>\n Two findings stuck with me.<\/p>\n First, there’s no single magic number of examples. The optimal count is per-model. What helped GPT-4o wasn’t the same as what helped a smaller LLaMA. So any blog post (including this one) that tells you “use five examples” is guessing. You have to measure it on your model and your task.<\/p>\n Second, they got their best results not by cramming in more examples but by combining TF-IDF selection with a stratified sampling approach and tuning the quantity per model. In other words, the win came from picking better examples and stopping at the right point, not from volume. That matches my experience more than any prompting guide I’ve read.<\/p>\n I want to be careful here. This is one paper on a specific set of classification tasks. It’s not a law of nature. But it lines up with a thing I keep seeing in my own work, and one clean study that confirms a suspicion is worth more to me than ten confident threads that don’t cite anything.<\/p>\n I don’t have a rigorous mechanism for you, just a working theory that’s survived contact with a lot of prompts.<\/p>\n Examples cost context budget, and they cost attention. Every example you add is more tokens the model has to weigh against the actual question. If your examples are near-duplicates, the extra ones don’t add information, they just add mass that pulls the model toward whatever surface pattern they share. And if you hand-pick “hard” examples because you’re proud of catching the edge cases, you can accidentally shift the model’s sense of what a typical case looks like. I did exactly that. My tricky refund examples taught the model that ambiguous tickets are usually refunds, which is not true.<\/p>\nClassify each support ticket as: refund, billing, or technical.\n\nTicket: "I was charged twice this month, please reverse one."\nLabel: billing\n\nTicket: "The app crashes every time I open the export screen."\nLabel: technical\n\nTicket: "I want my money back for the annual plan I never used."\nLabel: refund\n\nTicket: "My card got charged after I cancelled. Fix this."\nLabel:\n<\/code><\/pre>\nThe over-prompting trap<\/h2>\n
What the research actually measured<\/h2>\n
Why I think this happens<\/h2>\n