Short version for the impatient: most of the LLM calls in your app don’t need an LLM. If you want to know why, and how to prove it on your own workload, read on.<\/p>\n
I spent a stupid amount of money last year routing every request in a side project through a frontier model. Classification, extraction, a bit of routing logic, all of it went to the biggest, priciest thing available, because that was the path of least resistance. When I finally looked at the bill and then at what those calls were actually doing, I felt a little sick. A model with a PhD in everything was being asked to decide whether an email was a receipt.<\/p>\n
So this is my case for small language models, or SLMs, and more importantly for treating “small versus large” as a routing decision instead of a religious one.<\/p>\n
There’s no committee-approved cutoff, but in practice people say SLM for models roughly in the 0.5B to 8B parameter range, the kind you can run on a single GPU or even a laptop, and LLM for the frontier hosted models with hundreds of billions of parameters. The interesting question was never “which is smarter.” Obviously the big one is smarter. The question is which one is correct enough for a specific task at a fraction of the cost and latency.<\/p>\n
And the research keeps landing on the same uncomfortable answer: for narrow, well-defined tasks, a small model that’s been pointed at the right problem does shockingly well. A recent paper on fine-tuning small models to generate MiniZinc, a niche constraint language the big models fumble, got small models up to high execution accuracy on a task where off-the-shelf performance was near zero. You can read the Learn2Zinc study<\/a> for the specifics, but the takeaway is that “small and focused” beat “large and general” on the thing that actually mattered.<\/p>\n Here’s the mistake I made, written as code so it stings a little more:<\/p>\n Every request, same model, no matter how trivial. It worked. That’s the seductive part. It worked well enough that I never questioned it until the invoice did the questioning for me. Correctness hid the waste, because when the output is right you don’t go looking for how much you overpaid to get it.<\/p>\n The fix isn’t “switch everything to a small model and hope.” Small models fail in ways that are annoying precisely because they look confident while doing it. The fix is to let the small model handle what it’s good at and escalate the rest.<\/p>\n The version of this I like most is collaborative inference, where a small model does the work and hands off to a large one only when it’s out of its depth. A neat paper on this, PyroDash<\/a>, trains the small model to emit a control signal when it wants help, so the handoff is learned rather than bolted on from outside. Across their math reasoning tests they held accuracy close to the big-model baseline while cutting cost meaningfully, because the expensive model only got pulled in for the genuinely hard tokens.<\/p>\n You don’t need their training setup to get most of the benefit. A confidence-gated cascade captures a lot of it:<\/p>\n On the workload I moved over, something like 70% of requests never touched the frontier model, and the ones that did arrived with a first-draft answer that made the big model’s job easier. Latency dropped for the common case because the small model ran close to the app. The bill dropped for the obvious reason.<\/p>\n The clearest win is on-device and edge work, where you literally can’t call a hosted giant on every keystroke. The Octopus<\/a> work fine-tuned small models for calling software APIs on-device and reported better function-calling accuracy than a much larger general model, while staying fast enough to run locally. Function calling, intent detection, structured extraction, routing: these are narrow, repetitive, and forgiving of a fine-tuned specialist. They’re exactly where paying frontier prices makes the least sense.<\/p>\n The other quiet win is privacy. A model running on your hardware means the data never leaves. For anything touching user records or internal documents, that’s not a nice-to-have, it changes what you’re even allowed to build.<\/p>\n Here’s the catch I want to be honest about: an off-the-shelf small model is often mediocre. The wins in the research above didn’t come from raw small models, they came from small models pointed hard at one task through fine-tuning. That word scares people off because it used to mean a GPU cluster and a research team.<\/p>\n It doesn’t anymore. For a lot of narrow tasks you can fine-tune a 7B model on a few thousand labeled examples in an afternoon, on rented hardware, for less than a nice dinner. The Learn2Zinc results are a good reminder of the shape of this: they collected the errors the models actually made and trained on fixing exactly those, rather than dumping a giant generic dataset at the problem. Small, targeted training data on a small model beats a big model with no context surprisingly often. The unlock isn’t size, it’s specificity.<\/p>\n If you can’t or won’t fine-tune, that’s fine, but then be honest that you’re comparing a generalist small model to a generalist large one, and the large one will usually win. The small-model advantage shows up when you’re willing to specialize.<\/p>\n I’ll argue against my own thesis for a second, because the SLM enthusiasm gets oversold too. Open-ended reasoning, long-context synthesis, anything where the failure mode is “subtly wrong and you won’t notice until it’s shipped,” lean on the frontier model. The cost of a small model being confidently incorrect on a hard task usually dwarfs whatever you saved. If you can’t cheaply verify the output, don’t gamble it on the cheap model.<\/p>\n The honest framing is a cascade, not a coronation. Small model first because it’s cheap and often enough. Big model as the escalation path for the cases that earn it. If you’re deciding where to actually run these things, I compared the local serving options in my vLLM versus Ollama writeup<\/a>, which is the natural next step once you’ve decided a small model belongs in your stack.<\/p>\n Pull your last thousand LLM calls and bucket them by task. I’d bet a real amount of money that a big slice is classification, extraction, or routing, the stuff a fine-tuned 7B model eats for breakfast. Take the single highest-volume, most boring task and try a small model behind a confidence gate, with the frontier model as fallback. Measure accuracy and cost side by side.<\/p>\nThe trap of defaulting to big<\/h2>\n
# what I actually shipped: everything goes to the expensive model\ndef handle(request):\n return frontier_llm.complete(\n model="big-expensive-model",\n prompt=build_prompt(request),\n )\n<\/code><\/pre>\nThe pattern that actually works: let the small model ask for help<\/h2>\n
def handle(request):\n # 1. small model takes the first swing, cheap and local\n result = slm.complete(build_prompt(request))\n\n # 2. only escalate when the small model is unsure\n if result.confidence < THRESHOLD or result.asked_for_help:\n result = frontier_llm.complete(\n model="big-expensive-model",\n prompt=build_prompt(request, hint=result.text),\n )\n return result\n<\/code><\/pre>\nWhere small models genuinely shine<\/h2>\n
Fine-tuning is the part people skip<\/h2>\n
When you should just use the big model<\/h2>\n
What to do this week<\/h2>\n