Confession: for about six months, roughly 40% of every classification prompt I sent to an LLM was a list of categories. Not instructions. Not the thing being classified. Just an enum, 600-odd strings long, shipped on every single request, so the model would return a label I could actually store in a database column.
It worked. It also felt stupid, and it got more expensive every time someone added a category.
Then I read Doug Turnbull’s Don’t classify. Hallucinate!, which flips the problem in a way I hadn’t considered, and which I’ve now been running in anger for a couple of weeks. Short version: stop telling the model what your categories are. Let it invent one. Then snap the invention to the closest real category using embeddings.
I like it. I also think it has failure modes that nobody’s talking about yet, so I’ll get to those too.
The giant enum I was shipping on every request
The standard approach to zero shot classification with an LLM is structured outputs. You define the legal values as a literal type, hand the schema to the API, and the provider constrains decoding so the model can’t return anything outside it.
from typing import Literal
from pydantic import BaseModel, Field
Category = Literal[
"Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables",
"Furniture / Bedroom Furniture / Dressers & Chests",
"Decor & Pillows / Decorative Pillows & Blankets / Throw Pillows",
# ... 600 more
]
class QueryClassification(BaseModel):
category: Category = Field(description="Best category for this query")
This is correct and I’m not going to tell you it’s wrong. The output is guaranteed legal, which matters when you’re writing to a foreign key.
The problems are practical. Every request carries the full vocabulary, so your input token count scales with your taxonomy rather than your query. There’s an upper bound on how large a schema the provider will accept, which OpenAI documents and which you will eventually hit. And in my experience the model reasons worse when most of its context is a wall of near-identical strings. It starts pattern matching on surface text instead of thinking about the query.
Ask for a category that doesn’t exist
The trick is to remove the vocabulary entirely and ask the model to invent a plausible category, giving it a handful of examples purely to communicate the shape.
prompt = f"""Invent a novel, never-seen-before product category that best fits
this search query. Categories look like:
Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables
Kitchen & Tabletop / Kitchen Organization / Food Storage & Canisters
Baby & Kids / Toddler & Kids Bedroom Furniture / Kids Beds
Query: {query}
"""
For “brown coffee table” a small model happily returns something like Furniture / Living Room / Tables / Coffee. That category does not exist in the taxonomy. It’s wrong.
It’s also close enough to be useful, because the resolution step is a nearest neighbour lookup against embeddings of your real categories:
import numpy as np
from sentence_transformers import SentenceTransformer
encoder = SentenceTransformer("all-MiniLM-L6-v2")
real = load_taxonomy() # ~600 strings
real_vecs = encoder.encode(real, normalize_embeddings=True)
def resolve(guess: str, k: int = 3):
v = encoder.encode([guess], normalize_embeddings=True)[0]
scores = real_vecs @ v # cosine, vectors are normalised
top = np.argsort(-scores)[:k]
return [(real[i], float(scores[i])) for i in top]
resolve("Furniture / Living Room / Tables / Coffee")
# [('Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables', 0.88),
# ('Furniture / Living Room Furniture / Coffee Tables & End Tables / End & Side Tables', 0.79),
# ('Furniture / Office Furniture / Desks', 0.44)]
Six hundred vectors is a few megabytes of RAM and the lookup is a single matrix multiply. On my laptop it’s well under a millisecond. The expensive part, the LLM call, now carries a prompt of maybe 80 tokens instead of 4,000, and because the task got easier you can drop to a cheaper model.
This is HyDE with the labels swapped in
If the shape feels familiar, it should. It’s the same move as HyDE, the 2022 retrieval paper by Gao and colleagues: given a query, have the model write a fake document that answers it, embed the fake document, and use that embedding to retrieve real documents. The generated text is not the answer. It’s a better query vector.
Here the generated category isn’t the label. It’s a better query vector for finding the label.
I find that framing useful because it tells you where the technique’s strength comes from. You’ve changed the model’s job from “select from a list you can barely fit in context” to “write something in this format”, and generative models are much better at the second task. Simon Willison picked up the same post and made the point that it lets you push the work down to smaller models, which is the real cost story.
Where it quietly goes wrong
Here’s the part I’d want to know before putting this in front of customers.
Nearest neighbour always returns something. There is no “none of the above” in a dot product. If a query genuinely doesn’t belong in your taxonomy, you’ll get a confident wrong label with a mediocre score rather than an error. The fix is a similarity threshold below which you return unknown, but picking that threshold requires labelled examples, which is exactly the work you were hoping to skip. I ended up hand-labelling 300 rows to find mine. It was worth it. It wasn’t free.
Near-duplicate leaves are the second problem. “Coffee Tables” and “End & Side Tables” live next to each other in the taxonomy and next to each other in embedding space. MiniLM does not reliably separate them, and neither did the larger encoder I tried. The structured-outputs version at least forced the model to reason about the distinction. The hallucinate-then-resolve version outsources that decision to cosine similarity, which does not know that a coffee table is low and an end table is tall.
The third issue is that hierarchical paths are bad embedding inputs. When you embed Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables, the word “Furniture” appears three times and dominates the vector. Everything under Furniture looks like everything else under Furniture. I got a measurable improvement by embedding only the leaf plus its immediate parent, and a further one by scoring the path segments separately and combining them. Neither is in the original post; both took an afternoon.
Fourth: general purpose encoders don’t know your jargon. If your catalogue distinguishes “sectional” from “modular sofa” and the encoder thinks they’re synonyms, no amount of prompt tuning fixes it. That’s a fine-tuning problem, or a synonym table, and it’s the same wall you eventually hit with retrieval. I went through this in more detail when I wrote about choosing between RAG and fine-tuning.
And finally, multi-label. A ranked list of similarities is not a set. If a query legitimately belongs in three categories you need a cutoff rule, and now you’re tuning two thresholds instead of one.
The option nobody in this thread mentions
Once you’ve hand-labelled those 300 rows to pick a threshold, you own something valuable, and it’s worth stopping to ask whether you still need the LLM at all.
You already have an encoder. You already have labelled examples. Fitting a logistic regression on top of the embeddings takes about six lines and runs in milliseconds on CPU with no API call:
from sklearn.linear_model import LogisticRegression
X = encoder.encode(train_queries, normalize_embeddings=True)
clf = LogisticRegression(max_iter=2000).fit(X, train_labels)
clf.predict(encoder.encode(["brown coffee table"], normalize_embeddings=True))
clf.predict_proba(...).max() # an actual calibrated confidence, for free
On the two datasets where I’ve compared them properly, a linear model over embeddings beat both LLM approaches on accuracy once I had more than about 40 labelled examples per class, at roughly a thousandth of the cost. It also gives you a probability you can threshold honestly, which is the thing the nearest-neighbour approach fakes with a cosine score.
The catch is coverage. A trained classifier can only emit classes it saw during training, so a taxonomy with a long tail of rare categories, or one that changes weekly, is a bad fit. That’s the actual dividing line, and it’s not “LLM versus not LLM”. It’s whether you have labels and whether your label set is stable. Zero shot classification earns its name when you have neither.
I’ve landed on using all three in the same system: the classifier handles the head, the hallucinate-and-resolve path handles queries the classifier is unsure about, and structured outputs stay in place for the handful of decisions where a wrong label is expensive enough to justify the tokens. That’s less elegant than picking a winner. It’s also what actually works.
How I’d test it before trusting it
Don’t take my word or Doug’s. The comparison is cheap to run.
Hand-label 200 to 500 examples that look like your real traffic, including some that don’t fit anywhere. Run both approaches over them and measure top-1 accuracy, cost per thousand classifications, and p95 latency. Then measure one more thing: the share of inputs whose best similarity score falls below your threshold. That number is your canary. When someone adds a category next quarter and forgets to rebuild the index, it moves before your accuracy does.
My results, for a taxonomy of about 600 leaves: accuracy within two points of structured outputs, cost down by roughly 80%, and latency better mostly because the prompt shrank. If your numbers come out eight points worse, the enum was doing real work and you should keep paying for it.
One operational note. Rebuild the embedding index whenever the taxonomy changes, and treat it as a deploy artifact rather than something you compute lazily at runtime. I keep the encode step in the same job that syncs the category table, which means the two can’t drift. Most of my client work now involves this kind of plumbing rather than model choice, and you can see the sort of thing I mean in my portfolio.
Try it on your smallest taxonomy this week
Pick the taxonomy you have with the fewest leaves. Write the two versions: the Literal enum you probably already have, and the hallucinate-plus-resolve pipeline, which is maybe forty lines including the encoder.
Run a hundred labelled rows through both and put the numbers in a table. That’s an afternoon, and at the end of it you’ll know whether the giant enum in your prompt is earning its tokens. Mine wasn’t.
