Short version for the impatient: how you phrase a prompt changes what you pay, and the effect lands harder on the output side than the input side. Numbers below.
I’ve spent about a year doing the boring kind of prompt cost work. Cache the system block. Trim the few-shot examples. Stop pasting an entire PDF in and hoping the model sorts it out. All real wins, all measurable, none of them surprising.
Then a paper turned up on arXiv that measured something I had never thought to measure. Not the content of the prompt. The tone of it. Same 570 questions, seven tones running from sycophantic to threatening, temperature pinned at zero, and a count of output tokens burned under each condition. The spread reached 44.3%.
I assumed I’d misread it. Read it again. Then I went and ran a cheap version on my own workload, because a 44% swing in billable output isn’t a rounding error. It’s a line item.
The number that made me re-run it
The paper is Understanding Tone-Dependent Inference Cost in Large Language Models. The setup is simple enough that I trust it: one fixed MMLU subset of 570 questions, seven prompt tones, temperature 0, then measure accuracy and output token length side by side.
Two results matter if you pay an API bill.
Output-token-length variation was substantially larger than accuracy variation, across every model they tested. Phrasing moved the bill more than it moved the answers. That’s the finding in one sentence, and it’s the part everyone skipped.
The second result is the one getting misread. For ChatGPT 4o and 5-nano, the rude tone came out dominant on the accuracy-versus-token-length frontier. For Gemini 2.5 Flash and Flash Lite, rude and neutral both sat on the Pareto-optimal frontier.
I’ve already seen this turned into “be mean to your model, science says so.” That is not the claim. Dominant on a Pareto frontier means no other tone in the test gave better accuracy at equal or lower token cost for that model. It’s a tradeoff statement. It says nothing about rudeness making a model smarter, and the accuracy differences were the small half of the story anyway.
Output tokens are where the money goes
Most providers charge more per output token than per input token, often several times more. Don’t take my word for the ratio, it moves. Check OpenAI’s current pricing when you’re doing the math, and whatever your own provider publishes.
Here’s the asymmetry that makes tone interesting. Input tokens are the ones you write, so they’re the ones everyone optimizes. You can count them before you send them. Output tokens get decided by the model at generation time, which makes them feel like weather. They aren’t. Prompt phrasing is one of the few levers you have on how much the model decides to say.
Reasoning models sharpen this, since thinking tokens bill as output. A prompt that nudges a model into a longer internal monologue charges you twice: once for the tokens, once for the latency.
What I actually measured
I didn’t reproduce the paper. I ran the small version of it against prompts I already had in production, which is what I’d suggest instead of trusting anyone’s MMLU numbers, the authors’ included.
import statistics
from anthropic import Anthropic
client = Anthropic()
TONES = {
"neutral": "{q}",
"polite": "Could you please help me with this? {q}",
"terse": "Answer only, no preamble. {q}",
"inviting": "Walk me through your thinking on this. {q}",
}
def output_tokens(prompt: str) -> int:
r = client.messages.create(
model="claude-sonnet-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
return r.usage.output_tokens
for name, template in TONES.items():
counts = sorted(output_tokens(template.format(q=q)) for q in REAL_QUESTIONS)
p95 = counts[int(len(counts) * 0.95)]
print(f"{name:9} median={statistics.median(counts):5.0f} p95={p95:5.0f}")
Fifty to a hundred real inputs per condition is enough to see a signal. Use your actual questions, not a benchmark.
My ordering didn’t match the paper’s, which I half expected given a different model and a completely different task distribution. The direction held. Phrasings that invited elaboration cost the most by a wide margin, and the gap between my cheapest and most expensive way of asking for the same thing was large enough to matter once you multiply by request volume.
The formatting habit I had to unlearn
Adjacent finding, same general theme. A benchmark of five models on multi-sensor hazard data tested whether structured tabular prompts beat plain prose. They didn’t. Tabular formatting showed no consistent advantage, and ChatGPT-4o did significantly better under prose (p = 0.001).
That one stung a bit, because I’d been converting prompts into neat markdown tables for two years on the assumption that structure helps the model. For a narrow task, on five models, on one dataset, it didn’t. I’m not throwing away every table on the strength of a single paper. I did stop treating “add more structure” as automatically correct, and started checking whether the table earns its token count.
Track the p95, not the average
This is the part I got wrong for months. I was watching average output tokens per endpoint and feeling fine about it.
Averages hide the tail, and on a high-volume endpoint the tail is your bill. One phrasing might have a median of 180 tokens and a p95 of 900 because it occasionally sends the model off writing an essay. Another sits at a median of 240 with a p95 of 310. The second looks worse on the dashboard and costs less in production.
Log usage.output_tokens on every call, tagged by prompt version. If you aren’t logging it you’re guessing, and the guess is usually optimistic. I went through the input-side version of this problem in my post on prompt caching, where the savings were at least attributable to something.
Where compression research fits, and where it doesn’t
There’s a parallel line of work on shrinking the input instead. ARC-Encoder trains an encoder that compresses context into continuous representations which replace token embeddings in the decoder, at roughly four to eight times fewer representations than text tokens. The reported results are strong.
I want to be honest about who can use it. If you call an API, you can’t. This needs access to the decoder’s embedding layer, so it’s self-hosted or nothing. Most people who read a summary of that paper and get excited are on a hosted API with no path to it.
Rephrasing a prompt costs nothing and works today on every provider. Less interesting as research, more useful as engineering. I keep having to remind myself of that split when I read arXiv on a Sunday.
What I’d change this week
Pick your three highest-volume prompts. For each one, write two alternate phrasings that ask for the same thing, one terse and one with an explicit length ceiling. Run fifty real inputs through each. Compare medians and p95s.
Then add the explicit length instruction to whichever wins. “Answer in at most three sentences” is the cheapest cost lever I know of, and it does more work than tone does.
One thing to skip: don’t get rude with your model to save money. The effect is small next to asking for brevity, and prompts end up in logs, in bug reports, and occasionally in front of a customer. If you’re auditing prompts anyway, the few-shot post is worth a read too, since example count and phrasing tend to be tangled together in practice.
Most of this ends up in client backends rather than blog posts for me, and there’s more about that on my work page. The measurement above takes an afternoon, though, and afterwards you’ll know your own numbers instead of someone else’s.
