{"id":724,"date":"2026-09-24T09:01:41","date_gmt":"2026-09-24T09:01:41","guid":{"rendered":"https:\/\/abrarqasim.com\/blog\/jev-benchmarks-read-carefully-where-444x-cheaper-comes-from\/"},"modified":"2026-09-24T09:01:41","modified_gmt":"2026-09-24T09:01:41","slug":"jev-benchmarks-read-carefully-where-444x-cheaper-comes-from","status":"publish","type":"post","link":"https:\/\/abrarqasim.com\/blog\/jev-benchmarks-read-carefully-where-444x-cheaper-comes-from\/","title":{"rendered":"Jev Benchmarks, Read Carefully: Where 444x Cheaper Comes From"},"content":{"rendered":"

When a launch page says “193.6x faster, 444.6x cheaper”, my first move is to stop reading the page and go find the chart the number came from. Nine times out of ten the chart is fine and the headline is doing something creative with it.<\/p>\n

With Jev, TypeSafe AI made this unusually easy, and I want to say that first. They published their workflow evals as a public site, with every model, every workflow, cost per case and seconds per case. They also wrote a “nuance” section under each claim in the launch post<\/a>, which is more self-criticism than most model launches manage in a year.<\/p>\n

So this isn’t a takedown. Jev is fast and very cheap, and on some tasks it’s as accurate as models that cost a hundred times more. But if you only read the headline, you’ll pick the wrong comparison, and the wrong comparison is how you end up putting it on a task where it loses.<\/p>\n

Where 444x comes from<\/h2>\n

The workflow evals overview<\/a> averages four workflows (security incidents, agent trace review, invoice processing, customer service) and plots each model’s accuracy against cost per case. Jev’s point is 67.8% accuracy at $0.0004 per case and 0.4 seconds.<\/p>\n

Here are the neighbours on that chart, all running the same workflow harness:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
Model (workflow mode)<\/th>\nAccuracy<\/th>\nCost per case<\/th>\nSeconds per case<\/th>\n<\/tr>\n<\/thead>\n
Jev<\/td>\n67.8%<\/td>\n$0.0004<\/td>\n0.4<\/td>\n<\/tr>\n
GPT-5.6 Terra<\/td>\n67.9%<\/td>\n$0.0304<\/td>\n10.1<\/td>\n<\/tr>\n
Luna (OpenAI)<\/td>\n66.8%<\/td>\n$0.0033<\/td>\n12.9<\/td>\n<\/tr>\n
DeepSeek V4 Flash<\/td>\n64.4%<\/td>\n$0.0059<\/td>\n51.9<\/td>\n<\/tr>\n
Sonnet 5<\/td>\n67.8%<\/td>\n$0.1174<\/td>\n78.1<\/td>\n<\/tr>\n
Opus 5<\/td>\n73.1%<\/td>\n$0.1761<\/td>\n37.8<\/td>\n<\/tr>\n
Sol (OpenAI)<\/td>\n74.1%<\/td>\n$0.0836<\/td>\n23.3<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Now do the division yourself. Opus 5’s cost over Jev’s cost is about 440x. Sonnet 5’s time over Jev’s time is about 195x. Those are very close to the headline numbers, which suggests (and I might be wrong about exactly how they averaged it) that the “444x cheaper” figure is roughly the Opus comparison and “193x faster” is roughly the Sonnet one.<\/p>\n

That’s not dishonest. The launch post itself says these are “on the higher end of real world gains”. But look at what else is on that table. Luna gets within one point of Jev’s accuracy at $0.0033. That’s about 8x more expensive than Jev, not 444x. Terra matches Jev’s accuracy almost exactly for about 76x the cost.<\/p>\n

And Opus and Sol are five or six points more accurate than Jev. That isn’t a rounding error. On a workflow that runs a million times a month, six points is sixty thousand decisions.<\/p>\n

So the honest summary is something like this: at Jev’s accuracy level, it’s roughly 8x cheaper and 30x faster than the cheapest LLM that matches it, and about 5 points behind the best models. That’s still a very good result. It’s just a different result from the one on the homepage.<\/p>\n

The invoice workflow is where Jev loses<\/h2>\n

Averages hide the interesting part. Break the chart into its four workflows and Jev’s position moves a lot.<\/p>\n

On invoice processing<\/a>, Jev scores 61.8%. Luna gets 67.8%, DeepSeek V4 Flash gets 69.8%, Terra gets 74.7%, and Sol gets 79.1%. Only Haiku 4.5 is below Jev there. That’s a 17 point gap to the top.<\/p>\n

On customer service, the story flips. Jev scores 76.0% at $0.0001 per case, DeepSeek V4 Flash scores 76.8% at $0.0029, and Sol tops out at 78.3%. Jev is basically at the frontier there.<\/p>\n

Why the difference? The invoice workflow reads the invoice, the purchase order, the contract, prior invoices and delivery records, then asks questions like whether a line was billed above its stated rate, or whether the invoice bills past the contract cap. TypeSafe’s harness already pushes “sums, dates, account numbers and statuses” into code. Even so, a lot of the remaining questions are one step removed from arithmetic.<\/p>\n

That lines up exactly with TypeSafe’s own jaggedness notes for jev-1.13<\/a>, which say the model struggles with numeric precision, date comparison and indirection. The docs and the evals agree with each other. I trust both more because of it.<\/p>\n

My takeaway: if your decisions are about tone, intent, category or policy fit, Jev looks strong. If your decisions are about whether numbers reconcile, keep more of the logic in code than you think you need, or use a bigger model for that branch.<\/p>\n

Accuracy against what, exactly<\/h2>\n

This is the part I’d want every reader to understand before quoting any of these percentages.<\/p>\n

The evals don’t use human-labelled ground truth. The reference answer for each question is the average of GPT-6 Astra and Claude Fable 5.1, both at high thinking. Every other model is scored on how closely it agrees with that pair.<\/p>\n

TypeSafe is upfront about why: they want to measure the model inside a fixed harness, not argue about labels. That’s a reasonable design choice. But it has two effects.<\/p>\n

First, “67.8% accuracy” means “agrees with two frontier models 67.8% of the time”, not “is right 67.8% of the time”. Sometimes both references are wrong, and a model that disagrees with them gets penalised for being right. TypeSafe acknowledges this, and notes that it probably flatters OpenAI and Anthropic models and underrates theirs and DeepSeek’s.<\/p>\n

Second, the workflows were written by people on TypeSafe’s model capabilities team. The post says they weren’t built to favour Jev, and that some bias could exist. I believe both halves of that sentence.<\/p>\n

An independent check helps here. A paper posted this week, Same Scores, Different Decisions<\/a>, compared Jev with nine language models on ContractNLI, a contract-inference dataset. Their summary: Jev had the lowest cost and median response time of anything tested, while hosted language models reached higher baseline accuracy. Same shape as TypeSafe’s own chart. Cheap and fast, a few points behind the top, and worth knowing which few points before you pick.<\/p>\n

I’ve done this kind of benchmark squinting before, when I looked at why API scores and chat app scores drift apart<\/a>. The pattern is the same every time: the number is real, and the setup around it decides what it means.<\/p>\n

The part of the chart everyone skips<\/h2>\n

There’s a second result on the evals site that I think matters more than Jev’s position, and almost nobody is quoting it.<\/p>\n

Every model was run two ways: as a decomposed workflow (many small typed questions, logic in code) and as a single prompt with the same policy written out. Averaged across the four workflows, every model did better in workflow mode. (Not on every single workflow: DeepSeek V4 Flash actually scored higher as a prompt on security incidents.) Haiku 4.5 went from 18.1% as a prompt to 53.6% as a workflow. Opus 5 went from 64.8% to 73.1%. Sonnet 5 went from 60.4% to 67.8%. Workflow mode was also cheaper and faster for nearly all of them, because the model does less reasoning per call.<\/p>\n

In other words, a big chunk of the improvement TypeSafe is showing comes from the architecture (small judgments, code in control) rather than from the model. You can get that gain today with whatever model you already pay for. Jev then makes the workflow shape dramatically cheaper to run, because asking thirty small questions costs about the same as asking one.<\/p>\n

I got this backwards for a while. I thought the pitch was “our model is smarter per dollar”. The more useful pitch is “decompose your decisions, and here’s a model built for exactly that shape”.<\/p>\n

Jevons was the right name<\/h2>\n

The model is named after William Stanley Jevons, the economist who noticed that more efficient steam engines made Britain burn more coal, not less. That’s the Jevons paradox, and TypeSafe’s FAQ is explicit that they expect the same thing to happen with decisions.<\/p>\n

I think they’re right, and I think it cuts both ways for developers. When a judgment costs $0.0001, you’ll start asking questions you’d never have paid a chat model to answer. Score every line of every support transcript. Check every RAG passage before it reaches the answering model. Rerank every search result. That’s the upside.<\/p>\n

The downside is that a 5 point accuracy gap, multiplied across ten times as many decisions, is ten times as many wrong decisions. Cheap judgments need the same care as expensive ones, and the confidence score is where that care goes. Act automatically when confidence is high, and send the low-confidence cases somewhere a human or a bigger model can look.<\/p>\n

I help clients make exactly this kind of cost versus accuracy call, and you can see some of that work on my portfolio<\/a>. The answer is almost never “the cheapest model everywhere” or “the best model everywhere”. It’s a split, decided per branch.<\/p>\n

What I’d actually do with these numbers<\/h2>\n

If you’re evaluating Jev this week, skip the homepage ratios and open the per-workflow charts on the evals site. Find the one closest to your own use case (triage, review, invoices, support) and note three numbers: Jev’s accuracy, the accuracy of the cheapest LLM within two points of it, and the gap to the best model. Then pick the one decision in your system where that gap matters least, usually a routing or categorisation step, and run Jev there first with a confidence threshold and a fallback. Leave anything that reconciles numbers on the model you already trust until you’ve measured it on your own data.<\/p>\n","protected":false},"excerpt":{"rendered":"

Jev’s 444x cheaper claim, checked against TypeSafe’s own evals: which model it’s compared to, where Jev loses on invoices, and what the accuracy numbers measure.<\/p>\n","protected":false},"author":2,"featured_media":723,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"","rank_math_description":"Jev's 444x cheaper claim, checked against TypeSafe's own evals: which model it's compared to, where Jev loses on invoices, and what the accuracy numbers measure.","rank_math_focus_keyword":"jev","rank_math_canonical_url":"","rank_math_robots":"","footnotes":""},"categories":[4],"tags":[10,807,812,813,810],"class_list":["post-724","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-evaluation","tag-jev","tag-llm-benchmarks","tag-llm-cost","tag-typesafe-ai"],"_links":{"self":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/724","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/comments?post=724"}],"version-history":[{"count":0,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/724\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media\/723"}],"wp:attachment":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media?parent=724"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/categories?post=724"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/tags?post=724"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}