Short version for the impatient: read the page called “Jev 1.13 jaggedness” before you read the quickstart. It’s the most useful thing in the TypeSafe AI docs, and it will save you from writing the three bugs I was about to write.
Some context. Earlier today I published a post about overbuilding a triage bot with a chat model, and in it I sketched what the Jev version might look like. I wrote that sketch from the launch post and Simon Willison’s notes, and I said so at the time. Then I sat down with the actual docs and found my guessed field names were wrong, my idea of what “confidence” means was half wrong, and one of the patterns I’d planned to use is specifically called out as unreliable.
So this is the corrected version. Real request shape, real SDK calls, and the edges TypeSafe itself admits to. I’m still waiting on an API key, so every response body below comes from TypeSafe’s documentation, not from my terminal. I’ll flag the places where that matters.
The request shape in one curl call
Jev has one endpoint. You POST a state (the thing being judged) and a map of questions, and you get back one typed answer per question. The quickstart shows this with a support ticket:
curl -X POST https://api.typesafe.ai/v1/systemone \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"state": "Hi, I have been trying to connect my Stripe account for 3 days and the integration keeps failing. I am losing sales. Please help ASAP.",
"model": "jev-latest",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle this",
"criteria": {
"billing": "Payment or subscription issues",
"technical": "Bugs or integration problems",
"sales": "Pricing or account questions"
}
},
"is_urgent": {
"type": "noul",
"instructions": "The message conveys urgency or time-sensitivity"
}
}
}'
And the documented response for the Choice question:
{
"department": {
"type": "choice",
"choice": "technical",
"confidence": 0.78,
"probabilities": { "technical": 0.85, "sales": 0.0, "billing": 0.15 }
}
}
The thing I got wrong in my sketch: questions are a keyed object, not a list, and the key is yours. The docs are explicit that the key is never sent to the model. So "is_urgent" tells the model nothing. Whatever you want judged has to be in instructions. I had been naming my keys descriptively and assuming that helped. It doesn’t.
The other surprise is how little ceremony there is. No system prompt, no temperature, no max tokens. Pricing is per input token only ($0.042 per million), and output is free because there’s no text being generated.
Choice, Score and Noul, and when I’d pick each
There are three question types. The Python SDK (pip install typesafe-sdk, Python 3.10 or newer) gives each one a class:
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
client = TypeSafeClient() # reads TYPESAFE_API_KEY
response = client.system_one(
state={
"ticket_message": "My flight was cancelled. Can I get a refund?",
"refund_policy": "Cancelled flights are eligible for a full refund.",
},
questions={
"refund_requested": Noul(
instructions="Does `ticket_message` request a refund?",
),
"request_type": Choice(
instructions="What is the main request in `ticket_message`?",
criteria={
"refund": "The customer wants money returned.",
"rebooking": "The customer wants a replacement flight.",
"information": "The customer is asking for information only.",
},
),
"frustration": Score(
instructions="How frustrated does the customer appear in `ticket_message`?",
criteria=[
"Calm and neutral.",
"Concerned but civil.",
"Very angry or using strong language.",
],
),
},
)
print(response.answers["refund_requested"].noul) # probability of yes
print(response.answers["request_type"].choice) # one of your keys
print(response.answers["frustration"].score) # a float along your levels
Notice the backticks around ticket_message in the instructions. That’s the documented way to point a question at one field of a structured state, and it’s a nice touch. You can hand Jev a whole ticket, order and policy object and aim each question at the part it should care about.
My rule of thumb after reading the primitives page:
Noul when the question is a clean yes or no and the probability itself is useful, like “does this message contain an email address”. Choice when there’s a fixed set of unordered options and each one maps to a code path. Score when the answer sits on a spectrum you can describe level by level.
The trap is using Noul for a spectrum. The docs give the example “Is this candidate strong in Python?”, and point out that a Noul of 0.5 means the model thinks yes and no are equally likely. It does not mean “medium skill”. If you want medium skill, write a Score with levels. I would absolutely have made this mistake, because 0.5 looks like “halfway” when you’re skimming logs at 11pm.
Confidence is math on the probabilities
I assumed confidence was a separate signal the model produced. It isn’t. The confidence page says it’s a statistic computed from how peaked the probability distribution is. All the mass on one option gives 1.0, an even spread gives 0.
Their interactive demo uses (n * largest - 1) / (n - 1) as an approximation for n options. I checked it against the quickstart response. Three options, largest probability 0.85: (3 x 0.85 – 1) / 2 = 0.775. The documented response says 0.78. So the approximation holds, at least on their example.
Two practical consequences.
First, Noul answers have no confidence field at all. A Noul is already a single probability, so your threshold goes directly on noul. I had a helper function that read .confidence off every answer. It would have thrown on the first Noul.
Second, because you get the full probabilities dict, you can compute your own measure. The docs say this outright. For a routing decision I care more about the gap between the top two options than about overall flatness, so I’ll probably compute top1 - top2 myself and threshold on that. The confidence page also shows a nice pattern where the threshold depends on the action: a read-only action runs above 0.5, while an irreversible one needs 0.9 plus a confirmation step. That’s the part I’d copy into any real integration.
The jaggedness page is the part to read twice
TypeSafe publishes a page listing where jev-1.13 is known to fail. Vendors rarely do this, and I’ll give them real credit for it. Here are the entries that changed my plans.
Counting. Jev does not count reliably, and the error grows with the size of the thing being counted. The recommended fix is to loop in code, ask one Noul per item, and sum the answers yourself. They show this with a list of words and a “is this a fruit” Noul per index.
Dates. Jev reads dates as text, so “which date is earlier” or “is this inside the settlement window” is unreliable. Their fix is to extract each part (day, month, year) as a Choice over a closed set, with a “not stated” option, then do the date arithmetic in code. This one stung, because half of my planned invoice checks were date comparisons.
Structural invariants. This is the one I found most interesting. The same refund question asked as a Noul returned 0.22, while asked as a yes/no Choice it returned 0.01 for yes. And on a different ticket, “is this a refund” plus “is this something other than a refund” summed to 1.19. So you can’t tune a threshold on a Noul and carry it over to a Choice, and you can’t assume a question and its negation are complementary. Pick one phrasing per decision and stick with it.
Adversarial content. State is treated as data, and text that argues for its own classification can move the answer. That matters a lot if you plan to use Jev as a filter on user input, which is exactly what I plan to do. More on that in a separate post.
Literal reading. Jev answers the words you wrote, not the intent behind them. Their advice is good general advice: when you catch yourself explaining what you really meant, that explanation is the missing half of the instruction.
Pin the model version before you tune anything
The quickstart uses jev-latest. The models page explains that aliases move when a release ships, so your answers can change without a deploy on your side. If you’ve tuned confidence thresholds against a specific version, pin the versioned ID (jev-1.13.0 right now) and upgrade on your own schedule. Every response includes the model field that actually answered, so log it next to each decision.
A few other limits from that page I’d write on a sticky note. Context is 64k tokens per request, with 32k for the state plus the longest single question. Input is text only, so images need to become text or structured fields first. English is the primary training language, and other languages work less well. Rate limits are listed at 250,000 tokens per second and 1,200 requests per minute, with a warning that they’re adjusting while demand is high.
And one thing I liked: there’s no fine-tuning. The same weights serve everyone, and you customise through state, instructions and criteria. That’s less flexible than training your own classifier, but it also means there’s no training pipeline to maintain. For the kind of client projects I usually take on, that trade is usually worth it.
Where this fits next to structured output
If you’ve been getting typed answers out of chat models with JSON schemas, you already know the failure modes. I wrote about a small model trained specifically for structured output a while back, and the lesson there was that valid JSON and a correct answer are different problems. Jev solves the first one by construction, since it can’t return an option you didn’t define. It doesn’t solve the second. A wrong answer with a correct type is still a wrong answer, and the confidence number is the only warning you get.
So I’d treat Jev as a replacement for the classification step, not for the whole pipeline. Anything that needs generated text still goes to a chat model. Anything that needs arithmetic stays in code.
What to do with this before you get a key
The playground at console.typesafe.ai works once you’re off the waitlist, but you can prepare now. Pick one decision your app makes today with a chat model or a pile of regexes. Rewrite it as one to five questions, using the rules above: one judgment per question, the full question text in instructions, a Score wherever you were tempted to use a Noul for “how much”, and no dates or counts left for the model. Then pull 50 real examples from your logs with the answer a human gave. When the key arrives you’ll have a test set on day one, and you can see whether the confidence numbers mean anything on your data instead of on theirs.
