Short version for the impatient: read the page called “Jev 1.13 jaggedness” before you read the quickstart. It’s the most useful thing in the TypeSafe AI docs, and it will save you from writing the three bugs I was about to write.<\/p>\n
Some context. Earlier today I published a post about overbuilding a triage bot with a chat model<\/a>, and in it I sketched what the Jev version might look like. I wrote that sketch from the launch post and Simon Willison’s notes, and I said so at the time. Then I sat down with the actual docs and found my guessed field names were wrong, my idea of what “confidence” means was half wrong, and one of the patterns I’d planned to use is specifically called out as unreliable.<\/p>\n So this is the corrected version. Real request shape, real SDK calls, and the edges TypeSafe itself admits to. I’m still waiting on an API key, so every response body below comes from TypeSafe’s documentation, not from my terminal. I’ll flag the places where that matters.<\/p>\n Jev has one endpoint. You POST a And the documented response for the Choice question:<\/p>\n The thing I got wrong in my sketch: questions are a keyed object, not a list, and the key is yours. The docs are explicit that the key is never sent to the model. So The other surprise is how little ceremony there is. No system prompt, no temperature, no max tokens. Pricing is per input token only ($0.042 per million), and output is free because there’s no text being generated.<\/p>\n There are three question types. The Python SDK ( Notice the backticks around My rule of thumb after reading the primitives page<\/a>:<\/p>\n Noul when the question is a clean yes or no and the probability itself is useful, like “does this message contain an email address”. Choice when there’s a fixed set of unordered options and each one maps to a code path. Score when the answer sits on a spectrum you can describe level by level.<\/p>\n The trap is using Noul for a spectrum. The docs give the example “Is this candidate strong in Python?”, and point out that a Noul of 0.5 means the model thinks yes and no are equally likely. It does not mean “medium skill”. If you want medium skill, write a Score with levels. I would absolutely have made this mistake, because 0.5 looks like “halfway” when you’re skimming logs at 11pm.<\/p>\n I assumed Their interactive demo uses Two practical consequences.<\/p>\n First, Noul answers have no Second, because you get the full TypeSafe publishes a page listing where jev-1.13 is known to fail<\/a>. Vendors rarely do this, and I’ll give them real credit for it. Here are the entries that changed my plans.<\/p>\n Counting. Jev does not count reliably, and the error grows with the size of the thing being counted. The recommended fix is to loop in code, ask one Noul per item, and sum the answers yourself. They show this with a list of words and a “is this a fruit” Noul per index.<\/p>\n Dates. Jev reads dates as text, so “which date is earlier” or “is this inside the settlement window” is unreliable. Their fix is to extract each part (day, month, year) as a Choice over a closed set, with a “not stated” option, then do the date arithmetic in code. This one stung, because half of my planned invoice checks were date comparisons.<\/p>\n Structural invariants. This is the one I found most interesting. The same refund question asked as a Noul returned 0.22, while asked as a yes\/no Choice it returned 0.01 for yes. And on a different ticket, “is this a refund” plus “is this something other than a refund” summed to 1.19. So you can’t tune a threshold on a Noul and carry it over to a Choice, and you can’t assume a question and its negation are complementary. Pick one phrasing per decision and stick with it.<\/p>\n Adversarial content. State is treated as data, and text that argues for its own classification can move the answer. That matters a lot if you plan to use Jev as a filter on user input, which is exactly what I plan to do. More on that in a separate post.<\/p>\n Literal reading. Jev answers the words you wrote, not the intent behind them. Their advice is good general advice: when you catch yourself explaining what you really meant, that explanation is the missing half of the instruction.<\/p>\n The quickstart uses A few other limits from that page I’d write on a sticky note. Context is 64k tokens per request, with 32k for the state plus the longest single question. Input is text only, so images need to become text or structured fields first. English is the primary training language, and other languages work less well. Rate limits are listed at 250,000 tokens per second and 1,200 requests per minute, with a warning that they’re adjusting while demand is high.<\/p>\n And one thing I liked: there’s no fine-tuning. The same weights serve everyone, and you customise through state, instructions and criteria. That’s less flexible than training your own classifier, but it also means there’s no training pipeline to maintain. For the kind of client projects I usually take on<\/a>, that trade is usually worth it.<\/p>\nThe request shape in one curl call<\/h2>\n
state<\/code> (the thing being judged) and a map of questions<\/code>, and you get back one typed answer per question. The quickstart<\/a> shows this with a support ticket:<\/p>\ncurl -X POST https:\/\/api.typesafe.ai\/v1\/systemone \\\n -H "Authorization: Bearer $TYPESAFE_API_KEY" \\\n -H "Content-Type: application\/json" \\\n -d '{\n "state": "Hi, I have been trying to connect my Stripe account for 3 days and the integration keeps failing. I am losing sales. Please help ASAP.",\n "model": "jev-latest",\n "questions": {\n "department": {\n "type": "choice",\n "instructions": "Which team should handle this",\n "criteria": {\n "billing": "Payment or subscription issues",\n "technical": "Bugs or integration problems",\n "sales": "Pricing or account questions"\n }\n },\n "is_urgent": {\n "type": "noul",\n "instructions": "The message conveys urgency or time-sensitivity"\n }\n }\n }'\n<\/code><\/pre>\n{\n "department": {\n "type": "choice",\n "choice": "technical",\n "confidence": 0.78,\n "probabilities": { "technical": 0.85, "sales": 0.0, "billing": 0.15 }\n }\n}\n<\/code><\/pre>\n\"is_urgent\"<\/code> tells the model nothing. Whatever you want judged has to be in instructions<\/code>. I had been naming my keys descriptively and assuming that helped. It doesn’t.<\/p>\nChoice, Score and Noul, and when I’d pick each<\/h2>\n
pip install typesafe-sdk<\/code>, Python 3.10 or newer) gives each one a class:<\/p>\nfrom typesafe_sdk import Choice, Noul, Score, TypeSafeClient\n\nclient = TypeSafeClient() # reads TYPESAFE_API_KEY\n\nresponse = client.system_one(\n state={\n "ticket_message": "My flight was cancelled. Can I get a refund?",\n "refund_policy": "Cancelled flights are eligible for a full refund.",\n },\n questions={\n "refund_requested": Noul(\n instructions="Does `ticket_message` request a refund?",\n ),\n "request_type": Choice(\n instructions="What is the main request in `ticket_message`?",\n criteria={\n "refund": "The customer wants money returned.",\n "rebooking": "The customer wants a replacement flight.",\n "information": "The customer is asking for information only.",\n },\n ),\n "frustration": Score(\n instructions="How frustrated does the customer appear in `ticket_message`?",\n criteria=[\n "Calm and neutral.",\n "Concerned but civil.",\n "Very angry or using strong language.",\n ],\n ),\n },\n)\n\nprint(response.answers["refund_requested"].noul) # probability of yes\nprint(response.answers["request_type"].choice) # one of your keys\nprint(response.answers["frustration"].score) # a float along your levels\n<\/code><\/pre>\nticket_message<\/code> in the instructions. That’s the documented way to point a question at one field of a structured state, and it’s a nice touch. You can hand Jev a whole ticket, order and policy object and aim each question at the part it should care about.<\/p>\nConfidence is math on the probabilities<\/h2>\n
confidence<\/code> was a separate signal the model produced. It isn’t. The confidence page<\/a> says it’s a statistic computed from how peaked the probability distribution is. All the mass on one option gives 1.0, an even spread gives 0.<\/p>\n(n * largest - 1) \/ (n - 1)<\/code> as an approximation for n options. I checked it against the quickstart response. Three options, largest probability 0.85: (3 x 0.85 – 1) \/ 2 = 0.775. The documented response says 0.78. So the approximation holds, at least on their example.<\/p>\nconfidence<\/code> field at all. A Noul is already a single probability, so your threshold goes directly on noul<\/code>. I had a helper function that read .confidence<\/code> off every answer. It would have thrown on the first Noul.<\/p>\nprobabilities<\/code> dict, you can compute your own measure. The docs say this outright. For a routing decision I care more about the gap between the top two options than about overall flatness, so I’ll probably compute top1 - top2<\/code> myself and threshold on that. The confidence page also shows a nice pattern where the threshold depends on the action: a read-only action runs above 0.5, while an irreversible one needs 0.9 plus a confirmation step. That’s the part I’d copy into any real integration.<\/p>\nThe jaggedness page is the part to read twice<\/h2>\n
Pin the model version before you tune anything<\/h2>\n
jev-latest<\/code>. The models page<\/a> explains that aliases move when a release ships, so your answers can change without a deploy on your side. If you’ve tuned confidence thresholds against a specific version, pin the versioned ID (jev-1.13.0<\/code> right now) and upgrade on your own schedule. Every response includes the model<\/code> field that actually answered, so log it next to each decision.<\/p>\nWhere this fits next to structured output<\/h2>\n