Jev, and the trouble with putting LLMs in production

AI chatbots are brilliant at writing. Real systems mostly need them to make small decisions, reliably, thousands of times a day. That mismatch causes a surprising amount of trouble. This is about the trouble, and about Jev, a new kind of AI model built to avoid it.

If you have used ChatGPT, Claude or Gemini, you have used a large language model, or LLM. You type something, it writes something back, and it is remarkably good at it. So it is natural to think: if it can write an essay, surely it can sort my inbox, route a complaint to the right team, or check a report for a missing number. And it can. Mostly. That "mostly" is what this piece is about.

Two of the projects on this site are built around LLMs. One of them, PAD Economic Analysis, reads a Project Appraisal Document (the long report written to justify funding a project) and works out whether the project's costs and benefits add up. The economics is the interesting part. But if you open the code, a surprising share of it exists for a different reason: to cope with the AI's answers. There is a file of a little over a thousand lines whose whole job is to reshape what the AI produced into something the dashboard can read. And there is a small helper whose description says, with admirable honesty, "Best-effort extraction of a JSON object/array from an LLM response."

That is normal. Almost everyone who has put an LLM into a real system has written code like this. Last month I wrote about AI agents: LLMs that use tools in a loop, making a small decision at every step. This piece is about why those decisions are harder than they look, and about Jev, a new model released by TypeSafe AI in September that tackles the problem from a different angle. No technical background needed. If you want to know how LLMs work underneath, I wrote about that in May, and about what AI actually is in June, but you don't need either to follow along.

In one minute
The problem Jev tackles
The problem An LLM answers by writing. But most of what a real system needs from it is a decision: which category, yes or no, how serious. So programmers ask a writer to tick a box, then write extra code to dig the tick out of its prose. Sometimes the tick is missing, misspelled, or in the wrong place.
The hidden problem Ask an LLM how sure it is and it will write a number like "90%". That number is just more writing. It says 90% whether it is right or wrong, so you can't use it to decide when to trust the answer and when to call a person.
What Jev does Jev can't write at all. You give it a question and the allowed answers. It picks one, and tells you how likely each answer is. No prose to dig through, and a confidence number designed to mean something.
LLM writes→ Jev decides→ Code acts
A few words used in this pieceplain-English glossary
LLM
Large language model. The technology behind ChatGPT, Claude and Gemini. It produces text one small piece at a time.
In production
Running for real: connected to real data, used thousands of times a day, with nobody checking each answer. The opposite of a demo.
JSON
A strict text format programs use to pass data around. It looks like {"category": "Broken services"}: a label, a colon, a value.
Parser
The bit of code that reads text and pulls out the values. If the text isn't in the expected format, the parser fails.
Confidence
A number between 0 and 1 saying how sure a system is. 0.9 means "90% sure".
Threshold
A cut-off. "If confidence is above 0.8, act automatically. Otherwise, ask a person."
01

It works in the demo. Then it meets real life.

Building the first version of an AI feature is genuinely fun. You write some instructions for the AI (a "prompt"), try a dozen examples, and the answers are good. Sometimes startlingly good. It feels finished.

Then you put it in production, and the questions change. They stop being "is it clever?" and become much more boring, and much more important. Can my program use every answer, not just most of them? Will it give the same answer tomorrow? When it's unsure, will I know? What does it cost, and how long does it take, at fifty thousand requests a day?

A demo only has to work once, in front of people who want it to. A production system has to work on the ten-thousandth case, at three in the morning, with nobody watching. Or at least know when it isn't working. That is a different problem, and how clever the AI is turns out to be only a small part of it.

02

The AI writes. Your program has to read.

Here is the root of the trouble. An LLM produces its answer by writing it, one small piece of a word at a time. That is the only thing it does. Even when all you want is a simple decision, what you get back is a piece of writing that usually contains one.

Imagine hiring a brilliant assistant to sort the post into five trays. Instead of dropping each letter into a tray, they write you a note: "This one belongs in the Broken services tray." Most notes are clear. Some say "Broken services, or possibly Fees." Some begin with "Great question!" Now and then they invent a sixth tray. You would soon need a second person whose only job is reading the notes and moving the letters.

In software, that second person is a parser: code that reads the AI's text and tries to pull out the answer. To make its life easier, you ask the AI to reply in JSON, the strict format from the glossary. And you ask firmly. Here is how the instructions in the appraisal tool end, and what the clean-up code does with the reply.

From the appraisal tool · the instruction, and the clean-upin plain English
The last line of the instructions sent to the AI
"…If a value is not stated, use null. Output ONLY valid JSON."
What the clean-up code does with the reply
  1. If the AI wrapped its answer in decorative markers (like ```json), strip them off.
  2. Find the first { and the last } and keep only what's between them, in case the AI added a chatty sentence before or after.
  3. Try to read what's left.
  4. If that fails, write a warning in the log and give up.
Each step handles a common way AI answers go wrong. None of them has anything to do with economics.

The capital letters in "Output ONLY valid JSON" are doing a lot of work. They are also a tell. You only shout at something when you're not sure it's listening. An instruction to an AI is a request, not a guarantee: it will be followed almost every time. Real systems live in the "almost".

Here is what that looks like. Development projects run a complaints line for the people they affect (formally, a "grievance redress mechanism"), and someone has to sort every message that comes in. Below is one such complaint, five allowed categories, and eight answers from an LLM asked to pick one. The answers are made up, but each one is a kind of mistake that clean-up code exists to handle. The code judging them in your browser follows exactly the steps above.

Demo 01 · Ask a writer to decidepress run and watch the clean-up code cope
One complaint, one question, eight answers
We ask the AI to pick one of five categories and say how sure it is, in the format {"category": …, "confidence": …}. Each answer goes through the clean-up steps, then a check that the category is one of the five.
The complaint
Allowed categories
Press run to send the same complaint to the AI eight times.
—Usable by the program
—Different answers
—How sure it said it was
Decorative markers, a chatty opening, a category that isn't on the list, the wrong kind of quote marks, a confidence of "high", an answer cut off halfway. None of these are exotic. And look at how sure it said it was: about 0.9, whichever category it picked.

There are two kinds of problem in there. The first is loud: an answer the program can't read. You notice it, you add another clean-up step, you move on. Annoying, but at least it's honest.

The second is quiet, and it's in the last column. The AI picked "Broken services" most of the time and "Fees & misconduct" some of the time, and said it was about 90% sure both times. They can't both be 90% likely. That number isn't a measurement. It's more writing: the AI writes "0.9" after the word "confidence" because that is what tends to come next in the text it learned from.

The appraisal tool asks for a confidence number like this too, and if the AI leaves it out, the code fills in 0.5. As defaults go, it's a sensible one. But neither number is something you would want to base a real decision on. Most AI systems carry a number like this, and most treat it as more meaningful than it is.

03

The usual fixes, and what they actually fix

None of this is news to people who build these systems, and there is a well-worn set of fixes. Each one genuinely helps. It's worth being precise about how.

Table · The usual fixeswhat each one solves
FixIn plain wordsSolvesDoesn't solve
Structured outputsForce the AI to always fill in the form correctlyMessy formatsWhether the answer, or its confidence, is right
RetriesIf the answer can't be read, ask againThe odd garbled answerCosts more and takes longer exactly when things go wrong
"Temperature zero"Turn the AI's randomness downMost of the run-to-run varietySame answer isn't guaranteed, and a consistent wrong answer is still wrong
An AI judgeA second AI marks the first one's workCatches some mistakesThe judge's marks are also just writing
A custom classifierA small model trained for one sorting jobReliable, fast, cheapNeeds lots of labelled examples, and only knows that one question
"Structured outputs" is the big one: most major AI providers now offer a mode that guarantees the reply matches your format. If you can use it, you should. It makes most of Demo 01's clean-up unnecessary. It doesn't touch the confidence problem.

Notice the pattern. Every fix makes the answer easier to read. None of them makes the AI's sense of its own certainty any more trustworthy.

"You can force an AI to fill in the form correctly. You can't force it to know how sure it should be."

04

How sure is "sure"?

There is a name for a confidence number you can trust: calibrated. A system is calibrated if, when it says "80% sure", it turns out to be right about 80% of the time. Weather forecasts are the classic example. On the days a good forecaster says there's a 70% chance of rain, it rains on about seven days in ten. They are wrong three times in ten, and that's fine, because they told you so in advance.

Why does this matter? Because almost every automated system has a threshold hidden inside it: if the AI is sure enough, act automatically; if not, ask a person. That only works if "sure" means something. Otherwise the threshold is a dial that isn't connected to anything.

Below are 100 decisions from two systems that are right equally often, about three times in four. Each dot is one decision. Red means it was wrong. The rows group the decisions by how sure the system said it was. Move the slider to choose when the system is allowed to act on its own, and watch which mistakes slip through.

Demo 02 · 100 decisionsswitch systems · move the slider
When it says "90% sure", how often is it right?
Same 100 cases, two systems. The highlighted rows are the ones the system would handle alone, without a person checking.
right wrong handled alone
90%
—Handled alone
—Of those, wrong
—Right overall
Made-up data, shaped like what you typically see. With the LLM's own estimates, almost everything claims 80% or more, and the red dots are spread through every row: wherever you set the slider, about one in four of the answers it handles alone are wrong. With a calibrated system, the red dots sink to the bottom rows. Set the slider high and it handles fewer cases alone, but gets almost all of them right.

This is the same lesson as the skewed study-hours line in What is AI, really? The model never says "I'm not sure about this one". It gives a clean, confident answer either way. Both systems above would get the same score in a headline accuracy test. Only one of them can safely be left alone.

05

Enter Jev: an AI that ticks boxes instead of writing

On 15 September, TypeSafe AI released Jev to early users. They call it a "System One" model, after the psychologist Daniel Kahneman's name for fast, intuitive thinking (as opposed to System Two, the slow and deliberate kind). The company name is a programmer's joke: in a "type-safe" programming language, a piece of code that promises to return a number can't hand you a paragraph instead.

The idea is simple. Jev doesn't write at all. You send it two things: the text you want judged, and your questions, each with its allowed answers. It sends back, for each question, how likely each allowed answer is, plus an overall confidence. That's it. Ask it to write an email and there is nothing it can give you.

It handles three kinds of question. Pick one chooses from a list you provide (TypeSafe calls this a Choice). Rate it places something on a scale you define, such as 1 for minor up to 4 for critical (a Score). Yes or no gives the probability that a statement is true (TypeSafe's name for this one is a Noul). You can ask several questions about the same text in one go. Here is what that looks like for the complaint from Demo 01.

Figure · A conversation with Jevsimplified
You send
The text"The water point near the school has been broken since the rains…"
Question 1 · pick oneWhich desk should handle it? Land & compensation, Health & environment, Broken services, Fees & misconduct, Not a complaint
Question 2 · yes or noIs someone's health or safety at risk?
Question 3 · rate itHow serious is it, from 1 (minor) to 4 (critical)?
You get back
Which desk?Broken services 47%, Fees & misconduct 39%, others 14%. Confidence 0.34: it's torn.
At risk?22% likely.
How serious?Most likely 2, moderate. Average 2.3 out of 4.
What you don't getAny writing. Nothing to clean up.
Illustrative numbers, not real Jev output. The real interface is a little more technical, but this is the whole idea: questions with fixed answers go in, likelihoods come out.

Back to the post. An LLM is the assistant who writes you a note about each letter. Jev is a sorting machine with five slots and a gauge: the letter can only go into one of the slots, and the gauge shows how torn the machine was. That has three consequences, and each one fixes a problem from earlier.

It can't answer outside your list. There is no "Broken services / Fees", no chatty opening, no half-finished answer. The loud problem from Demo 01 doesn't need cleaning up. It can't happen.

Its confidence is calculated, not written. If one answer is far more likely than the rest, confidence is high. If two are neck and neck, it's low. And TypeSafe says Jev is trained so that its likelihoods match how often it's actually right. In other words: calibrated, like the second system in Demo 02. That is the claim that matters most.

It's fast and cheap. An LLM might write a hundred words to deliver one category. Jev returns a handful of numbers. How much faster and cheaper is a question I'll come back to, because the headline figures deserve some scrutiny.

There is a personal irony here. In 2022, just before ChatGPT arrived, I trained a small model called DistilBERT to sort survey responses into categories (it's in the LLM essay). It was quick, cheap, and never invented a category. Its weakness was that it only knew the one question it had been trained on: asking a new one meant collecting labelled examples and training it all over again. From what TypeSafe has published, Jev works on a similar principle but takes the question on the spot. It's a sorting model you don't have to train.

06

LLM writes, Jev decides, code acts

Put the pieces together and you get the slogan that has followed Jev since launch. Give each job to the tool built for it.

Figure · Three jobs, three toolswho does what
LLMwrites

Summarises, translates, drafts replies. Anything a person will eventually read.

gives you: text
→
Jevdecides

Pick one, rate it, yes or no. Answers come back as likelihoods.

gives you: likelihoods
→
Codeacts

Plain rules: thresholds, escalations, a record of what happened. The only part allowed to do anything.

gives you: actions
The important box is the last one. The thresholds live in ordinary code, where people can read them, test them, and explain them to an auditor. Not hidden inside an AI's instructions.

Take the complaints line again. The LLM still has real work to do: translating a complaint that arrived in Hausa, summarising a long one for the case officer, drafting a polite acknowledgement. Jev answers the small questions: which desk, is anyone at risk, how serious. And a few plain rules turn those answers into an action. Pick a complaint below and move the thresholds.

Demo 03 · Jev decides, code actspick a complaint · move the thresholds
Three questions, one answer each, one action
Each complaint goes to Jev once, with three questions. The rules underneath are ordinary code that a project team writes and can change. The highlighted rule is the one that applies.
Which desk?pick oneconfidence —
Is someone's health or safety at risk?yes / no
yes
—
How serious?rate 1–4average — / 4
0.75
0.38
—
The numbers here are made up to show how Jev behaves, not captured from it. Try the last complaint: someone has slipped an instruction into their message, hoping to get it dismissed. Jev can't be tricked into writing anything, but notice it has been nudged towards "Not a complaint". That's why the final say, and a human safety net, belong in the rules.

Look at the appraisal tool with this in mind and some of its AI requests turn out not to be writing jobs at all. It asks the LLM to rate how complete its own work was: that's a "rate it" question. Whether a report states a rate of return at all is a "yes or no" question. The LLM is still the right tool for pulling cost tables out of a two-hundred-page document. It was never the right tool for marking its own homework.

07

Should you believe the numbers?

Every AI launch comes with impressive numbers, and Jev's are spectacular. They deserve the same treatment I gave the satellite data in the air quality piece: useful, and worth checking the receipt.

Table · The launch claimsand who measured them
ClaimFigureWho measured it
Faster than an LLM193.6×TypeSafe, on tasks it chose
Cheaper than an LLM444.6×TypeSafe, on tasks it chose
Accuracy on TypeSafe's test~68%TypeSafe; a few points below the best LLM it compared against
Faster, in early independent tests~1.2× to ~25×Others; depends on the task
As reported at and shortly after launch. One early critique pointed out that TypeSafe's test used answers from other AI models as the "correct" answers, so part of what it measures is agreement with those models rather than with the truth.

My read: the direction is real, the size is marketing. A model that returns five numbers will obviously be faster and cheaper than one that writes a paragraph to say one word. That's arithmetic, not a breakthrough. How much faster on your own work depends on what you compare it with, and early independent tests suggest "a lot less than 193 times, but sometimes still a lot". The accuracy figure is more interesting than it looks. Jev isn't claiming to be smarter than an LLM. It's claiming to be roughly as good, much quicker, and honest about when it's unsure. For sorting and triage, that's the right trade.

08

What Jev can't do

It can't write. Anything a person will read still needs an LLM, or a person. Jev works alongside an LLM, not instead of one.

"Never makes things up" is about format, not truth. You'll see this claim a lot. It's true that Jev can't invent an answer that isn't on your list. But it can still pick the wrong one from the list. A tidy wrong answer is still a wrong answer. It's easier for your program to read, and no comfort at all to the person whose complaint went to the wrong desk.

Its honesty is the big claim, and the least tested. Everything good about Jev depends on its confidence numbers meaning what they say. Until that has been checked by people other than TypeSafe, on tasks TypeSafe didn't choose, I would run my own version of Demo 02 before trusting any threshold.

It has weak spots. TypeSafe's own guidance reportedly flags double negatives, roundabout phrasing, and questions that need several steps of reasoning. Giving it more text isn't always better either: irrelevant detail can make its answers worse.

It can be nudged. It reads whatever text you give it, including text written by someone trying to game the system. It can't be tricked into writing anything, but it can be pushed towards an answer, as the last complaint in Demo 03 shows. Thresholds and a human safety net are the defence.

Your data leaves the building. Jev is a hosted service run by one company, still in early access. For a lot of uses that's fine. For complaints, health surveys, or anything covered by a data-sharing agreement, it's a conversation to have before you start, not after. And I have seen nothing yet on how it handles a survey answer written in Zarma or Hausa. After everything in the data trust essay, I would want to test that before trusting it.

09

The short version

Whether Jev itself lasts almost doesn't matter. The idea behind it will. Most of the trouble with putting LLMs into real systems comes from asking a writer to make decisions, then writing extra code to find out what it decided, and trusting a confidence number it simply made up. Separate the jobs. Let the model that writes, write. Let something built for decisions return decisions, with confidence numbers that mean something. And keep the rules that turn decisions into actions in plain code, where people can see them.

There's a small irony in where this has landed. In 2022, a general-purpose AI made my little sorting model feel like an antique. Four years later, one of the more interesting launches of the year is, in effect, a sorting model: one that reads, picks from a list, and refuses to improvise. The field moved fast, but not in a straight line, and some of the old ideas were waiting for it.

LLM writes, Jev decides, code acts. Swap in whatever names come next. Keep the separation.

Notes
1.Details on Jev come from TypeSafe AI's launch materials and documentation, and from early third-party write-ups, including Langfuse on using Jev for evals and DataSci Ocean's look at the benchmark claims. The product is moving quickly (versions, limits and access terms changed within weeks of launch), so check the current docs before relying on any specific number here.
2.All three demos are simulations. The eight answers in Demo 01 were written by hand to show common mistakes; the clean-up code that judges them is real. Demo 02 uses made-up data. The numbers in Demo 03 are illustrative, not captured from Jev. Confidence in Demo 03 is worked out from the likelihoods: 0 when every answer is equally likely, 1 when one answer has all of it. (For the curious: (k × pmax − 1) / (k − 1), for k answers. TypeSafe's exact formula may differ.)
3.The clean-up steps summarise a helper called _extract_json in the PAD Economic Analysis tool. Nothing in it is wrong. It is simply the kind of code you write when the thing on the other end is a writer.
4."System 1" and "System 2" come from Daniel Kahneman's Thinking, Fast and Slow (2011). The comparison is loose: Jev is "fast" in the sense that it answers in one go without writing anything out, not in any sense that it thinks.