AI chatbots are brilliant at writing. Real systems mostly need them to make small decisions, reliably, thousands of times a day. That mismatch causes a surprising amount of trouble. This is about the trouble, and about Jev, a new kind of AI model built to avoid it.
If you have used ChatGPT, Claude or Gemini, you have used a large language model, or LLM. You type something, it writes something back, and it is remarkably good at it. So it is natural to think: if it can write an essay, surely it can sort my inbox, route a complaint to the right team, or check a report for a missing number. And it can. Mostly. That "mostly" is what this piece is about.
Two of the projects on this site are built around LLMs. One of them, PAD Economic Analysis, reads a Project Appraisal Document (the long report written to justify funding a project) and works out whether the project's costs and benefits add up. The economics is the interesting part. But if you open the code, a surprising share of it exists for a different reason: to cope with the AI's answers. There is a file of a little over a thousand lines whose whole job is to reshape what the AI produced into something the dashboard can read. And there is a small helper whose description says, with admirable honesty, "Best-effort extraction of a JSON object/array from an LLM response."
That is normal. Almost everyone who has put an LLM into a real system has written code like this. Last month I wrote about AI agents: LLMs that use tools in a loop, making a small decision at every step. This piece is about why those decisions are harder than they look, and about Jev, a new model released by TypeSafe AI in September that tackles the problem from a different angle. No technical background needed. If you want to know how LLMs work underneath, I wrote about that in May, and about what AI actually is in June, but you don't need either to follow along.
{"category": "Broken services"}: a label, a colon, a value.Building the first version of an AI feature is genuinely fun. You write some instructions for the AI (a "prompt"), try a dozen examples, and the answers are good. Sometimes startlingly good. It feels finished.
Then you put it in production, and the questions change. They stop being "is it clever?" and become much more boring, and much more important. Can my program use every answer, not just most of them? Will it give the same answer tomorrow? When it's unsure, will I know? What does it cost, and how long does it take, at fifty thousand requests a day?
A demo only has to work once, in front of people who want it to. A production system has to work on the ten-thousandth case, at three in the morning, with nobody watching. Or at least know when it isn't working. That is a different problem, and how clever the AI is turns out to be only a small part of it.
Here is the root of the trouble. An LLM produces its answer by writing it, one small piece of a word at a time. That is the only thing it does. Even when all you want is a simple decision, what you get back is a piece of writing that usually contains one.
Imagine hiring a brilliant assistant to sort the post into five trays. Instead of dropping each letter into a tray, they write you a note: "This one belongs in the Broken services tray." Most notes are clear. Some say "Broken services, or possibly Fees." Some begin with "Great question!" Now and then they invent a sixth tray. You would soon need a second person whose only job is reading the notes and moving the letters.
In software, that second person is a parser: code that reads the AI's text and tries to pull out the answer. To make its life easier, you ask the AI to reply in JSON, the strict format from the glossary. And you ask firmly. Here is how the instructions in the appraisal tool end, and what the clean-up code does with the reply.
```json), strip them off.{ and the last } and keep only what's between them, in case the AI added a chatty sentence before or after.The capital letters in "Output ONLY valid JSON" are doing a lot of work. They are also a tell. You only shout at something when you're not sure it's listening. An instruction to an AI is a request, not a guarantee: it will be followed almost every time. Real systems live in the "almost".
Here is what that looks like. Development projects run a complaints line for the people they affect (formally, a "grievance redress mechanism"), and someone has to sort every message that comes in. Below is one such complaint, five allowed categories, and eight answers from an LLM asked to pick one. The answers are made up, but each one is a kind of mistake that clean-up code exists to handle. The code judging them in your browser follows exactly the steps above.
{"category": …, "confidence": …}. Each answer goes through the clean-up steps, then a check that the category is one of the five.There are two kinds of problem in there. The first is loud: an answer the program can't read. You notice it, you add another clean-up step, you move on. Annoying, but at least it's honest.
The second is quiet, and it's in the last column. The AI picked "Broken services" most of the time and "Fees & misconduct" some of the time, and said it was about 90% sure both times. They can't both be 90% likely. That number isn't a measurement. It's more writing: the AI writes "0.9" after the word "confidence" because that is what tends to come next in the text it learned from.
The appraisal tool asks for a confidence number like this too, and if the AI leaves it out, the code fills in 0.5. As defaults go, it's a sensible one. But neither number is something you would want to base a real decision on. Most AI systems carry a number like this, and most treat it as more meaningful than it is.
None of this is news to people who build these systems, and there is a well-worn set of fixes. Each one genuinely helps. It's worth being precise about how.
| Fix | In plain words | Solves | Doesn't solve |
|---|---|---|---|
| Structured outputs | Force the AI to always fill in the form correctly | Messy formats | Whether the answer, or its confidence, is right |
| Retries | If the answer can't be read, ask again | The odd garbled answer | Costs more and takes longer exactly when things go wrong |
| "Temperature zero" | Turn the AI's randomness down | Most of the run-to-run variety | Same answer isn't guaranteed, and a consistent wrong answer is still wrong |
| An AI judge | A second AI marks the first one's work | Catches some mistakes | The judge's marks are also just writing |
| A custom classifier | A small model trained for one sorting job | Reliable, fast, cheap | Needs lots of labelled examples, and only knows that one question |
Notice the pattern. Every fix makes the answer easier to read. None of them makes the AI's sense of its own certainty any more trustworthy.
"You can force an AI to fill in the form correctly. You can't force it to know how sure it should be."
There is a name for a confidence number you can trust: calibrated. A system is calibrated if, when it says "80% sure", it turns out to be right about 80% of the time. Weather forecasts are the classic example. On the days a good forecaster says there's a 70% chance of rain, it rains on about seven days in ten. They are wrong three times in ten, and that's fine, because they told you so in advance.
Why does this matter? Because almost every automated system has a threshold hidden inside it: if the AI is sure enough, act automatically; if not, ask a person. That only works if "sure" means something. Otherwise the threshold is a dial that isn't connected to anything.
Below are 100 decisions from two systems that are right equally often, about three times in four. Each dot is one decision. Red means it was wrong. The rows group the decisions by how sure the system said it was. Move the slider to choose when the system is allowed to act on its own, and watch which mistakes slip through.
This is the same lesson as the skewed study-hours line in What is AI, really? The model never says "I'm not sure about this one". It gives a clean, confident answer either way. Both systems above would get the same score in a headline accuracy test. Only one of them can safely be left alone.
On 15 September, TypeSafe AI released Jev to early users. They call it a "System One" model, after the psychologist Daniel Kahneman's name for fast, intuitive thinking (as opposed to System Two, the slow and deliberate kind). The company name is a programmer's joke: in a "type-safe" programming language, a piece of code that promises to return a number can't hand you a paragraph instead.
The idea is simple. Jev doesn't write at all. You send it two things: the text you want judged, and your questions, each with its allowed answers. It sends back, for each question, how likely each allowed answer is, plus an overall confidence. That's it. Ask it to write an email and there is nothing it can give you.
It handles three kinds of question. Pick one chooses from a list you provide (TypeSafe calls this a Choice). Rate it places something on a scale you define, such as 1 for minor up to 4 for critical (a Score). Yes or no gives the probability that a statement is true (TypeSafe's name for this one is a Noul). You can ask several questions about the same text in one go. Here is what that looks like for the complaint from Demo 01.
Back to the post. An LLM is the assistant who writes you a note about each letter. Jev is a sorting machine with five slots and a gauge: the letter can only go into one of the slots, and the gauge shows how torn the machine was. That has three consequences, and each one fixes a problem from earlier.
It can't answer outside your list. There is no "Broken services / Fees", no chatty opening, no half-finished answer. The loud problem from Demo 01 doesn't need cleaning up. It can't happen.
Its confidence is calculated, not written. If one answer is far more likely than the rest, confidence is high. If two are neck and neck, it's low. And TypeSafe says Jev is trained so that its likelihoods match how often it's actually right. In other words: calibrated, like the second system in Demo 02. That is the claim that matters most.
It's fast and cheap. An LLM might write a hundred words to deliver one category. Jev returns a handful of numbers. How much faster and cheaper is a question I'll come back to, because the headline figures deserve some scrutiny.
There is a personal irony here. In 2022, just before ChatGPT arrived, I trained a small model called DistilBERT to sort survey responses into categories (it's in the LLM essay). It was quick, cheap, and never invented a category. Its weakness was that it only knew the one question it had been trained on: asking a new one meant collecting labelled examples and training it all over again. From what TypeSafe has published, Jev works on a similar principle but takes the question on the spot. It's a sorting model you don't have to train.
Put the pieces together and you get the slogan that has followed Jev since launch. Give each job to the tool built for it.
Summarises, translates, drafts replies. Anything a person will eventually read.
gives you: textPick one, rate it, yes or no. Answers come back as likelihoods.
gives you: likelihoodsPlain rules: thresholds, escalations, a record of what happened. The only part allowed to do anything.
gives you: actionsTake the complaints line again. The LLM still has real work to do: translating a complaint that arrived in Hausa, summarising a long one for the case officer, drafting a polite acknowledgement. Jev answers the small questions: which desk, is anyone at risk, how serious. And a few plain rules turn those answers into an action. Pick a complaint below and move the thresholds.
Look at the appraisal tool with this in mind and some of its AI requests turn out not to be writing jobs at all. It asks the LLM to rate how complete its own work was: that's a "rate it" question. Whether a report states a rate of return at all is a "yes or no" question. The LLM is still the right tool for pulling cost tables out of a two-hundred-page document. It was never the right tool for marking its own homework.
Every AI launch comes with impressive numbers, and Jev's are spectacular. They deserve the same treatment I gave the satellite data in the air quality piece: useful, and worth checking the receipt.
| Claim | Figure | Who measured it |
|---|---|---|
| Faster than an LLM | 193.6× | TypeSafe, on tasks it chose |
| Cheaper than an LLM | 444.6× | TypeSafe, on tasks it chose |
| Accuracy on TypeSafe's test | ~68% | TypeSafe; a few points below the best LLM it compared against |
| Faster, in early independent tests | ~1.2× to ~25× | Others; depends on the task |
My read: the direction is real, the size is marketing. A model that returns five numbers will obviously be faster and cheaper than one that writes a paragraph to say one word. That's arithmetic, not a breakthrough. How much faster on your own work depends on what you compare it with, and early independent tests suggest "a lot less than 193 times, but sometimes still a lot". The accuracy figure is more interesting than it looks. Jev isn't claiming to be smarter than an LLM. It's claiming to be roughly as good, much quicker, and honest about when it's unsure. For sorting and triage, that's the right trade.
It can't write. Anything a person will read still needs an LLM, or a person. Jev works alongside an LLM, not instead of one.
"Never makes things up" is about format, not truth. You'll see this claim a lot. It's true that Jev can't invent an answer that isn't on your list. But it can still pick the wrong one from the list. A tidy wrong answer is still a wrong answer. It's easier for your program to read, and no comfort at all to the person whose complaint went to the wrong desk.
Its honesty is the big claim, and the least tested. Everything good about Jev depends on its confidence numbers meaning what they say. Until that has been checked by people other than TypeSafe, on tasks TypeSafe didn't choose, I would run my own version of Demo 02 before trusting any threshold.
It has weak spots. TypeSafe's own guidance reportedly flags double negatives, roundabout phrasing, and questions that need several steps of reasoning. Giving it more text isn't always better either: irrelevant detail can make its answers worse.
It can be nudged. It reads whatever text you give it, including text written by someone trying to game the system. It can't be tricked into writing anything, but it can be pushed towards an answer, as the last complaint in Demo 03 shows. Thresholds and a human safety net are the defence.
Your data leaves the building. Jev is a hosted service run by one company, still in early access. For a lot of uses that's fine. For complaints, health surveys, or anything covered by a data-sharing agreement, it's a conversation to have before you start, not after. And I have seen nothing yet on how it handles a survey answer written in Zarma or Hausa. After everything in the data trust essay, I would want to test that before trusting it.
Whether Jev itself lasts almost doesn't matter. The idea behind it will. Most of the trouble with putting LLMs into real systems comes from asking a writer to make decisions, then writing extra code to find out what it decided, and trusting a confidence number it simply made up. Separate the jobs. Let the model that writes, write. Let something built for decisions return decisions, with confidence numbers that mean something. And keep the rules that turn decisions into actions in plain code, where people can see them.
There's a small irony in where this has landed. In 2022, a general-purpose AI made my little sorting model feel like an antique. Four years later, one of the more interesting launches of the year is, in effect, a sorting model: one that reads, picks from a list, and refuses to improvise. The field moved fast, but not in a straight line, and some of the old ideas were waiting for it.
LLM writes, Jev decides, code acts. Swap in whatever names come next. Keep the separation.
_extract_json in the PAD Economic Analysis tool. Nothing in it is wrong. It is simply the kind of code you write when the thing on the other end is a writer.