Headstart AIMeasuring AI accuracy in production
1 You probably can't put an error bar on your model's live accuracy
Not for lack of effort. Most teams check their outputs somehow. But three things make a casual check produce a number you can't rely on.
Nobody grades the grader. A reviewer marks 200 outputs and finds 6 wrong. Is that a 3% error rate? Only if the reviewer is right every time, and nobody has measured them. One unmeasured judge gives you a number with no bound on it.
A checker who has seen the model's answer tends to agree with it. That is not carelessness, it is how people work. If the verdict is already on the screen, the review confirms rather than judges, and the result is not independent evidence.
Errors cluster, so casual sampling misses them. Model errors do not spread evenly. They concentrate on one lighting condition, one product type, one customer segment, usually the slices your test set under-represents. A random 50 item sample spread thin across everything will sail past a slice where the model is wrong 30% of the time and report that things look fine.
So what most teams have is test set accuracy from before the model met the real world, confidence scores that are the model's opinion of itself, and a spot check with no bound on it.
2 What a real measurement looks like
Four things have to be true before an error rate means anything.
The judge is blind. They never see what the model decided. Once you have seen an answer you tend to agree with it, so a reviewer who has been shown the verdict is confirming rather than judging.
The judges are measured against your standard. Known answer items, drawn from your own labelled examples, are mixed invisibly into the work. Every judge has a tracked accuracy against your definition of correct, and their votes are weighted by it. That is what turns "someone checked and found six errors" into a number with a bound on it.
Effort follows difficulty. Every item is judged by several people independently. Each vote carries a measurable amount of information, and a vote from a judge who is right 95% of the time tells you far more than one from a judge who is right 70%. So we keep gathering votes until there is enough information to settle the item, and stop when there is. Straightforward items settle quickly. Contested ones go to more judges, and to the strongest ones. The effort lands where the uncertainty is instead of being spread evenly across everything.
The two kinds of error are priced separately. A false pass and a false fail rarely cost the same. In our own pipeline a wrong approval poisons downstream training data while a wrong rejection costs one frame, so the bar for approving is set higher than the bar for rejecting. Your ratio will be different, and it is a setting rather than a default.
Your reviewer stays the standard throughout. What changes is what reaches them: the items the system could not settle confidently, plus a blind sample for measurement. Everything else is decided before it gets to their desk.
3 We built this for ourselves first
Our first customer was our own computer vision pipeline.
We run a CV system whose models produce visual judgements at a volume no one can review by hand. The domain and the client are under NDA, but the problem will be familiar: thousands of outputs, a standard that only we could define, and no way to know how often the model was getting it wrong. Every output was reviewed manually. That review was both the quality guarantee and the bottleneck.
So we built the layer described above and pointed it at ourselves. A crowd approaching 1,000 judges, each one measured against our own labelled examples. Every output judged independently by several of them, with the algorithm gathering votes until it has enough information to settle the item.
It now settles roughly 95% of items outright. Our reviewer sees the contested remainder, plus a small blind sample drawn deliberately across different conditions. That sample is not overhead. It is what produces the error rate: without it we would have automation and no measurement, which is the position most pipelines are already in.
Two things changed. The volume reaching the reviewer collapsed, and what does reach them is worth their attention: genuinely contested cases rather than a queue of obvious passes. And for the first time the pipeline has an error rate with a bound on it rather than a general feeling that things are probably fine.
That system runs in production today. Everything on this page describes something we operate, not something we propose.
4 Why it has to keep running
An audit measures one sample, at one moment, on one kind of content. It is valid for exactly that.
Models change. They get retrained, prompts and thresholds get tuned, upstream data shifts. Any of those can move the error rate, and none of them announce themselves. The failure mode is not a sudden break, it is a slow slide that nobody notices because nothing looks different from the inside.
Content changes too, and this one matters more than it sounds. A measurement made on one type of content does not transfer to another. A new product type, a new site, a new customer segment: each is a different problem for the model, and its error rate has to be established separately rather than inherited. In our own system, content the judges had not been trained on ran at roughly double the error rate of content they had. We now treat every new stream as its own project, with its own examples, its own known answer items, and its own audit.
There is a statistical reason as well. A single sample gives you a wide bound. Zero errors in a hundred items sounds perfect, but with no errors observed the 95% upper bound on the true rate is about three in a hundred, so all you have really established is that the rate is probably under 3%. Keep sampling and the bound tightens week by week, until the number is precise enough to act on. A one off audit gives you a snapshot. Sampling continuously gives you a measured rate, a trend, and something that raises its hand when it moves.
The machinery is already running either way. Not switching it off is the cheapest part of this.
5 Where to start
The first step is a conversation, not a purchase. We want to understand how you check your model's outputs today: who looks at them, how many, and what happens when one turns out to be wrong. Some pipelines do not need this. That conversation tells us both.
If it looks like a fit, the first piece of work is a single audit. You export a sample of decisions your model has already made, chosen across the conditions worth testing. We judge them blind. You get a measured error rate with a bound on it, broken down by condition rather than averaged into one number, and a view of where the errors concentrate. It takes about two weeks and requires no integration. Audits start at £2,000 depending on sample size.
If the errors are worth watching, the same sampling continues: a fresh blind sample of live decisions every week, the error rate tracked over time, and an alert when it moves. That runs from £3,000 a month.
If you want to know what your real error rate is, the conversation takes twenty minutes.