Testing AI Is Harder Than Ever. Because It's Not a Test.

Most of software never had to grade work with no single right answer. The fields that always did have run this playbook for fifty years.

Air Canada’s support bot invented a bereavement-fare policy that did not exist. A customer relied on it, booked his flight, applied for the refund the bot had promised him, and was turned down because the policy was never real. He took the airline to tribunal, and the airline lost.

The headline cases are the loud ones, but the quieter write-ups from this year are the ones worth reading. They describe agents that follow their instructions cleanly in week one and, by week six, have renegotiated those instructions with themselves. The thread running through all of it is unglamorous: almost none of these failures register as errors to the systems watching them. They return a valid response. They throw no exception. Monitoring built to catch the absence of an answer has nothing to say about an answer that is fluent, well-formed, and wrong.

I ran into the same wall on a far smaller stage, building an advisor agent, and the first real test I wrote for it worked perfectly. It had one rule it could never break: research a figure before stating it, and never assert a number it hadn’t looked up. That either happened or it didn’t. Clean pass, clean fail, nothing to argue about.

Then I encountered a bigger problem.

As the agent got more capable, its memory and its skills got spread across more systems, and the first thing that broke was the voice. What had sounded like a person started sounding like a manual. That failure was relatively easy to repair — but then generated another, harder problem. With the voice restored, it drifted: plain on the first exchange, faintly professorial by the third, talking like a philosophy seminar by the sixth. No single response was obviously wrong, but the trajectory meant that someone having a conversation over time with the agent would feel the slope.

The cause wasn’t the model changing under me — though that’s also a real failure for production agents (say, a provider ships an update under a stable model and behavior shifts overnight). Mine was the opposite kind: the model still held but the context moved. My own prior turns were the strongest examples in front of it, so a slightly ornate answer on turn one became the example that turn two imitated.

A slope isn’t something you catch immediately by reading transcripts. So I seeded the agent’s instructions with a handful of examples of the voice I wanted, then scored each run on how far it wandered from them, grading the spread of outputs rather than any one of them. And somewhere in building it, I realized the research test and the voice score were not two versions of the same job — they were different jobs.

Three different jobs, three different evaluations.

For most of its history, a piece of software took a defined input and produced a defined output, and testing meant checking the second against the first. Put a language model inside that software and the output stops being a single value and becomes a spread of values: same input, a distribution of possible responses, no single correct one to check against. A pass/fail gate has nothing fixed to measure a distribution against.

So the one job becomes three, and only one of them is still a gate.

Some things stay binary: the research must always happen first, the output must be valid. Those are gates, and they still block the build exactly as before. Some things are invariants that have to hold across the whole spread, whatever words come out: the agent concedes uncertainty when it has no grounds to be sure; the answer stays the same when only an irrelevant detail of the question changes. Those are property tests. And some things are matters of judgment: is the voice right, did the joke land, would this put a customer off. No assertion settles those, because those are gauges on a scale. A gauge gets read, and watched over time.

It’s worth stating that software has not been uniformly blind to this. Machine learning has graded output with no single right answer for decades — translations scored against references, models tuned on precision and recall, human labels reconciled for agreement. Property-based testing has asserted invariants across a range of inputs for years. Performance engineers have always shipped against a distribution of latencies, not a single number. The pieces existed. Yet they lived in the ML lab and a few specialized corners, and the great deterministic middle of software (the CRUD apps, the business logic, the SaaS that runs on defined input and defined output) built its entire quality model on the assumption it would never need them. That middle is the latecomer.

It built its quality model around a specific bet: that testing is mechanical enough for the person who wrote the code to also write the test. Over the last decade the bet won. Separate QA got folded into engineering, “throw it over the wall” became an embarrassment, and the coder owning the test became the default. That arrangement holds for gates because a gate is mechanical, and catches the fault no matter who wrote it. But a gauge needs a reader whose judgment is independent of the making: a second party, or a rubric fixed in advance so the score isn’t just the maker’s gut.

The strain of managing AI quality is already in the numbers.

Developers working with AI assistants merge far more code, carrying something like 1.7 times the defects of code written by hand. QA headcount, widely written off a few years ago, has grown faster than developer headcount over the same stretch. And the field is openly split on where the judgment work should go: back to a revived quality function, or out to the robots, with LLMs grading other LLMs.

The real answer is both, but applied in an order. The robots scale the gauge: an LLM can score a thousand voice samples against a rubric overnight in a way no human panel can. But it cannot own the verdict, because you have no reason to trust its scores until you’ve checked them against a human whose judgment you already trust. That requires organizations to calibrate the judge first, against a person, the way clinical raters are checked against each other before their scores are allowed to count. The teams doing this well are the ones who decided on purpose where a human stays in the loop, instead of discovering the answer after a tribunal.

When you evaluate voice drift, the judgment takes two forms. First is calibrating the scale itself: what makes an output a 4 and not a 5, a 3 and not a 4. Second is whether to put a gate on the gauge at all, and where. The mistake is to gate on the average, because the average lies. A voice whose mean score looks fine but whose worst five percent collapses into generic-assistant filler is exactly the one a user stops trusting. So you gate on the tail, the fraction of outputs that breach the bar, not on the score of a typical one.

Which raises a question the industry is asking backward. It isn’t whether we need testers again. It’s who in the building already knows how to read a gauge — and some of them have been doing it in plain sight. NPS was never a number you passed or failed; it was a user-generated gauge, on which teams read and argued about where to draw the line. Whether your organization strove for an NPS of 7 or 8 was a matter of judgment. Likewise, “good enough to ship” was always a gauge reading disguised as a decision. Product is one function that’s lived with this; so has anyone who's owned an SLA or a support-CSAT target. The capability the org suddenly needs — owning a threshold on a distribution and defending it — is one that teams have practiced for years under a different name.

I spent years running quality as a function before I ever built an agent: the launch frameworks, the named go/no-go conditions, the framework for deciding when a thing met the bar that customers expected. That apparatus for judgment already existed, and I’d built versions of it. It had simply never been pointed at a language model.

Other industries built this apparatus long ago.

A pharmaceutical release lab is an evaluation harness built in stainless steel. Standardized sample pulls, calibrated instruments, spec limits applied automatically, and a literal gate at the end: the batch does not ship unless its measured distribution sits inside spec. That is the same move: a gate riding on the tail of a gauge, not on any single reading.

A factory floor runs the other half — automated measurement on the line, the output distribution charted in real time, an alarm when it drifts past its limits. That is drift monitoring, built decades before anyone needed it for a chatbot.

And standardized essay grading, the machinery behind every large-scale exam, long ago built distributed rater pools, calibration sets fed to graders before live scoring, double-scoring with adjudication when two raters disagree, and lately an automated scorer running alongside the humans with its agreement to them monitored. That last one is LLM-as-judge, with the operational scaffolding already bolted on and fifty years of practice behind it.

When you catch yourself asking “is this good?” about an output that varies and has no single right answer, the field that has answered that question for fifty years is medicine, or sensory science, or competitive judging. Wine has calibration standards. Gymnastics drops the high and low score and decomposes the rest. Clinical trials have independent panels and analysis plans written before the data arrives. Ask who has graded your kind of problem for half a century, then go read what they built.

Two things borrow cleanly. The first is calibrating the judge against a human: you validate your scorer against a person you trust before you let the scorer’s numbers mean anything, the same inter-rater check a clinical study runs on its raters.

The second aspect to borrow is recognizing the difference between a gate that blocks and a gauge that gets read. A gate asks whether a thing is present or absent, and a script can run it. A gauge asks how good, across a spread, and it needs a calibrated reader. Confuse the two and you get the worst of both: force voice into a pass/fail check and you’ll set it so loose it never catches anything, or so tight nothing ever ships and your own builders start routing around their own test.

Which brings me back to the voice score that reads 4.1/5 on the first turn and verges on the esoteric by the fifth. The binary check was never going to tell me whether the voice was good — no more than a check would help customer service agents who silently renegotiate their instructions after weeks of use. In judgment, the deciding factor is whether the person on the other end can trust what they’re reading. Trust is no longer a layer on top of the product; it is the product. And judging trust — reading the gauge — is no longer a layer on top of the work; it is the work.

I write Close Tack to give voice to the patterns leaders miss — the ones they're moving too fast to notice.

Click below to get the next one in your inbox.