Every year, a manager sits down in December and scores twelve months of someone’s work.
The technology industry has spent the last two years learning that a judge scoring open-ended work is unreliable until it’s been calibrated. The lesson arrived because we started building AI systems that were open-ended (you don’t know exactly what a chatbot will respond with) and evaluating the range of open-ended possibilities required grading.
That grading turned out to be harder than most people expected. What almost nobody has pointed out is that the same problem describes a process companies already run: the performance review.
A standard that drifts
Essays of similar quality can score differently based on a slow drift, even with a consistent rubric. In the performance review, particularly in organizations where entire teams are graded at once, the standard under which someone is evaluated can drift at two stages, and neither one is typically checked.
The first stage begins when the manager sits down and starts drafting her team’s reviews. One type of drift happens through contrast. A manager who has just written three reviews for a strong team writes the fourth against a bar the prior three quietly raised (the fourth person isn’t scored against the rubric, they’re scored against whoever came before them). Another type comes from fatigue. The manager writing her eighth review at eleven at night is not the same judge who wrote the first (the rubric hasn’t changed, but the person applying it has). These scores arrive at the calibration meeting already drifted, and nobody unpacks them.
The second stage is the calibration room, which is the step most companies believe is the fix. Yet it has its own drift problem. I spent years in these rooms, once watched a VP ask the right question: “How do we know we’re rating the same way we did six months ago?” Nobody had an answer, because the system wasn’t measuring it. The room calibrated at the end of the cycle and never checked whether its own standard moved during the year, which means the process designed to catch drift is subject to the same failure it’s supposed to prevent.
Every one of these has the same fifty-year-old fix (borrowed from a standard practice in large-scale educational assessment): feed known-score papers into the flow mid-session and measure how far the standard has slid.
When I graded college papers as an adjunct, I found myself applying a version of this by necessity: grade the 12th paper, then go back and re-grade the 1st with the score covered up. If my score on the re-grade was different from what I’d initially assigned, I knew my standard had shifted between the 1st and 12th paper. The AI evaluation community adopted the same method: inserting known-score outputs into an LLM judging run and flagging when scores drift from the expected mark. Most performance reviews have neither version.
Drift moves the standard without the judge noticing. The next failure leaves the standard steady, but measures the wrong thing.
When the judge measures the wrong thing
In a 2026 study, a University of Cambridge-led team tested three frontier AI judges on over 750 undergraduate essays from three UK universities, already graded by human examiners. All three systems were oversensitive to linguistic features, awarding higher marks for length, vocabulary range, and sentence complexity, regardless of the academic quality of the argument underneath. The judge systematically credited the surface of sophistication over the substance.
Put a manager in that sentence. The articulate self-advocate outscores the quiet high performer. Visible activity outscores quiet impact. The confident narrative outscores the correct story, because there is no independent way to check. Research in applied psychology (Scullen, Mount & Goff, 2000) has found that more than half the variation in performance ratings is attributable to the rater, not the person being rated. Ratings tell you more about the judge than the judged.
The only thing that catches a judge who consistently scores the wrong feature is a second, independent rater whose disagreements you take seriously: someone who lands on a different score for the same work and makes you explain the gap. Most calibration panels don't do this. The manager presents their rating, the room reacts to it, and the number that was written at the desk anchors the outcome.
What to do before your next review cycle
None of these require a new system, a new form, or HR approval. They require a director who decides, before the cycle opens, that the instrument is worth checking. Without that, the calibration exercise ossifies into a distribution exercise: identify your bottom ten percent, assign the names, move on. Nobody reads the rubric anymore.
Score a sample yourself before the meeting. Score a sample yourself before the meeting. Pull four or five performance summaries from the previous cycle — cases the room already calibrated and finalized. Strip the names and scores, rate them against the rubric fresh. If your scores match what the room agreed to last time, your standard held. If they don't, it drifted between cycles, and you know before the new round starts. Then do the same with a handful of this cycle's summaries to catch drift within the current session.
Treat your calibration room like a multi-judge panel. The room already has managers from different parts of the company — that's its strength. The problem is how it runs: one manager presents their rating, the room reacts, and the original score becomes the starting point. Flip the order. When a flagged case comes up, present the work summary before revealing the employee's name, their manager, or the score. Managers from different functions carry different assumptions about what "impact" looks like, and the disagreements (when they score independently first) reveal which features each judge is actually weighting. A panel that hears the first judge's score and works from there is just one judge with an audience.
Check whether your own managers are calibrated to each other. Take a handful of the anonymized summaries from your prep and hand the same ones to two or three of your managers. Have them score independently against the rubric. Count how often they land within a point. Your managers report to you and presumably share the same standards — but do they? The AI community runs this test routinely before trusting any judge: if two judges can’t agree on the same work, you don’t average their scores and move on. You figure out why they disagree. If your managers are calibrated, you’ll see it in the numbers.
Every company that would not dream of shipping a product on an uncalibrated measurement runs its entire compensation and promotion apparatus on a judge nobody has ever validated. The tech industry built the calibration apparatus for the machines within eighteen months of needing it, because the scores were going into a product and somebody would notice.
The scores from the December review go into someone’s mortgage. Run the calibration before you sign off on them.