An evaluation usually asks a smaller question than the one people later attach to it. For a customer support assistant, that question is often simple: did the response match the expected answer? The operational question is messier. Did the customer solve the problem? Did they come back angry? Did the answer create work for a human team later?
Those are different questions. A support assistant can answer common password-reset questions well and still fail on the messy account problems that create the most escalations. If the evaluation counts both misses the same way, it has already made a product decision without saying so.
That is how a correct evaluation becomes misleading. It answers the question built into the test, then the score gets treated as evidence for something broader: launch readiness, business risk, customer impact, operational load.
The problem is not measurement. The problem is forgetting that every metric defines “good,” and that definition may be much narrower than the decision it is being used to support.
The easy thing to measure is rarely the thing that matters
Every evaluation answers an easier question than the one the team actually needs answered. That does not mean the team was careless. The cheap question is local and immediate: did this output match this reference answer on this case? The harder question shows up later: did this answer solve the problem, push a customer toward churn, or concentrate failures where failure hurts most?
Those questions pull in different directions. The gap does not close because the team runs the same evaluation more carefully. It closes when someone notices the substitution and limits what the test is allowed to claim.
That is what happens in the support assistant example. The 98% was not fake. It was an honest answer to the wrong question.
Choosing what to measure is deciding what “good” means
A metric is not a neutral instrument pointed at a system. It is a definition, fixed in advance, of what counts as success. Somebody chose that definition, even if it did not feel like a choice at the time.
“98% accuracy” sounds like a fact about the model. It also depends on an earlier choice: every support case in the evaluation set counts equally, and answering ninety-eight out of a hundred is “good” regardless of which two went wrong. Nobody sat down and said, “we have decided that mishandling a confused enterprise customer is the same as giving a slightly clumsy answer to a routine FAQ.” The evaluation made that trade anyway by treating every case as interchangeable.
A team that asks, before computing any number, “what does a miss cost us, and does that cost vary by case?” builds a different evaluation. Not a tidier version of the same one. A different one.
What you measure decides what looks good
A customer asks about a duplicate invoice charge. The assistant sends the standard refund-policy answer and never addresses the duplicate. Same response, two ways to evaluate it.
What they measured
Did the response match the approved answer?
The wording matched the refund-policy template, so the response passes. Looks correct. But the evaluation only checks wording. It never asks whether the customer's actual problem got solved, so that gap stays invisible.
What mattered
Did the customer actually solve the problem?
The customer reopened the ticket. Enterprise account, open billing dispute. The answer never explained how to reverse the duplicate charge. The problem was not solved. On a routine FAQ this might be forgettable, but a high-value account with an unresolved charge is a different story.
The insider problem
There is another reason the substitution goes unnoticed. When the people deciding what counts as “good” also built the system, the evaluation can start to confirm the build instead of testing it.
This is not dishonesty. It is closer to a blind spot. A team that built a content moderation classifier absorbs, over hundreds of small implementation decisions, a working sense of what a “miss” looks like. That sense can narrow until it matches the cases the system already handles well. Nobody sets out to define misses narrowly. They just spend months looking at the system through the habits it has taught them.
An evaluation designed by someone who was never in the room tends to ask a different question. Not always a smarter one. Just a less familiar one, which is sometimes exactly what the system needs.
Same score, different eyes
The builder, the reviewer, and the statistician look at the same 98% label match. They notice different things.
Held-out evaluation
98%
label match
Reviewer view
Which misses share the same cause, and are any of them expensive?
Checks the miss distribution before anyone uses the score to decide what ships.
Evaluation design is experiment design
Designing a valid evaluation is closer to designing an experiment than writing a test suite. Many engineering teams were never trained for that, because it is not really an engineering skill.
A real experiment starts by deciding what would have to be true for the result to mean something. What is held constant? What would count as evidence against the assumption being tested? What can the result claim once it comes back? Assembling examples and computing a pass rate produces a number. It does not make that number meaningful just because someone ran the calculation.
This is why adding more test cases rarely fixes the problem. More cases answer the same question with more precision. They do not make it a better question. The support team could have evaluated ten times as many conversations and arrived at the same 98%, with a tighter interval around the same proxy.
Why this requires having seen it before
This pattern is easy to describe in hindsight and hard to catch in advance. Recognition comes from having seen it before. A team designing its first evaluation has one main data point: its own system. So it discovers each failure mode for the first time in production, after the cost has already landed.
A customer support system scores well on “did this response match an expected answer” and still drives customers to escalate in frustration. A content moderation classifier hits its precision target while missing a category of harm nobody thought to test for. A lead-routing model sends valuable edge cases to the same queue as routine requests. Inside one company, these look unrelated. After you have seen enough of them, they start to look like the same few mistakes wearing different clothes.
One-off mistake or systemic failure?
Mark whether the incident is an isolated test issue or part of a repeated evaluation failure.
1 of 4
Incident
A support system matched the expected answer. Customers still opened follow-up tickets because they could not act on the response.
The fluency the role actually requires
Put the last few points together and the role gets awkward. A good evaluation needs statistics, product judgment, and domain risk in the same conversation. Most career paths train people in one of those. An engineer may know how to build a statistically sound test and still not know which two percent of cases would hurt the business if missed. A product leader may know exactly what failure costs and still struggle to turn that into a checkable criterion. A measurement specialist may be strong on confidence intervals and weak on both the product and the domain.
Teams that are strong in one area can still be blind in the others. The gap often stays hidden until production exposes it, when fixing it costs the most.
The question worth asking before you trust a score
This is not an argument against measurement. It is about what happens after the score comes back. A number from an evaluation is evidence for a specific claim. You still have to know what that claim is, and whether it was the one you needed.
Before the next score gets treated as settled, ask a smaller question first: what would this number have to mean for us to trust it this much, and who decided that meaning?
That question does not have one right answer. It does have an answer every time. Most teams never ask it.