Evals: the unglamorous work that separates systems from demos
An eval set is a collection of real inputs with known good answers, run against your AI system automatically on every change. It is the closest thing LLM engineering has to a test suite, and its presence is the fastest way we know to tell a production system from a demo. When we meet an AI project in trouble, the first question is not about the model. It is: how do you know whether yesterday’s change made the system better or worse? Teams with an answer have evals. Teams without one have opinions.
This note is about LLM evaluation for production business systems, not academic benchmarks: why hand testing collapses, what an eval set looks like when it is not a research project, and how much of this effort your system actually needs.
Why "we test it by hand" fails at the second prompt change
Hand testing works exactly once. Before the first release someone tries twenty inputs, the outputs look good, everyone moves on. Then the first complaint arrives and someone adjusts the prompt to fix it. Does the fix break any of the original twenty? Nobody rechecks, because rechecking is tedious, so the honest answer is: unknown. By the fifth change you are steering a system whose behavior no one can describe.
LLM systems punish this harder than normal software because changes are not local. A sentence added to improve refund questions can degrade warranty questions, and there is no compiler, no type system, nothing that catches the regression except a test you run on purpose. There is also a selection problem: the person testing by hand built the system, and tests the inputs they imagined. Users specialize in inputs nobody imagined.
What an eval set actually looks like
Smaller than teams fear. For a typical business system: 50 to 200 examples drawn from real tickets, real invoices, real questions, each paired with what a good output looks like. A spreadsheet and a script are a legitimate version one; the tooling can grow later.
Three things matter more than size. The examples come from production reality, not from a workshop whiteboard. The set includes the ugly cases: ambiguous requests, incomplete inputs, questions out of scope, questions the system should refuse to answer. And the set grows by a standing rule: every real failure found in production becomes an eval case, permanently. Each bug gets one free occurrence.
Grading without a research lab
Structured outputs grade themselves: an extracted invoice total either matches the known answer or it does not, which is one reason we push designs toward structured outputs wherever possible. For open-ended answers, a second model can grade against a rubric you wrote: did it cite a source, did it stay in scope, is the recommendation one of the allowed ones. Judging models with models sounds circular, and it works in practice if you calibrate it: have a human grade a sample, check that the judge agrees often enough, and recheck that agreement periodically. Keep genuine human review for the small set of cases where nuance is the product.
What evals change about iteration
Everything about its pace. Each change, whether prompt, retrieval logic or model version, runs the set before it merges, so regressions surface before users find them. Model swaps become an afternoon instead of a leap of faith: when a cheaper or better model ships, you run the set, read the diff and decide with numbers, which also changes how you negotiate with vendors. Debugging gets targeted: when a RAG assistant starts answering wrongly, the eval set localizes whether retrieval broke or generation did. And arguments end faster, because "I think it got worse" becomes a score.
How much evaluation a production system needs
Proportional to stakes, and you can start in a day: pull 50 real examples, define good, script the run, record the baseline. An internal assistant with a human reading every output needs modest coverage. A system that acts on its own, faces customers, or lives inside a regulated flow needs more cases, stricter rubrics and a higher bar before changes ship.
The failure modes sit at the two extremes. Zero evals means steering blind, which is how systems rot in the dark. Eval perfectionism means six months building a measurement harness before shipping any value, which kills projects just as reliably. The 50-example spreadsheet beats both.
Evals are also what the rest of the discipline stands on: they are the evidence that lets a prototype graduate to production honestly, and they pair with monitoring once the system is live as the offline and online halves of knowing what your system does. In our custom AI builds, the eval set starts in week one, on the client’s real data, because a system you cannot score is a system you cannot improve.