AI Agent Evals: How to Tell If Your Agent Actually Works
I never ship an AI employee without an eval set. Here is how I turn "it feels worse lately" into a number I can act on.
An AI agent eval is a systematic test that measures how well an agent performs on a specific task. Each eval pairs a task (a scenario you put the agent in) with a grader (logic that scores the result), turning a vague "it feels worse lately" into a number you can act on. You build evals by collecting representative tasks, writing graders (deterministic code for objective checks, an LLM-as-a-judge for quality), and re-running them on every change, so you can prove an improvement instead of guessing at one.
That is the whole discipline in one paragraph. The rest of this article shows how it works in practice, using one concrete example I keep coming back to: a slide-generation agent, improved through five versions, where each change is driven by what the evals revealed. I come at this from a factory background (mechanical engineering, then automation work at Bosch) where the rule is simple. Inputs must lead to predictable outputs. If you can't measure the output you don't have a process, you have a hope.
Why does measuring AI agents matter now?
Most teams can now build an AI agent. Far fewer can answer the only question that matters once it's running: is it actually working, and did the last change make it better or worse? That gap has a cost. Gartner forecast in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. In my experience, the projects that survive are the ones that can measure themselves. The ones that quietly fail are the ones that drifted because nothing was watching.
Evals are how you watch. They are the bridge from "it seems to work" to "we know it works, and here is how we know." When I build an AI employee for a client, the eval set ships with it on day one. You don't need another AI subscription. You need AI you can actually measure, and no measurement means no handover.
What is an AI agent eval?
Strip away the jargon and an eval is two things bolted together:
- a task: a specific scenario, like "generate a five-slide deck on salary negotiation," and
- a grader: logic that encodes your expectation and judges the result, like "there are exactly five slides" or "the text is readable, scored 0 to 5."
Put simply, an eval is a unit test for output that is not deterministic. A normal unit test checks that 2 + 2 returns 4. An eval checks something softer and harder to pin down: is this slide deck any good? It turns the answer into a score you can track across versions. Without that, quality lives in your head as a feeling. With it, quality becomes a value: comparable over time, attributable to a cause, and impossible to argue with when a change makes things worse. If you want the wider picture of what reliable agents look like, I wrote about that in trustworthy AI agents.
Why aren't public benchmarks enough?
Public benchmarks measure generic capability. Your specific job is a different question. When a new model ships, it arrives with a scorecard: SWE-bench Verified for coding, OSWorld and τ-bench for tool use, ARC-AGI for reasoning. These are evals too, and they are genuinely useful for comparing models against each other on generic capability. SWE-bench Verified, for instance, gives an agent real GitHub issues and grades a fix by running the test suite; Anthropic notes that models climbed from roughly 40% to over 80% on it in about a year.
But a public leaderboard tells you almost nothing about your use case. Your agent doesn't do "coding in general." It builds your slide decks, triages your leads, drafts your newsletters, all against your standards and your idea of done. A model that tops a benchmark can still produce output your users reject, because the benchmark measured a different job. The rule every serious team converges on: build your own evals, and benchmark the models against your work rather than the other way around.
What are the three types of graders?
A grader is simply how you judge an output, and you have three families that trade off against each other: deterministic code, an LLM judge, and a human expert.
Code graders are deterministic logic: a string match, a regex, a count. Did it produce a file? Are there exactly five slides? How many emojis appear? These are fast, cheap, and perfectly repeatable, but brittle and blind to nuance: they can confirm a deck exists, not that it's good. Default to them whenever an answer is objectively checkable; LLM judges on objective tasks tend to be needlessly unreliable.
Model graders, the LLM-as-a-judge, score an output against a rubric: "rate this slide's readability from 0 to 5." This is the only practical way to grade soft qualities like coherence, layout, and taste. It handles judgment calls that no count can reach, and it runs across hundreds of cases without you watching, at the cost of being non-deterministic, costing money, and needing careful calibration. The technique is more reliable than it sounds: OpenAI reports that strong LLM judges can reach over 80% agreement with human evaluators, about the same level humans reach with each other. Two refinements raise the ceiling further: pairwise comparison (ask which of two outputs is better, since models judge relative quality better than absolute scores) and multi-judge consensus (several judges vote, majority wins, buying back some determinism).
Human graders are subject-matter experts reviewing output directly. They catch what no model or count will, but they are the slowest and most expensive, so you use them least: for spot-checks, A/B comparisons, and, critically, to calibrate your model graders. Anthropic's guidance is that once a system is reliable, occasional human review is enough. That is also the point where you can stop babysitting the agent and let it run.
The craft is matching the grader to the question. "How many words are on this slide?" is countable, so use a code grader. "Is the text overlapping or spilling off the edge?" is a judgment call, so use a model grader or a human eye.
How do you build evals? A worked example
The clearest way to learn this is to watch it on a real agent. Take a simple one: an agent whose entire brief is "you are a slide-generation agent; when the user gives you a topic, create a PowerPoint file; you have a shell with python-pptx." Here is the loop, version by version. (The scores below come from a public Anthropic engineering session's live demo. They are illustrative of the method, not formal benchmarks.)
Version 1, the baseline. Run it and you get five slides, technically: overlapping text, scattered emojis, the same teal on every slide. No one would send it to a client. This is the moment to ask the question that builds an eval set: looking at what failed, what should we measure? The failures write the list: code graders for slide count, emoji count, shapes per slide (a clutter proxy), tiny fonts, text-heavy slides; model graders for color, image, layout, and text quality, each scored 0 to 5. Run them, and the first lesson lands immediately: that genuinely poor deck scored between 2.8 and 4.0 on the judges. Look at the deck, then the scores, and they don't match. The graders aren't generous because the agent is good; they're miscalibrated. An uncalibrated judge will lie to you politely.
Version 2, teach it taste. Add typography and density rules to the prompt: font sizes, breathing room, left-aligned text, and explicitly "avoid the AI tells": no decorative emojis. The decks get visibly cleaner. But the scorecard now reports an emoji count of 20 on a deck with none visible, and a "text-heavy" flag on slides a human would call fine. The agent improved; the eval revealed its own bugs. So you fix the grader, not the agent, which is the move most teams miss.
Version 3, demand evidence. Require a real chart on every slide. The decks become grounded instead of waffling through bullets. Better, but the image judge returns "3.8" and nothing else. A bare number you can't act on, which is a flaw in how the judge was asked (more on that below).
Version 4, make it check its own work. Add a QA loop, instructed adversarially: assume there are problems; your job is to find them; approach QA as a bug hunt, not a confirmation step; render the deck to images, inspect every slide, fix, re-render, and don't stop until you've completed at least one fix-and-verify cycle. The agent becomes its own critic, and the judges climb to 4.2 to 4.4.
Version 5, change the engine. Throw away the elaborate prompt, return to the simple one, and run it on a stronger model. The output is the best of all five: no emojis by default, readable fonts, uniformly high scores. Two lessons land here. Sometimes the right lever is the model rather than the prompt, and your evals are exactly what tell you so. And when every judge scores everything high, your evals have stopped discriminating; they've saturated, which is the cue to make them harder.
What mistakes make evals lie to you?
Building evals is easy. Building evals you can trust is where the discipline lives. Three failure modes account for most of the pain.
Vague rubrics. "Rate this 0 to 5" gives a judge nothing to anchor on, so it drifts to a polite middle. That is exactly why that first bad deck scored 3.5. The fix is to anchor the rubric with examples: show the judge what a 0 looks like and what a 5 looks like. This calibration is fiddly and slow, and it is where most of an eval set's quality actually comes from.
The ordering trap. If you ask a judge for a score and then its reasoning, you get a worse score. Language models are autoregressive (each token is shaped by the ones before it), so a judge that commits to "4" first will spend its reasoning defending the 4, however bad the output. Flip the order: ask for reasons first, then the score. Anthropic and LangChain both recommend having the judge reason before it grades, precisely because the number then becomes a conclusion instead of a rationalization. As Anthropic puts it, "Don't grade the path the agent took, grade what it produced."
Saturation. Evals are a living artifact, not ground truth you set once. As the agent improves, graders that once separated good from bad stop discriminating, and everything passes. That's saturation: the eval no longer gives you information you can act on, and it's time for a harder one. Budget for refreshing evals, the same way you budget for building them. For high-stakes judgments with no clean right answer, like summarizing a legal case, combine multi-judge consensus with adversarial refutation (one agent surfaces issues, a second tries to refute each), because graders hallucinate too. I dig into that failure pattern in confidently wrong.
How many eval examples do you need?
Fewer than most teams expect. Anthropic's guidance is that 20 to 50 examples is often enough to start, because early changes have large effect sizes and small sets are cheap to re-run. Begin with one workflow you care about, an honest answer to "what does good look like?", and the smallest eval that encodes it: a code grader for the thing that must always be true, and one calibrated judge for the thing that matters most. Run it before and after your next change. That single number, moving in the right direction, is the entire idea. Grow toward a hundred or so cases as the workflow goes to production.
Frequently asked questions
What is an eval in AI?
An eval is a systematic test that measures how well an AI system performs on a specific task. It combines a task (the scenario) with a grader (logic that scores the output), producing a number you can track over time. Evals are the standard way teams turn subjective impressions of quality into measurable, repeatable signals.
What is the difference between a code grader and an LLM-as-a-judge?
A code grader uses deterministic logic (string matches, counts, regex) and is fast, cheap, and perfectly repeatable, but blind to nuance. An LLM-as-a-judge uses a model to score output against a rubric, which is the only practical way to grade subjective qualities like coherence and layout, at the cost of being non-deterministic and needing calibration. Use code graders for objectively checkable things and model graders for quality judgments.
Are LLM-as-a-judge evaluations reliable?
They can be, when calibrated. OpenAI reports that strong LLM judges reach over 80% agreement with human evaluators, comparable to the agreement between two humans. Reliability depends on anchoring the rubric with examples, asking the judge to reason before it scores, controlling for known biases like position and verbosity, and validating against human judgment periodically.
How many examples do I need to start evaluating an AI agent?
Often just 20 to 50. Anthropic's guidance is that small eval sets are enough early on because changes have large effect sizes and the sets are cheap to re-run. Start small with the cases that matter most, then grow toward roughly 100 as the agent moves to production.
Why aren't benchmarks like SWE-bench enough?
Public benchmarks measure generic capability and are useful for picking a model, but they don't measure your specific use case, standards, or definition of "done." A model can top a leaderboard and still produce output your users reject. You need your own evals, run against your own work.
Sources and further reading
- Anthropic: Demystifying evals for AI agents (engineering guide).
- OpenAI: Evaluation best practices.
- Gartner: Press release, June 2025: over 40% of agentic AI projects will be canceled by end of 2027.
- SWE-bench (Jimenez et al., Princeton, 2023): swebench.com.
- The worked example is drawn from a public Anthropic engineering session on evals (slide-generation agent). Scores quoted are from that session's live demo and are illustrative of the method, not formal benchmarks.
Mid-market AI workflows rarely fail because the model wasn't capable. They fail because nothing measured whether the workflow was doing its job, so it drifted quietly until someone noticed. That's why every AI employee I run ships with its own eval set: code graders for what must always be true, calibrated judges for quality, and a refresh cadence so the evals don't go quiet. We keep the system honest, the same way we run our own agency on it. There's one honest boundary in this. If you can't define what "good" looks like, if you can't write the eval, that use case isn't ready to ship, and I'll tell you so. If you want help drawing that line for your own workflows, start a conversation.
Christoph Sauerborn is the founder of Brixon AI. He builds AI employees for capacity-constrained service firms, and runs his own agency on them. Mechanical engineer by training (RWTH Aachen), former Industry 4.0 engineer at Bosch. More about how I work.