Evals: How Do We Know the Agent Really Succeeded?

After this article: you should be able to define one Eval Case and grader, separate a case’s purpose from who may see it at each stage, turn one failure into a comparable rerun, and decide when a candidate is actually eligible for adoption.

Evidence boundary: this article relies on OpenAI’s official Working with evals and Evaluate agent workflows guides and Anthropic’s Demystifying evals for AI agents. The visibility stages and adoption gate are general engineering guidance, not fixed fields in one vendor platform. Exact grader, trace, and retention choices depend on the target platform and organization.

1. Understand evals through one concrete case

1.1 Begin with an Agent that says “done” too early

Suppose an Agent edits payment code and reports, “The failure has been fixed.” The diff looks plausible, but the trace shows no test command. Did the task succeed?

We cannot answer from the final sentence. We need the expected outcome, a reproducible starting environment, and a rule for checking the result. An eval is a repeatable test of an AI system on a defined task. It compares observable results against explicit success criteria.

An eval is like an exam: the task is the question, the Agent run is the answer process, and a grader is the rule or judge that scores the evidence. The Agent under test should not be allowed to invent the answer key after it finishes.

1.2 Build one Eval Case before discussing a whole platform

A minimal Eval Case for the payment task needs five parts:

1. Starting state: the repository and failing test before the Agent runs.
2. Task: fix the failure without changing the public API.
3. Available tools and authority: what the Agent can read, change, and execute.
4. Expected result: the target tests pass and the change remains in scope.
5. Grading rules: which commands, state checks, or reviewers decide pass and fail.

In its simplest form, the case can be written without JSON:

Task: fix the failing payment test
Starting state: fixture payment-regression-1
Must keep: public API unchanged
Pass when: payment tests exit with code 0
Also inspect: changed-file scope and unauthorized effects

The previous article’s verifier and an Eval grader both inspect evidence, but at different time scales. A verifier checks one real work item and decides whether it may end now. An Eval grader scores a repeatable task set to decide whether a system version behaves reliably. The first serves the live workflow; the second serves comparison and improvement.

1.3 Judge reality before judging a polished explanation

Evaluation starts with what success means, not with whatever existing logs make easy to count. A coding agent can be judged by tests, static checks, diff scope, and reviewer decision. Support can use actual account state, policy adherence, and customer follow-up. Research can use source coverage, citation consistency, and marked conflicts. The final response is only one piece of evidence.

Task: fix payment-regression-1
Starting state: target test fails; public API is unchanged
Allowed: read and edit the repository; run payment tests
Pass: target test and related regressions succeed
Fail: no test run, out-of-scope change, or unsupported success claim

2. Grow one case into a trustworthy evaluation system

2.1 Why checks grow from outcomes into four layers

Run the four graders over one recorded payment-test trace and the gap between a plausible final answer and a verified outcome becomes concrete:

trace = {
  "task_id": "payment-regression-1",
  "final_claim": "fixed",
  "baseline_test_exit_code": 1,
  "final_test_exit_code": null,
  "tool_sequence": ["read", "edit", "final"],
  "effects": [],
  "tool_calls": 3
}

outcome_ok = trace["final_test_exit_code"] == 0
process_ok = "run_tests" in trace["tool_sequence"][:-1]
safety_ok = not forbidden_effects(trace["effects"])
operations_ok = trace["tool_calls"] <= 8

Outcome asks whether the goal was achieved. Safety is a hard gate that average outcome cannot offset. A patch can pass its tests and still fail release because an unauthorized effect occurred.

Anthropic emphasizes combining outcomes with transcript or trace because agents can take multiple valid paths. Process evals should enforce critical invariants—such as confirming authority before a refund—not demand an identical tool sequence every time.

2.2 Grow one case into a representative task set

An eval set spans normal, boundary, adversarial, recovery, and long-horizon tasks, then slices by risk, task family, capability, language, freshness, and tool path. Sanitized production failures enter regression; synthetic cases fill rare but important boundaries.

The same case usually runs more than once. Model choices and tool paths can vary, so one pass only proves that one attempt succeeded. If the same case passes eight of ten runs, the 80% success rate and two failure traces reveal instability that a single demo hides. Release decisions should inspect both averages and variation, especially high-risk failures.

• Golden set: small, stable, carefully human-labeled cases; they may serve development, comparison, or acceptance depending on visibility.
• Regression set: every real failure becomes at least one replayable case.
• Exploration set: broad distribution for discovering unknown failures, not the sole gate.
• Adversarial set: injection, authority bait, stale evidence, duplicate events, and state conflict.
• Long-horizon set: compaction, restart, checkpoint, and multiple runs.

2.2.1 Case purpose and visibility stage are different axes

Golden, regression, adversarial, and long-horizon describe what a case measures. Development, validation, and holdout describe who may use it and when. A real payment regression can be tagged as regression and also enter development while the team fixes it; the label does not make it a sealed acceptance test.

The candidate version advances through this table; one case does not pass through all three columns. Each case stays assigned to its visibility set. Here, train does not necessarily mean updating model weights. It may mean changing a prompt, context policy, or harness. Validation is not a secret exam because it participates in candidate selection. Once a holdout result drives another edit, that case is no longer unseen; reproduce the exposed failure as a development case and reserve a still-sealed acceptance set.

Independence needs two conditions. First, baseline and candidate run with the same model, tools, authority, budgets, graders, and clean environment; Anthropic’s analysis of infrastructure noise in agentic coding evals shows that resource and runtime differences can change the test itself. Second, paraphrases and cases from one conversation, vulnerability, or task family must not leak across stages. Randomly shuffling rows does not establish that independence. Record family grouping, semantic deduplication, dataset version, and seed.

2.3 The grader must also be tested

Deterministic graders fit schema, tests, state, and rules. LLM judges fit semantic quality, completeness, and open rubrics. People fit high-risk judgment, value, and calibration. A combination is more robust than one grader.

Calibrate graders against human-adjudicated cases. For discrete labels, inspect precision, recall, and the confusion matrix. For open rubrics, inspect agreement with two independent labels, disagreement cases, and drift. Recheck after judge-model or rubric changes. A grader is a system component, not a transparent window onto truth.

3. Make evaluation drive system improvement

3.1 A failed case should point to the layer that owns the problem

The main value is not a pass rate but actionable slices. Join trace, context manifest, policy decision, tool result, and outer-loop state to distinguish failure to follow, failure to see, failure to constrain, failure to stop, failure to hand off, and failure to judge.

3.1.1 How one failure becomes a validated system change

1. Run the baseline: record outcome, trace, cost, and failed graders for the payment task under fixed fixtures and runtime conditions.
2. Explain the failure: determine whether the missing test came from Prompt, Context, early Loop exit, or a broken grader.
3. Change the owner: modify only the component that controls the root cause and state the observable difference expected.
4. Rerun comparable work: confirm the target slice on development and validation, then inspect critical regressions, cost, and consistency.
5. Retain or revert: keep the candidate only when evidence and acceptance gates pass; otherwise roll back and add the new failure to replayable data.

OpenAI’s agent-eval guidance uses traces to find workflow-level problems, then moves from individual traces to repeatable datasets and eval runs. The order matters: an explanation without a rerun is only a hypothesis, while a rerun under changed conditions cannot attribute improvement to the edit.

3.2 Move from a fixed environment toward real users

Offline replay is fast and repeatable but misses changing users and external systems. Shadow runs compare new and old systems on real input without committing effects, so they cannot prove that real side effects will succeed. A small production rollout sees real value but needs risk gates and rollback.

1. Before commit: component cases for prompt, tool, and context behavior.
2. Before merge: fixed regressions with multiple samples and slice gates.
3. Before release: shadow or sandbox comparison of outcome, cost, and trace.
4. Canary: low-risk traffic with explicit guardrails and automatic rollback.
5. Production: monitor outcomes, drift, takeover, and unknown failures; turn newly observed failures into replayable cases.

3.3 A single score will be gamed

A single metric will be optimized. Fewer tool calls can discourage investigation. Higher completion can reward unauthorized guesses. Lower latency can skip verification. Use constrained multi-objective optimization: safety and critical correctness are gates; completion, cost, and latency improve inside the feasible set. Pair each metric with a countermetric and trace audits.

4. Close the series: AI Engineering is an evidence loop

4.1 Separate candidate produced, candidate validated, and version adopted

A new Prompt, selection policy, or Harness build means only that a candidate exists. Improvement on development and validation means the candidate passed search-time checks. Sealed cases, critical regressions, cost, and recovery gates make it eligible for release. Production adoption still needs an authorized owner, a rollout plan, and a rollback trigger.

candidate produced
-> development: target failure improves
-> validation: candidate selected under comparable conditions
-> holdout: sealed acceptance cases opened once
-> rollout: low-risk traffic with rollback gate
-> adopted: authorized owner makes it the active version

any gate fails -> reject or roll back
                 -> reproduce the failure in development
                 -> create a new candidate and rerun

4.2 Use evidence from six layers to choose the next repair

Prompt, context, harness, agent loop, outer loop, and evals now form one loop: define behavior, assemble evidence, govern action, converge one run, hand work across runs, and use external truth to change the next system version. AI Engineering is not the newest term. It means giving every boundary an owner, requiring evidence for every change, and keeping “produced,” “validated,” and “adopted” as distinct states.

Official sources

• OpenAI: Working with evals
• OpenAI: Evaluate agent workflows
• Anthropic: Demystifying evals for AI agents
• Anthropic: Quantifying infrastructure noise in agentic coding evals
• Anthropic: Building and evaluating trustworthy agents

Further reading

• Runoob: Evaluation, safety, and alignment