step 14 / 19Part IV Agents

S14 Agent Evals, Data and Optimization

What this chapter covers

S8 Evals Methodology covers single answers, and S12 The Agent Loop and Harness introduced a minimal agent eval. Here you will evaluate whole trajectories, repeated trials, simulated users and environment settings. Even machine resources can shift scores by several percentage points.

Once the eval is trustworthy, use traces to find failures, build a data flywheel, optimize prompts, distill a small router and test harness components through ablation. S15 Multi-agent Systems will use these evals to decide whether multiple agents help.

After this chapter you can

When reading a benchmark score, name the reasons it might be off (flawed tasks, contamination, infra settings, eval awareness). (lecture and self-check; the core exercise measures one infra variable)
Design “outcome first, then trajectory” evaluation for an agent, and use a transition failure matrix to find the step where it most often breaks. (core)
Build a rerunnable agent eval with isolated trials, a balanced task set and simulated users, and report the environment variables. (core; simulated users in the advanced exercise)
Run an ablation on a harness component and use a confidence interval to judge whether it’s worth keeping. (you can do one as part of the core exercise; the full version is in the advanced one)
Run one round of prompt optimization with a DSPy-style optimizer and validate on data that wasn’t used for optimization. (advanced)

Lecture

First, the terms. When you evaluate an agent:

Task: an input plus a success criterion, e.g. “Write the names of all notes containing TODO into todo.txt.”
Trial: one attempt by the agent at the task. The agent gives different results each time, so you run the same task several times.
Environment: where the agent acts, e.g. a copy of a notes folder, a set of tools, a database.
Trace / transcript: the complete record of one trial, including every tool call, its arguments and result, and the final reply (← tracing in S6 LLM APIs and Application Basics).
Grader: the code, judge or person that decides whether a trial succeeded.

The running environment comes from the S12 The Agent Loop and Harness core exercise: 20 Markdown notes, exactly 6 containing TODO, and five tools: list_files, read_file, grep, write_file, delete_file. This simplified notes assistant uses only file tools, so code can check each task’s result.

The TODO task illustrates grading one trial, combining repeated trials and controlling the environment. Later sections on trace clustering, prompt optimization and routers return to Sam’s full notes assistant.

★ What does agent quality cover?

Looking only at success rate misses a lot. Kaggle’s “Agent Quality” whitepaper divides agent quality into four dimensions (four pillars):

Dimension	Question it asks	Notes-assistant example
effectiveness	Did it achieve the user’s goal?	Does `todo.txt` contain exactly those 6 file names?
efficiency	How many tokens, how much time, how many steps, how much money?	Done in 3 steps, or did it read all 20 files one by one?
robustness	Does it hold up when tools fail, inputs are vague or the environment changes?	Does it retry after `grep` times out once?
safety	Did it do anything it shouldn’t?	Did it also delete files that “look like duplicates” while it was at it?

An agent with a 90% success rate can still be unsuitable for release if each task costs $10 or it sometimes deletes the wrong file. Kapoor et al. 2024 criticized agent research that reports accuracy without cost. Report the four dimensions separately, then choose hard thresholds for your product, usually including safety. A total score can make a cheap but unsafe system look equivalent to an expensive safe one.

★ Grade the outcome or the steps?

Grade mainly on the final outcome, and use the trajectory to find which step went wrong. This order is called outside-in: first treat the agent as a black box and only ask whether the goal was achieved; if it failed, open up the trajectory.

Why the outcome comes first. There’s usually more than one path to the goal, and rigidly checking steps marks correct solutions as wrong. And the outcome must be checked from the state of the environment, not from what the agent says: check the contents of todo.txt, not the agent’s “Done” reply. The example in Anthropic’s “Demystifying evals for AI agents”: for a booking task, query the database to see whether the booking exists.

Trajectory match compares the agent’s sequence of tool calls against a reference trajectory. There are four common ways to compare: exact match, same order (extra steps in between allowed), any order, and tool-call precision / recall. Precision is “what fraction of the agent’s calls appear in the reference trajectory” (repeated calls each count), and recall is “what fraction of the reference trajectory’s steps the agent performed.”

Walk through the TODO task. The reference trajectory is list_files → grep("TODO") → write_file(todo.txt), and here are two trials:

	Trial 1	Trial 2
Tool calls	`list_files` → `grep("TODO")` → `read_file(meeting-03.md)` → `write_file(todo.txt)`	`grep("TODO")` → `write_file(todo.txt)` → `read_file(todo.txt)`
Agent’s reply	“Wrote 6 file names”	“Wrote 6 file names and read them back to confirm”
Actual contents of `todo.txt`	5 file names (one missed when copying)	6 file names, all correct
Same order (extra steps allowed)	Matches	Doesn’t match (missing `list_files`)
Precision / recall	3/4 = 0.75 / 3/3 = 1.0	2/3 ≈ 0.67 / 2/3 ≈ 0.67
Outcome grader	Fail	Pass

By trajectory, trial 1 looks better; by outcome, it’s the other way around. Trial 1’s steps were all right; it went wrong copying the grep results into the file, dropping one, and its reply still claims it wrote 6. Trial 2 skipped the unnecessary list_files and even added a verification step, yet trajectory matching penalizes it.

When you still need trajectory match. Two cases: diagnosing failures (as with trial 1, where you need to know it broke at the copying step), and when the path itself is a requirement, e.g. “always ask before deleting a file” or “always confirm the recipient before sending email.” For the latter, a single rule checked against the trace is enough; you don’t need to compare against a whole reference trajectory.

A transition failure matrix (Hamel’s “AI Evals FAQ”) tallies failures by “the step where it broke.” First divide the flow into states, e.g. understand the task → find the target → modify → verify → report (states that aren’t needed can be skipped). Record one entry per failure: the row is the last state that succeeded, the column is the first state that failed. Say one version fails 20 of 75 trials:

Last success ＼ First failure	Find the target	Modify	Verify
Understand the task	4	1
Find the target		10	1
Modify			4

The cells sum to 4 + 1 + 10 + 1 + 4 = 20, equal to the number of failures. The biggest cell is “find the target → modify” with 10: it found the right files but wrote them wrong, the same kind of failure as trial 1. Fixing this first (e.g. requiring in the harness that it read back and compare after writing) is far more effective than a vague “tweak the prompt.” It adds location information to a plain “failure reason tally”: the same “wrong file name” calls for different fixes when it happens while finding the target versus while copying. This is the error analysis from S8 Evals Methodology extended to agents.

There are three kinds of grader: code checks (comparing file contents, querying a database), LLM judges (← judge calibration in S8 Evals Methodology) and agent-as-a-judge (Zhuge et al. 2024). In the last one, the evaluator is itself an agent: it can read files, run code and operate the product, then score based on evidence. It suits checking “the thing that was built,” for example whether a weekly-report note written by the agent has the right format and whether its links open. Use code wherever code can check it; align judges and agent judges with human labels first.

Trial 1 claims to have written 6 entries, but only 5 exist. A grader reading only the reply would incorrectly mark it successful. Check the environment.

Before you build: determining states from a trace

Write state detection as code that takes a JSONL trace and outputs “the last state reached successfully” and “the first failed state.” For example, the success condition for “find the target” could be: the result of some grep or read_file contains all 6 target file names.
Write each state’s success condition in advance. Don’t add them after you see failures, or the matrix will be swayed by your after-the-fact explanations.
Trials the grader marked successful don’t go into the matrix, but reading a few successful traces is still useful for spotting “lucky successes” (e.g. a file name was missed, but it happened to be outside what the check covers).
When reading failed transcripts, label each one “the agent really got it wrong” or “the grader misjudged.” If the grader misjudges a lot, fix the grader before worrying about the agent.

★ How do you score results that vary every run?

An agent’s results vary from trial to trial, so you run each task k times and then decide how to combine the k results into one number. S8 Evals Methodology introduced two ways: pass@k is the fraction of tasks where at least one of k trials succeeds; pass^k is the fraction where all k succeed.

Which one depends on the use case. If results can be verified automatically and you can let the agent try several times and pick the right one (e.g. code with tests), pass@k is meaningful. If the user gets the agent’s result directly every time, as when the notes assistant collects TODOs, look at pass^k: it measures “you can trust it every time.”

Three notes-assistant tasks, 5 trials each:

Task	5 results	Success rate	At least one success in 5	All 5 succeed
A: list TODO file names	✓✓✓✓✓	5/5	Yes	Yes
B: merge 3 pairs of duplicate notes	✓✓✗✓✓	4/5	Yes	No
C: “Delete that file” (should ask which one first)	✗✗✓✗✗	1/5	Yes	No

Average success rate: (5 + 4 + 1) ÷ 15 ≈ 67%
pass@5: all 3 tasks succeed at least once, 3/3 = 100%
pass^5: only A succeeds every time, 1/3 ≈ 33%

The same data can be reported as 100%, 67% or 33%. When you report, always state which one and what k is.

The gap grows with k. Assuming independent trials and a per-trial success rate of 0.8: pass@3 = 1 − 0.2³ = 0.992, pass^3 = 0.8³ = 0.512. The same agent looks nearly perfect by pass@3 and only a bit better than a coin flip by pass^3.

With few trials, these numbers are very uncertain. A success rate from 15 tasks × 5 trials needs a bootstrap confidence interval (← S8 Evals Methodology). When comparing two versions, run both on the same tasks and look at the interval of the paired difference: entirely above 0 means the new version is better, entirely below 0 means it’s worse, and if it contains 0 you can’t tell them apart. Don’t just see which point estimate is bigger. A high pass@k does not mean the agent is reliable. Task C counts as a “success” under pass@5, but in 4 out of 5 trials the user got an agent that deleted the wrong file or guessed blindly.

Going deeper: estimating pass@k and pass^k from n trials

Run each task n times with c successes, and you want the value for some k ≤ n. Plugging the success rate p̂ = c/n directly into 1 − (1 − p̂)^k or p̂^k is biased. The usual unbiased estimate is “draw k of the n trials without replacement”:

pass@k = 1 − C(n − c, k) / C(n, k) (Chen et al. 2021)
pass^k = C(c, k) / C(n, k) (τ-bench, Yao et al. 2024)

Example: n = 5, c = 4, k = 3. C(1, 3) = 0, so pass@3 = 1; pass^3 = C(4, 3) / C(5, 3) = 4/10 = 0.4. Plugging p̂ = 0.8 into p̂³ gives 0.512, which is different. The overall score is the average of the per-task estimates.

★ How do you make every trial fair?

Agents change their environment, so evaluation has to provide an environment you can reset, and check the environment’s state at the end. This is called environment-based eval. Three things make it fair.

Every trial starts from a clean environment (isolated trials). For the notes assistant, each trial copies the folder fresh from the original. Leftover state causes cascading failures and can also inflate scores: “Demystifying evals” mentions seeing a model read the git history left by a previous trial and take advantage of it.

Balance positive and negative cases. Test both “should act” and “shouldn’t act.” If every task is “please go edit the files,” the best strategy is to always edit right away, and the eval pushes the agent in that direction. So include negative cases: when information is missing it should ask first (“Delete that file” doesn’t say which), or when the request itself makes no sense (“Rewrite the nonexistent budget.md as three paragraphs”). The grader for negative cases checks that the folder is byte-for-byte identical to the original. Report success rates for positive and negative cases separately: 80% overall could be 95% on positives and only 50% on negatives.

Simulated users. Conversational agents need user replies to move forward. In the negative case above, when the notes assistant asks “Which file should I delete?”, someone has to answer “draft-plan.md.” The approach is to have an LLM play the user according to a persona (a simulated user), e.g. “busy, speaks in half-sentences” or “gives the wrong date.” τ-bench (Yao et al. 2024) uses this to evaluate retail and airline customer service; τ²-bench (Barres et al. 2025) goes further and lets the user act on the environment too (dual-control), for example the user has to change a setting on their own phone.

A simulated user can be too cooperative, forget its goal or invent facts outside its persona. Those mistakes affect the agent’s score. Read sample conversations to check that the simulation behaves plausibly.

★ Can machine settings change the score?

Settings like machine resources and time limits directly change scores (infra noise), so they’re experimental variables and belong in the report.

Anthropic’s “Infrastructure noise in agentic coding evals” (2026-02-05) found on Terminal-Bench 2.0 that changing only the container’s resource settings produced a success-rate gap of about 6 percentage points between the most generous and the most restrictive settings. They make three recommendations: treat resource and time limits as experimental variables to report, run multiple times at different times of day, and be skeptical of leaderboard gaps under 3 percentage points.

Applied to the notes assistant: change the tool timeout from 10 seconds to 2, and grep on a large folder may occasionally time out; the agent then has to retry or give up, and the success rate may drop a few percentage points without a single line of the agent changing. Step 4 of the core exercise measures exactly this gap and writes it up as one line, “infra variables for this eval.” Infra differences are not noise that averages out. They are a systematic shift: if two versions run under different settings, the whole gap might come from the settings.

★ Does a higher score always mean better?

Reward hacking is a phenomenon introduced in S4 Post-training and Reasoning (you don’t need to have read S4): the thing being optimized learns to push up “the measurement” instead of actually achieving the goal. In agent evals, it shows up from two directions.

The agent exploits holes in the environment. Reading the previous trial’s git history, above, is one example. Another is eval awareness: the model recognizes it’s being evaluated. Anthropic’s “Eval awareness in Claude Opus 4.6’s BrowseComp performance” (2026-03-06) documented 2 of 1,266 questions where the model suspected it was being tested, figured out which benchmark it was, and then went to decrypt the answers. The article concludes that when running evals on the open web, evaluation integrity has to be treated as an ongoing adversarial problem.

Optimization exploits holes in the grader. The prompt optimization later in this chapter repeatedly adjusts the prompt to push up some metric. If the summary judge prefers answers with many bullet points, the optimizer quickly learns to “always list 10 points”: the judge score goes up while users find it wordier.

Have graders inspect environment state, hold out test data from optimization, and read outputs regularly. A higher score supports an improvement only when it holds on that test data and the outputs pass inspection.

★ What extra pitfalls do agent benchmarks have?

Public benchmarks age: tasks leak into training data (contamination) and get solved to near-perfect scores (saturation) (← S8 Evals Methodology). Agent benchmarks add flaws in the tasks themselves. Two recent examples:

SWE-bench Verified: OpenAI assembled this human-verified subset with the SWE-bench authors in 2024. In 2026-02, OpenAI announced it would no longer evaluate on it, citing contamination (the tasks come from public repos, and for some of them models can reproduce the original fix from memory) and flawed tasks: they took the 138 tasks that o3 couldn’t solve consistently across 64 runs, had engineers check each one, and found substantive problems with the tests or task descriptions in 59.4% of them. OpenAI now recommends SWE-bench Pro instead.
τ-bench: SABER (Cuadron et al. 2025) showed that τ-bench’s airline and retail tasks contain labeling errors and unclear specs that lower the achievable ceiling, and released a corrected τ-Bench Verified.

So whenever you see an agent benchmark score, first ask three questions (benchmark hygiene): Has anyone audited the tasks? Have they leaked? What were the infra settings? A 2-point gap between two agents is already smaller than the infra gap measured above.

METR time horizon (Kwa et al. 2025) describes capability differently: how long human experts typically take on the tasks a model completes with 50% probability. The paper found this length has doubled roughly every 7 months since 2019. It’s easier to interpret than a single benchmark score, but the tasks are concentrated in software and research work, and the authors spell out the limits on external validity. A high benchmark score only describes performance on that set of tasks with that setup; your notes assistant needs its own evals.

★ What if there are too many traces to read?

After launch, you’ll have more traces than you can read. Cluster analysis helps you decide what to read first; a data flywheel connects what you find back into evals and improvements.

Cluster analysis turns traces (or summaries of them) into embeddings, clusters them, and then has you read a few from each cluster to find which clusters have unusually many failures and which represent usage you never thought of. Hamel’s FAQ also lists clustering as one sampling method for picking traces to review by hand. An illustrative example: the notes assistant has accumulated 2,000 traces, split into 12 clusters. The overall thumbs-down rate is 8%, but the “relative dates” cluster (“last week’s meeting,” “the note from the day before yesterday”) has 150 traces and a 30% thumbs-down rate. Reading 15 of them shows the agent often treats “last week” as the past 7 days rather than the previous calendar week, so you add 10 relative-date tasks to the eval.

A data flywheel turns use of the system into fuel for improving it automatically:

User feedback and the traces collected since S6 LLM APIs and Application Basics → error analysis and clustering → new eval tasks and failure categories → changes to prompts, tools or harness → confirmation with evals → new traces after launch.

Connect feedback to traces and turn trace failures into eval tasks. Once a capability eval is stable, retain it as a regression eval (← S8 Evals Methodology). S17 Product and AI UX covers the feedback interface. Clustering helps prioritize what to read; a person must still inspect traces to establish a failure category, such as “relative dates” in S8 Evals Methodology.

★ Can a better prompt be found automatically?

Prompt optimization lets a program search for better prompts automatically: you provide a metric and examples, and the optimizer repeatedly generates candidate prompts, scores them on the examples and keeps the good ones.

DSPy (Khattab et al. 2023) has you write LLM programs as modules with input/output signatures (e.g. “user request → intent”), and an optimizer then tunes each module’s prompt. Two commonly used optimizers:

MIPROv2 (Opsahl-Ong et al. 2024) searches each module’s instruction text and few-shot examples together.
GEPA (Agrawal et al. 2025) has the model reflect in text on execution trajectories, propose prompt changes, and keep candidates that each do well on different tasks. The GEPA paper reports beating GRPO (a training method that tunes model weights directly with reinforcement learning; deeper: S4 Post-training and Reasoning) and MIPROv2 while using far fewer rollouts.

There are two prerequisites: a reliable metric (a judge calibrated per S8 Evals Methodology, or a code check), and separate training and test data. An illustrative example: intent classification for the notes assistant (“search notes,” “create note,” “send email”), 300 labeled requests split into 200 for training and 100 for testing. Before optimization: 81% on training, 80% on test; after: 93% on training, 86% on test. The real improvement is the 6 points on test, not the 12 on training; the gap between them is the part where the optimizer fit the training data.

The optimizer is designed to raise scores on its optimization data, so use held-out test data to report results. Include before-and-after outputs to check for the reward hacking described above.

Before you build: preparing for a round of prompt optimization

Write the metric function first: it takes an example and the model output and returns a score from 0 to 1. If you use a judge as the metric, first check how well it agrees with human labels (← S8 Evals Methodology).
Set a budget cap first. The optimizer calls the model many times; a common approach is to optimize with a small model and validate on the target model.
Don’t look at the test data at all until optimization is finished, and don’t use it to pick “the best round,” or it becomes part of the training data. If you need to choose, split off a separate validation set.
After optimization, read 10–20 outputs to make sure no metric-pleasing pattern like “always list 10 points” has appeared.

★ Can routing go to a cheap small model?

Agents often first classify a request to choose a flow, model or tool. You can initially use a strong model for this, then replace it with a cheaper small classifier once you have enough traces. This is router distillation.

How it works. For every routing decision, record the request, the strong model’s decision, and whether it turned out to be correct (did the user rephrase, did the task get done, or a spot check). Use these “strong model’s judgment + after-the-fact confirmation” labels to train a small classifier (e.g. embeddings plus logistic regression, a simple classification method), or fine-tune a small model. This is distillation: having a small model learn a large model’s outputs (← S4 Post-training and Reasoning; application-route readers see the “Training stages” and “What fine-tuning can and can’t do” sections of the Model Crash Course).

An illustrative example: the notes assistant records 600 routing decisions, which after spot-checking and correction are split into 480 for training and 120 for testing. On the test data, the strong model routes with 97% accuracy and the small classifier with 94%; but the small classifier takes only tens of milliseconds and costs almost nothing, while the strong model waits on an API call every time. The errors are concentrated between “send email” (email_send) and “create an email draft” (email_draft). These two have different consequences (sending is high risk, ← S10 Tools and MCP), so you can hand back to the strong model only when the classifier is unsure between these two.

Inspect which classes account for the 3-point loss in accuracy. Confusing search with note creation may lead the user to ask again; routing a request to send email has different consequences. S18 Deployment and Architecture covers model routing and cascades in deployment.

★ Is this harness component worth keeping?

An ablation removes one harness component at a time and compares the difference on the same tasks with the same number of trials, to test whether the component is worth keeping.

S12 The Agent Loop and Harness said every harness component represents an assumption about “what the model can’t do on its own”; an ablation is how you test that assumption. Things you can remove include compaction, skills, an evaluator, a particular permission rule or a stop condition.

An illustrative example: the notes assistant, 20 tasks × 5 trials, succeeds 72/100 with compaction and 64/100 without. That’s an 8-point difference, but the 95% confidence interval is about −5 to +21 points. It contains 0, so you can’t say compaction helps. Both versions ran on the same tasks, so compare them paired (→ S19 Real Agent Systems and the Capstone); if the paired interval still contains 0, add tasks, add trials, or switch to tasks that bring out this component more (e.g. long weekly reports).

Going deeper: how this interval is computed and where it's off

Treat each version’s 100 trials as two independent samples: the standard error is √(0.72×0.28/100 + 0.64×0.36/100) ≈ 0.066, and times 1.96 that’s about ±13. This formula is a rough approximation that ignores two things. First, the 5 trials of one task are correlated, so when estimating a single version, a bootstrap by task usually gives a wider interval than this formula. Second, both versions ran on the same tasks, so the comparison should be paired: each time, sample 20 tasks with replacement, carrying both versions’ results along with each task, and compute the difference. Task difficulty cancels out between the versions, so the paired difference’s interval is usually narrower than the one above.

Model–harness fit: a component’s value depends on the model. Anthropic found context resets necessary for one generation and unnecessary overhead for the next. Repeat ablations after changing models, especially for components added to compensate for an older model’s weaknesses.

○ Can you train an agent directly?

You can also train the agent directly: SFT on successful trajectories, or RL so the model learns to use tools in an environment. For example, Search-R1 (Jin et al. 2025) uses RL to train a model to call search during reasoning (← GRPO / RLVR in S4 Post-training and Reasoning). This chapter’s environment-based evals double as RL training environments: the grader is the reward, so the reward hacking discussed above gets even worse during training. This needs GPUs and the background from M4 SFT → LoRA → DPO → GRPO on a Small Qwen3 (what the codes mean), and is optional. For self-improvement methods like STaR and ADAS, and tree search across multiple trajectories, start with Stanford CS329A.

Exercises

Concepts route (no code): after “Try it first”, you can skip to the self-check.

Try it first (about 10 minutes)

1. The TODO task’s reference trajectory is list_files → grep → write_file. One trial called grep → read_file × 2 → write_file, and todo.txt contains exactly the 6 correct file names. Compute its tool-call precision and recall, and state the outcome grader’s verdict.

Check your answer

The agent made 4 calls, of which grep and write_file are in the reference trajectory: precision = 2/4 = 0.5. It performed 2 of the 3 reference steps: recall = 2/3 ≈ 0.67. The outcome grader says pass, because the environment state (todo.txt) is fully correct; a low trajectory score doesn’t mean it did the job wrong.

2. A coding agent tries each task up to 3 times, runs unit tests each time, and gives the user the first version that passes. Should this scenario use pass@3 or pass^3? What about the notes assistant’s “collect TODOs”?

Check your answer

Coding uses pass@3: tests automatically pick out the correct version, the user only sees the one that passed, and one success in 3 is enough. Once “collect TODOs” is live, there are no tests to pick the right version for the user; every result goes straight to the user, so use pass^3.

Core: A trustworthy trajectory eval

Goal: build a rerunnable evaluation of an existing agent, with a report of the variables, and use trajectories to find the step where it most often breaks.

Steps

Use the agent from the S12 The Agent Loop and Harness core exercise and its folder of 20 Markdown notes (if you have not completed it, first build the S12 core agent, its 10 tasks and graders; recreating only the folder is not enough). Reuse that exercise’s 10 tasks and graders directly, and add 5 “shouldn’t act” tasks for 15 tasks total. “Shouldn’t act” means information is missing and it should ask first (e.g. “Delete that file” doesn’t say which), or the request itself makes no sense (e.g. “Rewrite the nonexistent budget.md as three paragraphs”). The graders for these 5 tasks check that the folder contents are byte-for-byte identical to the original.
Define the agent flow as 4–6 states (e.g. understand the task → find the target → read → modify → verify → report), and write a program that determines from a trace which state each run reached.
Prepare two versions (e.g. two system prompts or two models) and run each on 15 tasks × 5 trials. If you choose “with / without a harness component” as the two versions (← S12 The Agent Loop and Harness; e.g. the rule that stops after 3 consecutive failures), this step doubles as an ablation. Start every trial from a clean copy.
Take one of the versions, change just one environment setting (tool timeout or step limit), and run another round.
Build the transition failure matrix; read at least 10 transcripts from failed runs and label each one as the agent really getting it wrong or the grader misjudging.

Acceptance criteria

3 rounds, 225 runs in total (2 versions + 1 environment variable × 15 tasks × 5 trials), each with a trace and a grader result.
For each round, report the success rate, a 95% bootstrap confidence interval, pass^5, and the success rates for positive and negative cases separately.
Comparison of the two versions: report the confidence interval of the success-rate difference (A − B); write “A is better than B” only if the interval is entirely above 0. Entirely below 0 means A is worse; if it contains 0, write that you can’t tell them apart.
The difference between the environment-variable round and the original (with an interval), written as one line, “infra variables for this eval.”
The transition failure matrix’s cells sum to the number of failures; each of the 10 transcripts you read has a label, and you report how many the grader misjudged.

Hints

TypeScript: store traces as JSONL and analyze with Node scripts; a bootstrap is a dozen lines to write yourself, or use simple-statistics.
Python: pandas for tables, numpy or scipy.stats.bootstrap for the bootstrap.
Cost: 225 runs on a small cloud model usually costs a few dollars to $10–20; Ollama is free, and with Ollama, raise the context length first (see S6 LLM APIs and Application Basics). Check the whole pipeline with 3 tasks × 2 trials first.
If you get stuck:
- Reread: outcome first, then trajectory, pass@k and pass^k, environment-based evals, infra noise
- Further: Anthropic’s “Demystifying evals for AI agents”; “How do I evaluate agentic workflows?” in Hamel’s FAQ

Note: you can start with 10 tasks × 3 trials comparing just two versions, go through the whole process once with this minimal version, and then fill in the rest.

Advanced: System project A14: Agent evals and optimization

Goal: build a complete agent eval for your system, and improve it once with each of three methods, with a confidence interval on every conclusion.