- A hands-on workshop showing how evals turn vague "vibes" about agent quality into actionable, repeatable measurements, demonstrated by hill-climbing a slide-generation agent through successive iterations.
- The speaker walks through the three grader types — code-based, model-based (LLM judge), and human — and uses a mix of them (emoji count, cluttered slides, font size, plus rubric judges for color/layout/text) to diagnose and fix a deliberately bad slide deck.
- Iterative improvements (better system prompt, requiring diagrams, adding an adversarial QA loop, then switching to a smarter model) steadily raise scores, while the saturating judge scores reveal that evals themselves are living artifacts that must be recalibrated.
Evals for taste: Hill-climbing a slide-generation agent
- Evals are systematic tests made of tasks plus grading logic; they force you to define what success looks like and convert reactive, production-only firefighting into proactive, pre-launch confidence.
- Public benchmarks (SWE-bench, Terminal-Bench, OSWorld, ARC-AGI) measure general capability but rarely your specific use case — so build your own evals to benchmark models and pick the right one for the job.
- Three grader types trade off cost vs nuance: code-based graders (string/regex/count) are fast, cheap, deterministic but brittle; model-based judges (rubrics, pairwise comparison, multi-judge consensus) are flexible and nuanced but non-deterministic and need calibration; human graders are highest quality but slow and expensive.
- The slide agent was improved by reading eval output and updating the system prompt — adding explicit typography sizes, layout/density rules, and "avoid AI tells" instructions (no thin accent lines, no decorative emojis) based on observed failure modes.
- Requiring every slide to include a generated diagram/chart, then adding an adversarial QA loop ("assume there are problems, your job is to find them; inspect every slide image, fix, re-render, re-inspect") produced clearly better, more grounded decks and raised judge scores into the 4.2–4.4 range.
- LLM judges must output their reasoning BEFORE the score, never after — because the model is autoregressive, anchoring on a number first makes it rationalize that number; ask for pros and cons first, then decide.
- Evals are living artifacts: a judge giving uniformly high scores (e.g. a "5" on image quality for a deck with no images) signals saturation or miscalibration, meaning the grader must be revised to keep measuring something useful.
Eval — A systematic test measuring how well an AI system performs on a specific domain or use case, made of tasks and grading logic.
Grader — A mechanism that judges an output; can be code-based, model-based, or human.
Code-based grader — A deterministic check (string match, regex, count) similar to a unit test; fast and cheap but brittle.
Model-based grader (LLM judge) — A grader that uses an LLM with a rubric to score nuanced qualities like coherence or layout.
Pairwise comparison — Asking a model which of two outputs it prefers and why, useful when "good" is hard to define absolutely.
Multi-judge consensus — Running several judges independently and taking the majority to add determinism to model-based grading.
Rubric — The criteria and guidance given to a judge describing what to evaluate and what good looks like.
Saturation — When an eval stops producing relevant, actionable information (e.g. uniformly high scores).
QA loop — A generator–critic pattern where a critic agent adversarially hunts for problems and feeds fixes back to the creator agent.
Hill-climbing — Iteratively making and measuring small changes to steadily improve an agent's performance.
Good afternoon, everyone. I hope you all had a wonderful lunch. There's so many of you — I'm actually kind of surprised by this. Happy to see that there's that much interest in talking about evals. I personally am a big fan of anything evals related, but I know that's not everyone's cup of tea. So today's session is really going to be about evals, and my goal is for you all to be inspired afterwards to build evals — to think "okay, evals are actually really useful," and to get a better sense of how to think about building them, what useful types of evals exist, and how to use them to make better agents. The way we're going to do this is by building a slide generation agent and finding out what good evals look like, what we want to measure, and how to build better agents based on the feedback.
First we need to set the stage: what are evals? Evals are systematic tests that measure how well an AI system performs on a specific domain or use case. They give you information about the quality of the results — what it did well, what it was not good at, how you can improve. Evals are made up of tasks that define certain scenarios, which then encode certain expectations through the grading logic. If you're building an AI agent and you want to make sure the output adheres to a certain quality, evals are a way to encode that behavior so that if your evals fail, you know your agent is not behaving the way intended. Evals are the bridge between "it seems to work" or "it kind of feels worse today" and something actionable. Vibes have their place for a general sense check, but they're not actionable, and that's what you want out of evals.
When we release a model, we always ship an accompanying benchmark scorecard. There are usual suspects: SWE-bench, which measures agentic coding abilities; Terminal-Bench; tool use and agent evals like Tau-bench and OSWorld; and reasoning and knowledge evals like ARC-AGI 2. These give a general sense of how good the model is and how much we improved over previous versions. But if you're building an agentic system, these generic benchmarks don't say much about your very specific use case. That's why we always say: build your own evals, benchmark the different models, benchmark your AI agent, and make sure you're using the right model for the job.
Why are evals important? Without evals, you've all been in the scenario where an agent is working fine, and then a customer says "it's not really up to par after this new model switch, something is off." It's very hard to do anything with that — you're flying blind, in a reactive loop, only catching issues in production. Fixing one issue might create more down the line. It's annoying to distinguish genuine feedback from noise. And most importantly, there's no way to verify improvements or regressions. With evals, you have clarity: you must define what success looks like. Building these evals forces you to formalize what you expect your agent to do. It lets you iterate on optimal agent configs, adopt new models faster with clarity on what's better and what's not, and make problems visible before launch.
How do evals fit in? Originally, with prompt engineering, the flow was: develop your test cases (the evals), write a prompt, test it against the tasks, refine, run again, refine until you're confident, then ship the polished prompt. Over time, systems have gotten more complex with agents, tool calls, and skills — more levers to pull, which makes it even more important to have evals that force you to identify what you can change and what impacts the system positively.
When you create evals, there are a few graders. A grader is a way to judge the output. One is a code-based grader, pretty similar to a unit test — a string match, regex, or fuzzy match, a strict static check. The advantage is it's fast, cheap, deterministic, but the big drawback is it's brittle and lacks nuance. These deterministic checks force a deterministic behavior, which is sometimes exactly what we want — for example, you want to make sure there is a slide deck present. But for nuanced things, like the quality of the slide deck, you can't easily encode that. That's why we also have model-based graders, which are rubric-based reasoning. You ask things like "is this slide high quality?" or "is this text coherent?" Pairwise comparison is underrated: you give the model two outputs and ask which it prefers and why — interesting for scenarios where there's no clear definition of better. Multi-judge consensus is best of three: three judges score independently, majority wins. It introduces more determinism, since an LLM is non-deterministic and might say "great" some times and "not good" others. This is flexible, scalable, nuanced, but non-deterministic, costs more money, and requires calibration, which is not easy at all. Finally, the most expensive are human graders. These are the ones you'll use the least because they're incredibly expensive and slow, but they're the highest quality and most nuanced — good for A/B testing and spot checking.
Let me give you an overview of the repo. We have an agent.yaml where you define your agent — this uses the managed agent, the same thing as the earlier session. We give it a system prompt: "You are a slide generation agent, and when the user gives you a topic, create a PowerPoint file at this location. You have a shell with Python PPTX pre-installed." And then an environment with the packages needed.
The first question I have for the audience: we want to make a slide generation agent. What do you think is a good eval? What are you trying to measure? [Audience: number of words on slides; overlapping/overspilling text.] I like these two examples because they immediately give you a different sense of how to use the types of graders. The number of words on a slide is quantifiable — you can count it with a deterministic code grader. Whether it's overlapping or overspilling is harder to encode in code, so for that you'd use a model grader.
We've defined a few graders for you in two directories: code and judge. The code ones are deterministic — for example, emoji count, where we count the number of emojis in the slide deck, because we noticed it's quite prevalent. If I open the initial agent's slide deck, slide one through five — I think we can all agree this is not the best slide deck you've ever seen, but it's a good start. It does five slides, which is what we asked for. Given these slides, is there anything we'd never want? [Audience: no teal.] We see overlap of words, weird coloring, a horizontal artifact. Based on this, you decide what graders to define. We noticed emoji count is prevalent. Another is cluttered slides — how many shapes are on the slides; counting the number of slides (we always ask for five); slides with images, small fonts, text-heavy slides. This is arbitrarily chosen — it really depends from use case to use case what makes a good grader. The way I think about it: if a grader gives you no useful information, you should not have it in your eval. For each scenario you test, you should be able to say what information you want, what part of the system you're testing, and how you can act on it if it degrades. We also have judge graders, for example the color judge which judges color contrast and gives a score from zero to five. Same with image, layout, text. The judge's system prompt says: "Please evaluate the slide based on each of the following criteria. Text: the title should be simple and clear to indicate the main points. For main content, avoid too many texts and keep words concise. Use a consistent and readable font size, style, and color." And it goes on.
We have a script that automatically scores your slide deck. At the top it lists slide count, slides with image, text-heavy slides, cluttered slides, small font size, and so on, plus the judges scoring zero to five for text, image, layout, and color. You can immediately note these scores are quite high — between 2.8 and 4 — which is high given the slide deck we've seen. That's the calibration that needs to happen. One thing I want to stress: it's not because you've set up your evals once that they're now the ground truth. Evals can evolve. They need to be a living artifact. Over time we need to make sure the evals we create are still measuring something useful. If you hear people talk about saturation of evals, that's when the eval is not giving relevant, actionable information anymore.
The first thing we want is a more polished agent. So we update the system prompt. Instead of just "you are a slide generation agent, make a slide deck," we give more information about expectations on typography: slide title should be this size, section header this size, body this size, caption this size. We give information on layout and density — keep body text concise, leave breathing room, left-align paragraphs. And avoid AI-generated tells: never use thin accent lines in titles, and don't pepper slides with emojis as decorative icons. This is based on what we saw in our eval — emoji count of four, small font slides four, cluttered slides two, text-heavy slides. The new agent's result: slide one is immediately more enjoyable, no overlapping stuff, no dollar sign, cleaner, more consistent coloring. But now we run the score again and see emoji count of 20, and I'm wondering where they are — I haven't seen them, wondering if it's a mistake. We improved on cluttering. This shows the value of human review, because now I'm arguing this is not as text-heavy as the grader claims, which means something is wrong with how we're grading. So we'd go back, change the grader, and make sure it better reflects what we want to measure. This calibration of how your agent should behave and how your judges should judge is really fickle — you should spend proper time on it.
Let's say one of our requirements is that the agent always includes diagrams. We update the system prompt: "every slide must include at least one generated diagram or chart inserted as an actual image." The result is interesting — I'm not a fan of an image on the opening slide, but it's what we defined, so I'll let it slide. It's a nice graph about no negotiation versus active negotiation, showing the gap widens over time. This looks way better just by being grounded in actual facts instead of waffling through the slide deck. The score: no emojis, no cluttered slides, still text-heavy and small font, but with images we accept those. The image judge says 3.8 out of five, but it doesn't give us a lot to go off — just a random number.
One thing that works generally well across every use case is adding a QA loop. For coding this is intuitive: one agent writes the code, a second agent critiques it — "this is bad, this introduces a bug, this is not up to standards" — and that feedback goes back to the creation agent, which makes changes, then goes back to the critic, and the loop continues until both sides agree it can ship. So we say: "require QA loop. Assume there are problems, and your job is to find them. Approach QA as a bug hunt, not a confirmation step." We're actively instructing the agent to behave adversarially. Then: "after writing the deck, convert it to images. Inspect every slide image yourself. Fix the issues, re-render, re-inspect, and do not stop until you've completed at least one fixed and verified cycle." This is immediately a lot better — the image is bigger, more readable from a distance, with a source listed. More clearly structured. We take a look at the score and see all the judges are now higher than before, in the 4.2 to 4.4 range. We're on a good track.
You can keep doing this, but sometimes — and this is where it gets nuanced — you can just go to a smarter model. With models getting smarter over time, you expect them to figure this out on their own. So in the last one we switched our model from Sonnet 4.6 to Opus 4.7, and gave it the simple initial prompt again. The result is significantly better than the Sonnet one — more structured. The score is telling: Opus just does not use any emojis. It knows emojis are probably not the right place for a slide deck about salary increase. It has fewer small font slides because it has innate knowledge that it should be readable. Then the judge graders: 4.4, a five for images — do we even have an image in this one? We don't, but we got a five. Layout judge 4.2, color judge 4.8, title-body coherence 4.4. These are extremely high scores, which shows we might not be measuring the right thing.
The code graders are straightforward — emoji count just counts emojis. But with the judging, what we've done is actually quite problematic. We say "give a score from zero to five" and describe the criteria, but it has nothing to anchor on. It doesn't really know what good or bad looks like. So there's a trade-off between what the model knows and what we need to give it. In this case, it would help to say "this is a bad example, here's a zero, these are telltale signs of an extremely badly formatted slide deck," and express different ranges over time. Even then, the model decided to output a five for a deck with no images — what do we do with that number? One way is to always ask your judge graders to give reasons for their conclusion. And be very cautious about ordering. I've had it happen while setting this up: I asked for the number first, then the reasons. It said "it's a four, and the reasons are these." But an LLM works autoregressively — if it's anchored on that four, it will do anything to argue why it should be a four, even if it should be a one. It needs to justify the four. So you turn it around: first ask for a bunch of reasons, pros, cons, reasons it should be high, reasons it should be bad, and then based on all those reasons make the final decision. That goes back to the QA loop too — you can have multiple agents doing verification, one finding issues and another refuting them. For example, summarizing a legal case: an agent would love to jump to conclusions, and the grader might catch that it's unclear or glossing over facts. You can apply multiple techniques, multiple graders, to see which issues keep popping up, because a grader might also hallucinate, especially in nuanced scenarios.
What I've shown you today is just a small introduction to how evals can help you, but it's definitely not the end. Forty-five minutes for a session on evals is quite short because it can get really deep. I started this session talking about benchmarks, which in the end are just evals. Every model provider cares so much about benchmarks because it's one of the most important things when building models — we need to find what we're failing at, what we're good at, and how to make the model better in future generations. It's the same when building applications using AI agents: finding what works, finding what doesn't, iterating, and making sure the changes you make have a positive influence on your final outputs. Thank you guys so much. This is all the time I have.
TL;DR
- A hands-on workshop showing how evals turn vague "vibes" about agent quality into actionable, repeatable measurements, demonstrated by hill-climbing a slide-generation agent through successive iterations.
- The speaker walks through the three grader types — code-based, model-based (LLM judge), and human — and uses a mix of them (emoji count, cluttered slides, font size, plus rubric judges for color/layout/text) to diagnose and fix a deliberately bad slide deck.
- Iterative improvements (better system prompt, requiring diagrams, adding an adversarial QA loop, then switching to a smarter model) steadily raise scores, while the saturating judge scores reveal that evals themselves are living artifacts that must be recalibrated.
Takeaways
- Evals are systematic tests made of tasks plus grading logic; they force you to define what success looks like and convert reactive, production-only firefighting into proactive, pre-launch confidence.
- Public benchmarks (SWE-bench, Terminal-Bench, OSWorld, ARC-AGI) measure general capability but rarely your specific use case — so build your own evals to benchmark models and pick the right one for the job.
- Three grader types trade off cost vs nuance: code-based graders (string/regex/count) are fast, cheap, deterministic but brittle; model-based judges (rubrics, pairwise comparison, multi-judge consensus) are flexible and nuanced but non-deterministic and need calibration; human graders are highest quality but slow and expensive.
- The slide agent was improved by reading eval output and updating the system prompt — adding explicit typography sizes, layout/density rules, and "avoid AI tells" instructions (no thin accent lines, no decorative emojis) based on observed failure modes.
- Requiring every slide to include a generated diagram/chart, then adding an adversarial QA loop ("assume there are problems, your job is to find them; inspect every slide image, fix, re-render, re-inspect") produced clearly better, more grounded decks and raised judge scores into the 4.2–4.4 range.
- LLM judges must output their reasoning BEFORE the score, never after — because the model is autoregressive, anchoring on a number first makes it rationalize that number; ask for pros and cons first, then decide.
- Evals are living artifacts: a judge giving uniformly high scores (e.g. a "5" on image quality for a deck with no images) signals saturation or miscalibration, meaning the grader must be revised to keep measuring something useful.
Vocabulary
Eval — A systematic test measuring how well an AI system performs on a specific domain or use case, made of tasks and grading logic.
Grader — A mechanism that judges an output; can be code-based, model-based, or human.
Code-based grader — A deterministic check (string match, regex, count) similar to a unit test; fast and cheap but brittle.
Model-based grader (LLM judge) — A grader that uses an LLM with a rubric to score nuanced qualities like coherence or layout.
Pairwise comparison — Asking a model which of two outputs it prefers and why, useful when "good" is hard to define absolutely.
Multi-judge consensus — Running several judges independently and taking the majority to add determinism to model-based grading.
Rubric — The criteria and guidance given to a judge describing what to evaluate and what good looks like.
Saturation — When an eval stops producing relevant, actionable information (e.g. uniformly high scores).
QA loop — A generator–critic pattern where a critic agent adversarially hunts for problems and feeds fixes back to the creator agent.
Hill-climbing — Iteratively making and measuring small changes to steadily improve an agent's performance.
Transcript
Good afternoon, everyone. I hope you all had a wonderful lunch. There's so many of you — I'm actually kind of surprised by this. Happy to see that there's that much interest in talking about evals. I personally am a big fan of anything evals related, but I know that's not everyone's cup of tea. So today's session is really going to be about evals, and my goal is for you all to be inspired afterwards to build evals — to think "okay, evals are actually really useful," and to get a better sense of how to think about building them, what useful types of evals exist, and how to use them to make better agents. The way we're going to do this is by building a slide generation agent and finding out what good evals look like, what we want to measure, and how to build better agents based on the feedback.
First we need to set the stage: what are evals? Evals are systematic tests that measure how well an AI system performs on a specific domain or use case. They give you information about the quality of the results — what it did well, what it was not good at, how you can improve. Evals are made up of tasks that define certain scenarios, which then encode certain expectations through the grading logic. If you're building an AI agent and you want to make sure the output adheres to a certain quality, evals are a way to encode that behavior so that if your evals fail, you know your agent is not behaving the way intended. Evals are the bridge between "it seems to work" or "it kind of feels worse today" and something actionable. Vibes have their place for a general sense check, but they're not actionable, and that's what you want out of evals.
When we release a model, we always ship an accompanying benchmark scorecard. There are usual suspects: SWE-bench, which measures agentic coding abilities; Terminal-Bench; tool use and agent evals like Tau-bench and OSWorld; and reasoning and knowledge evals like ARC-AGI 2. These give a general sense of how good the model is and how much we improved over previous versions. But if you're building an agentic system, these generic benchmarks don't say much about your very specific use case. That's why we always say: build your own evals, benchmark the different models, benchmark your AI agent, and make sure you're using the right model for the job.
Why are evals important? Without evals, you've all been in the scenario where an agent is working fine, and then a customer says "it's not really up to par after this new model switch, something is off." It's very hard to do anything with that — you're flying blind, in a reactive loop, only catching issues in production. Fixing one issue might create more down the line. It's annoying to distinguish genuine feedback from noise. And most importantly, there's no way to verify improvements or regressions. With evals, you have clarity: you must define what success looks like. Building these evals forces you to formalize what you expect your agent to do. It lets you iterate on optimal agent configs, adopt new models faster with clarity on what's better and what's not, and make problems visible before launch.
How do evals fit in? Originally, with prompt engineering, the flow was: develop your test cases (the evals), write a prompt, test it against the tasks, refine, run again, refine until you're confident, then ship the polished prompt. Over time, systems have gotten more complex with agents, tool calls, and skills — more levers to pull, which makes it even more important to have evals that force you to identify what you can change and what impacts the system positively.
When you create evals, there are a few graders. A grader is a way to judge the output. One is a code-based grader, pretty similar to a unit test — a string match, regex, or fuzzy match, a strict static check. The advantage is it's fast, cheap, deterministic, but the big drawback is it's brittle and lacks nuance. These deterministic checks force a deterministic behavior, which is sometimes exactly what we want — for example, you want to make sure there is a slide deck present. But for nuanced things, like the quality of the slide deck, you can't easily encode that. That's why we also have model-based graders, which are rubric-based reasoning. You ask things like "is this slide high quality?" or "is this text coherent?" Pairwise comparison is underrated: you give the model two outputs and ask which it prefers and why — interesting for scenarios where there's no clear definition of better. Multi-judge consensus is best of three: three judges score independently, majority wins. It introduces more determinism, since an LLM is non-deterministic and might say "great" some times and "not good" others. This is flexible, scalable, nuanced, but non-deterministic, costs more money, and requires calibration, which is not easy at all. Finally, the most expensive are human graders. These are the ones you'll use the least because they're incredibly expensive and slow, but they're the highest quality and most nuanced — good for A/B testing and spot checking.
Let me give you an overview of the repo. We have an agent.yaml where you define your agent — this uses the managed agent, the same thing as the earlier session. We give it a system prompt: "You are a slide generation agent, and when the user gives you a topic, create a PowerPoint file at this location. You have a shell with Python PPTX pre-installed." And then an environment with the packages needed.
The first question I have for the audience: we want to make a slide generation agent. What do you think is a good eval? What are you trying to measure? [Audience: number of words on slides; overlapping/overspilling text.] I like these two examples because they immediately give you a different sense of how to use the types of graders. The number of words on a slide is quantifiable — you can count it with a deterministic code grader. Whether it's overlapping or overspilling is harder to encode in code, so for that you'd use a model grader.
We've defined a few graders for you in two directories: code and judge. The code ones are deterministic — for example, emoji count, where we count the number of emojis in the slide deck, because we noticed it's quite prevalent. If I open the initial agent's slide deck, slide one through five — I think we can all agree this is not the best slide deck you've ever seen, but it's a good start. It does five slides, which is what we asked for. Given these slides, is there anything we'd never want? [Audience: no teal.] We see overlap of words, weird coloring, a horizontal artifact. Based on this, you decide what graders to define. We noticed emoji count is prevalent. Another is cluttered slides — how many shapes are on the slides; counting the number of slides (we always ask for five); slides with images, small fonts, text-heavy slides. This is arbitrarily chosen — it really depends from use case to use case what makes a good grader. The way I think about it: if a grader gives you no useful information, you should not have it in your eval. For each scenario you test, you should be able to say what information you want, what part of the system you're testing, and how you can act on it if it degrades. We also have judge graders, for example the color judge which judges color contrast and gives a score from zero to five. Same with image, layout, text. The judge's system prompt says: "Please evaluate the slide based on each of the following criteria. Text: the title should be simple and clear to indicate the main points. For main content, avoid too many texts and keep words concise. Use a consistent and readable font size, style, and color." And it goes on.
We have a script that automatically scores your slide deck. At the top it lists slide count, slides with image, text-heavy slides, cluttered slides, small font size, and so on, plus the judges scoring zero to five for text, image, layout, and color. You can immediately note these scores are quite high — between 2.8 and 4 — which is high given the slide deck we've seen. That's the calibration that needs to happen. One thing I want to stress: it's not because you've set up your evals once that they're now the ground truth. Evals can evolve. They need to be a living artifact. Over time we need to make sure the evals we create are still measuring something useful. If you hear people talk about saturation of evals, that's when the eval is not giving relevant, actionable information anymore.
The first thing we want is a more polished agent. So we update the system prompt. Instead of just "you are a slide generation agent, make a slide deck," we give more information about expectations on typography: slide title should be this size, section header this size, body this size, caption this size. We give information on layout and density — keep body text concise, leave breathing room, left-align paragraphs. And avoid AI-generated tells: never use thin accent lines in titles, and don't pepper slides with emojis as decorative icons. This is based on what we saw in our eval — emoji count of four, small font slides four, cluttered slides two, text-heavy slides. The new agent's result: slide one is immediately more enjoyable, no overlapping stuff, no dollar sign, cleaner, more consistent coloring. But now we run the score again and see emoji count of 20, and I'm wondering where they are — I haven't seen them, wondering if it's a mistake. We improved on cluttering. This shows the value of human review, because now I'm arguing this is not as text-heavy as the grader claims, which means something is wrong with how we're grading. So we'd go back, change the grader, and make sure it better reflects what we want to measure. This calibration of how your agent should behave and how your judges should judge is really fickle — you should spend proper time on it.
Let's say one of our requirements is that the agent always includes diagrams. We update the system prompt: "every slide must include at least one generated diagram or chart inserted as an actual image." The result is interesting — I'm not a fan of an image on the opening slide, but it's what we defined, so I'll let it slide. It's a nice graph about no negotiation versus active negotiation, showing the gap widens over time. This looks way better just by being grounded in actual facts instead of waffling through the slide deck. The score: no emojis, no cluttered slides, still text-heavy and small font, but with images we accept those. The image judge says 3.8 out of five, but it doesn't give us a lot to go off — just a random number.
One thing that works generally well across every use case is adding a QA loop. For coding this is intuitive: one agent writes the code, a second agent critiques it — "this is bad, this introduces a bug, this is not up to standards" — and that feedback goes back to the creation agent, which makes changes, then goes back to the critic, and the loop continues until both sides agree it can ship. So we say: "require QA loop. Assume there are problems, and your job is to find them. Approach QA as a bug hunt, not a confirmation step." We're actively instructing the agent to behave adversarially. Then: "after writing the deck, convert it to images. Inspect every slide image yourself. Fix the issues, re-render, re-inspect, and do not stop until you've completed at least one fixed and verified cycle." This is immediately a lot better — the image is bigger, more readable from a distance, with a source listed. More clearly structured. We take a look at the score and see all the judges are now higher than before, in the 4.2 to 4.4 range. We're on a good track.
You can keep doing this, but sometimes — and this is where it gets nuanced — you can just go to a smarter model. With models getting smarter over time, you expect them to figure this out on their own. So in the last one we switched our model from Sonnet 4.6 to Opus 4.7, and gave it the simple initial prompt again. The result is significantly better than the Sonnet one — more structured. The score is telling: Opus just does not use any emojis. It knows emojis are probably not the right place for a slide deck about salary increase. It has fewer small font slides because it has innate knowledge that it should be readable. Then the judge graders: 4.4, a five for images — do we even have an image in this one? We don't, but we got a five. Layout judge 4.2, color judge 4.8, title-body coherence 4.4. These are extremely high scores, which shows we might not be measuring the right thing.
The code graders are straightforward — emoji count just counts emojis. But with the judging, what we've done is actually quite problematic. We say "give a score from zero to five" and describe the criteria, but it has nothing to anchor on. It doesn't really know what good or bad looks like. So there's a trade-off between what the model knows and what we need to give it. In this case, it would help to say "this is a bad example, here's a zero, these are telltale signs of an extremely badly formatted slide deck," and express different ranges over time. Even then, the model decided to output a five for a deck with no images — what do we do with that number? One way is to always ask your judge graders to give reasons for their conclusion. And be very cautious about ordering. I've had it happen while setting this up: I asked for the number first, then the reasons. It said "it's a four, and the reasons are these." But an LLM works autoregressively — if it's anchored on that four, it will do anything to argue why it should be a four, even if it should be a one. It needs to justify the four. So you turn it around: first ask for a bunch of reasons, pros, cons, reasons it should be high, reasons it should be bad, and then based on all those reasons make the final decision. That goes back to the QA loop too — you can have multiple agents doing verification, one finding issues and another refuting them. For example, summarizing a legal case: an agent would love to jump to conclusions, and the grader might catch that it's unclear or glossing over facts. You can apply multiple techniques, multiple graders, to see which issues keep popping up, because a grader might also hallucinate, especially in nuanced scenarios.
What I've shown you today is just a small introduction to how evals can help you, but it's definitely not the end. Forty-five minutes for a session on evals is quite short because it can get really deep. I started this session talking about benchmarks, which in the end are just evals. Every model provider cares so much about benchmarks because it's one of the most important things when building models — we need to find what we're failing at, what we're good at, and how to make the model better in future generations. It's the same when building applications using AI agents: finding what works, finding what doesn't, iterating, and making sure the changes you make have a positive influence on your final outputs. Thank you guys so much. This is all the time I have.