- Margo van Laar, an applied AI engineer at Anthropic in London, walks through prompting best practices across two practical scenarios: maintaining/migrating an existing production prompt, and building a new agent from scratch.
- Using a Meridian Mobile customer-support bot with a five-case eval, she fixes failure modes one by one — applying general hygiene (XML structure, output contracts), removing redundant patches, and giving the model tools instead of telling it to try harder.
- For the new scheduling agent, she compares models (Sonnet 4.6, Opus 4.7, adaptive thinking) and a generate-evaluate-repair agentic loop, showing the loop solves all cases with lower tokens and latency.
The prompting playbook
- Evaluations are the starting point and provide the rigor to know whether a prompt change actually correlates to a performance improvement; without them you can't tell if a migrated model is behaving differently or is simply less capable.
- A good eval suite covers three case types: a control case that should always pass, edge cases where the model has failed before, and capability-boundary cases where it should hand off to a human or refuse.
- Apply general hygiene first: remove redundant content (e.g. text copied from a website, references to hero images and cookies, telling a bot it's a human), and add structure with XML tags separating role, guidelines, policy, and tone.
- An output contract improves format consistency; combine prompt instructions (XML output tags) with harness changes like a stop sequence, and use structured outputs for complex schemas.
- Models can withhold information they have access to, not just hallucinate; a redundant patch ("never give wrong details, point to the URL") meant for an older model caused over-fitting, so give a balanced view and treat the customer data as source of truth, using version control for defensive changes.
- Instructions don't add capability — telling the model "always calculate correctly" doesn't make it better at mental math; the fix is to give it a tool (a calculate-proration tool with a defined schema and implementation).
- State both sides of a trade-off: a one-sided instruction ("avoid escalating, it costs $8") makes the model over-fit to not escalating; adding the cost of getting it wrong (refunds, customer trust) fixes the billing-escalation case, and this matters more as models get better at making trade-offs themselves.
- For a new agent, model and harness matter as much as the prompt: Opus 4.7 with adaptive thinking reliably passed but tripled tokens and latency, while a generate-evaluate-repair loop of three simple prompts solved all cases with lower cost and allowed soft constraints to be added at runtime.
Evaluation (eval) — A suite of test cases used to rigorously measure whether a prompt change improves model performance.
Control case — An unambiguous test case the model should always pass.
Edge case — A test case representing behavior where the model has failed before.
General hygiene — Baseline prompt cleanup: removing redundancy and adding clear structure.
Output contract — A defined output format the model must follow, often enforced with XML tags and stop sequences.
Stop sequence — A harness setting that tells the model to stop generating when a token (e.g. a closing tag) is detected.
Over-fitting (to a patch) — When a model follows a previous defensive instruction too literally, harming current behavior.
Tool use — Giving the model a defined tool (schema plus implementation) to perform tasks like calculation reliably.
Adaptive thinking — Letting the model decide how much reasoning to use to solve a problem.
Generate-evaluate-repair loop — An agentic pattern of separate generator, evaluator, and repair prompts working in a loop.
Hello everyone. Thank you so much for joining me this afternoon in the breakout room, the last session today of Code with Claude. My name is Margo van Laar. I am an applied AI engineer at Anthropic here in London. And this afternoon we're going to be talking about the prompting playbook. Prompting is arguably one of the first skills, if not the first skill, that we had to learn as engineers when we first started to work with LLMs. And even now it continues to be one of the most critical skills to building effective AI systems.
So today we're going to discuss some best practices in the context of two practical scenarios that you're probably encountering at work. The first is where you have an existing prompt in production that you've been maintaining for some time, and possibly you're migrating it to a new model or making a change to the architecture, and for some reason it's no longer working as well. The second scenario is where we're building an entirely new agentic use case from the ground up, and we need to build the prompt from zero to one.
In order to illustrate these best practices, I don't just want to give you a list of do's and don'ts. I want to walk through a practical example that's been inspired by real prompts that I've seen some of our customers work with who are building on Claude. The prompt we'll look at today is a miniaturized example. The prompts you're working with are probably a lot longer and more complex, but it's representative of common problems you might encounter when maintaining a prompt. Imagine we have a prompt that multiple people have been collaborating on, contributing to. There's no clear owner. It covers a lot of different areas like policy, tone, processes. We have some patches for previous models that we've migrated to all mixed together. It's built up and it's complex. And when we're migrating to a new model, we're finding that suddenly a lot of our test cases are no longer working as well as we expected.
So what's actually going on here? In order to start unpacking that question, we need a starting point, and that starting point is evaluations. We need evaluations to provide that rigor to understand whether a change to our prompt is actually correlating to an improvement in its performance. We have different models which have different capabilities and behaviors. When you migrate to a different model, your system might no longer be working as well for two reasons. First, the new model might be capable but behaving differently, and therefore we can tune our prompting to fix that behavior. The second case is where the model we're changing to isn't as capable, and no amount of prompting is going to fix that. So we need an eval suite to act as a way of testing that regression.
In the example we're going to look at today, it's a miniaturized example. We'll have five test cases in our eval. In reality you'll have a lot more, but the key thing is that it's representative of three key cases. The three key cases include having a control case, which should always pass — something we know the model handles well, it's unambiguous. The second is edge cases, cases where we've seen the model fail before; by including instructions into the prompt, we make sure that same behavior doesn't slip through again. And finally and critically, we need to make sure the model has a good understanding of the extent of its capabilities, where it should be handing off to a human, or where it should be point-blank refusing to answer a request.
In the example today, we'll be using a prompt for a customer support bot for a Telco company called Meridian Mobile. These are the five test cases. We have a simple control case looking at what's the data limit in the basic plan. We're also looking at edge cases such as its ability to do calculations, like calculating proration bills — if I switch my plan halfway through the month, what will my bill look like? We want to check it's accurately addressing key questions covered by our policy. We need to make sure it's escalating to a human whenever there is a billing error. And finally, we want to make sure our model isn't withholding any information it has access to, which it should be handing over to the customer.
So what we're going to do is take our prompt and run it on our V0 of the eval. We'll see what our failure modes are and systematically target those failure modes one at a time to see if we can resolve them by prompting. And along the way, we'll learn about the antipatterns and traps to avoid. This is representative of how we'd apply these best techniques in practice — we are rarely writing a prompt from scratch, we're often debugging an existing prompt. Best practice before we start targeting failure modes specifically is to apply our general prompting 101 best practices, applying general hygiene to clean up before we do the eval run.
Let's have a look at the example. Before we look at the prompt, this is just a vibe-coded web app that I've made for the presentation so we can look at how we're iterating on the prompt. I can easily run my evals on all five test cases and inspect the results. This is a pretty good first pass at a prompt. We've defined the bot's role at the top. When we scroll down, we've given it some data, some information on how to reason over the answers, critical instructions around the tone it should use, how to do calculations, etc. And finally we're passing in our customer account context and our user message. So let's have a look at how our first pass at the evals did. Our control case, all of our test cases have passed, which is what we expect for this unambiguous test case. But it's performing pretty poorly in these other areas.
Before we zoom in on those specific failure modes, let's do some general cleanup of our prompt. There's a couple of oddities here already. The first one is we're telling the bot that it's a human, which just isn't true. There's clearly some information here copied directly from a website — the key giveaway is a reference to a hero image. There are even references to cookies at the bottom. So we need to remove redundant information. When we look at the instructions, they're all grouped into one big paragraph — some reasoning, instructions about the role, critical instructions, without a real way of unpacking policy from guidelines from tone. So I preempted some changes, and this is a diff view. What we've done is add some structure — XML tags to define the role, separate general guidelines, separate policy, separate tone of voice. If we run that eval on this updated prompt, we should hopefully see an improvement. We can see just by clearing up the prompt, we've already improved the model's performance on this prepaid scenario. There's an interesting regression in that fifth hotspot case, and I don't want to worry too much about that now — there's going to be some natural variance in different runs, and we'll come back to it.
So what did we learn? Simply clearing up the prompt with a better structure and role description has improved performance. This is a best practice you can return to at any stage of writing and maintaining your prompt, especially as prompts get more complex. A general rule of thumb I like to follow: if you're reading a prompt and you can't tell guidelines from policy from data, most likely the model isn't able to either.
Before looking at those cases in more detail, there's a bit more general cleanup we can do, specifically creating an output contract. This is a key best practice if you're struggling with output format consistency. In this case we have a customer support bot, we want it to reply in a conversational tone, so it's unlikely to be a big issue here, but it's something to bear in mind for more complex output structures like nested JSONs. We've added a section at the end where we've defined an output format telling it to use XML tags. But the prompt is not always the most effective way of handling issues. We can also change things in the harness — we've added a stop sequence to the API call, which detects that closing XML tag and tells the model to stop generating at that point. When I run the eval, I don't necessarily expect clear improvement, but it's a general best practice, especially with more complex output schemas. One thing to point out: if you have a more complex output schema, structured outputs can be incredibly helpful to ensure consistency in a more programmatic way.
After the cleanup, we now have two test cases consistently passing, but three key failure modes: the proration, the billing error, and the hotspot. Let's isolate these one by one. First, the hotspot question: how much hotspot data is on my unlimited plan? We expect the model to state directly the amount of hotspot data the customer has. This is slightly complex because the customer test case is on a legacy plan, so the current policy doesn't apply. The customer data we're feeding includes the amount of hotspot data — they have 5 GB — but they also have a grandfathered plan. So what we're seeing the model tell the customer is the unlimited plan includes 4 GB, but since you're on a legacy plan, you should go check this out yourself. Let's look at the prompt to see why the model is deflecting to the customer account URL rather than giving the information. Originally it said: we changed our plans recently, the policy doc shows the current plan data, customers on grandfathered plans have different rates, never give a customer the wrong plan details, instead point them to the URL. It's clear this latter instruction — never give wrong information — is what the bot has been optimizing for. You might recognize this as similar to a patch you might have introduced in a previous model to avoid giving the wrong information. As our models have evolved, they've gotten much better at instruction following, so instructions like these have now become redundant and are being over-fitted to. So we'll tell the model instead to give a balanced view — customers on grandfathered plans have different allowances, but it's captured in the customer information given, and that is the accurate source of truth. Running the eval, all our test cases are passing.
So what did we learn? We worry a lot about hallucinations or the invention of facts, but the opposite can also happen — the model can withhold information it actually has access to. This is likely a result of a patch we introduced for a previous model, and a best practice is to use version control: wherever we make defensive changes in the prompt, we track the reason why. Sometimes they're necessary, but in the future these changes can produce unwanted effects, so we can backtrack on them.
The next failing test case is the proration calculation, where a customer asks: what if I upgrade to the 30 GB plan, what will my next bill be? We want the model to perform a calculation and return exactly what their next bill would be, rather than a vague output. If we look at what the model is returning, it's reasoning through it, doing mental math here and there, but not really giving a concrete answer, and I wouldn't rely on this. If we look at the prompt: don't ever give a customer a vague answer, critical, always calculate any prorated amounts correctly. Telling the model to do a good job isn't particularly helpful when we don't give it the capability to actually do a good job. We want to avoid the model doing mental math. So we're going to give the model a tool — whenever you're doing calculations, please use the calculate-proration tool. To introduce that tool, we need to introduce it into the API, define the tool schema which tells the model what it does and when to use it, and finally implement the tool, the maths behind the calculation. Running that eval, all the test cases are now passing. It's clearly done the maths using the tool and returned the correct response. The key lesson: instructions don't add capability. Telling the model it's critical to do a calculation right doesn't make it better at mental maths. The correct approach was to give it a tool, giving it the ability to reason over harder problems and using tools to execute reliably.
Now we have one final failing test case, the billing error. There is a billing conflict, and we want the agent to escalate this to a human. What we're seeing instead is it's trying to explain to the customer what the reason might be and diagnose the problem itself. Let's look at the prompt: avoid escalating or transferring to a care specialist unless absolutely necessary as it costs approximately $8 and counts against our team's fast contract resolution. This is only giving one side of the story — we're telling it the cost of escalating, which means it's going to over-fit to not escalating. And there's a clear conflict between what we've defined in the eval versus what we're telling it. The fix is to give it both sides: it costs $8 to escalate, but if you get this wrong, it's going to cost a refund as well as customer trust. Again we observed how the model optimizes for a goal, and this is a common instruction, similar to the one earlier where we didn't want it to over-fit. As models become more intelligent, we need to remember to state both sides of the trade-offs because our models are becoming better themselves at making those trade-offs. Running our final test case, all our evals are now passing.
So overall, we looked at applying general hygiene principles, removing redundant instructions which were initially intended as patches for previous model behavior, and giving it tools to do certain tasks reliably.
Now there's one other scenario we introduced at the start: building a new agent from scratch. The example here is an agent whose purpose is to create a week-long retail staff schedule based on employee availability and other constraints. When building a new agent from scratch, we need to consider not just the prompt, but also the model and the harness. In this next example, we're going to compare a number of approaches to explore the impact of those three different areas. I've vibe-coded up this web app so we can walk through the problem. We have eight employees. On the right, we have this schedule we need to staff with the headcount, and our constraints that must be satisfied in every scenario. Because we have these hard rules, rather than using an LLM judge like in the previous case to do the grading, we can use a Python function which programmatically checks for every schedule how many violations were made.
To begin with, we want to start simple. We're going to use a simple prompt, the bare bones we think we'll need, with model Sonnet 4.6, to see how it performs and how we're going to hill climb against that. Here is our baseline prompt. We've already applied some of that general hygiene using XML tags, and we've given it an output format now that we're giving a schedule, asking it to output a JSON which, if we don't give that structure, might lead to parsing errors downstream. When we run the simple model on a first iteration, all cases fail. In our test set, we're doing five trials, and these numbers show how many violations were made in each trial. In the output, it's made a decent attempt at reasoning through the problem, but it's burning a lot of tokens and clearly not checking its work.
So let's try a larger model, one we know is better at reasoning. We'll run it through Opus 4.7 instead, keeping everything else the same. Interestingly, whilst all test cases are still failing, the overall number of violations Opus has made has reduced significantly from Sonnet 4.6. So we're possibly onto something — this isn't good enough to ship because it's still failing, but giving it more reasoning capability is helping drive it towards a better result. What we'll try next is Opus with adaptive thinking, so it can decide for itself how much reasoning it needs. No change to the prompt, just a change to the API. This now seems to reliably generate compliant schedules, but it requires a lot more tokens — we're tripling the number of tokens and tripling the latency. So we want to optimize that cost-latency trade-off a bit more. This is latency 100 seconds. Opus 4.7 hasn't magically gotten much faster since the last time you used it.
What we haven't tried yet is using Sonnet 4.6, a smaller model, but with a better prompt. We looked a lot at prompt optimization in that last section. I've added a couple of details to the prompt, in particular how to reason through this problem, and most critically telling it to check its work before outputting it. When I ran that eval, it passes in two out of five cases. The failure modes we're seeing are actually not violations of the scheduling requirements, but the model hasn't been able to finish the task within the output limit we set. Whilst we could increase the max tokens to get all five passing, we're using even more tokens and this run has even higher latency. So this is probably not the route we want to go down.
As a final pass, we want to look at doing this a little more agentically. We're going to use this generate-evaluate-repair loop, where the generator now creates a first draft of the schedule. Then we have a separate prompt which reports any specific violations that it made — not programmatically checking it, but checking it with an LLM. So we're checking for every rule and providing evidence of every violation. And we then have a third repair prompt which receives any violations and tries to make targeted fixes. So we have three very simple prompts, but they're now running independently rather than trying to do everything in one large prompt. In this case our agentic approach has solved all of our test cases with a much lower number of tokens and a lower latency than trying Sonnet 4.6 with a better prompt.
So going forward, it seems there's two appropriate approaches: using Opus 4.7 with adaptive thinking or using this agentic loop. We'd probably want to do a bit more optimization on this loop to make it more efficient, but there's one key benefit from using this generate-evaluate-repair loop: you can put in soft requirements at runtime. In the evaluation prompt, we can say Harry doesn't like working with Sally, so as much as possible try and separate them, or we need a third shift on Wednesday. So you're not having to make changes to the Python function doing the evaluation in the back end every time to satisfy soft constraints which might depend on a case-by-case basis.
So to wrap up, pulling all of those learnings together: we looked at two scenarios I as an engineer see most in my day-to-day — maintaining a prompt and migrating to a new model with different behaviors, and building a new use case from scratch. We saw that general hygiene principles can immediately uplift performance against a set of evals, and that we need those evals to rigorously see any impacts of changing our prompt. Then we saw the process of targeting failure modes one by one, adding structure, avoiding long ban lists. And finally with our new agentic bot, we saw the impact of splitting into three separate prompt systems — rather than using one prompt to address everything, isolating different tasks where it's easy and repeatable to separate out the steps. Thank you so much for attending this afternoon. I hope you have a fantastic rest of your day.
TL;DR
- Margo van Laar, an applied AI engineer at Anthropic in London, walks through prompting best practices across two practical scenarios: maintaining/migrating an existing production prompt, and building a new agent from scratch.
- Using a Meridian Mobile customer-support bot with a five-case eval, she fixes failure modes one by one — applying general hygiene (XML structure, output contracts), removing redundant patches, and giving the model tools instead of telling it to try harder.
- For the new scheduling agent, she compares models (Sonnet 4.6, Opus 4.7, adaptive thinking) and a generate-evaluate-repair agentic loop, showing the loop solves all cases with lower tokens and latency.
Takeaways
- Evaluations are the starting point and provide the rigor to know whether a prompt change actually correlates to a performance improvement; without them you can't tell if a migrated model is behaving differently or is simply less capable.
- A good eval suite covers three case types: a control case that should always pass, edge cases where the model has failed before, and capability-boundary cases where it should hand off to a human or refuse.
- Apply general hygiene first: remove redundant content (e.g. text copied from a website, references to hero images and cookies, telling a bot it's a human), and add structure with XML tags separating role, guidelines, policy, and tone.
- An output contract improves format consistency; combine prompt instructions (XML output tags) with harness changes like a stop sequence, and use structured outputs for complex schemas.
- Models can withhold information they have access to, not just hallucinate; a redundant patch ("never give wrong details, point to the URL") meant for an older model caused over-fitting, so give a balanced view and treat the customer data as source of truth, using version control for defensive changes.
- Instructions don't add capability — telling the model "always calculate correctly" doesn't make it better at mental math; the fix is to give it a tool (a calculate-proration tool with a defined schema and implementation).
- State both sides of a trade-off: a one-sided instruction ("avoid escalating, it costs $8") makes the model over-fit to not escalating; adding the cost of getting it wrong (refunds, customer trust) fixes the billing-escalation case, and this matters more as models get better at making trade-offs themselves.
- For a new agent, model and harness matter as much as the prompt: Opus 4.7 with adaptive thinking reliably passed but tripled tokens and latency, while a generate-evaluate-repair loop of three simple prompts solved all cases with lower cost and allowed soft constraints to be added at runtime.
Vocabulary
Evaluation (eval) — A suite of test cases used to rigorously measure whether a prompt change improves model performance.
Control case — An unambiguous test case the model should always pass.
Edge case — A test case representing behavior where the model has failed before.
General hygiene — Baseline prompt cleanup: removing redundancy and adding clear structure.
Output contract — A defined output format the model must follow, often enforced with XML tags and stop sequences.
Stop sequence — A harness setting that tells the model to stop generating when a token (e.g. a closing tag) is detected.
Over-fitting (to a patch) — When a model follows a previous defensive instruction too literally, harming current behavior.
Tool use — Giving the model a defined tool (schema plus implementation) to perform tasks like calculation reliably.
Adaptive thinking — Letting the model decide how much reasoning to use to solve a problem.
Generate-evaluate-repair loop — An agentic pattern of separate generator, evaluator, and repair prompts working in a loop.
Transcript
Hello everyone. Thank you so much for joining me this afternoon in the breakout room, the last session today of Code with Claude. My name is Margo van Laar. I am an applied AI engineer at Anthropic here in London. And this afternoon we're going to be talking about the prompting playbook. Prompting is arguably one of the first skills, if not the first skill, that we had to learn as engineers when we first started to work with LLMs. And even now it continues to be one of the most critical skills to building effective AI systems.
So today we're going to discuss some best practices in the context of two practical scenarios that you're probably encountering at work. The first is where you have an existing prompt in production that you've been maintaining for some time, and possibly you're migrating it to a new model or making a change to the architecture, and for some reason it's no longer working as well. The second scenario is where we're building an entirely new agentic use case from the ground up, and we need to build the prompt from zero to one.
In order to illustrate these best practices, I don't just want to give you a list of do's and don'ts. I want to walk through a practical example that's been inspired by real prompts that I've seen some of our customers work with who are building on Claude. The prompt we'll look at today is a miniaturized example. The prompts you're working with are probably a lot longer and more complex, but it's representative of common problems you might encounter when maintaining a prompt. Imagine we have a prompt that multiple people have been collaborating on, contributing to. There's no clear owner. It covers a lot of different areas like policy, tone, processes. We have some patches for previous models that we've migrated to all mixed together. It's built up and it's complex. And when we're migrating to a new model, we're finding that suddenly a lot of our test cases are no longer working as well as we expected.
So what's actually going on here? In order to start unpacking that question, we need a starting point, and that starting point is evaluations. We need evaluations to provide that rigor to understand whether a change to our prompt is actually correlating to an improvement in its performance. We have different models which have different capabilities and behaviors. When you migrate to a different model, your system might no longer be working as well for two reasons. First, the new model might be capable but behaving differently, and therefore we can tune our prompting to fix that behavior. The second case is where the model we're changing to isn't as capable, and no amount of prompting is going to fix that. So we need an eval suite to act as a way of testing that regression.
In the example we're going to look at today, it's a miniaturized example. We'll have five test cases in our eval. In reality you'll have a lot more, but the key thing is that it's representative of three key cases. The three key cases include having a control case, which should always pass — something we know the model handles well, it's unambiguous. The second is edge cases, cases where we've seen the model fail before; by including instructions into the prompt, we make sure that same behavior doesn't slip through again. And finally and critically, we need to make sure the model has a good understanding of the extent of its capabilities, where it should be handing off to a human, or where it should be point-blank refusing to answer a request.
In the example today, we'll be using a prompt for a customer support bot for a Telco company called Meridian Mobile. These are the five test cases. We have a simple control case looking at what's the data limit in the basic plan. We're also looking at edge cases such as its ability to do calculations, like calculating proration bills — if I switch my plan halfway through the month, what will my bill look like? We want to check it's accurately addressing key questions covered by our policy. We need to make sure it's escalating to a human whenever there is a billing error. And finally, we want to make sure our model isn't withholding any information it has access to, which it should be handing over to the customer.
So what we're going to do is take our prompt and run it on our V0 of the eval. We'll see what our failure modes are and systematically target those failure modes one at a time to see if we can resolve them by prompting. And along the way, we'll learn about the antipatterns and traps to avoid. This is representative of how we'd apply these best techniques in practice — we are rarely writing a prompt from scratch, we're often debugging an existing prompt. Best practice before we start targeting failure modes specifically is to apply our general prompting 101 best practices, applying general hygiene to clean up before we do the eval run.
Let's have a look at the example. Before we look at the prompt, this is just a vibe-coded web app that I've made for the presentation so we can look at how we're iterating on the prompt. I can easily run my evals on all five test cases and inspect the results. This is a pretty good first pass at a prompt. We've defined the bot's role at the top. When we scroll down, we've given it some data, some information on how to reason over the answers, critical instructions around the tone it should use, how to do calculations, etc. And finally we're passing in our customer account context and our user message. So let's have a look at how our first pass at the evals did. Our control case, all of our test cases have passed, which is what we expect for this unambiguous test case. But it's performing pretty poorly in these other areas.
Before we zoom in on those specific failure modes, let's do some general cleanup of our prompt. There's a couple of oddities here already. The first one is we're telling the bot that it's a human, which just isn't true. There's clearly some information here copied directly from a website — the key giveaway is a reference to a hero image. There are even references to cookies at the bottom. So we need to remove redundant information. When we look at the instructions, they're all grouped into one big paragraph — some reasoning, instructions about the role, critical instructions, without a real way of unpacking policy from guidelines from tone. So I preempted some changes, and this is a diff view. What we've done is add some structure — XML tags to define the role, separate general guidelines, separate policy, separate tone of voice. If we run that eval on this updated prompt, we should hopefully see an improvement. We can see just by clearing up the prompt, we've already improved the model's performance on this prepaid scenario. There's an interesting regression in that fifth hotspot case, and I don't want to worry too much about that now — there's going to be some natural variance in different runs, and we'll come back to it.
So what did we learn? Simply clearing up the prompt with a better structure and role description has improved performance. This is a best practice you can return to at any stage of writing and maintaining your prompt, especially as prompts get more complex. A general rule of thumb I like to follow: if you're reading a prompt and you can't tell guidelines from policy from data, most likely the model isn't able to either.
Before looking at those cases in more detail, there's a bit more general cleanup we can do, specifically creating an output contract. This is a key best practice if you're struggling with output format consistency. In this case we have a customer support bot, we want it to reply in a conversational tone, so it's unlikely to be a big issue here, but it's something to bear in mind for more complex output structures like nested JSONs. We've added a section at the end where we've defined an output format telling it to use XML tags. But the prompt is not always the most effective way of handling issues. We can also change things in the harness — we've added a stop sequence to the API call, which detects that closing XML tag and tells the model to stop generating at that point. When I run the eval, I don't necessarily expect clear improvement, but it's a general best practice, especially with more complex output schemas. One thing to point out: if you have a more complex output schema, structured outputs can be incredibly helpful to ensure consistency in a more programmatic way.
After the cleanup, we now have two test cases consistently passing, but three key failure modes: the proration, the billing error, and the hotspot. Let's isolate these one by one. First, the hotspot question: how much hotspot data is on my unlimited plan? We expect the model to state directly the amount of hotspot data the customer has. This is slightly complex because the customer test case is on a legacy plan, so the current policy doesn't apply. The customer data we're feeding includes the amount of hotspot data — they have 5 GB — but they also have a grandfathered plan. So what we're seeing the model tell the customer is the unlimited plan includes 4 GB, but since you're on a legacy plan, you should go check this out yourself. Let's look at the prompt to see why the model is deflecting to the customer account URL rather than giving the information. Originally it said: we changed our plans recently, the policy doc shows the current plan data, customers on grandfathered plans have different rates, never give a customer the wrong plan details, instead point them to the URL. It's clear this latter instruction — never give wrong information — is what the bot has been optimizing for. You might recognize this as similar to a patch you might have introduced in a previous model to avoid giving the wrong information. As our models have evolved, they've gotten much better at instruction following, so instructions like these have now become redundant and are being over-fitted to. So we'll tell the model instead to give a balanced view — customers on grandfathered plans have different allowances, but it's captured in the customer information given, and that is the accurate source of truth. Running the eval, all our test cases are passing.
So what did we learn? We worry a lot about hallucinations or the invention of facts, but the opposite can also happen — the model can withhold information it actually has access to. This is likely a result of a patch we introduced for a previous model, and a best practice is to use version control: wherever we make defensive changes in the prompt, we track the reason why. Sometimes they're necessary, but in the future these changes can produce unwanted effects, so we can backtrack on them.
The next failing test case is the proration calculation, where a customer asks: what if I upgrade to the 30 GB plan, what will my next bill be? We want the model to perform a calculation and return exactly what their next bill would be, rather than a vague output. If we look at what the model is returning, it's reasoning through it, doing mental math here and there, but not really giving a concrete answer, and I wouldn't rely on this. If we look at the prompt: don't ever give a customer a vague answer, critical, always calculate any prorated amounts correctly. Telling the model to do a good job isn't particularly helpful when we don't give it the capability to actually do a good job. We want to avoid the model doing mental math. So we're going to give the model a tool — whenever you're doing calculations, please use the calculate-proration tool. To introduce that tool, we need to introduce it into the API, define the tool schema which tells the model what it does and when to use it, and finally implement the tool, the maths behind the calculation. Running that eval, all the test cases are now passing. It's clearly done the maths using the tool and returned the correct response. The key lesson: instructions don't add capability. Telling the model it's critical to do a calculation right doesn't make it better at mental maths. The correct approach was to give it a tool, giving it the ability to reason over harder problems and using tools to execute reliably.
Now we have one final failing test case, the billing error. There is a billing conflict, and we want the agent to escalate this to a human. What we're seeing instead is it's trying to explain to the customer what the reason might be and diagnose the problem itself. Let's look at the prompt: avoid escalating or transferring to a care specialist unless absolutely necessary as it costs approximately $8 and counts against our team's fast contract resolution. This is only giving one side of the story — we're telling it the cost of escalating, which means it's going to over-fit to not escalating. And there's a clear conflict between what we've defined in the eval versus what we're telling it. The fix is to give it both sides: it costs $8 to escalate, but if you get this wrong, it's going to cost a refund as well as customer trust. Again we observed how the model optimizes for a goal, and this is a common instruction, similar to the one earlier where we didn't want it to over-fit. As models become more intelligent, we need to remember to state both sides of the trade-offs because our models are becoming better themselves at making those trade-offs. Running our final test case, all our evals are now passing.
So overall, we looked at applying general hygiene principles, removing redundant instructions which were initially intended as patches for previous model behavior, and giving it tools to do certain tasks reliably.
Now there's one other scenario we introduced at the start: building a new agent from scratch. The example here is an agent whose purpose is to create a week-long retail staff schedule based on employee availability and other constraints. When building a new agent from scratch, we need to consider not just the prompt, but also the model and the harness. In this next example, we're going to compare a number of approaches to explore the impact of those three different areas. I've vibe-coded up this web app so we can walk through the problem. We have eight employees. On the right, we have this schedule we need to staff with the headcount, and our constraints that must be satisfied in every scenario. Because we have these hard rules, rather than using an LLM judge like in the previous case to do the grading, we can use a Python function which programmatically checks for every schedule how many violations were made.
To begin with, we want to start simple. We're going to use a simple prompt, the bare bones we think we'll need, with model Sonnet 4.6, to see how it performs and how we're going to hill climb against that. Here is our baseline prompt. We've already applied some of that general hygiene using XML tags, and we've given it an output format now that we're giving a schedule, asking it to output a JSON which, if we don't give that structure, might lead to parsing errors downstream. When we run the simple model on a first iteration, all cases fail. In our test set, we're doing five trials, and these numbers show how many violations were made in each trial. In the output, it's made a decent attempt at reasoning through the problem, but it's burning a lot of tokens and clearly not checking its work.
So let's try a larger model, one we know is better at reasoning. We'll run it through Opus 4.7 instead, keeping everything else the same. Interestingly, whilst all test cases are still failing, the overall number of violations Opus has made has reduced significantly from Sonnet 4.6. So we're possibly onto something — this isn't good enough to ship because it's still failing, but giving it more reasoning capability is helping drive it towards a better result. What we'll try next is Opus with adaptive thinking, so it can decide for itself how much reasoning it needs. No change to the prompt, just a change to the API. This now seems to reliably generate compliant schedules, but it requires a lot more tokens — we're tripling the number of tokens and tripling the latency. So we want to optimize that cost-latency trade-off a bit more. This is latency 100 seconds. Opus 4.7 hasn't magically gotten much faster since the last time you used it.
What we haven't tried yet is using Sonnet 4.6, a smaller model, but with a better prompt. We looked a lot at prompt optimization in that last section. I've added a couple of details to the prompt, in particular how to reason through this problem, and most critically telling it to check its work before outputting it. When I ran that eval, it passes in two out of five cases. The failure modes we're seeing are actually not violations of the scheduling requirements, but the model hasn't been able to finish the task within the output limit we set. Whilst we could increase the max tokens to get all five passing, we're using even more tokens and this run has even higher latency. So this is probably not the route we want to go down.
As a final pass, we want to look at doing this a little more agentically. We're going to use this generate-evaluate-repair loop, where the generator now creates a first draft of the schedule. Then we have a separate prompt which reports any specific violations that it made — not programmatically checking it, but checking it with an LLM. So we're checking for every rule and providing evidence of every violation. And we then have a third repair prompt which receives any violations and tries to make targeted fixes. So we have three very simple prompts, but they're now running independently rather than trying to do everything in one large prompt. In this case our agentic approach has solved all of our test cases with a much lower number of tokens and a lower latency than trying Sonnet 4.6 with a better prompt.
So going forward, it seems there's two appropriate approaches: using Opus 4.7 with adaptive thinking or using this agentic loop. We'd probably want to do a bit more optimization on this loop to make it more efficient, but there's one key benefit from using this generate-evaluate-repair loop: you can put in soft requirements at runtime. In the evaluation prompt, we can say Harry doesn't like working with Sally, so as much as possible try and separate them, or we need a third shift on Wednesday. So you're not having to make changes to the Python function doing the evaluation in the back end every time to satisfy soft constraints which might depend on a case-by-case basis.
So to wrap up, pulling all of those learnings together: we looked at two scenarios I as an engineer see most in my day-to-day — maintaining a prompt and migrating to a new model with different behaviors, and building a new use case from scratch. We saw that general hygiene principles can immediately uplift performance against a set of evals, and that we need those evals to rigorously see any impacts of changing our prompt. Then we saw the process of targeting failure modes one by one, adding structure, avoiding long ban lists. And finally with our new agentic bot, we saw the impact of splitting into three separate prompt systems — rather than using one prompt to address everything, isolating different tasks where it's easy and repeatable to separate out the steps. Thank you so much for attending this afternoon. I hope you have a fantastic rest of your day.