- Public benchmarks are only directional; a small, well-designed private eval built from your own representative tasks is far more useful for deciding whether a new model is right for your use case.
- The right model isn't the cheapest or fastest per token — it's the one that is cheapest per successful outcome, and counterintuitively more intelligent models like Opus can finish faster and with fewer tokens than smaller ones.
- You have many dials — thinking, effort, prompt caching, and context engineering — that let you move along the cost/accuracy frontier or shift the whole frontier, often unlocking Opus quality at Sonnet cost.
Picking the right model
- Frame model choice through three pillars: model quality (task completion rate / accuracy), latency (critical for customer-facing use cases), and cost. Build an eval around all three.
- An eval is composed of tasks — the atomic unit — each with inputs and success criteria. Treat it like a math exam: the working (the steps the agent took) matters as much as the final answer.
- Combine LLM-as-a-judge graders (robust to syntactic variation, e.g. different-but-equivalent SQL) with deterministic code-based graders (e.g. assert a specific tool was always called) per task.
- Avoid three common eval failure modes: mistaking noise for signal (run each task multiple times), infra failures (separate API/tool-call failures from genuine model failures by reading transcripts), and silent saturation (keep the dataset representative by feeding production traces back in).
- Read your transcripts. Set up observability (LangSmith, BrainTrust, etc.) so you can see exactly what the model saw and how it behaved — it once exposed Claude extracting answers from git history instead of solving the coding task.
- From the 4.6-class models onward, adaptive thinking lets the model decide how much to think (a scratchpad / system-two thinking before acting), while the effort parameter tells Claude how much to write across thinking, tool calls, and responses. The two are independent dials.
- Prompt caching pays 1/10 the list price of cached input tokens, effectively giving Opus quality at Sonnet cost; aim for an 80–90% cache hit rate, and use an append-only messages array (avoid dynamic variables like a datetime in the system prompt that break the cache).
- Context hygiene matters more than complex multi-agent orchestration: cleaning a tool response (markdown over JSON, simpler timestamps, deduplicating articles) cut tokens 66.4% in one case and, in a web-search case, cut input tokens 77% and cost 65% while accuracy rose 9%.
Eval — A curated dataset of tasks with inputs and success criteria used to measure how well a model performs on your specific workload.
Task — The atomic unit of an eval: a single test with inputs, expected outcome, and the expected intermediate steps.
LLM-as-a-judge — Using a language model to grade another model's output, tolerant of syntactic differences as long as the meaning or result matches.
Adaptive thinking — A capability (4.6-class onward) where the model itself decides how much to reason before acting, acting as a scratchpad for system-two thinking.
Effort — A parameter telling Claude how much to write/work across thinking, tool calls, and responses; independent of thinking.
Prompt caching — Reusing a precomputed, pre-cached prompt prefix to pay roughly 1/10 the list price of those input tokens.
Cache hit rate — The fraction of input tokens served from the prompt cache; best systems reach 80–90%.
Context engineering — Shaping the data fed to the model (cleaner tool responses, deduplication) to reduce tokens, lower cost, and improve accuracy.
Pareto frontier — The cost-vs-accuracy trade-off curve along which you select, or which you shift entirely with caching and context engineering.
Cheapest per successful outcome — The decision metric that values cost per completed task rather than cost per token.
My name's Lucas. I'm on our applied AI team, and today we're going to be talking about picking the right model. Conceptually this seems very easy, but the more we dig into it, the more difficult the problem tends to be. Consider the following scenario. We at Anthropic have just launched a new model and, as usual, there's a lot of noise. Alongside the launch we release a model card, prompting guides, and benchmark results. You'll also see a lot of hot takes on X — from "AGI is here" all the way to "Anthropic is cooked" — and various posts giving takes on the quality of our new model. But the key question for you is: what does that mean for your use case? What does it mean for your business? Should you drop this new model into your product — will it be an uplift? Or if you're building greenfield, which model should you even start with? To address that, we want to build a repeatable process: something you can come back to time and again that gives you a clear yes/no decision on when to select a new model. In other words, we want to build an eval.
Coming back to the original problem, conceptually it's simple: when you need more intelligence you reach for Opus; when you need lower latency or lower cost you reach for Haiku; and when you want a balance of the two you can use Sonnet. But what about effort levels? Should we reach for Sonnet with max thinking? Opus with low thinking? Haiku versus Sonnet with no thinking? And this is just Anthropic — many of you will be comparing across other providers too. So how do we frame this decision? There are really three pillars most folks think through. The first is model quality: how well does it actually perform on your task — task completion rate, accuracy, and other metrics. Number two is latency, which becomes very important for customer-facing use cases. And number three is cost, a big consideration for many customers. Through the lens of quality, cost, and latency we can build an eval to make an informed decision on which model to use.
There are three things to take away today. Number one: a small, well-designed eval will be much more important for assessing which model to use than any public benchmark. Number two: the model that's right for your use case is not necessarily the one that's cheapest or fastest per token, but the one that is cheapest per successful outcome. Number three: we have quite a lot of knobs and dials to move up and down the cost/accuracy frontier with fine-grained control — or to shift that curve entirely.
Let me reflect on what public benchmarks do and don't tell us. Directionally they give insight into whether a model has improved at coding or other tasks. Our releases often include results on things like SWE-bench Verified or BrowseComp — how good is the model generally at coding, or at research-type tasks. However, none of these really match your use case. A coding agent in production often needs to research a niche part of an SDK on the web and then implement it in code, so already we're crossing two benchmarks, and your workloads are usually more heterogeneous. You might have a coding task that looks very different from SWE-bench — maybe in languages that don't even show up there. So public benchmarks are somewhat directional, but for your specific workload it's incredibly important to build your own evals.
How do we do that? You'll hear us at Anthropic talk a lot about evals — we have other workshops on this, so this is an express tour. We build an eval composed of a set of tasks, where a task is the atomic unit: a test with inputs and success criteria, and we build a dataset of these. The heuristic I've used over the last few months is to think of an eval like a math exam at school: you have your question, the answer you need to get right, but it's also important to show the working in between. Especially for agentic tasks, this is exactly what we should think about. So we compose a dataset with a set of inputs; we want to check the agent reaches the final outcome correctly, but also that it took the right steps. The working is as important as the outcome.
Consider a real example: a customer-service agent. We might build an eval where an LLM-as-a-judge checks the final response matches what's expected, but we also use an LLM-as-a-judge to make sure the agent queried the database correctly to find the customer's details. LLM-as-a-judge is helpful because if your agent is writing SQL it can be syntactically different, and the judge notices that — as long as the same data is pulled it's robust to that. You can also use deterministic, code-based evals to ensure the agent always calls a specific tool to search your internal routines, or always adds an argument to localize the search to the customer's country. So you build a set of graders per task. You'll do a lot of hard yards creating these datasets, defining the right solutions, outcomes, and steps. But it's one of the highest-leverage things you'll do — in a world where we're automating a lot with AI, taking the time to build the eval dataset is one of the best uses of your human time.
The TL;DR is that we want a reference set of evals that forms the basis of a much more data-driven decision on picking the right model. We've been building evals at Anthropic for a while, and it's worth touching on common gotchas and failure modes. Number one is mistaking noise for signal: once you've defined your eval, run it a number of times on every task and make sure the result holds. If you see a lot of variance, that's a signal the task isn't well defined or your evals aren't fully aligned. Number two is infra failures: you might see numbers that look off — maybe really down for Opus — and when you dig into the transcripts you notice lots of API failures or tool-call failures. We should separate those from our actual evaluation of the model. These are infra issues, not model issues, so digging into transcripts to figure out where a failure happened is easy to overlook but a very good use of your time. Number three is silent saturation: have a dataset that's actually representative of production data and the types of questions a human might ask. Generate a feedback loop — once you've launched, collect traces, see what people are asking, see where your agent is failing, and put those back into your eval set so you get a representative, diverse sample.
It's worth calling out that every model has its own quirky behavioral changes. We release a prompting guide with every new model and put a lot of time into it — it's worth reading, or even feeding to Claude and asking it to update your prompts accordingly. For example, I was working on a tool within Claude AI: with Opus 4.5 that tool was drastically under-triggered, and with Opus 4.6 the tool was drastically over-triggered with the exact same prompt. So you'll need to do some fine-tuning and adjusting of your prompts based on the model you're using. Claude performs a little differently between variants, and differently than GPT, so there's some hand-tweaking too.
The other big takeaway: you really need to read the transcripts of what the agent or model is doing at different points. Make this as stupidly easy as possible — set up good observability, whether LangSmith, BrainTrust, or other platforms, so that if you're building an agent everything is traced: the system prompts, the tool call the agent made, the tool result it gets back. You want to dig in and at any point see exactly what the model saw and how it behaved, because that's your way to debug any issues. For example, we ran an eval on Claude Code and saw Claude performing very well on a coding benchmark. When we dug into the transcripts, we found Claude was going into the git history, seeing what it did in previous trials, and extracting the answer from there. If we'd just looked at the headline metrics we'd have thought we made a huge improvement — only by digging into transcripts do you see the underlying patterns and the real things that need fixing. The closer you can get to the raw data, the better.
To recap: building a private eval is extremely important for a data-informed model decision; we did an express loop of building an eval and its common failure modes. Now, what tools and dials are available to move up and down that frontier, trading off cost and quality or latency and quality? Let's start with a story. We had a code-fix pipeline internally, a pretty simple task. The team used Haiku 4.5 with no thinking and scored 92%. They wanted 100%, so they turned thinking on and got there. They also reran the eval with Sonnet and Opus, and both scored 100% as well — but counterintuitively took way less time. We weren't cost-constrained here; we just wanted it to run as fast as possible. This is interesting: you'd think Haiku would be much faster, but some of the more intelligent models can be much more time-efficient because they do things in fewer turns. They plan more strategically and don't need to spend as much time researching to validate that what they'll do is correct. So we can play around with these configs, thinking, and effort to extract maximum value for our specific use case.
It's worth touching on what these are. From Sonnet 4.6 or our 4.6-class of models onward we have adaptive thinking: the model itself decides how much it needs to think for a given task. This is a scratchpad to think before it acts — system-two thinking. Then we have effort, which tells Claude how much to write across thinking, tool calls, and responses. You can have low thinking with high effort, or no thinking but still use effort parameters. Effort tells Claude how much work to put into the task; thinking gives Claude a scratchpad to think before it acts. We can use both configs to have fine-grained control over where we are on the accuracy/cost curve.
Another counterintuitive example: when we released Opus 4.5, what we saw — Opus 4.5 is the orange line on screen — is that it completed tasks with higher accuracy than Sonnet and with significantly fewer output tokens. If you were just going off the vibe of "smaller model runs faster," you'd have chosen Sonnet, not knowing Opus can complete things faster and with fewer tokens. You can also see the Opus spread between different levels of effort, letting you make nice trade-offs: am I optimizing for fewer output tokens or for accuracy? Higher effort gives higher accuracy; lower effort gives lower latency. So the effort parameter gives more control than selecting a model alone.
What excites me is shifting that frontier entirely. There are a couple of hacks to not just move along the curve but shift it. The first is prompt caching. We have guides online — I'd encourage you to read them. When you're using prompt caching, you're using a prefix of a prompt that's saved, precomputed, and pre-cached, and you pay 1/10 the list price of input tokens. This means you can get Opus quality at Sonnet cost, or Sonnet quality at Haiku cost. It's one of the most effective strategies we use across our products — extensively in Claude Code and in Cowork. A lot of the best AI systems we see have prompt-cache hit rates around 80% or 90%, so that's your goal. This unlocks whole new use cases: if we pay 1/10 the list price of input tokens, suddenly we can afford Opus where we couldn't, or whole new budgets open up. In our APIs and SDKs we return token metrics — not only raw tokens consumed but the prompt-cache tokens on the input — so it's very easy to measure your cache hit rate. Measure it, hill-climb it, and get it as high as possible. The other main thing with prompt caching: especially if you're building an agent, use an append-only strategy. If you have a system prompt, don't have too many dynamic variables in it. The common failure mode is putting a datetime variable in the system prompt — with every turn the time ticks up and you break the cache. Treat everything in your messages array as immutable, and append only. Tarik from our Claude Code team has written and talked about this extensively, so go look at how he implemented it in Claude Code.
The second thing is context hygiene and context engineering. My hot take is people spend too much time on super-complex multi-agent orchestration and not enough doing the simple thing that works: good context hygiene and engineering. If we improve the token efficiency of our tool responses, we save a lot of tokens going into the model, saving cost and improving latency. There's also a secondary effect: because we give Claude cleaner, more efficient data, Claude's responses tend to be better and more accurate. For example, say we have a tool that returns sports data — Premier League scores. We made a few changes: first, use markdown instead of JSON; second, instead of full inefficient datetime stamps, a very simple one; and we added a day of the week so Claude more easily understands what day each game is, rather than having to reason about it. Just cleaning up this response gave a 66.4% reduction in tokens from the tool response. And that compounds every turn — your tool response shows up in every single turn of the conversation. These small hygiene things make a huge impact. A couple more examples: working on a web-search use case with a customer, we deduplicated articles returned from the same or different searches. Taking the articles and deduplicating them before returning to Claude led to a 77% reduction in input tokens, a 65% reduction in cost, and Claude's accuracy actually went up 9% because it's reasoning over less data. These metrics sound plucked out of thin air, but we can measure them because we built evals to begin with. If you can save 65% cost just by good context engineering, that opens up using a more intelligent model or running whole new use cases for your budget. So don't just take an API response and pipe it back through your tool — be more thoughtful. Clean up the JSON response before passing it to Claude; like a human, make it as simple and easy to read as possible. You'll reduce tokens and get the secondary accuracy benefit.
Three main things to take away before the workshop. Number one: a small, well-designed eval will teach you so much more about which model to use than any public benchmark could, so invest time in building it. Number two: the right model isn't the cheapest per token — it's the cheapest per successful outcome. Once you've built the eval, run Claude with multiple models and configurations, see the Pareto frontier, and choose where you want to be. Number three: use the dials — effort, thinking, prompt caching, and context engineering — to have fine-grained control over where you end up on that frontier, or to shift the frontier entirely. Do those three things and you'll be in a good place choosing the right model, even unlocking new use cases and accessing higher intelligence at lower cost.
Now I want to move into a workshop you can follow along with. Go to cwc26.short.gy/workshops. What we'll do — and you can do this in your own time — is we've built a skill that audits your existing evals and can also sweep the eval you've set up, instrumenting it to run across multiple models, thinking off and on, and multiple effort levels. It takes those results, plots them, saves them, and formats them nicely so you can see your eval performance across different models, thinking levels, and effort levels. Here we're doing it with Tau-bench, a benchmark of customer-service agent use cases, focusing on the airline side. In the README you'll see how to download and set up Tau-bench yourself; you can give a prompt to Claude and it'll set it up for you. I've already downloaded Tau-bench for time. The next part is to run the skill, instrument the code to run the eval across different models, thinking, and effort levels, and then run the eval itself.
Claude has loaded the skill, and in the interest of time I'll skip to the results. Here's one I ran earlier — the results from running this sweep. We've very easily run the benchmark across all three models — Haiku, Opus 4.7, Sonnet 4.6 — with thinking off and on and across different effort levels, and the results were surprising. The first chart shows pass rate per task against average output tokens per task. As we might expect, Opus 4.7 with thinking on and high effort has the highest pass rate — but it does so with fewer tokens than Sonnet consumes for the same use case. The next chart: even though Opus was best on high effort, it's also clearly the most expensive. If we optimize for pure pass rate, we'd pick Opus with high thinking; but if we optimize for cost it's different, because Haiku with thinking on performed similarly to Sonnet with thinking on and high effort. The final chart shows something similar but interesting: Opus with high effort and thinking on performed faster, with lower latency, than Sonnet with a similar level of thinking. What these charts really tell us is they give us the data to make an informed decision on which model and config to choose. We can identify non-intuitive properties of the models at different thinking levels and trade off our criteria — latency, cost, or model quality.
We're over time, so I'll end with the core takeaways. If you want to pick the right model, build an eval for your use case — for those who work with the Anthropic applied AI team, we're here to help you build those. Number two: optimize for what you care about in your use case — once you're making data-driven decisions, you can pick whether you're optimizing for intelligence, latency, or cost. Number three: you can shift the curve entirely by doing good context engineering and using strategies like prompt caching to get more out of the model. Thank you very much.
TL;DR
- Public benchmarks are only directional; a small, well-designed private eval built from your own representative tasks is far more useful for deciding whether a new model is right for your use case.
- The right model isn't the cheapest or fastest per token — it's the one that is cheapest per successful outcome, and counterintuitively more intelligent models like Opus can finish faster and with fewer tokens than smaller ones.
- You have many dials — thinking, effort, prompt caching, and context engineering — that let you move along the cost/accuracy frontier or shift the whole frontier, often unlocking Opus quality at Sonnet cost.
Takeaways
- Frame model choice through three pillars: model quality (task completion rate / accuracy), latency (critical for customer-facing use cases), and cost. Build an eval around all three.
- An eval is composed of tasks — the atomic unit — each with inputs and success criteria. Treat it like a math exam: the working (the steps the agent took) matters as much as the final answer.
- Combine LLM-as-a-judge graders (robust to syntactic variation, e.g. different-but-equivalent SQL) with deterministic code-based graders (e.g. assert a specific tool was always called) per task.
- Avoid three common eval failure modes: mistaking noise for signal (run each task multiple times), infra failures (separate API/tool-call failures from genuine model failures by reading transcripts), and silent saturation (keep the dataset representative by feeding production traces back in).
- Read your transcripts. Set up observability (LangSmith, BrainTrust, etc.) so you can see exactly what the model saw and how it behaved — it once exposed Claude extracting answers from git history instead of solving the coding task.
- From the 4.6-class models onward, adaptive thinking lets the model decide how much to think (a scratchpad / system-two thinking before acting), while the effort parameter tells Claude how much to write across thinking, tool calls, and responses. The two are independent dials.
- Prompt caching pays 1/10 the list price of cached input tokens, effectively giving Opus quality at Sonnet cost; aim for an 80–90% cache hit rate, and use an append-only messages array (avoid dynamic variables like a datetime in the system prompt that break the cache).
- Context hygiene matters more than complex multi-agent orchestration: cleaning a tool response (markdown over JSON, simpler timestamps, deduplicating articles) cut tokens 66.4% in one case and, in a web-search case, cut input tokens 77% and cost 65% while accuracy rose 9%.
Vocabulary
Eval — A curated dataset of tasks with inputs and success criteria used to measure how well a model performs on your specific workload.
Task — The atomic unit of an eval: a single test with inputs, expected outcome, and the expected intermediate steps.
LLM-as-a-judge — Using a language model to grade another model's output, tolerant of syntactic differences as long as the meaning or result matches.
Adaptive thinking — A capability (4.6-class onward) where the model itself decides how much to reason before acting, acting as a scratchpad for system-two thinking.
Effort — A parameter telling Claude how much to write/work across thinking, tool calls, and responses; independent of thinking.
Prompt caching — Reusing a precomputed, pre-cached prompt prefix to pay roughly 1/10 the list price of those input tokens.
Cache hit rate — The fraction of input tokens served from the prompt cache; best systems reach 80–90%.
Context engineering — Shaping the data fed to the model (cleaner tool responses, deduplication) to reduce tokens, lower cost, and improve accuracy.
Pareto frontier — The cost-vs-accuracy trade-off curve along which you select, or which you shift entirely with caching and context engineering.
Cheapest per successful outcome — The decision metric that values cost per completed task rather than cost per token.
Transcript
My name's Lucas. I'm on our applied AI team, and today we're going to be talking about picking the right model. Conceptually this seems very easy, but the more we dig into it, the more difficult the problem tends to be. Consider the following scenario. We at Anthropic have just launched a new model and, as usual, there's a lot of noise. Alongside the launch we release a model card, prompting guides, and benchmark results. You'll also see a lot of hot takes on X — from "AGI is here" all the way to "Anthropic is cooked" — and various posts giving takes on the quality of our new model. But the key question for you is: what does that mean for your use case? What does it mean for your business? Should you drop this new model into your product — will it be an uplift? Or if you're building greenfield, which model should you even start with? To address that, we want to build a repeatable process: something you can come back to time and again that gives you a clear yes/no decision on when to select a new model. In other words, we want to build an eval.
Coming back to the original problem, conceptually it's simple: when you need more intelligence you reach for Opus; when you need lower latency or lower cost you reach for Haiku; and when you want a balance of the two you can use Sonnet. But what about effort levels? Should we reach for Sonnet with max thinking? Opus with low thinking? Haiku versus Sonnet with no thinking? And this is just Anthropic — many of you will be comparing across other providers too. So how do we frame this decision? There are really three pillars most folks think through. The first is model quality: how well does it actually perform on your task — task completion rate, accuracy, and other metrics. Number two is latency, which becomes very important for customer-facing use cases. And number three is cost, a big consideration for many customers. Through the lens of quality, cost, and latency we can build an eval to make an informed decision on which model to use.
There are three things to take away today. Number one: a small, well-designed eval will be much more important for assessing which model to use than any public benchmark. Number two: the model that's right for your use case is not necessarily the one that's cheapest or fastest per token, but the one that is cheapest per successful outcome. Number three: we have quite a lot of knobs and dials to move up and down the cost/accuracy frontier with fine-grained control — or to shift that curve entirely.
Let me reflect on what public benchmarks do and don't tell us. Directionally they give insight into whether a model has improved at coding or other tasks. Our releases often include results on things like SWE-bench Verified or BrowseComp — how good is the model generally at coding, or at research-type tasks. However, none of these really match your use case. A coding agent in production often needs to research a niche part of an SDK on the web and then implement it in code, so already we're crossing two benchmarks, and your workloads are usually more heterogeneous. You might have a coding task that looks very different from SWE-bench — maybe in languages that don't even show up there. So public benchmarks are somewhat directional, but for your specific workload it's incredibly important to build your own evals.
How do we do that? You'll hear us at Anthropic talk a lot about evals — we have other workshops on this, so this is an express tour. We build an eval composed of a set of tasks, where a task is the atomic unit: a test with inputs and success criteria, and we build a dataset of these. The heuristic I've used over the last few months is to think of an eval like a math exam at school: you have your question, the answer you need to get right, but it's also important to show the working in between. Especially for agentic tasks, this is exactly what we should think about. So we compose a dataset with a set of inputs; we want to check the agent reaches the final outcome correctly, but also that it took the right steps. The working is as important as the outcome.
Consider a real example: a customer-service agent. We might build an eval where an LLM-as-a-judge checks the final response matches what's expected, but we also use an LLM-as-a-judge to make sure the agent queried the database correctly to find the customer's details. LLM-as-a-judge is helpful because if your agent is writing SQL it can be syntactically different, and the judge notices that — as long as the same data is pulled it's robust to that. You can also use deterministic, code-based evals to ensure the agent always calls a specific tool to search your internal routines, or always adds an argument to localize the search to the customer's country. So you build a set of graders per task. You'll do a lot of hard yards creating these datasets, defining the right solutions, outcomes, and steps. But it's one of the highest-leverage things you'll do — in a world where we're automating a lot with AI, taking the time to build the eval dataset is one of the best uses of your human time.
The TL;DR is that we want a reference set of evals that forms the basis of a much more data-driven decision on picking the right model. We've been building evals at Anthropic for a while, and it's worth touching on common gotchas and failure modes. Number one is mistaking noise for signal: once you've defined your eval, run it a number of times on every task and make sure the result holds. If you see a lot of variance, that's a signal the task isn't well defined or your evals aren't fully aligned. Number two is infra failures: you might see numbers that look off — maybe really down for Opus — and when you dig into the transcripts you notice lots of API failures or tool-call failures. We should separate those from our actual evaluation of the model. These are infra issues, not model issues, so digging into transcripts to figure out where a failure happened is easy to overlook but a very good use of your time. Number three is silent saturation: have a dataset that's actually representative of production data and the types of questions a human might ask. Generate a feedback loop — once you've launched, collect traces, see what people are asking, see where your agent is failing, and put those back into your eval set so you get a representative, diverse sample.
It's worth calling out that every model has its own quirky behavioral changes. We release a prompting guide with every new model and put a lot of time into it — it's worth reading, or even feeding to Claude and asking it to update your prompts accordingly. For example, I was working on a tool within Claude AI: with Opus 4.5 that tool was drastically under-triggered, and with Opus 4.6 the tool was drastically over-triggered with the exact same prompt. So you'll need to do some fine-tuning and adjusting of your prompts based on the model you're using. Claude performs a little differently between variants, and differently than GPT, so there's some hand-tweaking too.
The other big takeaway: you really need to read the transcripts of what the agent or model is doing at different points. Make this as stupidly easy as possible — set up good observability, whether LangSmith, BrainTrust, or other platforms, so that if you're building an agent everything is traced: the system prompts, the tool call the agent made, the tool result it gets back. You want to dig in and at any point see exactly what the model saw and how it behaved, because that's your way to debug any issues. For example, we ran an eval on Claude Code and saw Claude performing very well on a coding benchmark. When we dug into the transcripts, we found Claude was going into the git history, seeing what it did in previous trials, and extracting the answer from there. If we'd just looked at the headline metrics we'd have thought we made a huge improvement — only by digging into transcripts do you see the underlying patterns and the real things that need fixing. The closer you can get to the raw data, the better.
To recap: building a private eval is extremely important for a data-informed model decision; we did an express loop of building an eval and its common failure modes. Now, what tools and dials are available to move up and down that frontier, trading off cost and quality or latency and quality? Let's start with a story. We had a code-fix pipeline internally, a pretty simple task. The team used Haiku 4.5 with no thinking and scored 92%. They wanted 100%, so they turned thinking on and got there. They also reran the eval with Sonnet and Opus, and both scored 100% as well — but counterintuitively took way less time. We weren't cost-constrained here; we just wanted it to run as fast as possible. This is interesting: you'd think Haiku would be much faster, but some of the more intelligent models can be much more time-efficient because they do things in fewer turns. They plan more strategically and don't need to spend as much time researching to validate that what they'll do is correct. So we can play around with these configs, thinking, and effort to extract maximum value for our specific use case.
It's worth touching on what these are. From Sonnet 4.6 or our 4.6-class of models onward we have adaptive thinking: the model itself decides how much it needs to think for a given task. This is a scratchpad to think before it acts — system-two thinking. Then we have effort, which tells Claude how much to write across thinking, tool calls, and responses. You can have low thinking with high effort, or no thinking but still use effort parameters. Effort tells Claude how much work to put into the task; thinking gives Claude a scratchpad to think before it acts. We can use both configs to have fine-grained control over where we are on the accuracy/cost curve.
Another counterintuitive example: when we released Opus 4.5, what we saw — Opus 4.5 is the orange line on screen — is that it completed tasks with higher accuracy than Sonnet and with significantly fewer output tokens. If you were just going off the vibe of "smaller model runs faster," you'd have chosen Sonnet, not knowing Opus can complete things faster and with fewer tokens. You can also see the Opus spread between different levels of effort, letting you make nice trade-offs: am I optimizing for fewer output tokens or for accuracy? Higher effort gives higher accuracy; lower effort gives lower latency. So the effort parameter gives more control than selecting a model alone.
What excites me is shifting that frontier entirely. There are a couple of hacks to not just move along the curve but shift it. The first is prompt caching. We have guides online — I'd encourage you to read them. When you're using prompt caching, you're using a prefix of a prompt that's saved, precomputed, and pre-cached, and you pay 1/10 the list price of input tokens. This means you can get Opus quality at Sonnet cost, or Sonnet quality at Haiku cost. It's one of the most effective strategies we use across our products — extensively in Claude Code and in Cowork. A lot of the best AI systems we see have prompt-cache hit rates around 80% or 90%, so that's your goal. This unlocks whole new use cases: if we pay 1/10 the list price of input tokens, suddenly we can afford Opus where we couldn't, or whole new budgets open up. In our APIs and SDKs we return token metrics — not only raw tokens consumed but the prompt-cache tokens on the input — so it's very easy to measure your cache hit rate. Measure it, hill-climb it, and get it as high as possible. The other main thing with prompt caching: especially if you're building an agent, use an append-only strategy. If you have a system prompt, don't have too many dynamic variables in it. The common failure mode is putting a datetime variable in the system prompt — with every turn the time ticks up and you break the cache. Treat everything in your messages array as immutable, and append only. Tarik from our Claude Code team has written and talked about this extensively, so go look at how he implemented it in Claude Code.
The second thing is context hygiene and context engineering. My hot take is people spend too much time on super-complex multi-agent orchestration and not enough doing the simple thing that works: good context hygiene and engineering. If we improve the token efficiency of our tool responses, we save a lot of tokens going into the model, saving cost and improving latency. There's also a secondary effect: because we give Claude cleaner, more efficient data, Claude's responses tend to be better and more accurate. For example, say we have a tool that returns sports data — Premier League scores. We made a few changes: first, use markdown instead of JSON; second, instead of full inefficient datetime stamps, a very simple one; and we added a day of the week so Claude more easily understands what day each game is, rather than having to reason about it. Just cleaning up this response gave a 66.4% reduction in tokens from the tool response. And that compounds every turn — your tool response shows up in every single turn of the conversation. These small hygiene things make a huge impact. A couple more examples: working on a web-search use case with a customer, we deduplicated articles returned from the same or different searches. Taking the articles and deduplicating them before returning to Claude led to a 77% reduction in input tokens, a 65% reduction in cost, and Claude's accuracy actually went up 9% because it's reasoning over less data. These metrics sound plucked out of thin air, but we can measure them because we built evals to begin with. If you can save 65% cost just by good context engineering, that opens up using a more intelligent model or running whole new use cases for your budget. So don't just take an API response and pipe it back through your tool — be more thoughtful. Clean up the JSON response before passing it to Claude; like a human, make it as simple and easy to read as possible. You'll reduce tokens and get the secondary accuracy benefit.
Three main things to take away before the workshop. Number one: a small, well-designed eval will teach you so much more about which model to use than any public benchmark could, so invest time in building it. Number two: the right model isn't the cheapest per token — it's the cheapest per successful outcome. Once you've built the eval, run Claude with multiple models and configurations, see the Pareto frontier, and choose where you want to be. Number three: use the dials — effort, thinking, prompt caching, and context engineering — to have fine-grained control over where you end up on that frontier, or to shift the frontier entirely. Do those three things and you'll be in a good place choosing the right model, even unlocking new use cases and accessing higher intelligence at lower cost.
Now I want to move into a workshop you can follow along with. Go to cwc26.short.gy/workshops. What we'll do — and you can do this in your own time — is we've built a skill that audits your existing evals and can also sweep the eval you've set up, instrumenting it to run across multiple models, thinking off and on, and multiple effort levels. It takes those results, plots them, saves them, and formats them nicely so you can see your eval performance across different models, thinking levels, and effort levels. Here we're doing it with Tau-bench, a benchmark of customer-service agent use cases, focusing on the airline side. In the README you'll see how to download and set up Tau-bench yourself; you can give a prompt to Claude and it'll set it up for you. I've already downloaded Tau-bench for time. The next part is to run the skill, instrument the code to run the eval across different models, thinking, and effort levels, and then run the eval itself.
Claude has loaded the skill, and in the interest of time I'll skip to the results. Here's one I ran earlier — the results from running this sweep. We've very easily run the benchmark across all three models — Haiku, Opus 4.7, Sonnet 4.6 — with thinking off and on and across different effort levels, and the results were surprising. The first chart shows pass rate per task against average output tokens per task. As we might expect, Opus 4.7 with thinking on and high effort has the highest pass rate — but it does so with fewer tokens than Sonnet consumes for the same use case. The next chart: even though Opus was best on high effort, it's also clearly the most expensive. If we optimize for pure pass rate, we'd pick Opus with high thinking; but if we optimize for cost it's different, because Haiku with thinking on performed similarly to Sonnet with thinking on and high effort. The final chart shows something similar but interesting: Opus with high effort and thinking on performed faster, with lower latency, than Sonnet with a similar level of thinking. What these charts really tell us is they give us the data to make an informed decision on which model and config to choose. We can identify non-intuitive properties of the models at different thinking levels and trade off our criteria — latency, cost, or model quality.
We're over time, so I'll end with the core takeaways. If you want to pick the right model, build an eval for your use case — for those who work with the Anthropic applied AI team, we're here to help you build those. Number two: optimize for what you care about in your use case — once you're making data-driven decisions, you can pick whether you're optimizing for intelligence, latency, or cost. Number three: you can shift the curve entirely by doing good context engineering and using strategies like prompt caching to get more out of the model. Thank you very much.