- Ben and Jeff from Anthropic's applied AI team run a hands-on "agent battle" workshop where participants build and deploy a managed agent to mine the most diamonds in Minecraft.
- The exercise teaches three things: building/deploying a managed agent, understanding the impact of agent configuration (system prompt, model string, skills, MCPs), and hill-climbing on evals.
- Rules reward token efficiency: each run is five minutes of mining, only the top run per person counts, ties are broken by the best diamonds-to-tokens ratio — pushing participants to hone the system prompt rather than just pick the heaviest model.
Agent Battle: Mine the most diamonds in 45 minutes
- The workshop has three goals: learn to build and deploy a managed agent (hosted on Anthropic's infrastructure), understand the impact of agent configuration, and learn to hill-climb on evals the way Anthropic improves its own agents internally.
- Agent configuration means the system prompt, the model string, and the skills and MCPs plugged in to elicit the desired behavior.
- Hill-climbing on evals is the internal process of measuring an agent's impact, understanding its behavior, and iteratively making changes to improve — the same loop used for production agents.
- Battle rules: a 35-minute build/experiment window, only one accepted run per person (top run counts), each run is five minutes of mining, a roughly one-minute eval set for quick iteration, and the most diamonds wins.
- Ties are settled by token efficiency (diamonds-to-tokens ratio), so success is not just mining the most diamonds but honing the system prompt rather than throwing in the heaviest-weight model.
- The harness ships ready to use: a Minecraft clone connects to a Mineflayer bot exposed through MCP tools (e.g. mine block, jump, go near things) rather than visuals.
- Everyone starts from the same seed and start kit, so there's no optimization in the starting conditions; the levers live in my_agent.py where you adjust the model, the system prompt (initially empty), and a shipped or custom skill, plus an optional MCP server.
- The live leaderboard showed roughly 19 diamonds as the upper echelon, with one competitor breaking 19 with about a minute left; a Claude Code setup skill was provided to help with environment setup and conference Wi-Fi issues.
Managed agent — An agent hosted on Anthropic's infrastructure with much of the configuration pre-set, released a few weeks before the workshop.
Agent configuration — The combination of system prompt, model string, skills, and MCPs that shapes an agent's behavior.
Hill-climbing on evals — Iteratively measuring, understanding, and improving an agent's performance against an evaluation set.
System prompt — The instructions given to the agent that strongly influence its behavior; here it starts empty for participants to tune.
Token efficiency — The ratio of useful output (diamonds) to tokens spent, used as the tiebreaker.
Mineflayer — A bot framework the Minecraft clone connects to, controlled via MCP tools instead of visuals.
MCP tools — Model Context Protocol tools (e.g. mine block, jump, go near) the agent uses to act in the game.
Seed — The fixed Minecraft world-generation value everyone starts from, removing starting-condition variance.
What's going on everyone? Let's get settled in. This is going to be an agent battle and we're going to be tight on time, so we're going to get right to it. My name is Ben. I'm accompanied here by Jeff. We're on the applied AI team here at Anthropic, which means we help folks like yourselves squeeze the most juice out of the Claude ecosystem. Today we're going to be mining for diamonds in Minecraft. You might have played Minecraft. I did in the 2000s, 2010s, but in the 2020s, we have agents play Minecraft for us. And that's what we're going to be doing today.
What are we actually going to accomplish here today? There's really three things. Number one, we're going to learn how to build and deploy a managed agent. This is something that we released a few weeks ago. It's hosted on our infrastructure and we've set up a lot of the configuration for you, but your job is going to be to get the agent into peak diamond-mining condition. Number two is we're going to understand the impact of an agent configuration. That's the system prompt, the model string, the skills and MCPs that you're plugging into your agent to get it to elicit the behavior that you're actually looking for. And then number three, you heard Will talk about this in the last workshop session if you were here: we're going to learn how to hill-climb on evals. This is how we improve all of our agents internally. If you've deployed one into prod, you understand the process of measuring its impact, understanding its behavior, and then making changes to iteratively improve.
This is an agent battle, but there are some rules to the battle. Number one is we're going to have a timer for about 35 minutes in which you're going to be able to build and experiment with your agents. We're only going to accept one run per person. So you can submit as many as you'd like, but we're only taking your top run. Each run is five minutes of the agent mining for diamonds. You can kill the run at any point with just a quick Control-C in the terminal. We have an eval set that takes approximately one minute if you just want to quickly iterate on the agents, and whoever has the most diamonds at the end of 35 minutes wins. We're going to have a leaderboard up here to show you where you're at. There's also going to be a chat where your agents can talk to one another. And if anyone is in a tie at the end of the workshop, we're going to be settling that tie based on token efficiency. So this is not just mine the most diamonds — it is get the best diamonds-to-tokens ratio. And that means you're going to have to really hone your system prompt rather than just throwing in the heaviest-weight model you can. Now, I'm going to hand it over to Jeff to talk through the logistics of how the harness actually works.
Hello. Okay. So the harness — we're actually shipping quite a bit for you to get started. What you're going to primarily be thinking about is how can I optimize my experience in trying to mine as many diamonds as possible. We're giving you a couple different tools to accomplish this. The main structure is that you're going to be running through a Minecraft clone that connects to a Mineflayer bot. If you're not familiar with Mineflayer, you're not going to be relying on visuals. There's a series of MCP tools that are shipped directly with that, such as mine block, jump, go near things, something along those lines. You don't have to think too much about this. The main levers that you're going to be focused on are basically along the lines of how do I optimize this run. Everyone's going to start from the same seed, so there's no real optimization that needs to happen there. And every time you reset, you'll have the same start kit, the same seed to go with that.
Okay, so the agent that you'll be operating out of — there's a my_agent.py that's included in the repo. So if you go to /agent-battle, you'll have access to the repo and that's where you'll be running this. my_agent.py is where everything should really be taking place. You can adjust the model, you can adjust the system prompt, which is currently empty, and then you have the opportunity to use a skill that's shipped directly from Anthropic, or you can replace it with your own skill. And then we also are including an MCP server if you wish to adjust these things. So these are the parameters that you have available to adjust. So try things out. Run evals. We'll also be circulating around the room, so if you run into anything or want to discuss, then feel free to do so. Okay, this has largely already been covered, so I want to make sure that we have as much time as possible to get started. So we can actually move over to starting the countdown. Looks like several people have already joined. I'm going to reset the time now and you may begin. So these will all go away. It's completely refreshed, and the time has begun now. So feel free to get started, and if you have questions, we'll be around.
It looks like we already have a run with 10. We have several other participants who are getting started. Just to give slightly more context: within the repo there's a Claude Code skill that you can use to help with setup. If anyone's having an issue, that should hopefully help with the process. A couple people announced that they were having issues with Cloudflare. I think it may be due to the fact that there's so many people trying to connect via the conference Wi-Fi. It's a bit small, but I did put a command up here that found that it worked for at least one person, so feel free to give that a try if you're running into any issues. About five minutes remaining. We have a three-way tie. And for some reason, the first person is showing as zero tokens, which is highly suspicious. So we may have a two-way tie. It seems like 19 might be the upper echelon of what's possible, at least so far. We have time for about one more run. We actually have somebody who's broken 19 with only one minute 20 seconds to go. Wow, you have to reveal your technique. 20 seconds. 10, 9, 8, 7, 6, 5, 4, 3, 2, one. Time's up.
So thank you everybody for participating. It looks like we have a clear winner now and would love for everybody who was able to mine 19 diamonds to come find Ben and I. Because our second place has zero tokens, we don't know who actually won second and third place yet. So we'll invite everybody who is one through five to come up. We don't know what they'll win yet. You guys can smile in front of your screens. Nice job, everybody.
TL;DR
- Ben and Jeff from Anthropic's applied AI team run a hands-on "agent battle" workshop where participants build and deploy a managed agent to mine the most diamonds in Minecraft.
- The exercise teaches three things: building/deploying a managed agent, understanding the impact of agent configuration (system prompt, model string, skills, MCPs), and hill-climbing on evals.
- Rules reward token efficiency: each run is five minutes of mining, only the top run per person counts, ties are broken by the best diamonds-to-tokens ratio — pushing participants to hone the system prompt rather than just pick the heaviest model.
Takeaways
- The workshop has three goals: learn to build and deploy a managed agent (hosted on Anthropic's infrastructure), understand the impact of agent configuration, and learn to hill-climb on evals the way Anthropic improves its own agents internally.
- Agent configuration means the system prompt, the model string, and the skills and MCPs plugged in to elicit the desired behavior.
- Hill-climbing on evals is the internal process of measuring an agent's impact, understanding its behavior, and iteratively making changes to improve — the same loop used for production agents.
- Battle rules: a 35-minute build/experiment window, only one accepted run per person (top run counts), each run is five minutes of mining, a roughly one-minute eval set for quick iteration, and the most diamonds wins.
- Ties are settled by token efficiency (diamonds-to-tokens ratio), so success is not just mining the most diamonds but honing the system prompt rather than throwing in the heaviest-weight model.
- The harness ships ready to use: a Minecraft clone connects to a Mineflayer bot exposed through MCP tools (e.g. mine block, jump, go near things) rather than visuals.
- Everyone starts from the same seed and start kit, so there's no optimization in the starting conditions; the levers live in my_agent.py where you adjust the model, the system prompt (initially empty), and a shipped or custom skill, plus an optional MCP server.
- The live leaderboard showed roughly 19 diamonds as the upper echelon, with one competitor breaking 19 with about a minute left; a Claude Code setup skill was provided to help with environment setup and conference Wi-Fi issues.
Vocabulary
Managed agent — An agent hosted on Anthropic's infrastructure with much of the configuration pre-set, released a few weeks before the workshop.
Agent configuration — The combination of system prompt, model string, skills, and MCPs that shapes an agent's behavior.
Hill-climbing on evals — Iteratively measuring, understanding, and improving an agent's performance against an evaluation set.
System prompt — The instructions given to the agent that strongly influence its behavior; here it starts empty for participants to tune.
Token efficiency — The ratio of useful output (diamonds) to tokens spent, used as the tiebreaker.
Mineflayer — A bot framework the Minecraft clone connects to, controlled via MCP tools instead of visuals.
MCP tools — Model Context Protocol tools (e.g. mine block, jump, go near) the agent uses to act in the game.
Seed — The fixed Minecraft world-generation value everyone starts from, removing starting-condition variance.
Transcript
What's going on everyone? Let's get settled in. This is going to be an agent battle and we're going to be tight on time, so we're going to get right to it. My name is Ben. I'm accompanied here by Jeff. We're on the applied AI team here at Anthropic, which means we help folks like yourselves squeeze the most juice out of the Claude ecosystem. Today we're going to be mining for diamonds in Minecraft. You might have played Minecraft. I did in the 2000s, 2010s, but in the 2020s, we have agents play Minecraft for us. And that's what we're going to be doing today.
What are we actually going to accomplish here today? There's really three things. Number one, we're going to learn how to build and deploy a managed agent. This is something that we released a few weeks ago. It's hosted on our infrastructure and we've set up a lot of the configuration for you, but your job is going to be to get the agent into peak diamond-mining condition. Number two is we're going to understand the impact of an agent configuration. That's the system prompt, the model string, the skills and MCPs that you're plugging into your agent to get it to elicit the behavior that you're actually looking for. And then number three, you heard Will talk about this in the last workshop session if you were here: we're going to learn how to hill-climb on evals. This is how we improve all of our agents internally. If you've deployed one into prod, you understand the process of measuring its impact, understanding its behavior, and then making changes to iteratively improve.
This is an agent battle, but there are some rules to the battle. Number one is we're going to have a timer for about 35 minutes in which you're going to be able to build and experiment with your agents. We're only going to accept one run per person. So you can submit as many as you'd like, but we're only taking your top run. Each run is five minutes of the agent mining for diamonds. You can kill the run at any point with just a quick Control-C in the terminal. We have an eval set that takes approximately one minute if you just want to quickly iterate on the agents, and whoever has the most diamonds at the end of 35 minutes wins. We're going to have a leaderboard up here to show you where you're at. There's also going to be a chat where your agents can talk to one another. And if anyone is in a tie at the end of the workshop, we're going to be settling that tie based on token efficiency. So this is not just mine the most diamonds — it is get the best diamonds-to-tokens ratio. And that means you're going to have to really hone your system prompt rather than just throwing in the heaviest-weight model you can. Now, I'm going to hand it over to Jeff to talk through the logistics of how the harness actually works.
Hello. Okay. So the harness — we're actually shipping quite a bit for you to get started. What you're going to primarily be thinking about is how can I optimize my experience in trying to mine as many diamonds as possible. We're giving you a couple different tools to accomplish this. The main structure is that you're going to be running through a Minecraft clone that connects to a Mineflayer bot. If you're not familiar with Mineflayer, you're not going to be relying on visuals. There's a series of MCP tools that are shipped directly with that, such as mine block, jump, go near things, something along those lines. You don't have to think too much about this. The main levers that you're going to be focused on are basically along the lines of how do I optimize this run. Everyone's going to start from the same seed, so there's no real optimization that needs to happen there. And every time you reset, you'll have the same start kit, the same seed to go with that.
Okay, so the agent that you'll be operating out of — there's a my_agent.py that's included in the repo. So if you go to /agent-battle, you'll have access to the repo and that's where you'll be running this. my_agent.py is where everything should really be taking place. You can adjust the model, you can adjust the system prompt, which is currently empty, and then you have the opportunity to use a skill that's shipped directly from Anthropic, or you can replace it with your own skill. And then we also are including an MCP server if you wish to adjust these things. So these are the parameters that you have available to adjust. So try things out. Run evals. We'll also be circulating around the room, so if you run into anything or want to discuss, then feel free to do so. Okay, this has largely already been covered, so I want to make sure that we have as much time as possible to get started. So we can actually move over to starting the countdown. Looks like several people have already joined. I'm going to reset the time now and you may begin. So these will all go away. It's completely refreshed, and the time has begun now. So feel free to get started, and if you have questions, we'll be around.
It looks like we already have a run with 10. We have several other participants who are getting started. Just to give slightly more context: within the repo there's a Claude Code skill that you can use to help with setup. If anyone's having an issue, that should hopefully help with the process. A couple people announced that they were having issues with Cloudflare. I think it may be due to the fact that there's so many people trying to connect via the conference Wi-Fi. It's a bit small, but I did put a command up here that found that it worked for at least one person, so feel free to give that a try if you're running into any issues. About five minutes remaining. We have a three-way tie. And for some reason, the first person is showing as zero tokens, which is highly suspicious. So we may have a two-way tie. It seems like 19 might be the upper echelon of what's possible, at least so far. We have time for about one more run. We actually have somebody who's broken 19 with only one minute 20 seconds to go. Wow, you have to reveal your technique. 20 seconds. 10, 9, 8, 7, 6, 5, 4, 3, 2, one. Time's up.
So thank you everybody for participating. It looks like we have a clear winner now and would love for everybody who was able to mine 19 diamonds to come find Ben and I. Because our second place has zero tokens, we don't know who actually won second and third place yet. So we'll invite everybody who is one through five to come up. We don't know what they'll win yet. You guys can smile in front of your screens. Nice job, everybody.