- Replit's president and head of AI, Michele Catasta, argues that agent evaluation must shift from a one-time "boolean gatekeeper" score to a continuous engine that ships a better agent every single day.
- Replit runs two pillars: offline benchmarks (their new open-source VibeBench, which builds whole apps from PRDs and grades them with browser-driving automated evaluators) used before shipping, and online evaluation (AB testing plus semantic trace clustering, named Telescope) used after shipping.
- The talk emphasizes that AB tests rarely give crystal-clear results, so human taste and product philosophy remain essential, and recent long-context reasoning models (Opus 4.5+) finally make trace-based feedback loops practical.
Evaluating and improving Replit Agent at scale
- Vibe coding at Replit is an extreme case: users go from a natural-language prompt to a working app with no chosen framework and no tests, so traditional developer-style evals fundamentally must change.
- Because models, prompts, and tool definitions change constantly (multiple agent releases per day to millions of users), evaluation has to be continuous rather than a single human-produced score.
- SweetBench/HumanEval follow a fixed protocol (apply a patch, run existing tests) that doesn't fit vibe coding where there are no tests and you build from an empty repo; what matters is functional correctness — "does the app do what the user asked?"
- VibeBench takes a PRD as input, builds the app end-to-end, and uses automated evaluators that read the code, open a browser, and step through a natural-language test plan; pairings range from zero-to-one to "slop on slop" (building features on unverified agent code) to parallel task-decomposition-plus-merge.
- Notable VibeBench results: a ~2x gap between frontier and open-weight models, and most models do worse when extending their own code — hence always add a testing step between features.
- The online pillar (Telescope) clusters millions of daily production traces semantically, embeds failure summaries, classifies them with an LLM, and retrains clusters nightly to catch rare long-tail failures (e.g. 1% occurrence) invisible to a DataDog dashboard.
- Telescope's loop: discover problems from clustered traces, auto-cut PRs with a coding agent, run VibeBench as a litmus test (drop of 10 points = bad change), AB test controversial changes, and ship clear wins — about 90% aided by an agent in the loop.
- AB tests rarely give a clean signal (e.g. +7% run duration but 8% cheaper), so human taste, hypothesis formulation, and product philosophy determine what to prioritize and whether to launch.
Vibe coding — Going from a plain natural-language specification straight to a working application, without the user choosing frameworks or writing tests.
VibeBench — Replit's open-source benchmark for end-to-end vibe coding: a PRD goes in, an app is built, and automated evaluators grade functional correctness.
PRD — Product requirements document; here a long English prompt describing how to build an application, used as benchmark input.
Automated evaluator — An AI agent that reads a codebase, opens a browser, and steps through a natural-language test plan to score whether the app works.
Functional correctness — Whether the app actually does what the user asked, as opposed to whether code passes pre-written tests.
Telescope — Replit's internal system that clusters production traces, discovers problems, and auto-generates fixes.
Trace clustering — Grouping production traces by semantic similarity (via embeddings and an LLM) to surface recurring failure modes.
AB testing — Running multiple agent versions in production simultaneously to compare metrics like duration, cost, and sentiment.
Slop on slop — Building new agent-written features on top of earlier unverified agent code; the most challenging benchmark scenario.
Hi everyone. I'm Michele Catasta, president and head of AI at Replit. Today I'm going to be talking about both evaluating and improving on a daily basis Replit Agent at scale. As you know, Replit is a vibe-coding platform for knowledge workers. We're one of the top players in the space and we've been literally battling with this problem for the last almost 2 years. Vibe coding has a very broad definition. It ranges all the way from being used by software developers, but in our specific case it's even more of an extreme definition where you start from just a natural language specification of what the user wants and literally nothing else. So the user expects to go from a prompt to a working application, but they don't tell us what kind of framework they want to use. They don't write tests. They just expect things to work after our agent runs. So it means that a lot of the evaluations and a lot of the ways in which we've been building agents in the past, especially for software developers, has to fundamentally change.
Now, how do we track internally the fact that the agent gets better on a daily basis and is actually aligned with what our users want? This is especially challenging because if you think about it, models are changing almost constantly, and if anything the pace of evolution is even faster than it was a year ago. So we are always sitting on shifting grounds. Models change fast. We have to change our prompts, system prompts, user prompts very often to comply with the new features that we're building in product. Of course, the tool definitions are changing as well. We're adding more of them, we're simplifying, and we're shipping constantly. Every single day we have multiple releases of Replit Agent, which we put in front of millions of users. So my argument today for this talk will be that we have to fundamentally rethink how we do evaluations.
On the left side, you can see how evals look like for basically since we've been working on AI. You run your evaluation, they produce a single score, and that score is produced by a human. And what the human decides based on the score is, my agent harness is better than it was before or it regressed, or my model is better than before, or we should be doing more post-training. That's how we've been doing this for a long time. I argue that we should move more and more towards a continuous setting of evaluations, where especially when you actually have something that runs in production, and in our case it generates millions of traces every day, there is a lot of signal to be extracted from the traces themselves. So what we do is we have a combination of offline evals, very similar to SweetBench and other benchmarks that are very famous in the space, as well as online evaluations based on the system usage.
What do we want our evals to capture? Fundamentally, first of all, they need to optimize what our users care about, rather than just telling us if the agent is editing code correctly. They should also highlight immediately what is breaking, and they should inform us where what we should be shipping next in terms of improvement for our agent.
What goes into this black box? We're building a new system that allows us to do these constant improvements. The system has two pillars. On one end, we have the standard old-school benchmarks. They are usually employed before shipping a new version of the agent. Think of them sort of as a boolean flag. It's a gatekeeper before deciding if you should be launching a new version or not. So if there is a major regression on an offline benchmark, of course you stop your release. If you see no changes or positive changes, you know that you can go on. But the part that is more intriguing and is what is helping us to make constant progress is the fact that we do a lot of AB testing, and we also cluster all the traces that we obtain in production and we try to gain insights from them. So in this case, the online pillar actually works after you ship a version of the agent and it forces you to react as fast as possible. And the loop here is, once I have these two pillars in place, you reflect on the results that both of them are giving you, you likely change the code once again, run your eval, rinse and repeat. And this is the cycle that we are going through every single day at Replit.
I mentioned SweetBench and I want to give you a quick understanding why we work on something more powerful than SweetBench, or at least more tailored towards our use case. Something like SweetBench, HumanEval, they played a key role in our space to allow us to hill-climb them and make models more powerful. But eventually, they all follow the same protocol. You make sure that the code that is generated complies with the user prompt, then you apply the patch on your repository, you run the test. If the tests are passing, you have a higher score on the benchmark. This does not reflect what happens in vibe coding. As I was mentioning before, users are not writing the tests. They often start from a completely empty code base. So there is not a scenario where you can just apply patches. You're building things from the ground up. So what we need to capture instead is: does the app do what the user asked? There is a functional correctness gap between what SweetBench provides and what we were working on.
And today on stage, I'm launching VibeBench. It's a new public benchmark for vibe coding end-to-end that we worked on at Replit for several months. The input is just a PRD. It's literally like a long prompt that describes how to build an application. Rather than crafting them from synthetic data, we pick 20 real-world traces from usage of Replit, just plain English specifications without any implementation constraints. Then, we have a harness that builds the application end-to-end. So it goes from an empty repo into something functional. And then the key insight is, rather than stopping the benchmark here and then having humans evaluate the output of those PRDs, we built automated evaluators. This is what allows you to go from a benchmark that you run maybe on a weekly basis into something that you can run every single time you literally have a new PR merge in your repository. So as you can imagine, we have AI running all the evaluations on our behalf.
The setup is the following. You have an input, you have an implementation strategy, and then you have a set of our wired evaluations that you run on your app. And we came up with five different pairings, but in reality there are a lot more pairings that you could come up with yourself. The most basic pairing is the input is the PRD, and we build the application single-shot, zero to one. Then, in order to go from less complicated to more advanced, we have something called VibeBench Ref, or the reference implementation, where we start from something already working and we build a feature on top of that. Let's make it more advanced. We have VibeBench Vibe where we start from an agent MVP and then we build a new feature on top of that. Maybe this is something that in the lingo you would call slop on slop. So if we have something that's been built by the agent, not verified, and then you had more agent code written on top of that, and then you run the evaluations to see if it's working or not. You can keep making this arbitrarily more complex. For example, in Agent 4 that we launched a couple of months ago, we completely hide the complexity of doing task decomposition and running agents in parallel and then merging all the patches together. So we can also take as an input the PRD, we do the task decomposition, we run all of that in parallel and then we run the evals after that. As you can imagine the parallel-plus-merge scenario is far more challenging for a standard coding agent. You can even start from a buggy application, build a feature on top of that and see how much your agent struggles to make it work correctly.
So the key question or the complexity here is how do you actually do the grading? That's where we spent most of our effort. It turns out that when we were working on our previous version of the agent, we put quite a lot of effort in building an automated app-testing feature which literally doesn't know anything about the app per se. Because when it comes to vibe coding, we don't give any level of guardrails to our users. They can decide the language they want to use, they can decide a different framework. So the evaluator has to be totally agnostic on how the implementation looks like. And what our evaluator agent does is it reads the code base, it then opens a browser and points it to the application that our agent has built, and then step-by-step goes through our testing plan. And even the testing plan is expressed in natural language and the actions are like "open the admin dashboard and log in with a certain account and click on this toggle." And if any of those steps fails, then we collect all of them together and we generate a score. So we go through all these test plans and then we decide ultimately if the score is good or not. When it comes to SweetBench, we always had a fixed surface. We know the repositories, we know the task hardness, we know exactly how to make this work. In VibeBench, the surface is completely green field and that's why it took us several months of work to get there.
We're announcing it today. You can find the entire benchmark available open source on vibebench.ai. It's going to be presented a couple of weeks from today at the conference on AI and agentic systems. Peter, the lead of the project and first author of the paper, is sitting here. I'm just going to give you a sneak peek of the most notable results. First of all, we are noticing almost a 2x gap between the frontier models and the open-weights models. The reason why I want you to pay attention there is because we know that every single model builder is hill-climbing specific benchmarks. That's why we release this open source. I want the entire community to embrace it across open weights and closed weights and make sure that we make progress overall on vibe coding as well. The other one, possibly not very surprising, is most models do a worse job when extending their own code. So the slop-on-slop or VibeBench Vibe scenario that I was describing to you before is by far the most challenging. And that's the reason why I always advocate to have a testing step in between every time you create a new feature, otherwise you keep building on shaky foundations and eventually your vibe coding application is going to fail.
Let's move to the next step. We talked about offline eval, I talked to you about VibeBench, but as you recall there were two main pillars that we care about. And the reason why we also care about the online one is because there is a great difference in terms of volume of signal that we can collect. So on the offline one right now we're working with roughly 20 applications. The reason why we made it open is because we are more than welcoming contributions from the community. So more applications, harder prompts, we can keep making this benchmark arbitrarily harder so that we make our agent's life harder over time. But when it comes to Replit Engine instead, we're collecting millions of sessions every day and they're very valuable because they capture what our users actually do on the platform. And they're completely unscripted, the agent is always running.
So how do we distill useful information out of them? Well, it turns out that we run a lot of AB tests. It's basically our way to keep ourselves honest because VibeBench only tells us part of the story. And I invite every agent builder out there to start to invest on their AB testing infrastructure as soon as possible. It's the best way in which you can make steady progress. We ask a wide variety of questions for ourselves. We built a lot of different instrumentation and metrics in our agent so that, for instance, we can find out: is there a normal span ongoing or is the agent running for longer than expected? Or we constantly keep track of the user sentiment. It's a very easy thing to collect because every time you receive a prompt, you can do sentiment analysis on it and then perceive if the user is getting frustrated or not. And last but not least, this is very idiosyncratic to Replit. Our users can also publish their application on our product. So it's a very strong positive signal if they decide that whatever they built is worth sharing with their colleagues or putting in public in front of everyone.
So all these signals clustered together give us something like this. If you have ever done an AB test in your life, I'm sure this looks familiar. And it might look familiar especially because not everything is either green or red. This is the harsh truth about running an AB test. They never give you a crystal-clear signal of what you should be doing. So in this case, for example, the average run duration of our agent went up by 7%, but conversely it's 8% cheaper, and we saw the fluctuation in terms of positive and negative sentiment. What shall we do in this case? This is where human taste and product philosophy still plays a key role. You will very rarely get a crystal-clear configuration of the results of your AB test.
And to make our life a bit easier on how to generate these AB test candidates, what we do is we cluster all the traces that we receive every single day and night, and we try to identify first of all which clusters are capturing normal nominal behavior of the agent. And as you can imagine, there are plenty of them because the vast majority of times the application is actually successful, but of course there is a long tail of problems that happen to surface. In case we find a problematic cluster, then what do we do? We embed all the failure summaries. We cluster them by type so that we find out the different failures that are happening at a certain point in time. Then we pass them through an LLM to classify exactly what is going on. And the fact that we do this not based on regex on logs or very deterministic techniques is what actually makes a difference, because you will be able to cluster things that are semantically close to each other even though the agent doesn't give you always the same type of output. And once we find these cluster configurations, we are forced to retrain them every single night because we are running several versions of the engine in parallel. We ship many versions every single day, so we can't live with a fixed cluster configuration. And aside from also helping us to identify the failure modes, another interesting feature is the fact that once we retrain those clusters, we can go back and see if after we believe we fixed a problem, a certain cluster has actually disappeared. So say you have a certain tool failure that is happening only a limited amount of time. So not something that will show up in your DataDog dashboard, for example, not a failure that happens 50% of the times, but maybe it was happening 1% of the times in certain specific conditions. And from here, you will start to realize, okay, I shipped my new PR and that cluster has disappeared. So you start to have some evidence that you mitigated the problem.
The technology that we built internally was called Telescope and it's a fairly simple loop. First of all, we start by discovering the problems. And this is based on the trace clustering that I was talking about before. Then, based on that, we create code changes. So it's completely automated. We basically cut PRs with a coding agent based on the information we got from the trace, based on information we collected from the logs, from all the different dashboards that we had. Then we evaluate if the change we generated, first of all, is a breaking change. So we rerun VibeBench. VibeBench is kind of like a litmus test. If the score drops by 10 points, we decide that the change is bad. If it's a controversial change that we believe could affect also negatively our agent, we run an AB test. So we put it in front of the users and we try to find out what are the trade-offs. If it's a clear sure shot, then we just ship it in production. And a few times when we believe that the hypothesis was correct and maybe the PR was not perfect, we keep iterating on the problem. Maybe we run another AB test until we are in a good configuration and then we actually ship it. If you think about it, this is very similar to the work you do as an AI engineer all the time, but 90% of it is now aided by an agent in the loop, rather than you having to sweat about every single step of the process.
Let me give you an example of what we actually experienced once in production. Replit Agent immediately starts an execution environment the first time you submit a prompt. And of course, to set up everything, especially with the level of complexity we built, requires quite a few seconds. We had a degradation in the long tail where our setup time was longer than expected and our agent was ready to roll before the environment was fully set up. As you know, agents have become really eager to fix problems. So Replit Agent went on a tangent and started to try to fix the environment on the spot. It turns out that by the nature of agents not being deterministic, every single debugging session looked a bit different. So if it tried to find out this long-tail problem just by grabbing the logs, it would have almost not appeared. But the moment we cluster all the traces with the semantic layer, then we started to figure out that this problem was happening quite often, and we were immediately able to create a patch and fix the issue. In this case, we didn't have to run an AB test because it was a pure regression case. But in case it was a problem that required testing, of course we would have put an additional step in between.
And even though I'm a big fan of trying to optimize as much as possible the life of an AI engineer with agents, I do think that there is a lot of intellectual work to be done by humans still in the kind of loop that I presented to you. For example, whenever we find one of these clusters, as a team, we always focus on formulating a hypothesis on what could possibly go wrong there. Because the truth is when running in production, you're going to get a really large amount of candidates of what you could be fixing. So you need to prioritize where you put effort, where you put resources. So if the hypothesis is intriguing enough, then we move forward, and we don't just let the agent write a PR without any kind of supervision. We actually try to feed some of our understanding of what is wrong. So we give it more guidance. And ultimately, every single time we decide to work on a PR and do an AB test, we're sort of shaping the hill that we try to optimize. Because if we only care about making the product more affordable, we're going to try to fix all the anomalous spend problems. So all these choices really determine the product philosophy of what you want to put in front of your users. And then, as I was showing you on the AB test dashboard, if there is not a clear result, ultimately, if you decide to launch or not, is still a choice that a human does. Oftentimes, that's on me at Replit.
In closing, what I want you to have as a takeaway today is don't think of evaluation just as this last check before shipping. It shouldn't be just a boolean flag, but rather think of this as an engine that allows you to ship a better agent every single day. Thanks everyone. And I'm going to invite Hannah on stage so we can have a chat about how we've been working together in the last few months.
Awesome. So my name is Hannah. I'm part of the Applied AI team at Anthropic, and I work with Michele on the Replit agent and on everything that they build with Claude. So I want to ask you a few questions about what you built here. My first question is: VibeBench is something that we've seen a couple times. You've used it with us to give us feedback on research models that we're testing. But what was your vision from the beginning to make something that you were going to open source and make available to the community? I think a lot of people may be trying to build evals like this for themselves where they see holes in the public benchmarks, but I think this is kind of a unique approach.
Yeah, I've been asked this often, why don't you keep this as a private eval of your company? Fundamentally, I believe that we should try to give back as much as possible to the community. And we all have to benefit from creating public evals. It helps you to make better models. It helps us to make a better agent. It helps everyone to create better products. And I don't believe in competing on evaluations. I come from a research background where everything should be open. So we'll keep doing this for as long as we can at Replit. And I really invite people to collaborate on VibeBench. I tried to instill this idea in the community several times when I was giving talks in the past and at a certain point I realized we have to build it ourselves rather than just trying to invite others to do it.
Well, I'm really excited to see what people do with that. I also want to ask you about the second pillar, Telescope. This is a pretty sophisticated system that you've built. It sounds incredibly useful. It sounds like something you've tuned quite a lot over time. For people in the room who might be trying to build something similar like this, what lessons did you learn along the way?
So first of all, if you tried to do this even, I don't know, 6 months ago, and you got discouraged because you didn't get a return on your investment, definitely try it again. The same inflection point that you experienced with coding agents back, say, in November with Opus 4.5 and similar frontier technology, it also reflects to what I just talked about now. So the fact that long context and models that are really capable of reasoning on a lot of content means that you can practically inject an entire trace of your agent into Opus and get fairly sophisticated feedback about it. So at the same time you should really be investing in collecting all the signal that you can from your agent as well as the environment where you're executing it. I had to run fast during the talk, but I couldn't go in depth on all the different signals that we bring into Telescope. It is not purely what's coming from the trace, it's also the feedback that we collect in product. We have a feedback form, so every time the agent doesn't behave correctly and the user complains, we have the user point of view, we have the trace of the agent, we have everything we instrument in the platform in DataDog. Unsurprisingly, all of that together, put in context, helps even more to pinpoint where the issues are.
Yeah, I think that idea of building the clusters, like you were saying that a single trace, it's very hard to debug what might be going wrong there, but when you are able to group them all together and get these other signals in, really reveals a lot more information and something actionable. Makes a ton of sense.
Yeah, it does, and I think it also makes the life of an agent builder less overwhelming because it really is when you start to get a lot of usage, the amount of feedback you receive is actually overwhelming. And it's a good signal. It means that people care about what you're doing, but at the same time it's hard to prioritize what's actually important versus what doesn't move the needle. What I showed today is also a way for us to try to understand, okay, this cluster keeps showing on a daily basis and it's fairly large and it has a lot of volume, so we should actually fix that first, and then there is the long tail of problems to fix that will always be there with agents given the fact that they're non-deterministic.
Yeah, that really makes me think about the third thing I wanted to ask you, which is what you said about taste. I think this is a really interesting point that the taste of the AI engineer is very important. Could you talk about how your team develops that taste?
I think we all brewed it over time because when we launched these, more than a year and a half ago, agents were way less powerful than they are today. Maybe not much taste to be applied. It was more like a survival game just to make sure it was barely functional. And now that they're becoming so powerful and you have such a wide variety of choices you can make, then you start to develop a taste that should really be in tune with your actual user base. Like if you're building an agent for software developers, maybe 80% of the choices that we make every day in the team will be almost the polar opposite of what we do. And you always want to keep in mind that it's being used by people very different from you. Especially in our case, like everyone at Replit is very technical and we are giving the product to knowledge workers who have never written a single line of code. So it's a very good trade-off to learn.
Awesome. Thank you for sharing that. I'm sure people are going to have lots of questions for you and for Peter, who is one of the authors of VibeBench. So everyone please come find them and thank you so much for coming.
TL;DR
- Replit's president and head of AI, Michele Catasta, argues that agent evaluation must shift from a one-time "boolean gatekeeper" score to a continuous engine that ships a better agent every single day.
- Replit runs two pillars: offline benchmarks (their new open-source VibeBench, which builds whole apps from PRDs and grades them with browser-driving automated evaluators) used before shipping, and online evaluation (AB testing plus semantic trace clustering, named Telescope) used after shipping.
- The talk emphasizes that AB tests rarely give crystal-clear results, so human taste and product philosophy remain essential, and recent long-context reasoning models (Opus 4.5+) finally make trace-based feedback loops practical.
Takeaways
- Vibe coding at Replit is an extreme case: users go from a natural-language prompt to a working app with no chosen framework and no tests, so traditional developer-style evals fundamentally must change.
- Because models, prompts, and tool definitions change constantly (multiple agent releases per day to millions of users), evaluation has to be continuous rather than a single human-produced score.
- SweetBench/HumanEval follow a fixed protocol (apply a patch, run existing tests) that doesn't fit vibe coding where there are no tests and you build from an empty repo; what matters is functional correctness — "does the app do what the user asked?"
- VibeBench takes a PRD as input, builds the app end-to-end, and uses automated evaluators that read the code, open a browser, and step through a natural-language test plan; pairings range from zero-to-one to "slop on slop" (building features on unverified agent code) to parallel task-decomposition-plus-merge.
- Notable VibeBench results: a ~2x gap between frontier and open-weight models, and most models do worse when extending their own code — hence always add a testing step between features.
- The online pillar (Telescope) clusters millions of daily production traces semantically, embeds failure summaries, classifies them with an LLM, and retrains clusters nightly to catch rare long-tail failures (e.g. 1% occurrence) invisible to a DataDog dashboard.
- Telescope's loop: discover problems from clustered traces, auto-cut PRs with a coding agent, run VibeBench as a litmus test (drop of 10 points = bad change), AB test controversial changes, and ship clear wins — about 90% aided by an agent in the loop.
- AB tests rarely give a clean signal (e.g. +7% run duration but 8% cheaper), so human taste, hypothesis formulation, and product philosophy determine what to prioritize and whether to launch.
Vocabulary
Vibe coding — Going from a plain natural-language specification straight to a working application, without the user choosing frameworks or writing tests.
VibeBench — Replit's open-source benchmark for end-to-end vibe coding: a PRD goes in, an app is built, and automated evaluators grade functional correctness.
PRD — Product requirements document; here a long English prompt describing how to build an application, used as benchmark input.
Automated evaluator — An AI agent that reads a codebase, opens a browser, and steps through a natural-language test plan to score whether the app works.
Functional correctness — Whether the app actually does what the user asked, as opposed to whether code passes pre-written tests.
Telescope — Replit's internal system that clusters production traces, discovers problems, and auto-generates fixes.
Trace clustering — Grouping production traces by semantic similarity (via embeddings and an LLM) to surface recurring failure modes.
AB testing — Running multiple agent versions in production simultaneously to compare metrics like duration, cost, and sentiment.
Slop on slop — Building new agent-written features on top of earlier unverified agent code; the most challenging benchmark scenario.
Transcript
Hi everyone. I'm Michele Catasta, president and head of AI at Replit. Today I'm going to be talking about both evaluating and improving on a daily basis Replit Agent at scale. As you know, Replit is a vibe-coding platform for knowledge workers. We're one of the top players in the space and we've been literally battling with this problem for the last almost 2 years. Vibe coding has a very broad definition. It ranges all the way from being used by software developers, but in our specific case it's even more of an extreme definition where you start from just a natural language specification of what the user wants and literally nothing else. So the user expects to go from a prompt to a working application, but they don't tell us what kind of framework they want to use. They don't write tests. They just expect things to work after our agent runs. So it means that a lot of the evaluations and a lot of the ways in which we've been building agents in the past, especially for software developers, has to fundamentally change.
Now, how do we track internally the fact that the agent gets better on a daily basis and is actually aligned with what our users want? This is especially challenging because if you think about it, models are changing almost constantly, and if anything the pace of evolution is even faster than it was a year ago. So we are always sitting on shifting grounds. Models change fast. We have to change our prompts, system prompts, user prompts very often to comply with the new features that we're building in product. Of course, the tool definitions are changing as well. We're adding more of them, we're simplifying, and we're shipping constantly. Every single day we have multiple releases of Replit Agent, which we put in front of millions of users. So my argument today for this talk will be that we have to fundamentally rethink how we do evaluations.
On the left side, you can see how evals look like for basically since we've been working on AI. You run your evaluation, they produce a single score, and that score is produced by a human. And what the human decides based on the score is, my agent harness is better than it was before or it regressed, or my model is better than before, or we should be doing more post-training. That's how we've been doing this for a long time. I argue that we should move more and more towards a continuous setting of evaluations, where especially when you actually have something that runs in production, and in our case it generates millions of traces every day, there is a lot of signal to be extracted from the traces themselves. So what we do is we have a combination of offline evals, very similar to SweetBench and other benchmarks that are very famous in the space, as well as online evaluations based on the system usage.
What do we want our evals to capture? Fundamentally, first of all, they need to optimize what our users care about, rather than just telling us if the agent is editing code correctly. They should also highlight immediately what is breaking, and they should inform us where what we should be shipping next in terms of improvement for our agent.
What goes into this black box? We're building a new system that allows us to do these constant improvements. The system has two pillars. On one end, we have the standard old-school benchmarks. They are usually employed before shipping a new version of the agent. Think of them sort of as a boolean flag. It's a gatekeeper before deciding if you should be launching a new version or not. So if there is a major regression on an offline benchmark, of course you stop your release. If you see no changes or positive changes, you know that you can go on. But the part that is more intriguing and is what is helping us to make constant progress is the fact that we do a lot of AB testing, and we also cluster all the traces that we obtain in production and we try to gain insights from them. So in this case, the online pillar actually works after you ship a version of the agent and it forces you to react as fast as possible. And the loop here is, once I have these two pillars in place, you reflect on the results that both of them are giving you, you likely change the code once again, run your eval, rinse and repeat. And this is the cycle that we are going through every single day at Replit.
I mentioned SweetBench and I want to give you a quick understanding why we work on something more powerful than SweetBench, or at least more tailored towards our use case. Something like SweetBench, HumanEval, they played a key role in our space to allow us to hill-climb them and make models more powerful. But eventually, they all follow the same protocol. You make sure that the code that is generated complies with the user prompt, then you apply the patch on your repository, you run the test. If the tests are passing, you have a higher score on the benchmark. This does not reflect what happens in vibe coding. As I was mentioning before, users are not writing the tests. They often start from a completely empty code base. So there is not a scenario where you can just apply patches. You're building things from the ground up. So what we need to capture instead is: does the app do what the user asked? There is a functional correctness gap between what SweetBench provides and what we were working on.
And today on stage, I'm launching VibeBench. It's a new public benchmark for vibe coding end-to-end that we worked on at Replit for several months. The input is just a PRD. It's literally like a long prompt that describes how to build an application. Rather than crafting them from synthetic data, we pick 20 real-world traces from usage of Replit, just plain English specifications without any implementation constraints. Then, we have a harness that builds the application end-to-end. So it goes from an empty repo into something functional. And then the key insight is, rather than stopping the benchmark here and then having humans evaluate the output of those PRDs, we built automated evaluators. This is what allows you to go from a benchmark that you run maybe on a weekly basis into something that you can run every single time you literally have a new PR merge in your repository. So as you can imagine, we have AI running all the evaluations on our behalf.
The setup is the following. You have an input, you have an implementation strategy, and then you have a set of our wired evaluations that you run on your app. And we came up with five different pairings, but in reality there are a lot more pairings that you could come up with yourself. The most basic pairing is the input is the PRD, and we build the application single-shot, zero to one. Then, in order to go from less complicated to more advanced, we have something called VibeBench Ref, or the reference implementation, where we start from something already working and we build a feature on top of that. Let's make it more advanced. We have VibeBench Vibe where we start from an agent MVP and then we build a new feature on top of that. Maybe this is something that in the lingo you would call slop on slop. So if we have something that's been built by the agent, not verified, and then you had more agent code written on top of that, and then you run the evaluations to see if it's working or not. You can keep making this arbitrarily more complex. For example, in Agent 4 that we launched a couple of months ago, we completely hide the complexity of doing task decomposition and running agents in parallel and then merging all the patches together. So we can also take as an input the PRD, we do the task decomposition, we run all of that in parallel and then we run the evals after that. As you can imagine the parallel-plus-merge scenario is far more challenging for a standard coding agent. You can even start from a buggy application, build a feature on top of that and see how much your agent struggles to make it work correctly.
So the key question or the complexity here is how do you actually do the grading? That's where we spent most of our effort. It turns out that when we were working on our previous version of the agent, we put quite a lot of effort in building an automated app-testing feature which literally doesn't know anything about the app per se. Because when it comes to vibe coding, we don't give any level of guardrails to our users. They can decide the language they want to use, they can decide a different framework. So the evaluator has to be totally agnostic on how the implementation looks like. And what our evaluator agent does is it reads the code base, it then opens a browser and points it to the application that our agent has built, and then step-by-step goes through our testing plan. And even the testing plan is expressed in natural language and the actions are like "open the admin dashboard and log in with a certain account and click on this toggle." And if any of those steps fails, then we collect all of them together and we generate a score. So we go through all these test plans and then we decide ultimately if the score is good or not. When it comes to SweetBench, we always had a fixed surface. We know the repositories, we know the task hardness, we know exactly how to make this work. In VibeBench, the surface is completely green field and that's why it took us several months of work to get there.
We're announcing it today. You can find the entire benchmark available open source on vibebench.ai. It's going to be presented a couple of weeks from today at the conference on AI and agentic systems. Peter, the lead of the project and first author of the paper, is sitting here. I'm just going to give you a sneak peek of the most notable results. First of all, we are noticing almost a 2x gap between the frontier models and the open-weights models. The reason why I want you to pay attention there is because we know that every single model builder is hill-climbing specific benchmarks. That's why we release this open source. I want the entire community to embrace it across open weights and closed weights and make sure that we make progress overall on vibe coding as well. The other one, possibly not very surprising, is most models do a worse job when extending their own code. So the slop-on-slop or VibeBench Vibe scenario that I was describing to you before is by far the most challenging. And that's the reason why I always advocate to have a testing step in between every time you create a new feature, otherwise you keep building on shaky foundations and eventually your vibe coding application is going to fail.
Let's move to the next step. We talked about offline eval, I talked to you about VibeBench, but as you recall there were two main pillars that we care about. And the reason why we also care about the online one is because there is a great difference in terms of volume of signal that we can collect. So on the offline one right now we're working with roughly 20 applications. The reason why we made it open is because we are more than welcoming contributions from the community. So more applications, harder prompts, we can keep making this benchmark arbitrarily harder so that we make our agent's life harder over time. But when it comes to Replit Engine instead, we're collecting millions of sessions every day and they're very valuable because they capture what our users actually do on the platform. And they're completely unscripted, the agent is always running.
So how do we distill useful information out of them? Well, it turns out that we run a lot of AB tests. It's basically our way to keep ourselves honest because VibeBench only tells us part of the story. And I invite every agent builder out there to start to invest on their AB testing infrastructure as soon as possible. It's the best way in which you can make steady progress. We ask a wide variety of questions for ourselves. We built a lot of different instrumentation and metrics in our agent so that, for instance, we can find out: is there a normal span ongoing or is the agent running for longer than expected? Or we constantly keep track of the user sentiment. It's a very easy thing to collect because every time you receive a prompt, you can do sentiment analysis on it and then perceive if the user is getting frustrated or not. And last but not least, this is very idiosyncratic to Replit. Our users can also publish their application on our product. So it's a very strong positive signal if they decide that whatever they built is worth sharing with their colleagues or putting in public in front of everyone.
So all these signals clustered together give us something like this. If you have ever done an AB test in your life, I'm sure this looks familiar. And it might look familiar especially because not everything is either green or red. This is the harsh truth about running an AB test. They never give you a crystal-clear signal of what you should be doing. So in this case, for example, the average run duration of our agent went up by 7%, but conversely it's 8% cheaper, and we saw the fluctuation in terms of positive and negative sentiment. What shall we do in this case? This is where human taste and product philosophy still plays a key role. You will very rarely get a crystal-clear configuration of the results of your AB test.
And to make our life a bit easier on how to generate these AB test candidates, what we do is we cluster all the traces that we receive every single day and night, and we try to identify first of all which clusters are capturing normal nominal behavior of the agent. And as you can imagine, there are plenty of them because the vast majority of times the application is actually successful, but of course there is a long tail of problems that happen to surface. In case we find a problematic cluster, then what do we do? We embed all the failure summaries. We cluster them by type so that we find out the different failures that are happening at a certain point in time. Then we pass them through an LLM to classify exactly what is going on. And the fact that we do this not based on regex on logs or very deterministic techniques is what actually makes a difference, because you will be able to cluster things that are semantically close to each other even though the agent doesn't give you always the same type of output. And once we find these cluster configurations, we are forced to retrain them every single night because we are running several versions of the engine in parallel. We ship many versions every single day, so we can't live with a fixed cluster configuration. And aside from also helping us to identify the failure modes, another interesting feature is the fact that once we retrain those clusters, we can go back and see if after we believe we fixed a problem, a certain cluster has actually disappeared. So say you have a certain tool failure that is happening only a limited amount of time. So not something that will show up in your DataDog dashboard, for example, not a failure that happens 50% of the times, but maybe it was happening 1% of the times in certain specific conditions. And from here, you will start to realize, okay, I shipped my new PR and that cluster has disappeared. So you start to have some evidence that you mitigated the problem.
The technology that we built internally was called Telescope and it's a fairly simple loop. First of all, we start by discovering the problems. And this is based on the trace clustering that I was talking about before. Then, based on that, we create code changes. So it's completely automated. We basically cut PRs with a coding agent based on the information we got from the trace, based on information we collected from the logs, from all the different dashboards that we had. Then we evaluate if the change we generated, first of all, is a breaking change. So we rerun VibeBench. VibeBench is kind of like a litmus test. If the score drops by 10 points, we decide that the change is bad. If it's a controversial change that we believe could affect also negatively our agent, we run an AB test. So we put it in front of the users and we try to find out what are the trade-offs. If it's a clear sure shot, then we just ship it in production. And a few times when we believe that the hypothesis was correct and maybe the PR was not perfect, we keep iterating on the problem. Maybe we run another AB test until we are in a good configuration and then we actually ship it. If you think about it, this is very similar to the work you do as an AI engineer all the time, but 90% of it is now aided by an agent in the loop, rather than you having to sweat about every single step of the process.
Let me give you an example of what we actually experienced once in production. Replit Agent immediately starts an execution environment the first time you submit a prompt. And of course, to set up everything, especially with the level of complexity we built, requires quite a few seconds. We had a degradation in the long tail where our setup time was longer than expected and our agent was ready to roll before the environment was fully set up. As you know, agents have become really eager to fix problems. So Replit Agent went on a tangent and started to try to fix the environment on the spot. It turns out that by the nature of agents not being deterministic, every single debugging session looked a bit different. So if it tried to find out this long-tail problem just by grabbing the logs, it would have almost not appeared. But the moment we cluster all the traces with the semantic layer, then we started to figure out that this problem was happening quite often, and we were immediately able to create a patch and fix the issue. In this case, we didn't have to run an AB test because it was a pure regression case. But in case it was a problem that required testing, of course we would have put an additional step in between.
And even though I'm a big fan of trying to optimize as much as possible the life of an AI engineer with agents, I do think that there is a lot of intellectual work to be done by humans still in the kind of loop that I presented to you. For example, whenever we find one of these clusters, as a team, we always focus on formulating a hypothesis on what could possibly go wrong there. Because the truth is when running in production, you're going to get a really large amount of candidates of what you could be fixing. So you need to prioritize where you put effort, where you put resources. So if the hypothesis is intriguing enough, then we move forward, and we don't just let the agent write a PR without any kind of supervision. We actually try to feed some of our understanding of what is wrong. So we give it more guidance. And ultimately, every single time we decide to work on a PR and do an AB test, we're sort of shaping the hill that we try to optimize. Because if we only care about making the product more affordable, we're going to try to fix all the anomalous spend problems. So all these choices really determine the product philosophy of what you want to put in front of your users. And then, as I was showing you on the AB test dashboard, if there is not a clear result, ultimately, if you decide to launch or not, is still a choice that a human does. Oftentimes, that's on me at Replit.
In closing, what I want you to have as a takeaway today is don't think of evaluation just as this last check before shipping. It shouldn't be just a boolean flag, but rather think of this as an engine that allows you to ship a better agent every single day. Thanks everyone. And I'm going to invite Hannah on stage so we can have a chat about how we've been working together in the last few months.
Awesome. So my name is Hannah. I'm part of the Applied AI team at Anthropic, and I work with Michele on the Replit agent and on everything that they build with Claude. So I want to ask you a few questions about what you built here. My first question is: VibeBench is something that we've seen a couple times. You've used it with us to give us feedback on research models that we're testing. But what was your vision from the beginning to make something that you were going to open source and make available to the community? I think a lot of people may be trying to build evals like this for themselves where they see holes in the public benchmarks, but I think this is kind of a unique approach.
Yeah, I've been asked this often, why don't you keep this as a private eval of your company? Fundamentally, I believe that we should try to give back as much as possible to the community. And we all have to benefit from creating public evals. It helps you to make better models. It helps us to make a better agent. It helps everyone to create better products. And I don't believe in competing on evaluations. I come from a research background where everything should be open. So we'll keep doing this for as long as we can at Replit. And I really invite people to collaborate on VibeBench. I tried to instill this idea in the community several times when I was giving talks in the past and at a certain point I realized we have to build it ourselves rather than just trying to invite others to do it.
Well, I'm really excited to see what people do with that. I also want to ask you about the second pillar, Telescope. This is a pretty sophisticated system that you've built. It sounds incredibly useful. It sounds like something you've tuned quite a lot over time. For people in the room who might be trying to build something similar like this, what lessons did you learn along the way?
So first of all, if you tried to do this even, I don't know, 6 months ago, and you got discouraged because you didn't get a return on your investment, definitely try it again. The same inflection point that you experienced with coding agents back, say, in November with Opus 4.5 and similar frontier technology, it also reflects to what I just talked about now. So the fact that long context and models that are really capable of reasoning on a lot of content means that you can practically inject an entire trace of your agent into Opus and get fairly sophisticated feedback about it. So at the same time you should really be investing in collecting all the signal that you can from your agent as well as the environment where you're executing it. I had to run fast during the talk, but I couldn't go in depth on all the different signals that we bring into Telescope. It is not purely what's coming from the trace, it's also the feedback that we collect in product. We have a feedback form, so every time the agent doesn't behave correctly and the user complains, we have the user point of view, we have the trace of the agent, we have everything we instrument in the platform in DataDog. Unsurprisingly, all of that together, put in context, helps even more to pinpoint where the issues are.
Yeah, I think that idea of building the clusters, like you were saying that a single trace, it's very hard to debug what might be going wrong there, but when you are able to group them all together and get these other signals in, really reveals a lot more information and something actionable. Makes a ton of sense.
Yeah, it does, and I think it also makes the life of an agent builder less overwhelming because it really is when you start to get a lot of usage, the amount of feedback you receive is actually overwhelming. And it's a good signal. It means that people care about what you're doing, but at the same time it's hard to prioritize what's actually important versus what doesn't move the needle. What I showed today is also a way for us to try to understand, okay, this cluster keeps showing on a daily basis and it's fairly large and it has a lot of volume, so we should actually fix that first, and then there is the long tail of problems to fix that will always be there with agents given the fact that they're non-deterministic.
Yeah, that really makes me think about the third thing I wanted to ask you, which is what you said about taste. I think this is a really interesting point that the taste of the AI engineer is very important. Could you talk about how your team develops that taste?
I think we all brewed it over time because when we launched these, more than a year and a half ago, agents were way less powerful than they are today. Maybe not much taste to be applied. It was more like a survival game just to make sure it was barely functional. And now that they're becoming so powerful and you have such a wide variety of choices you can make, then you start to develop a taste that should really be in tune with your actual user base. Like if you're building an agent for software developers, maybe 80% of the choices that we make every day in the team will be almost the polar opposite of what we do. And you always want to keep in mind that it's being used by people very different from you. Especially in our case, like everyone at Replit is very technical and we are giving the product to knowledge workers who have never written a single line of code. So it's a very good trade-off to learn.
Awesome. Thank you for sharing that. I'm sure people are going to have lots of questions for you and for Peter, who is one of the authors of VibeBench. So everyone please come find them and thank you so much for coming.