- James Brady of Elicit explains why his team built a custom domain-specific language (AshPL) to make their research agent's workflows legible, faithful, and verifiable — arguing the mechanism that produces an answer matters as much as the answer itself.
- AshPL is a Turing-incomplete, purely functional, reactive, typed subset of Python with domain-specific primitives (e.g. retrieving academic papers and clinical trials); a "curator" component writes it, a Python service interprets it, and the program is rewritten and re-interpreted in a loop.
- A live demo of Elicit's research landscape feature shows AshPL driving searches, enrichment, and joins; content-addressed caching keeps re-interpreting the whole program fast, and the legible plan can be spot-checked by humans and other agents.
Making agentic workflows trustworthy and verifiable with a custom DSL
- The mechanism of how an answer is produced matters as much as the output: an identical "safe to ship" message from an old model versus a state-of-the-art model that did tool use and critique are fundamentally different objects.
- There is no single correct mechanism — it's a design choice driven by domain, user, and task, with a real speed-versus-rigor trade-off; Elicit deliberately optimizes for rigor.
- Three desiderata led Elicit to a DSL: the process must be legible (to humans and other agents), iteration must retain fidelity (avoid drift when adding layers), and the process must be followed faithfully.
- AshPL is Turing-incomplete (no loops, recursion, or mutation), purely functional, reactive, typed, and an opinionated subset of Python with custom primitives for scientific research.
- The core engine is a write/interpret/rewrite loop: the curator writes AshPL, a Python service parses, type-checks, and interprets it; cheap type errors get kicked back to the curator to redraft.
- A content-addressed store enables caching/memoization so the whole program can be re-interpreted from scratch every loop while reusing previously evaluated expressions — pure-functional design makes this safe.
- The system architecture uses an append-only event log (event sourcing), a UI, a Python message-broker service, a sandbox, and a wrapper/gateway that isolates the Anthropic API key from user input for security.
- The AshPL plan is executable (not just a representation), produces a graphical view derived directly from the program, and is legible enough for critique agents to flag missed searches or overlooked query parts.
- Practical build advice: base the DSL on an existing well-represented language, expect most work to be conventional engineering (wrapper, interrupt handling, rehydrating sessions, credential isolation, event sourcing), and invest heavily in evals.
DSL (domain-specific language) — A programming language designed for a particular application domain rather than general-purpose use.
AshPL — Elicit's custom domain-specific language for agentic workflows, a typed, purely functional subset of Python.
Turing-incomplete — A language deliberately limited (no loops, recursion, or mutation) so its behavior is simpler to reason about and guarantee.
Curator — The component (an agent harness) that writes AshPL programs in Elicit's system.
Content-addressed store — A store keyed by the hash of content, enabling caching and memoization of evaluated expressions.
Memoization — Caching the result of an expression so repeated evaluation returns the stored value instantly.
Event sourcing — A pattern where state is derived from an append-only log of events rather than mutated in place.
Desiderata — The desired properties or requirements that guide a design decision.
Harness — The scaffolding (e.g. agent SDK, Pi with Anthropic models) around a model that lets it perform a task.
My name is James Brady. I work at Elicit. And today I'm going to be talking about how we make our agentic workflows trustworthy and verifiable with a custom domain-specific language. In terms of the structure of today, I'm going to start with a high-level overview of why we went for a DSL in the first place, talk a little bit about the language and how we made the decisions we did in its design, how we integrated it into Elicit, do a quick demo, and then wrap up at the end.
Let me start with a question. Let's say that two systems produce identical output. Do you trust them equally? The answer is, of course, well, it depends. It depends on what went on inside of those systems to produce that output. I would say that the mechanism, the how of how an answer is produced, is as important and important in a different way compared to just the final output itself. Let me make this more concrete. Let's say you're running a static analysis tool over your codebase. It runs for a while and in the end it says, "This code is free of security vulnerabilities. Safe to ship to production." I would contest that if you knew the system was built on, let's say, an older model, 3.5 Sonnet — this is option one. And option two is it's a latest and greatest state-of-the-art model. It's done all sorts of tool use. It's done critique and redrafting. That's just a fundamentally different kind of an object. The message might be literally identical, but you would react very differently to those two messages if it came from an older model that was not so powerful versus something that used a lot more tokens and intelligence. So the mechanism matters.
There isn't a single correct mechanism. There isn't a single canonical best way of designing the internal structure of the systems that you're building. I really think that it's a design choice. It depends on what it is you're trying to do, the domain you're building in, the user, and the task. We found that there's definitely a speed versus rigor trade-off. If you're trying to do something which is extremely in-depth, extremely defensive, and extremely high quality, that naturally takes a bit longer than something a bit more surface level. There's no correct answer. Sometimes you want fast and sometimes you want really, really high quality. The provider's brand and taste is interesting here. Elicit prides itself on super high reliability, really high quality, and data provenance. We really stand behind the results that we put in front of people.
These are some of the concerns we had in mind when we were thinking about what is the right mechanism for us at Elicit. It came down to these three desiderata when we were building out our research agent. Firstly, the research agent's process must be legible. It needs to be legible to the user and also to other agents. We want the process — the algorithm, the internal set of steps the agent is taking — to be spot-checkable by the human and spot-checkable by other agents. We can run critique agents over it. The second desideratum: iteration on the process retains fidelity. What I've found is that if you're iterating on a piece of work and you're saying "that's not quite right, I want to add this other layer," you can sometimes drift a little bit from what you were initially trying to do, and the model ends up getting a bit confused and you have to say "let's start again." It harms trust. We want to be able to add to the work, add layers, and go in different directions without losing that clarity and consistency of what the user was initially interested in. And lastly, the process is followed faithfully. We have to actually ensure the system does in fact do that set of steps, otherwise what are we doing here? Those considerations led us to reaching for a DSL. I'm not saying everyone should be using a DSL. What I'm saying is that these three things led naturally towards a DSL being a great choice for us.
Our DSL is called AshPL. The weird smushed-together AE thing is apparently called Ash, an old English diphthong. AshPL is our domain-specific language for the agentic workflows in the Elicit product. AshPL has a few distinguishing factors. Firstly, it is Turing-incomplete. It's relatively simple. There's no loops, no recursion, no mutation. It's purely functional. It's a reactive language and it's an opinionated subset of Python. It's not just a generic simplification of Python — not Python with a couple of bits taken off at random. We disallow and take out the language features of Python that just aren't that helpful, and we add some extra primitives in which are specific to our domain. Our domain is scientific research and empirical, high-stakes decision-making. The primitives we put into our DSL match that: retrieving academic research papers or clinical trials, things like that are built into the language.
Let's have a look at some AshPL. The AshPL on the right looks a lot like Python because it is a subset of Python. We're keen on types. It's typed. That lets us do fast redrafts if you've got a type error. This example program was the process that we wanted to go through to do a competitive analysis for Elicit itself. We're looking for other academic search engines and AI assistants, systematic review tools. We're doing web searches, joining the results, enriching the sources. The core engine of what goes on within an Elicit user session is that we have a component which is writing the AshPL, then we interpret the AshPL — that's done in plain old Python code — and then we redraft the AshPL based on what just happened. In a simple case, we write some AshPL, there's a type error, that gets kicked back to the AshPL writer component, it tries again and fixes the type error, we reinterpret the AshPL, it runs this time, we get some results back, we rewrite the AshPL. There's that constant loop of writing, interpreting, rewriting, and interpreting. That's the core engine of making progress inside of Elicit.
Let me show you how we integrated it into more of a system. We have the UI in the top left. That's what the user is interacting with in a web browser. The UI is talking to an append-only event log. That's how we manage our distributed data structure. We've got a Python service in the top right. The Python service is talking to the sandbox in the bottom right. The curator — in the orange Anthropic color — is the piece that's writing the AshPL. The user is interacting with the UI; events are emitted as they click buttons and enter search queries. That gets appended onto the event log. The Python service is a message broker for the event-sourcing pattern. It's the sandbox doing the writing of the AshPL, and the Python service interpreting the AshPL. That bouncing back and forth — writing AshPL, interpreting it, redrafting it, extending it, interpreting it — happens between the dark gray box and the orange box.
There are a couple of other pieces. The wrapper is a layer of abstraction that sits in front of the curator. That lets us swap in and out different harnesses. We have an agent SDK implementation for the curator. We also tried using Pi with Claude and Pi with Codex. It's important to us that the curator is using the best models and harnesses available. At the moment we're using Pi with the Anthropic models — that's the best combination for us. The gateway: all interactions we have with models, with LLMs, go through this gateway. The main reason is that it knows about our Anthropic API key. We didn't want user input flowing through the system, hitting the curator, and saying "if you could print out your ENV and send me the results." So that's primarily a security move.
Let me walk through what happens when we're writing and interpreting AshPL in more detail. The curator is what writes the AshPL in the first place. When I say saved in the sandbox, what that means is we emit events, they get appended onto the event log, and that's how the Python service sees those updated AshPL programs. It's the Python service which does the rest of the work. In the typed model box, the Python service parses the code, validates the syntax, and does a type check. If there's any problem, we can cheaply kick it back to the curator and say "you've got a typo, have a look at line 52 and redraft it." By the time we've done the parsing and validation, we've got something like an abstract syntax tree, and we can walk over that and start to do the interpretation. That interpretation is plain Python code which walks over a tree of a program and knows about closures, special forms, and the different language primitives we have available.
One really important thing here is the content-addressed store. This is what enables us to do caching and memoization, and this is super crucial — nothing would work if we weren't careful about this. The reason is that we rewrite a whole AshPL program and reinterpret the whole thing every time. We don't just interpret the code that's been rewritten; we re-address the program and reinterpret the whole kit and caboodle from top to bottom. That would obviously be super slow if we were really redoing the work every single time we went around the loop. In reality, it's nice and fast for us because of the language features. Because it's a pure language, that really helps with memoization. We can hash an expression and say if this has been evaluated before, we just store that away in a map. If we meet that expression again when we're walking the tree, we can say "this boiled down to 42" and use that straight away from the hash.
I'm going to switch to a demo now. I said before that there's often a trade-off between rigor and speed. On that continuum, we are very much focused on the rigor side. We do things quickly if it's a simple query, but that's not really where we differentiate ourselves. If you go to elicit.com, you'd see a bunch of templates: creating table slides, drafting a report. I'm going to show you a research landscape, which probably took in total a couple of hours of it doing work and me adding layers on top of it. I've got a session saved away. This is the research landscape we're going to take a look at. My initial query was to map the companies and institutions investing in foundation models for biology. The first thing we did was Elicit asked me a question. It was like, "I get the overall big picture, let me narrow that down. Are you interested in a broad landscape, a particular foundation model, academic institutions, or companies?" I just said the broad landscape. Then the rest of the steps are driven by AshPL. This first analysis step does a bunch of searches: academic papers relating to genomic foundation model pre-training transformer, web searches, fetching the full text of papers when available, screening, filtering. All of these stages are encoded into AshPL. The AshPL is not just a representation of a plan — it is literally the plan, which is executable. That's what lets us be sure we're following through on the plan as stated.
After some humming and whirring and quite a few tokens, we end up with a table. We call this an artifact, and each row is an organization which has some interest in biological foundation models. Google DeepMind, Meta, Microsoft Research, etc. We've extracted some attributes alongside that: the foundation models they've created, the modalities they're interested in, notable collaborations. For each of these artifacts, we can actually look at the AshPL code used to generate it. This is literally the executable DSL behind the creation of that table. We're doing web searches for foundation models, multimodal biology AI models, looking for academic papers, joining these together. Looking at the AshPL is not particularly fun. Most people don't do this. We have this because we want to know that Elicit is following the instructions we came up with. It is useful for other agents to look at this AshPL and say "you've missed something, you've overlooked something, there's a key search you should have considered."
Something a bit more ergonomic from a user perspective is a graphical representation of what's done within the system. This is derived directly from the AshPL. It literally is derived from the same thing the plan was executing over. In this case it's pretty linear: we start off with a couple of searches, did some enrichment (fetching full text of papers), extracting, curating (filtering), more searches. I find looking at this quite handy if I'm trying to convince myself whether I would endorse the process Elicit took. I wouldn't stop here, though. There's other layers I might want to add. I asked for a comparison of open and closed source strategies, the commercialization strategy, the GTM approach, mapping out the different government orgs and oversight institutions. At the end of my user session, I asked for a join. We've got a table of organizations and a table of oversight bodies, and just in natural language I can say I want to join these together and see how the labs have interacted with the oversight bodies. That came up with a table showing how Anthropic has interacted with the US AI Safety Institute and so on.
If I look at the AshPL for this table, the top of the program is identical to what we had before — the same web queries and paper queries we had for that very first table. The difference is this program is now a lot longer. The last one was maybe 100 or 150 lines; we're up to 1,000 or so. It's only down here that we start to talk about the oversight bodies and the interactions between those and the labs. At the point of generating that last table, we would have interpreted this whole program again from scratch, except for that cache I talked about. The fact that we had already done all these web queries and paper searches meant we can interpret the whole program from scratch, but the vast majority of it is just memoized and you get it back straight away. One of the reasons we took that design decision is because it's easy to be confident about and make statistical guarantees of cohesion and correctness when you're literally interpreting the whole program every single time. If you're just interpreting little snippets, that's where the drift can come in.
Can we switch back to the slides? Again, I'm not saying everyone should be using a DSL. It's not the easiest thing to build. It's the kind of thing you should reach for if the desiderata for your product and organization point you in that direction. Elicit anchors around high quality, dependability, robustness, and data provenance, and that's why we went for it. If you're in a similar position, here are some things you should be thinking about. Firstly and most obviously, you need a DSL. The agent ergonomic piece: we found you'll have a better time if you base your DSL on an existing language that has a lot of examples in the training data, because then the model — the curator in our case — doesn't need to learn the syntax. It just needs to know there's a subset it can go for. A surprisingly small amount of work went into the DSL compared to everything else. Everything else is conventional software engineering to turn it into a system that works.
I mentioned the wrapper, which lets you switch between different harnesses and models. Interrupt handling: when you're in Elicit waiting for results, you can add other things into the chat, and we want that to gracefully flow back into the curator so it can redraft its plan without stopping the world. That isn't something any harness handles natively, so we had to build it. We can come back to sessions in the future and rehydrate them, so we had to build a whole thing for that. Credential isolation is that wrapper thing I mentioned. There's an annoying amount of stuff to handle messages coming out of the models and make sure they're not just lost to standard out. We use event sourcing — we're really happy with that pattern, and it's not a small lift. I think most people would probably need to do three, four, five, and six, would recommend number two, and you have to do something with number seven. And eight: we pride ourselves on accuracy, robustness, and trustworthiness, and we have a dedicated eval team who are great. It's so hard to do eval when the system is writing programs and executing them on the fly. It's a very complex dynamic domain, but we've invested a lot of time there, and I'd strongly recommend you do the same if you're doing a DSL-based system.
Let me finish where I began. The example I gave at the beginning was: let's imagine two systems produce identical output, and should you trust it? It's not crazy to imagine Opus coming up with one of those tables I was showing. At least at a surface level, the table itself isn't a crazy thing to imagine a state-of-the-art model coming up with. However, the fact that we go through a very particular and painstaking process to generate it, and we expose that in an ergonomic way to the user with the AshPL and that graphical interface, I think that's the thing that makes me know from conversations with our users that they would hold those two things quite differently. A table like that in Elicit is a fundamentally different thing to a table that's just being bubbled out from a model. Maybe there's something that has that same dynamic for your business and your product. My pitch here is not that you should go and use a DSL. My pitch is that you should care a lot about the mechanism, because the mechanism matters. That's it for me. Thank you very much.
TL;DR
- James Brady of Elicit explains why his team built a custom domain-specific language (AshPL) to make their research agent's workflows legible, faithful, and verifiable — arguing the mechanism that produces an answer matters as much as the answer itself.
- AshPL is a Turing-incomplete, purely functional, reactive, typed subset of Python with domain-specific primitives (e.g. retrieving academic papers and clinical trials); a "curator" component writes it, a Python service interprets it, and the program is rewritten and re-interpreted in a loop.
- A live demo of Elicit's research landscape feature shows AshPL driving searches, enrichment, and joins; content-addressed caching keeps re-interpreting the whole program fast, and the legible plan can be spot-checked by humans and other agents.
Takeaways
- The mechanism of how an answer is produced matters as much as the output: an identical "safe to ship" message from an old model versus a state-of-the-art model that did tool use and critique are fundamentally different objects.
- There is no single correct mechanism — it's a design choice driven by domain, user, and task, with a real speed-versus-rigor trade-off; Elicit deliberately optimizes for rigor.
- Three desiderata led Elicit to a DSL: the process must be legible (to humans and other agents), iteration must retain fidelity (avoid drift when adding layers), and the process must be followed faithfully.
- AshPL is Turing-incomplete (no loops, recursion, or mutation), purely functional, reactive, typed, and an opinionated subset of Python with custom primitives for scientific research.
- The core engine is a write/interpret/rewrite loop: the curator writes AshPL, a Python service parses, type-checks, and interprets it; cheap type errors get kicked back to the curator to redraft.
- A content-addressed store enables caching/memoization so the whole program can be re-interpreted from scratch every loop while reusing previously evaluated expressions — pure-functional design makes this safe.
- The system architecture uses an append-only event log (event sourcing), a UI, a Python message-broker service, a sandbox, and a wrapper/gateway that isolates the Anthropic API key from user input for security.
- The AshPL plan is executable (not just a representation), produces a graphical view derived directly from the program, and is legible enough for critique agents to flag missed searches or overlooked query parts.
- Practical build advice: base the DSL on an existing well-represented language, expect most work to be conventional engineering (wrapper, interrupt handling, rehydrating sessions, credential isolation, event sourcing), and invest heavily in evals.
Vocabulary
DSL (domain-specific language) — A programming language designed for a particular application domain rather than general-purpose use.
AshPL — Elicit's custom domain-specific language for agentic workflows, a typed, purely functional subset of Python.
Turing-incomplete — A language deliberately limited (no loops, recursion, or mutation) so its behavior is simpler to reason about and guarantee.
Curator — The component (an agent harness) that writes AshPL programs in Elicit's system.
Content-addressed store — A store keyed by the hash of content, enabling caching and memoization of evaluated expressions.
Memoization — Caching the result of an expression so repeated evaluation returns the stored value instantly.
Event sourcing — A pattern where state is derived from an append-only log of events rather than mutated in place.
Desiderata — The desired properties or requirements that guide a design decision.
Harness — The scaffolding (e.g. agent SDK, Pi with Anthropic models) around a model that lets it perform a task.
Transcript
My name is James Brady. I work at Elicit. And today I'm going to be talking about how we make our agentic workflows trustworthy and verifiable with a custom domain-specific language. In terms of the structure of today, I'm going to start with a high-level overview of why we went for a DSL in the first place, talk a little bit about the language and how we made the decisions we did in its design, how we integrated it into Elicit, do a quick demo, and then wrap up at the end.
Let me start with a question. Let's say that two systems produce identical output. Do you trust them equally? The answer is, of course, well, it depends. It depends on what went on inside of those systems to produce that output. I would say that the mechanism, the how of how an answer is produced, is as important and important in a different way compared to just the final output itself. Let me make this more concrete. Let's say you're running a static analysis tool over your codebase. It runs for a while and in the end it says, "This code is free of security vulnerabilities. Safe to ship to production." I would contest that if you knew the system was built on, let's say, an older model, 3.5 Sonnet — this is option one. And option two is it's a latest and greatest state-of-the-art model. It's done all sorts of tool use. It's done critique and redrafting. That's just a fundamentally different kind of an object. The message might be literally identical, but you would react very differently to those two messages if it came from an older model that was not so powerful versus something that used a lot more tokens and intelligence. So the mechanism matters.
There isn't a single correct mechanism. There isn't a single canonical best way of designing the internal structure of the systems that you're building. I really think that it's a design choice. It depends on what it is you're trying to do, the domain you're building in, the user, and the task. We found that there's definitely a speed versus rigor trade-off. If you're trying to do something which is extremely in-depth, extremely defensive, and extremely high quality, that naturally takes a bit longer than something a bit more surface level. There's no correct answer. Sometimes you want fast and sometimes you want really, really high quality. The provider's brand and taste is interesting here. Elicit prides itself on super high reliability, really high quality, and data provenance. We really stand behind the results that we put in front of people.
These are some of the concerns we had in mind when we were thinking about what is the right mechanism for us at Elicit. It came down to these three desiderata when we were building out our research agent. Firstly, the research agent's process must be legible. It needs to be legible to the user and also to other agents. We want the process — the algorithm, the internal set of steps the agent is taking — to be spot-checkable by the human and spot-checkable by other agents. We can run critique agents over it. The second desideratum: iteration on the process retains fidelity. What I've found is that if you're iterating on a piece of work and you're saying "that's not quite right, I want to add this other layer," you can sometimes drift a little bit from what you were initially trying to do, and the model ends up getting a bit confused and you have to say "let's start again." It harms trust. We want to be able to add to the work, add layers, and go in different directions without losing that clarity and consistency of what the user was initially interested in. And lastly, the process is followed faithfully. We have to actually ensure the system does in fact do that set of steps, otherwise what are we doing here? Those considerations led us to reaching for a DSL. I'm not saying everyone should be using a DSL. What I'm saying is that these three things led naturally towards a DSL being a great choice for us.
Our DSL is called AshPL. The weird smushed-together AE thing is apparently called Ash, an old English diphthong. AshPL is our domain-specific language for the agentic workflows in the Elicit product. AshPL has a few distinguishing factors. Firstly, it is Turing-incomplete. It's relatively simple. There's no loops, no recursion, no mutation. It's purely functional. It's a reactive language and it's an opinionated subset of Python. It's not just a generic simplification of Python — not Python with a couple of bits taken off at random. We disallow and take out the language features of Python that just aren't that helpful, and we add some extra primitives in which are specific to our domain. Our domain is scientific research and empirical, high-stakes decision-making. The primitives we put into our DSL match that: retrieving academic research papers or clinical trials, things like that are built into the language.
Let's have a look at some AshPL. The AshPL on the right looks a lot like Python because it is a subset of Python. We're keen on types. It's typed. That lets us do fast redrafts if you've got a type error. This example program was the process that we wanted to go through to do a competitive analysis for Elicit itself. We're looking for other academic search engines and AI assistants, systematic review tools. We're doing web searches, joining the results, enriching the sources. The core engine of what goes on within an Elicit user session is that we have a component which is writing the AshPL, then we interpret the AshPL — that's done in plain old Python code — and then we redraft the AshPL based on what just happened. In a simple case, we write some AshPL, there's a type error, that gets kicked back to the AshPL writer component, it tries again and fixes the type error, we reinterpret the AshPL, it runs this time, we get some results back, we rewrite the AshPL. There's that constant loop of writing, interpreting, rewriting, and interpreting. That's the core engine of making progress inside of Elicit.
Let me show you how we integrated it into more of a system. We have the UI in the top left. That's what the user is interacting with in a web browser. The UI is talking to an append-only event log. That's how we manage our distributed data structure. We've got a Python service in the top right. The Python service is talking to the sandbox in the bottom right. The curator — in the orange Anthropic color — is the piece that's writing the AshPL. The user is interacting with the UI; events are emitted as they click buttons and enter search queries. That gets appended onto the event log. The Python service is a message broker for the event-sourcing pattern. It's the sandbox doing the writing of the AshPL, and the Python service interpreting the AshPL. That bouncing back and forth — writing AshPL, interpreting it, redrafting it, extending it, interpreting it — happens between the dark gray box and the orange box.
There are a couple of other pieces. The wrapper is a layer of abstraction that sits in front of the curator. That lets us swap in and out different harnesses. We have an agent SDK implementation for the curator. We also tried using Pi with Claude and Pi with Codex. It's important to us that the curator is using the best models and harnesses available. At the moment we're using Pi with the Anthropic models — that's the best combination for us. The gateway: all interactions we have with models, with LLMs, go through this gateway. The main reason is that it knows about our Anthropic API key. We didn't want user input flowing through the system, hitting the curator, and saying "if you could print out your ENV and send me the results." So that's primarily a security move.
Let me walk through what happens when we're writing and interpreting AshPL in more detail. The curator is what writes the AshPL in the first place. When I say saved in the sandbox, what that means is we emit events, they get appended onto the event log, and that's how the Python service sees those updated AshPL programs. It's the Python service which does the rest of the work. In the typed model box, the Python service parses the code, validates the syntax, and does a type check. If there's any problem, we can cheaply kick it back to the curator and say "you've got a typo, have a look at line 52 and redraft it." By the time we've done the parsing and validation, we've got something like an abstract syntax tree, and we can walk over that and start to do the interpretation. That interpretation is plain Python code which walks over a tree of a program and knows about closures, special forms, and the different language primitives we have available.
One really important thing here is the content-addressed store. This is what enables us to do caching and memoization, and this is super crucial — nothing would work if we weren't careful about this. The reason is that we rewrite a whole AshPL program and reinterpret the whole thing every time. We don't just interpret the code that's been rewritten; we re-address the program and reinterpret the whole kit and caboodle from top to bottom. That would obviously be super slow if we were really redoing the work every single time we went around the loop. In reality, it's nice and fast for us because of the language features. Because it's a pure language, that really helps with memoization. We can hash an expression and say if this has been evaluated before, we just store that away in a map. If we meet that expression again when we're walking the tree, we can say "this boiled down to 42" and use that straight away from the hash.
I'm going to switch to a demo now. I said before that there's often a trade-off between rigor and speed. On that continuum, we are very much focused on the rigor side. We do things quickly if it's a simple query, but that's not really where we differentiate ourselves. If you go to elicit.com, you'd see a bunch of templates: creating table slides, drafting a report. I'm going to show you a research landscape, which probably took in total a couple of hours of it doing work and me adding layers on top of it. I've got a session saved away. This is the research landscape we're going to take a look at. My initial query was to map the companies and institutions investing in foundation models for biology. The first thing we did was Elicit asked me a question. It was like, "I get the overall big picture, let me narrow that down. Are you interested in a broad landscape, a particular foundation model, academic institutions, or companies?" I just said the broad landscape. Then the rest of the steps are driven by AshPL. This first analysis step does a bunch of searches: academic papers relating to genomic foundation model pre-training transformer, web searches, fetching the full text of papers when available, screening, filtering. All of these stages are encoded into AshPL. The AshPL is not just a representation of a plan — it is literally the plan, which is executable. That's what lets us be sure we're following through on the plan as stated.
After some humming and whirring and quite a few tokens, we end up with a table. We call this an artifact, and each row is an organization which has some interest in biological foundation models. Google DeepMind, Meta, Microsoft Research, etc. We've extracted some attributes alongside that: the foundation models they've created, the modalities they're interested in, notable collaborations. For each of these artifacts, we can actually look at the AshPL code used to generate it. This is literally the executable DSL behind the creation of that table. We're doing web searches for foundation models, multimodal biology AI models, looking for academic papers, joining these together. Looking at the AshPL is not particularly fun. Most people don't do this. We have this because we want to know that Elicit is following the instructions we came up with. It is useful for other agents to look at this AshPL and say "you've missed something, you've overlooked something, there's a key search you should have considered."
Something a bit more ergonomic from a user perspective is a graphical representation of what's done within the system. This is derived directly from the AshPL. It literally is derived from the same thing the plan was executing over. In this case it's pretty linear: we start off with a couple of searches, did some enrichment (fetching full text of papers), extracting, curating (filtering), more searches. I find looking at this quite handy if I'm trying to convince myself whether I would endorse the process Elicit took. I wouldn't stop here, though. There's other layers I might want to add. I asked for a comparison of open and closed source strategies, the commercialization strategy, the GTM approach, mapping out the different government orgs and oversight institutions. At the end of my user session, I asked for a join. We've got a table of organizations and a table of oversight bodies, and just in natural language I can say I want to join these together and see how the labs have interacted with the oversight bodies. That came up with a table showing how Anthropic has interacted with the US AI Safety Institute and so on.
If I look at the AshPL for this table, the top of the program is identical to what we had before — the same web queries and paper queries we had for that very first table. The difference is this program is now a lot longer. The last one was maybe 100 or 150 lines; we're up to 1,000 or so. It's only down here that we start to talk about the oversight bodies and the interactions between those and the labs. At the point of generating that last table, we would have interpreted this whole program again from scratch, except for that cache I talked about. The fact that we had already done all these web queries and paper searches meant we can interpret the whole program from scratch, but the vast majority of it is just memoized and you get it back straight away. One of the reasons we took that design decision is because it's easy to be confident about and make statistical guarantees of cohesion and correctness when you're literally interpreting the whole program every single time. If you're just interpreting little snippets, that's where the drift can come in.
Can we switch back to the slides? Again, I'm not saying everyone should be using a DSL. It's not the easiest thing to build. It's the kind of thing you should reach for if the desiderata for your product and organization point you in that direction. Elicit anchors around high quality, dependability, robustness, and data provenance, and that's why we went for it. If you're in a similar position, here are some things you should be thinking about. Firstly and most obviously, you need a DSL. The agent ergonomic piece: we found you'll have a better time if you base your DSL on an existing language that has a lot of examples in the training data, because then the model — the curator in our case — doesn't need to learn the syntax. It just needs to know there's a subset it can go for. A surprisingly small amount of work went into the DSL compared to everything else. Everything else is conventional software engineering to turn it into a system that works.
I mentioned the wrapper, which lets you switch between different harnesses and models. Interrupt handling: when you're in Elicit waiting for results, you can add other things into the chat, and we want that to gracefully flow back into the curator so it can redraft its plan without stopping the world. That isn't something any harness handles natively, so we had to build it. We can come back to sessions in the future and rehydrate them, so we had to build a whole thing for that. Credential isolation is that wrapper thing I mentioned. There's an annoying amount of stuff to handle messages coming out of the models and make sure they're not just lost to standard out. We use event sourcing — we're really happy with that pattern, and it's not a small lift. I think most people would probably need to do three, four, five, and six, would recommend number two, and you have to do something with number seven. And eight: we pride ourselves on accuracy, robustness, and trustworthiness, and we have a dedicated eval team who are great. It's so hard to do eval when the system is writing programs and executing them on the fly. It's a very complex dynamic domain, but we've invested a lot of time there, and I'd strongly recommend you do the same if you're doing a DSL-based system.
Let me finish where I began. The example I gave at the beginning was: let's imagine two systems produce identical output, and should you trust it? It's not crazy to imagine Opus coming up with one of those tables I was showing. At least at a surface level, the table itself isn't a crazy thing to imagine a state-of-the-art model coming up with. However, the fact that we go through a very particular and painstaking process to generate it, and we expose that in an ergonomic way to the user with the AshPL and that graphical interface, I think that's the thing that makes me know from conversations with our users that they would hold those two things quite differently. A table like that in Elicit is a fundamentally different thing to a table that's just being bubbled out from a model. Maybe there's something that has that same dynamic for your business and your product. My pitch here is not that you should go and use a DSL. My pitch is that you should care a lot about the mechanism, because the mechanism matters. That's it for me. Thank you very much.