- Dylan from AirOps (a growth-marketing platform for AI search) explains how making agents accessible to non-technical personas like marketers is genuinely hard, and how chasing friction points is the path to production-grade, accessible agents.
- AirOps moved from a brittle node-based workflow builder (which hit a complexity ceiling and had a short shelf life as new models shipped) to the Claude Agent SDK, orchestrating agents through markdown files and skills; their new launch, AirOps Next, includes Quill (an agent for content marketers) and Playbooks (accessible, versioned skills).
- The two friction points they dialed in on were fitting the actual marketer workflow (document-based builder, transparency, control, enforced human review) and ensuring quality outputs through harness engineering — specialized tools and focused sub-agents (compliance check, writing, brand kit) to manage context efficiently.
How AirOps chases friction to build AI products with Claude
- AirOps is a growth-marketing platform for AI search (like SEO but for engines like ChatGPT, Gemini, Claude); it helps brands see how they appear, find gaps, take action, and measure impact.
- The old node-based workflow builder hit a complexity ceiling (teaching marketers liquid text and JSON), had a short shelf life as new Opus versions shipped, and broke when a change in step one silently affected variables referenced in step 20.
- The release of Opus 4.5 was the breaking point that justified investing heavily in agents, thanks to its tool-calling and instruction-following improvements; the Claude Agent SDK lets you orchestrate agents through markdown files and skills rather than programmatic node-routing.
- AirOps Next launched Quill (an agent "captain" for content marketers with access to AI search data, brand context, and brand kit) and Playbooks (a document-based, collaborative, versioned builder — essentially accessible skills); a Parallel case study saw a 130% citation-rate increase, 42% share-of-voice increase, and go-live in 1 week.
- Friction point 1 — "endless use cases force intentionality": be very intentional about which customer and problem you solve; the content-marketing workflow needs a document-based IDE, transparency (seeing which tool runs at each step), control, and enforced human review with governance and versioning.
- Friction point 2 — consistency and quality: harness engineering matters as much as the model ("a car needs more than its engine"); specialized tools (e.g. one page-analysis tool replacing many primitive tool calls) cut a tool's token consumption ~8% and improved speed.
- Sub-agents are crucial for quality: start with plain Claude doing its own tool calls, then add focused sub-agents — a compliance check, a writing agent with its own focused context window, and a brand-kit sub-agent that fetches context once and stores it as an artifact the main loop references.
- Context minimalism matters even with million-token windows: just because the window is larger doesn't mean you should use the whole thing; be efficient about what the model attends to. Next friction points AirOps is chasing are self-improvement/feedback loops (memory, "forgetting as a feature") and benchmarking content-creation agents.
AI search — Optimizing how a brand appears in answers from AI engines like ChatGPT, Gemini, and Claude; analogous to SEO for search engines.
Workflow builder — A node-based, drag-and-drop tool for orchestrating content steps and variables; AirOps' previous approach.
Claude Agent SDK — Anthropic's SDK that lets you orchestrate agents through markdown files and skills rather than programmatic node-routing.
Playbook — AirOps' document-based, collaborative, versioned builder; essentially an accessible skill for marketers.
Quill — AirOps' agent "captain" for content marketers, with access to AI search data, brand context, and brand kit.
Harness engineering — Building everything around the model (tools, context orchestration, sub-agents) to create a great agent — "the car around the engine."
Sub-agent — A focused agent spun off with its own context window for a specific job (compliance, writing, brand kit) to avoid polluting the main context.
Brand kit — A knowledge base / context layer holding everything to know about a brand, fetched once and stored as an artifact.
Human-in-the-loop — Enforced human review points within the agentic workflow, with governance, configurability, and accountability.
Hey, how's it going everyone? I'm Dylan. I work on the product team at AirOps. And I'm excited to walk you through how AirOps chases friction with building AI products with Claude. The main really big takeaway I want you guys to come away with is building agents and just making agents accessible is honestly a really hard problem. With developers it's a bit easier. People are used to all these different concepts, but when you try to make these accessible to personas like marketers, there are a lot of friction points. And I'm going to talk through some of the friction points that we have seen and battled with.
Just to start off, quick intro of who we are at AirOps. We are a growth-marketing platform for AI search. And AI search is kind of like SEO but for engines like ChatGPT, Gemini, Claude. Buyers are asking on Claude different questions of, "Hey, I want to buy these pair of sunglasses. How are you making sure that you are showing up for these searches?" So, we help brands see how they're appearing in search, identify gaps, take action on those gaps, whether that be creating content, refreshing content, and then being able to measure the impact of whether or not the actions that they're taking actually worked.
Quick agenda: I'm just going to walk through how we got here, our approach for agents, a quick run through of what we just launched last week with AirOps Next, and then two friction points that we really focus and dial in on for our launch when it comes to giving people the power of Claude and making that super accessible. And then closing with a couple of other friction points that we're really looking at for our next act.
How we got here? AirOps used to be — or still is, but — we were mainly focused on orchestrating content through a traditional workflow builder, so node-based style, kind of like an N8N where you can drag and drop these different nodes, you can orchestrate different variables and how things flow through this elaborate workflow. And with a workflow-builder style, and especially with our core customer and audience being marketers, you would hit this complexity ceiling where you're trying to teach a content marketer what liquid text is, what JSON is, and all these different concepts. And they also had this short shelf life, which as new models came out, Claude would release Opus 4.6, 4.7, it keeps on going, you would have to update these different steps. It changes the way that you're building this workflow, so customers are constantly having to go and update this workflow they just spent so much time building. And also if they update a step in step one, they don't realize that variables and outputs are referenced in step 20, so there's a bunch of complexity, and it was really brutal. And scaling enterprise use cases obviously then required someone technical to help really guide through this workflow-creating process. So our goal was to really lower this barrier to entry for content marketers to build and ship their ideas and the way they want to create content. And also while still maintaining that quality bar, which is a very big focus for them because brands don't want to just be pumping AI slop out, and having governance around keeping those enterprise standards.
So this is a quick look at what our workflow studio looked like. This use case was actually to create a content brief with internal links. As you can see, it gets kind of gnarly in there. Just for something that seemed pretty simple. It just goes to show how much thought goes into a lot of these workflows. So how can we take this kind of structure and create an agent experience while still harnessing the way that we use Claude Code and these other agent tools?
I think the first really breaking point when we decided to invest heavier into agents was really with the release of Opus 4.5. I think that's when a lot of people started to see how smart the models were in tool calling, being able to follow instructions, in a way that they weren't breaking the standards that they were setting for it. And at the same time we're trying to make building easier. And one of the ways that we do that is through this document-based-style builder. The first thing that we tried doing was actually taking this kind of like Google Doc and with an LLM compiling a workflow in the background, which was an interesting idea. It was a bit brittle, there's a lot of error points that can happen there, trying to turn these non-deterministic instructions into an actual workflow. Then after that was using a traditional agent orchestration framework where you're defining these nodes, you're letting an LLM decide different decision points. And we got to a point where we were getting pretty good outputs, but with those traditional frameworks, they honestly are pretty brittle. If I ever want to change the way I want to orchestrate different sub-agents, I basically have to make code changes. I actually have to go and change the way I'm routing these different nodes and steps together. And that's when we really decided to invest heavier into the Claude Agent SDK, where it's honestly pretty awesome that you can orchestrate agents just through markdown files and provide skills in different contexts, really just by manipulating this environment and harness rather than actually doing that programmatically through a traditional agent framework.
So that's what led to our launch which we just had last week with AirOps Next. Briefly, what did we launch? The first one was Quill, which was our branding of our agent captain for content marketers. Quill had access to all the data that we provide to teams, whether that be AI search data, the brand context and brand kit which houses literally everything about a brand. So Quill is following the instructions of how content should be created. And just making it easier throughout the UI to get people from insights they see in the dashboards to actual actions. Being able to take the findings and gaps and get to that in the very next step. The next one is Playbooks. So Playbooks is basically our new building experience. And for all you developers out there it's pretty much like a skill. That's very much what we grounded in — how can we make skills accessible — and marketers are all used to the document-based style. But we just allowed for collaboration on these Playbooks/skills and also governance and versioning. So you'll have people with 10 different versions of this playbook of how do I want to create this piece of content?
Just real quick, results that we've seen from customers. We did a case study with Parallel. We helped them produce and create content and they saw a 130% increase in citation rate, 42% increase in share of voice, and they were able to go live in 1 week, which for us is a huge accomplishment just because traditionally, especially because we work with these enterprise customers, it usually takes around a month at least with the workflow builder of constant feedback going back and forth, like "Hey, this is not really how I want to be speaking in my blog. Citations that are kind of being hallucinated or aren't right." So it was really incredible to be able to get to that acceptance criteria in such a short amount of time. And other quick customer quotes too. Incredible to see how Agents has moved what LLMs can do and how marketers are viewing using these more objective workflows. Animalz saying it felt more like a mid-level strategist for their team. And at Rock Content, being able to focus on off-boarding a lot of the tedious tasks and really focusing on where does my expertise come in and being able to add that value to the agent while it's creating that content and feeding that unique context.
So we were able to really accomplish this by focusing on two main friction points. When building and trying to bring high-quality Agents to these more highly professional and enterprise use cases, we really had to focus on how do we fit into the actual workflow of these content marketers? And how do we make sure that we ensure quality outputs through the way that we're building and orchestrating our harness?
So the first friction point I want to talk through is — I have it saying like "endless use cases force intentionality." A lot of us have probably been there too, when we first started using Claude Code or using an agent and we're like, this thing is really powerful. I can do a ton of different things. It's really easy to start sprawling into this spiral of so many different use cases. And not only on the product engineering side, but also for customers, how do we be very intentional about what's the actual problem that we're trying to solve here, and how do we really understand what is this workflow that we're trying to solve for people. So in my head now I have this mini Steve Ballmer head just going "marketers, marketers, marketers," just really trying to focus on who is my customer in this use case that I want to tackle.
Just a quick glimpse into what does this workflow look like for a content marketer. This is one example use case for content creation where they'll discover what they want to create on, they'll research the specific topic that they want to make sure that they're ranking in AI search or traditional search for, they'll draft a brief, draft the article, and then add any sort of internal linking and best practices when it comes to SEO and AEO. And throughout this whole process, there are different human review points. Human review comes up a ton, human-in-the-loop comes up a ton in content marketing, especially with when you're being cognizant about the content that you're pushing out. You really want to make sure that it is adhering to the way that your brand talks, to the information that you're serving out in the internet, and just giving you the best shot of also being surfaced within generative AI search.
So that whole process led us to focus on two parts when it comes to the content marketing flow. One is having a document-based kind of like IDE, which again is our playbook view. Marketers are super familiar with documents. They are used to Google Docs. They've probably been working with Docs for years and ages. Even before technology, you have this piece of paper that you're defining how you want to do something. So making that familiar versus this node-based workflow builder. Transparency was also really important. One thing that users actually really liked about the workflow builder was that they could actually see what tools were being used at each step. So how do you still have a document but at the same time when I read through, I can understand this is the exact tool that I'm using at this instant. This is the type of context I'm feeding at this point. And that goes hand in hand with control as well. How can I make sure that I'm still in control of this set of instructions or workflow that I'm building, especially now that it is not as deterministic because it is a document? So those are three areas that we focused on. And then on enforcing human review, how can we bring governance, configurability, accountability into an agentic workflow, which I think is pretty unique for us. I don't think a lot of brands tackle that problem. A lot of it with agents, also with coding, is kind of let it go, it will finish what's done with. And human review in that instance with coding is usually with PR reviews after it's literally finished the whole job. It's rarely ever while it's going through the actual coding process that it'll ask for feedback of taste in different areas. So that was another area that we really focused on.
With that, I want to jump into a quick demo of what this looks like within AirOps. The first thing I want to cover is our playbook. It's kind of like this skill, this natural-language builder where I can go in and either type in with a slash command and define different inputs, different outputs, and also all these different tools that content marketers can use and that they're super used to using. And when it comes to tools as well, you can add any MCP. So there's other outside connectors that you usually use, you can use those and access them. We also have the ability to schedule different triggers. So this gives this kind of always-on skill or playbook or agent that can do the certain action at either a scheduled cadence or based off webhooks. We also have monitor, which we partnered with Parallel on, when it comes to being able to put a query in, like saying "watch the internet" in a way. So when certain things happen, it would trigger off this playbook to then run. And then the last one is AO insights. So whenever a metric drops, let's say my citation rate dropped in the last week, it would trigger off one of these playbooks. It can go and do this research and come back to me with, "Hey, this is the reason why this happened."
And as I jump through into this process of someone basically creating this SOP of how they want to create this blog, if I come to the outline section, we enforce human review by adding the ability to assign different users at the end of each section. So as the agent goes through this whole playbook, it'll come to this section step and we have a tool in the background that will then fire off. And basically, since I'm assigned, I'm the only person that can actually unblock this agent. Other people can still leave comments on the outputs and artifacts and different feedback, but I'm this gatekeeper now that has to review this piece of content, and we do that to ensure that governance. And what that surfaces to users, one way is through our inbox. So we have an inbox within AirOps where every single time human review happens or different opportunities are surfaced, users can come in here and see those directly. So I can click on one of these items, it will open up our agent runs on the right side, basically this agent running through its whole process. I can see its thought traces. On the left side, all the different outputs and artifacts that are defined by me through that playbook. And I can then either edit this existing document, leave certain comments, and then kick it off and approve it. The other way that we surface this governance and human review is through our grid. So our grid is our way of orchestrating content at scale. In this specific example, I'm showing the way that you can collaborate. Augustine is also in this document with me. I'm able to edit it, also leave human review. If I close this out, I can see within the grid all these different outputs that are running. We're basically running skills at scale where each one of these rows is a specific job that I want to accomplish and I'm running that playbook. And I can also see human review at scale and click in these different cells to leave my feedback and ensure that all this content's talking in my right tone of voice.
The second friction point I want to talk about was, with agents, the biggest worry is around consistency, especially when you're coming from this workflow base. So how did we tackle going about producing quality outputs? How do we make sure that customers are also seeing that and being aligned to that as well? One of the ways that we visualize this — this graphic here was created and referenced by our VP of sales, which funny enough is actually one of the best explanations I feel like I've seen of harness engineering. On the product side we tried a couple of times, but it's just funny that from the go-to-market side it was the best way that we've been able to explain this, where you have this car where you then have an engine, the model obviously being Claude, Opus, or Sonnet, whatever engine that you want to pick for that. And everything else that goes around that, everything else that you build on top of it, is super important in terms of creating a super great agent.
So the two that I want to focus on, and the one that we focused on the most, was around tools and the way that we were orchestrating context. And the Claude Agent SDK and also Claude Managed Agents API have been super helpful in terms of being able to iterate on this quickly. Just making sure that we're programmatically setting these different sub-agents. They've been super instrumental in terms of us getting to a quality output.
So the first one I want to cover is around tools. The background on tools is you could give an agent a bunch of primitive tools. In this specific example, what we were always trying to accomplish with Claude and our agent was helping Claude understand what is wrong with a page on my website. We have different tools like access to traffic data, citation data, scrapers so I can find similar competitor pages. And we started off honestly with a skill of "here's the way of dissecting whether or not a page is losing, what's wrong with a specific page, are the schemas off, how's it compared to competitor pages." And they'll go on these safari trips and it's honestly a bit token-inefficient. So one thing that we wanted to focus on is how can we create specialized tools of jobs that Claude does over and over again, and how can we make that a bit more deterministic, where Claude can put a URL and immediately spit out everything about that page as well as different structured content gaps versus other similar pages in that space, as well as what should be my target keywords and my target prompts for it. The second tool that we made, in a workflow that we saw was super common with content marketers, was our page-versus tool, which was essentially being able to benchmark my page against top ranking pages in that space and finding what exactly is behind those pages and how can I close those gaps. So this is a really simple way of getting context efficiently. It's kind of like code mode in a way, being able to be more programmatic in terms of how we're fetching context versus looping through these different tool calls. Can I actually just produce code that will fetch exactly what I need in one loop.
The second one is through sub-agents. Sub-agents have definitely been instrumental and crucial in terms of getting to that quality of output. In general, what we tell users with playbooks, and honestly when you're first creating your agent harness, is that you actually just start off with Claude itself and just have it go through its own tool calls and really not try to make it too complex in all the context you're trying to give to it. So that's where we start off with, and we were reaching a couple of air spots when it came to the quality of the outputs we're getting. So what we did was decide to add on over time certain sub-agents. The first one was a compliance check. So being able to make sure that we're not polluting that main context window, just because context is honestly still a huge problem and will probably continue to be a problem when it comes to the tokens and text that you're attending to. So being able to spin off a sub-agent that will go through and have everything that needs to know about my brand and whether or not the content that I just produced is following those rules. It'll come back with a score whether or not it adhered to them, what were the things wrong with it, and then the agent can take that feedback and make edits. The second one we did was around writing. We first tried just using the regular Claude harness to actually write the content, but we found that it was better to spin off a sub-agent with its own very focused context window to focus solely on just writing that piece of content. So it's not distracted by any of the research that was created or any of maybe old compliance checks, just being able to very focus in on that one job. The next one is a brand-kit sub-agent. Brand kit for us is basically a knowledge base or context layer of everything you need to know about a brand. And this is something that we kick off at the beginning of all of our runs where we have a brand-kit sub-agent that will go and fetch all of the relevant context it needs and then stores it as an internal artifact. So then throughout the whole process our main agent loop can reference those artifacts versus having to use tools in our MCP to refetch that context. Because then what happens is, in different sub-agents you might actually have different brand context that was fetched over time. So just being able to fetch that up front, store it as an artifact, and then redirect the agent to always be referencing that same artifact. And then finally, just still being able to add custom ones. This is more for us internally where some of our solutions architects who work for our customers can spin off different sub-agents when needed, and just really helps with maintaining that context.
I think that's definitely been the biggest learning, is although context windows do continue to grow — you have a million context window with Opus 4.7 — but just still being really cognizant that just because it is larger doesn't mean that you should be using the whole thing. You should still be very efficient about what sorts of context you're letting the model attend to. And in terms of results, what we saw was actually an 8% decrease in fewer tokens consumption around at least that specific tool. So instead of that tool spinning off different primitives and going over and over again, you actually just have this one tool call that fetches a page and then all the results from it. The second one from those specialized tools was also speed. Beforehand, while it was making those 20 different tool calls to fetch all the context it needs, you just have this one entry point that spits out everything that you need. And then when it comes to the quality side, we had 10 enterprise customers while we were running this beta actually start publishing content in under 2 weeks. So they're able to self-serve and get to a quality of output through these agents, which used to be kind of a whole hand-holding experience.
So when it comes to building these agents, and with Claude being able to execute and making it easier to execute on things, it's really easy to think that there are no more problems to solve. But really every single time a problem is solved, that friction point always keeps moving. So those are the two that we were really chasing after. And there's a ton more that we also want to continue chasing, but I think overall it's a great thing to just continue to chase friction because that really is how you create production agents and make those more accessible to users outside of more technical spaces.
Just to quickly close out, the next two friction points that we're looking to battle and excited to show more learnings on are around self-improvement and feedback loops, which there were awesome talks yesterday around dreaming sequences and how — I think the most interesting one is actually how do you structure summaries of different traces? What's the best way of collecting the most relevant memories and also forgetting is actually a feature, being able to forget certain types of memories. And the last one is benchmarking content-creation agents. I think something really interesting about our space is that it isn't law, it isn't coding, it's not something that is really easy to say whether or not something's correct. There is a lot of taste that goes into a piece of content that you create and a lot of opinions of how I want certain content to be formatted and created. So what are the best ways that we are creating benchmarks so that every single time we do change our harness, we add a sub-agent or we decide to add a skill to our harness, how do we know that it's actually improving outputs and not just vibes that we're going after. But yeah, I'm around all day, would love to chat with you guys and hear about what you guys are building. I hope this was helpful. Thank you.
TL;DR
- Dylan from AirOps (a growth-marketing platform for AI search) explains how making agents accessible to non-technical personas like marketers is genuinely hard, and how chasing friction points is the path to production-grade, accessible agents.
- AirOps moved from a brittle node-based workflow builder (which hit a complexity ceiling and had a short shelf life as new models shipped) to the Claude Agent SDK, orchestrating agents through markdown files and skills; their new launch, AirOps Next, includes Quill (an agent for content marketers) and Playbooks (accessible, versioned skills).
- The two friction points they dialed in on were fitting the actual marketer workflow (document-based builder, transparency, control, enforced human review) and ensuring quality outputs through harness engineering — specialized tools and focused sub-agents (compliance check, writing, brand kit) to manage context efficiently.
Takeaways
- AirOps is a growth-marketing platform for AI search (like SEO but for engines like ChatGPT, Gemini, Claude); it helps brands see how they appear, find gaps, take action, and measure impact.
- The old node-based workflow builder hit a complexity ceiling (teaching marketers liquid text and JSON), had a short shelf life as new Opus versions shipped, and broke when a change in step one silently affected variables referenced in step 20.
- The release of Opus 4.5 was the breaking point that justified investing heavily in agents, thanks to its tool-calling and instruction-following improvements; the Claude Agent SDK lets you orchestrate agents through markdown files and skills rather than programmatic node-routing.
- AirOps Next launched Quill (an agent "captain" for content marketers with access to AI search data, brand context, and brand kit) and Playbooks (a document-based, collaborative, versioned builder — essentially accessible skills); a Parallel case study saw a 130% citation-rate increase, 42% share-of-voice increase, and go-live in 1 week.
- Friction point 1 — "endless use cases force intentionality": be very intentional about which customer and problem you solve; the content-marketing workflow needs a document-based IDE, transparency (seeing which tool runs at each step), control, and enforced human review with governance and versioning.
- Friction point 2 — consistency and quality: harness engineering matters as much as the model ("a car needs more than its engine"); specialized tools (e.g. one page-analysis tool replacing many primitive tool calls) cut a tool's token consumption ~8% and improved speed.
- Sub-agents are crucial for quality: start with plain Claude doing its own tool calls, then add focused sub-agents — a compliance check, a writing agent with its own focused context window, and a brand-kit sub-agent that fetches context once and stores it as an artifact the main loop references.
- Context minimalism matters even with million-token windows: just because the window is larger doesn't mean you should use the whole thing; be efficient about what the model attends to. Next friction points AirOps is chasing are self-improvement/feedback loops (memory, "forgetting as a feature") and benchmarking content-creation agents.
Vocabulary
AI search — Optimizing how a brand appears in answers from AI engines like ChatGPT, Gemini, and Claude; analogous to SEO for search engines.
Workflow builder — A node-based, drag-and-drop tool for orchestrating content steps and variables; AirOps' previous approach.
Claude Agent SDK — Anthropic's SDK that lets you orchestrate agents through markdown files and skills rather than programmatic node-routing.
Playbook — AirOps' document-based, collaborative, versioned builder; essentially an accessible skill for marketers.
Quill — AirOps' agent "captain" for content marketers, with access to AI search data, brand context, and brand kit.
Harness engineering — Building everything around the model (tools, context orchestration, sub-agents) to create a great agent — "the car around the engine."
Sub-agent — A focused agent spun off with its own context window for a specific job (compliance, writing, brand kit) to avoid polluting the main context.
Brand kit — A knowledge base / context layer holding everything to know about a brand, fetched once and stored as an artifact.
Human-in-the-loop — Enforced human review points within the agentic workflow, with governance, configurability, and accountability.
Transcript
Hey, how's it going everyone? I'm Dylan. I work on the product team at AirOps. And I'm excited to walk you through how AirOps chases friction with building AI products with Claude. The main really big takeaway I want you guys to come away with is building agents and just making agents accessible is honestly a really hard problem. With developers it's a bit easier. People are used to all these different concepts, but when you try to make these accessible to personas like marketers, there are a lot of friction points. And I'm going to talk through some of the friction points that we have seen and battled with.
Just to start off, quick intro of who we are at AirOps. We are a growth-marketing platform for AI search. And AI search is kind of like SEO but for engines like ChatGPT, Gemini, Claude. Buyers are asking on Claude different questions of, "Hey, I want to buy these pair of sunglasses. How are you making sure that you are showing up for these searches?" So, we help brands see how they're appearing in search, identify gaps, take action on those gaps, whether that be creating content, refreshing content, and then being able to measure the impact of whether or not the actions that they're taking actually worked.
Quick agenda: I'm just going to walk through how we got here, our approach for agents, a quick run through of what we just launched last week with AirOps Next, and then two friction points that we really focus and dial in on for our launch when it comes to giving people the power of Claude and making that super accessible. And then closing with a couple of other friction points that we're really looking at for our next act.
How we got here? AirOps used to be — or still is, but — we were mainly focused on orchestrating content through a traditional workflow builder, so node-based style, kind of like an N8N where you can drag and drop these different nodes, you can orchestrate different variables and how things flow through this elaborate workflow. And with a workflow-builder style, and especially with our core customer and audience being marketers, you would hit this complexity ceiling where you're trying to teach a content marketer what liquid text is, what JSON is, and all these different concepts. And they also had this short shelf life, which as new models came out, Claude would release Opus 4.6, 4.7, it keeps on going, you would have to update these different steps. It changes the way that you're building this workflow, so customers are constantly having to go and update this workflow they just spent so much time building. And also if they update a step in step one, they don't realize that variables and outputs are referenced in step 20, so there's a bunch of complexity, and it was really brutal. And scaling enterprise use cases obviously then required someone technical to help really guide through this workflow-creating process. So our goal was to really lower this barrier to entry for content marketers to build and ship their ideas and the way they want to create content. And also while still maintaining that quality bar, which is a very big focus for them because brands don't want to just be pumping AI slop out, and having governance around keeping those enterprise standards.
So this is a quick look at what our workflow studio looked like. This use case was actually to create a content brief with internal links. As you can see, it gets kind of gnarly in there. Just for something that seemed pretty simple. It just goes to show how much thought goes into a lot of these workflows. So how can we take this kind of structure and create an agent experience while still harnessing the way that we use Claude Code and these other agent tools?
I think the first really breaking point when we decided to invest heavier into agents was really with the release of Opus 4.5. I think that's when a lot of people started to see how smart the models were in tool calling, being able to follow instructions, in a way that they weren't breaking the standards that they were setting for it. And at the same time we're trying to make building easier. And one of the ways that we do that is through this document-based-style builder. The first thing that we tried doing was actually taking this kind of like Google Doc and with an LLM compiling a workflow in the background, which was an interesting idea. It was a bit brittle, there's a lot of error points that can happen there, trying to turn these non-deterministic instructions into an actual workflow. Then after that was using a traditional agent orchestration framework where you're defining these nodes, you're letting an LLM decide different decision points. And we got to a point where we were getting pretty good outputs, but with those traditional frameworks, they honestly are pretty brittle. If I ever want to change the way I want to orchestrate different sub-agents, I basically have to make code changes. I actually have to go and change the way I'm routing these different nodes and steps together. And that's when we really decided to invest heavier into the Claude Agent SDK, where it's honestly pretty awesome that you can orchestrate agents just through markdown files and provide skills in different contexts, really just by manipulating this environment and harness rather than actually doing that programmatically through a traditional agent framework.
So that's what led to our launch which we just had last week with AirOps Next. Briefly, what did we launch? The first one was Quill, which was our branding of our agent captain for content marketers. Quill had access to all the data that we provide to teams, whether that be AI search data, the brand context and brand kit which houses literally everything about a brand. So Quill is following the instructions of how content should be created. And just making it easier throughout the UI to get people from insights they see in the dashboards to actual actions. Being able to take the findings and gaps and get to that in the very next step. The next one is Playbooks. So Playbooks is basically our new building experience. And for all you developers out there it's pretty much like a skill. That's very much what we grounded in — how can we make skills accessible — and marketers are all used to the document-based style. But we just allowed for collaboration on these Playbooks/skills and also governance and versioning. So you'll have people with 10 different versions of this playbook of how do I want to create this piece of content?
Just real quick, results that we've seen from customers. We did a case study with Parallel. We helped them produce and create content and they saw a 130% increase in citation rate, 42% increase in share of voice, and they were able to go live in 1 week, which for us is a huge accomplishment just because traditionally, especially because we work with these enterprise customers, it usually takes around a month at least with the workflow builder of constant feedback going back and forth, like "Hey, this is not really how I want to be speaking in my blog. Citations that are kind of being hallucinated or aren't right." So it was really incredible to be able to get to that acceptance criteria in such a short amount of time. And other quick customer quotes too. Incredible to see how Agents has moved what LLMs can do and how marketers are viewing using these more objective workflows. Animalz saying it felt more like a mid-level strategist for their team. And at Rock Content, being able to focus on off-boarding a lot of the tedious tasks and really focusing on where does my expertise come in and being able to add that value to the agent while it's creating that content and feeding that unique context.
So we were able to really accomplish this by focusing on two main friction points. When building and trying to bring high-quality Agents to these more highly professional and enterprise use cases, we really had to focus on how do we fit into the actual workflow of these content marketers? And how do we make sure that we ensure quality outputs through the way that we're building and orchestrating our harness?
So the first friction point I want to talk through is — I have it saying like "endless use cases force intentionality." A lot of us have probably been there too, when we first started using Claude Code or using an agent and we're like, this thing is really powerful. I can do a ton of different things. It's really easy to start sprawling into this spiral of so many different use cases. And not only on the product engineering side, but also for customers, how do we be very intentional about what's the actual problem that we're trying to solve here, and how do we really understand what is this workflow that we're trying to solve for people. So in my head now I have this mini Steve Ballmer head just going "marketers, marketers, marketers," just really trying to focus on who is my customer in this use case that I want to tackle.
Just a quick glimpse into what does this workflow look like for a content marketer. This is one example use case for content creation where they'll discover what they want to create on, they'll research the specific topic that they want to make sure that they're ranking in AI search or traditional search for, they'll draft a brief, draft the article, and then add any sort of internal linking and best practices when it comes to SEO and AEO. And throughout this whole process, there are different human review points. Human review comes up a ton, human-in-the-loop comes up a ton in content marketing, especially with when you're being cognizant about the content that you're pushing out. You really want to make sure that it is adhering to the way that your brand talks, to the information that you're serving out in the internet, and just giving you the best shot of also being surfaced within generative AI search.
So that whole process led us to focus on two parts when it comes to the content marketing flow. One is having a document-based kind of like IDE, which again is our playbook view. Marketers are super familiar with documents. They are used to Google Docs. They've probably been working with Docs for years and ages. Even before technology, you have this piece of paper that you're defining how you want to do something. So making that familiar versus this node-based workflow builder. Transparency was also really important. One thing that users actually really liked about the workflow builder was that they could actually see what tools were being used at each step. So how do you still have a document but at the same time when I read through, I can understand this is the exact tool that I'm using at this instant. This is the type of context I'm feeding at this point. And that goes hand in hand with control as well. How can I make sure that I'm still in control of this set of instructions or workflow that I'm building, especially now that it is not as deterministic because it is a document? So those are three areas that we focused on. And then on enforcing human review, how can we bring governance, configurability, accountability into an agentic workflow, which I think is pretty unique for us. I don't think a lot of brands tackle that problem. A lot of it with agents, also with coding, is kind of let it go, it will finish what's done with. And human review in that instance with coding is usually with PR reviews after it's literally finished the whole job. It's rarely ever while it's going through the actual coding process that it'll ask for feedback of taste in different areas. So that was another area that we really focused on.
With that, I want to jump into a quick demo of what this looks like within AirOps. The first thing I want to cover is our playbook. It's kind of like this skill, this natural-language builder where I can go in and either type in with a slash command and define different inputs, different outputs, and also all these different tools that content marketers can use and that they're super used to using. And when it comes to tools as well, you can add any MCP. So there's other outside connectors that you usually use, you can use those and access them. We also have the ability to schedule different triggers. So this gives this kind of always-on skill or playbook or agent that can do the certain action at either a scheduled cadence or based off webhooks. We also have monitor, which we partnered with Parallel on, when it comes to being able to put a query in, like saying "watch the internet" in a way. So when certain things happen, it would trigger off this playbook to then run. And then the last one is AO insights. So whenever a metric drops, let's say my citation rate dropped in the last week, it would trigger off one of these playbooks. It can go and do this research and come back to me with, "Hey, this is the reason why this happened."
And as I jump through into this process of someone basically creating this SOP of how they want to create this blog, if I come to the outline section, we enforce human review by adding the ability to assign different users at the end of each section. So as the agent goes through this whole playbook, it'll come to this section step and we have a tool in the background that will then fire off. And basically, since I'm assigned, I'm the only person that can actually unblock this agent. Other people can still leave comments on the outputs and artifacts and different feedback, but I'm this gatekeeper now that has to review this piece of content, and we do that to ensure that governance. And what that surfaces to users, one way is through our inbox. So we have an inbox within AirOps where every single time human review happens or different opportunities are surfaced, users can come in here and see those directly. So I can click on one of these items, it will open up our agent runs on the right side, basically this agent running through its whole process. I can see its thought traces. On the left side, all the different outputs and artifacts that are defined by me through that playbook. And I can then either edit this existing document, leave certain comments, and then kick it off and approve it. The other way that we surface this governance and human review is through our grid. So our grid is our way of orchestrating content at scale. In this specific example, I'm showing the way that you can collaborate. Augustine is also in this document with me. I'm able to edit it, also leave human review. If I close this out, I can see within the grid all these different outputs that are running. We're basically running skills at scale where each one of these rows is a specific job that I want to accomplish and I'm running that playbook. And I can also see human review at scale and click in these different cells to leave my feedback and ensure that all this content's talking in my right tone of voice.
The second friction point I want to talk about was, with agents, the biggest worry is around consistency, especially when you're coming from this workflow base. So how did we tackle going about producing quality outputs? How do we make sure that customers are also seeing that and being aligned to that as well? One of the ways that we visualize this — this graphic here was created and referenced by our VP of sales, which funny enough is actually one of the best explanations I feel like I've seen of harness engineering. On the product side we tried a couple of times, but it's just funny that from the go-to-market side it was the best way that we've been able to explain this, where you have this car where you then have an engine, the model obviously being Claude, Opus, or Sonnet, whatever engine that you want to pick for that. And everything else that goes around that, everything else that you build on top of it, is super important in terms of creating a super great agent.
So the two that I want to focus on, and the one that we focused on the most, was around tools and the way that we were orchestrating context. And the Claude Agent SDK and also Claude Managed Agents API have been super helpful in terms of being able to iterate on this quickly. Just making sure that we're programmatically setting these different sub-agents. They've been super instrumental in terms of us getting to a quality output.
So the first one I want to cover is around tools. The background on tools is you could give an agent a bunch of primitive tools. In this specific example, what we were always trying to accomplish with Claude and our agent was helping Claude understand what is wrong with a page on my website. We have different tools like access to traffic data, citation data, scrapers so I can find similar competitor pages. And we started off honestly with a skill of "here's the way of dissecting whether or not a page is losing, what's wrong with a specific page, are the schemas off, how's it compared to competitor pages." And they'll go on these safari trips and it's honestly a bit token-inefficient. So one thing that we wanted to focus on is how can we create specialized tools of jobs that Claude does over and over again, and how can we make that a bit more deterministic, where Claude can put a URL and immediately spit out everything about that page as well as different structured content gaps versus other similar pages in that space, as well as what should be my target keywords and my target prompts for it. The second tool that we made, in a workflow that we saw was super common with content marketers, was our page-versus tool, which was essentially being able to benchmark my page against top ranking pages in that space and finding what exactly is behind those pages and how can I close those gaps. So this is a really simple way of getting context efficiently. It's kind of like code mode in a way, being able to be more programmatic in terms of how we're fetching context versus looping through these different tool calls. Can I actually just produce code that will fetch exactly what I need in one loop.
The second one is through sub-agents. Sub-agents have definitely been instrumental and crucial in terms of getting to that quality of output. In general, what we tell users with playbooks, and honestly when you're first creating your agent harness, is that you actually just start off with Claude itself and just have it go through its own tool calls and really not try to make it too complex in all the context you're trying to give to it. So that's where we start off with, and we were reaching a couple of air spots when it came to the quality of the outputs we're getting. So what we did was decide to add on over time certain sub-agents. The first one was a compliance check. So being able to make sure that we're not polluting that main context window, just because context is honestly still a huge problem and will probably continue to be a problem when it comes to the tokens and text that you're attending to. So being able to spin off a sub-agent that will go through and have everything that needs to know about my brand and whether or not the content that I just produced is following those rules. It'll come back with a score whether or not it adhered to them, what were the things wrong with it, and then the agent can take that feedback and make edits. The second one we did was around writing. We first tried just using the regular Claude harness to actually write the content, but we found that it was better to spin off a sub-agent with its own very focused context window to focus solely on just writing that piece of content. So it's not distracted by any of the research that was created or any of maybe old compliance checks, just being able to very focus in on that one job. The next one is a brand-kit sub-agent. Brand kit for us is basically a knowledge base or context layer of everything you need to know about a brand. And this is something that we kick off at the beginning of all of our runs where we have a brand-kit sub-agent that will go and fetch all of the relevant context it needs and then stores it as an internal artifact. So then throughout the whole process our main agent loop can reference those artifacts versus having to use tools in our MCP to refetch that context. Because then what happens is, in different sub-agents you might actually have different brand context that was fetched over time. So just being able to fetch that up front, store it as an artifact, and then redirect the agent to always be referencing that same artifact. And then finally, just still being able to add custom ones. This is more for us internally where some of our solutions architects who work for our customers can spin off different sub-agents when needed, and just really helps with maintaining that context.
I think that's definitely been the biggest learning, is although context windows do continue to grow — you have a million context window with Opus 4.7 — but just still being really cognizant that just because it is larger doesn't mean that you should be using the whole thing. You should still be very efficient about what sorts of context you're letting the model attend to. And in terms of results, what we saw was actually an 8% decrease in fewer tokens consumption around at least that specific tool. So instead of that tool spinning off different primitives and going over and over again, you actually just have this one tool call that fetches a page and then all the results from it. The second one from those specialized tools was also speed. Beforehand, while it was making those 20 different tool calls to fetch all the context it needs, you just have this one entry point that spits out everything that you need. And then when it comes to the quality side, we had 10 enterprise customers while we were running this beta actually start publishing content in under 2 weeks. So they're able to self-serve and get to a quality of output through these agents, which used to be kind of a whole hand-holding experience.
So when it comes to building these agents, and with Claude being able to execute and making it easier to execute on things, it's really easy to think that there are no more problems to solve. But really every single time a problem is solved, that friction point always keeps moving. So those are the two that we were really chasing after. And there's a ton more that we also want to continue chasing, but I think overall it's a great thing to just continue to chase friction because that really is how you create production agents and make those more accessible to users outside of more technical spaces.
Just to quickly close out, the next two friction points that we're looking to battle and excited to show more learnings on are around self-improvement and feedback loops, which there were awesome talks yesterday around dreaming sequences and how — I think the most interesting one is actually how do you structure summaries of different traces? What's the best way of collecting the most relevant memories and also forgetting is actually a feature, being able to forget certain types of memories. And the last one is benchmarking content-creation agents. I think something really interesting about our space is that it isn't law, it isn't coding, it's not something that is really easy to say whether or not something's correct. There is a lot of taste that goes into a piece of content that you create and a lot of opinions of how I want certain content to be formatted and created. So what are the best ways that we are creating benchmarks so that every single time we do change our harness, we add a sub-agent or we decide to add a skill to our harness, how do we know that it's actually improving outputs and not just vibes that we're going after. But yeah, I'm around all day, would love to chat with you guys and hear about what you guys are building. I hope this was helpful. Thank you.