Nathaniel Whittemore (NLW)
Appearances
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Today on the AI Daily Brief, a special interview with me on the future of AI agents. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show notes. Hello, friends. Welcome back to another AI Daily Brief. I am traveling this week, so we're doing a couple episodes that'll be different.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So an insurance company that we work with found that the threshold of tolerance that people had for a human agent being wrong when they were giving them information was like 5% of the time, 7% of the time, something like that. Whereas with a robot basically giving them that information, it was like less than 1%, right? People expect it to be absolutely perfect.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
And in certain cases, if you're in kind of like a... you know, highly regulated industry, the barrier is extra high because if you give bad advice, you know, if you think medicine, insurance, I mean, anything like that, the threshold is just extremely high.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So hallucination is one of those weird things where it's like, it's often just kind of funny and silly when it comes to consumers, but is a major detriment to how far into deployment and production certain use cases can be for big enterprises.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I think that evals are still actually underutilized. I think that a lot of companies, they're just now coming to understand what the full stack of things they have to do to actually implement these tools. And one of the things that's frustrating or can be frustrating is when you realize you have to... you can't just build the thing.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
You have to build this other set of infrastructure to support the thing, to make the thing work. And there's often a resistance to that. I mean, all of the custom build shops that are either building custom agents or helping deploy things, they always complain about how the budgets end when you get to evals and they don't want to put those into practice. So I think that even evals are there.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
To your point, I think that using them is a whole different issue. I think that the way that this will be most addressed in the short term, which is sort of has a, it's kind of good for a number of reasons. I think that you're going to see humans in the loop for much longer than they theoretically need to be to help solve and spot this. And I think that
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Human in the Loop is, in addition to it just being a solve for technical problems of AI, I also think it's a transitional tool to slow down the sort of like rate of full task and work replacement that AI could possibly do, right?
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Join over 9,000 global companies like Atlassian, Quora, and Factory who use Vanta to manage risk and prove security in real time. For a limited time, this audience gets $1,000 off Vanta at vanta.com slash nlw. That's v-a-n-t-a dot com slash nlw for $1,000 off.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
It creates a mechanism for, you know, continued human presence, even on areas that are being, you know, highly automated, which doesn't solve all the issues for, you know, job replacement and things like that.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
But I think that we will over-index on things like that even more than are necessary because society is going to have to find ways to slow down what AI could do from a replacement perspective.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
There's a lot of things. So if you look at what companies view as their biggest challenges with AI right now, kind of broadly speaking, The three that tend to come up are one, like data readiness and whatever complex set of things that that means. You know, like, is their data all, you know, in the same place? Is it ready to be used? I mean, this is a huge industry dealing with just that problem.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
The second is, you know, everything surrounding privacy, cybersecurity, all that sort of set of issues. And then the third is employee adoption and utilization. And usually it's sort of like in that order, roughly speaking. One of the things that you see constantly is Company X will have 10,000 Microsoft Copilot subscriptions, but only 33% of them are being used or something like that.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
And there's just no real support infrastructure. There's no enablement infrastructure surrounding that sort of utilization. And I think that that's just a market gap that is starting to be filled. I mean, this is sort of the space that Super plays in, obviously. But it just needs more people building more things in that space that can support adoption and implementation.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Uh, so, so that, that's, that's a big challenge. I think when it comes to agents, what we're going to see is sort of a lot of, um, we're going to have a misalignment of expectations. I think people are going to try to, uh, to sort of, or they're going to imagine that they can do more than they can to start.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Um, and I think that, you know, uh, they'll try to, you know, create these very complex systems right out of the gate that, that won't quite work. Yeah. There's probably a bunch of, a bunch more that I can think of as well.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I think that I would, at any given time when anyone is listening to this, I think it's really worthwhile to go try to do an audit of what tools are available in that space. Because this is so clearly such a massively juicy problem that, you know, there are startups now. There are going to be more startups in three months. There's going to be many more startups in six months.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
They're all going to be taking different approaches to this or slightly different approaches to this. There's a full... I mean, a full sort of spectrum of options of available support to the extent that people want to solve that now. Like there's companies that will come in and build things completely on premise. I mean, there's just a ton of options.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I think that one thing that's interesting that's happening with this sort of technology shift that maybe didn't in quite the same way in previous eras is companies are building a lot more than just buying off the shelf than they have in the past. So Menlo did a study, their enterprise adoption study, And between 23 and 24, the rate of build versus buy, there was a huge, huge shift.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So in 23, it was something like 80 buy, 20 build. And then last year, it was 53 buy, 47 build. So I mean, huge, huge shift. Now, I think that this boomerang's back. I think that what that reflects is verticalized solutions and verticalized agents not quite being ready for primetime yet.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
And so companies that are in those verticals or that are thinking about those functions, seeing the opportunity, racing to build it, you know, because there's all these frameworks that are available to them. But what I think will happen naturally is that
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
winners will emerge in the category that they had started to build in, and then they'll naturally shift back over to whatever sort of the market leader is. But it does create this whole interesting dynamic.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
And because of that, I think that one of the ways that people are more emboldened to solve some of these issues is if they can't get there with the available kind of security profiles of third-party vendors, there are options more than perhaps in the past that are accessible.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
for, you know, kind of rolling your own solution that, you know, that just has the kind of highest level of security that you can have.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Hey, it's great to be here. Thanks for having me.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I actually have a super strong point of view on this. You'll find this is a common thread for me. So you see a lot of kind of hand-wringing, I think, among people who have been in AI for a long time or who are sort of more technical experts on how mangled the definition of agent is as it's found its way into enterprise and stuff. And I actually think that we should not care about that.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I think that when people on average are talking about agents or referring to agents, they're bucketing AI into two categories. AI that I have to use and AI that does stuff for me, right? Like without me having to really, you know, tell it other than maybe that one first time. And obviously that's, you know, not super precise, but I think broadly it gets people kind of in the way to think about it.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Like particularly if you're, you know, an enterprise leader and you're thinking about whether you're going to deploy, you know, kind of a assistant style AI or agents, like they really kind of broadly bucket into those two categories. I also think that we've so rarely had as much
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
like narrative consolidation around a single term that's like kind of in the ballpark that the fact that everyone kind of knows this term and is there like trying to kind of like, you know, get into the nitty gritty between agent and automation, I just think is ultimately sort of a not particularly relevant pursuit.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I think what people are looking for when they're talking about agents is stuff that actually takes big chunks of work off the table for me, not just makes me do that work better.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I do have my podcast gear, so I will be recording some normal episodes. But for today, I'm sharing the first part of an interview that I did with another podcast, a great one called Tool Use, a couple of weeks ago about AI agents.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So we think a lot.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So the main product right now that Superintelligent is being hit up for is something we call the Agent Readiness Audit, which is basically an agentified process of looking across an organization's workflows and its procedures, its policies, to help them understand what they need to do to be ready to use agents and which agent use cases might be a good fit for them based on current capabilities.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
And I think that what we often end up kind of, you know, what ends up getting shared with them is, you know, we have these grand ideas of these sort of multi-agent workflows that are orchestrated perfectly and take, you know, giant chunks of tasks off. And that's really just not where things are.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Where things are right now is still in this sort of discrete task, you know, repetitive discrete task that you can do, that you have to do over and over and over again. And I think that the more that people and companies experiment, you know, with that in mind, the better suited they're going to be to actually, you know, taking advantage of where agents are now.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I think it's going to change dramatically over the course of this year, right? So really single purpose, very specific agents. I think the way that I think about it, you know, sort of from a personal perspective is, you know, we haven't really agentified a ton of the like podcast processes yet. You know, we use AI for a bunch of them, but they're sort of like not fully automated.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
I think that when it comes to building super intelligent, we're in the midst of going through and sort of totally reevaluating how everything gets done and actually trying to embed agentic workflows in how we work, right? So The way that we build the products is changing based on Cursor and different approaches there.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Obviously, this is the topic du jour, and because I was in the interviewee chair for this one, I got to riff a little bit more broadly around what I think the future of agents actually looks like than I otherwise normally would.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
The knowledge base that powers this agent readiness audit is a workflow that automates a set of different agents or automations or however you want to do it. There's a Zapier piece and a couple other pieces that all add up to automatically extracting information from the web around current agent capabilities that happens every day.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So we're kind of going one by one through all the things that we're doing and just asking which parts of this could be sort of supported by, augmented by, or replaced by an agent and trying to redesign on that basis.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
One of the things that often comes up, and this has been the case for some time and is sort of not agent related, is people underestimate the value of some very basic use cases. So we did a survey of super intelligent users a while back. And the number one use case for AI across this set of enterprise users was brainstorming, right?
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Basically making their work better by having ChatGPT act as like a consultant or a thought partner as they were thinking through things, right? And this is going to evolve over time. I think an interesting analogy is sort of like, imagine the marketing or social media that you guys do for the podcast, right?
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So what I'm going to do is I'm going to share a little more than half of this episode, and then I'll send you out a link to where you can find the rest of it on their feed, the guys over at Tool Use interview builders and entrepreneurs, and other folks who are actually using AI day in and day out about how they're using AI.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
You've probably shifted, I would imagine, from doing it totally raw yourself to now like partnering with ChatGPT on some of the copy and using mid-journey for some of the images. And so it's like now it's sort of this AI-enabled process.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
It's an AI-assisted process where maybe the time has gone down, but I bet the benefit is more just sort of like quality increase and cognitive load decrease for you guys.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
However, I would imagine that over the course of the next year or so, we'll all be able to actually, I think that, you know, a social media agent seems like one of the easiest to actually execute against, you know, how many tweets per day do you want? What do you want them to relate to? How much are they replies versus this?
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
You know, like you could kind of see how it comes together pretty quickly. What's the database that I have to pull from a previous messages. And, and, and so you're kind of going to see that, that process. And so if people are just getting started,
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Just using the assisted level AI to see how it makes their work better before they worry too much about even time-saving necessarily, I think is often a really good starting point.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Yeah.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So the thing that's closest right now in the super intelligent world is the automation of this knowledge base around current agent capabilities. So when we're trying to match this agent readiness audit, basically the way that it works is a company will come to us and we'll talk to them. And then we deploy a voice agent that does this interview.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Where, you know, we've customized the set of questions and we can do this, you know, a very small handful of times and just get a very kind of high level overview. Or we can deploy it all the way down to the employee level across hundreds or even thousands of people. Right. You get all of this information. And then based on the interviews with all those people.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So if that's interesting to you, I highly encourage you to go check out their show. So again, today's episode is an interview with me about the future of AI agents. Today's episode is brought to you by Vanta. That's where Vanta comes in. Businesses use Vanta to establish trust by automating compliance needs across over 35 frameworks like SOC 2 and ISO 27001.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
We're running that through this knowledge base of agentic opportunities that includes what the agent is, what it does, what use cases are related to it, what industries are related to it, what compliance regimes that it fits with, what tech stacks it matters to. So it's not just like the two or three kind of vectors that you might imagine. It's like
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
a database of like, I don't know, 20 or 50 rows or something like that around all this information that we're trying to gather. And we've really like hyper automated the process of ingesting that information. Now, we still have an additional layer of human interaction. So for example, like a lot of the subjective information that I get as relates to agents comes from Twitter slash X, right?
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Like people saying, oh, this sucks. Oh, this is great. And that's actually quite useful to try to benchmark like where a thing is when you're trying to give a company an expectation of whether they should be using it or not, you know, like If subjectively like half the sentiment on Twitter is that it's great and half the sentiment is bad, you can go in with the appropriate expectations.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
You know, we're not sure how ready for primetime it is. It might be like slightly overpromising or whatever. So it's not fully automated in the sense that there's things that are still much more valuable for kind of like the human to do. But it's getting there. Right. And I think that that's that's that's an important piece.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
When it comes to the podcast, there's nothing that's fully automated, although I did an experiment last week when I got sick of a much more automated process. So basically, I had pretty much lost my voice. And so I took a topic, used deep research to write a paper on it.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Um, I think the one that I ended up using, I tried a couple and the one I ended up using was, uh, what, what economic predictions in the era of AGI, basically how AGI is going to impact the economic landscape. Um, wrote a research paper on it with deep research and then fed that into Google's notebook LM and let them turn it into a podcast.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
And that's what I published that day, um, as an experiment, uh, It went over reasonably. It was sort of a cute idea. So I don't think I'm going to be turning back to that too frequently. When I think about where automations might come in the future for the AI Daily Brief, I think that there's...
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
There's probably not for my show because so much of it is like the context that I add implicitly around things. But, you know, news podcasts are going to be very easy to go from, you know, the automated feed that curates them and just turns it, you know, end-to-end pipeline into a podcast that gets pushed out, you know.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
It's a very, very sort of simple set of steps that, you know, each requires their own automation, but you could do it really effectively. Yeah.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Yeah. Well, I think it's part of the future, for sure. I think that we're going to have to do a lot of experimenting and iterating to figure out exactly how these things work. My sense is that most of the people who have had positive experiences with deep research have used it for...
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
uh for particular types of knowledge uh you know summarization that it was well suited to do and the people who've had bad experiences have started to figure out the jagged edges of where it's not so good right so it's very clear that like not having access to contemporary journals is a huge problem right it really like limits its ability to uh be super deep and contemporary when it comes to science or anything anything that requires access to you know journals that are behind paywalls
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
The other thing that I found is that when it comes to really like fast moving spaces, there are, it can be a challenge. So for example, this AGI thing that I did, it was mostly great, however, it was definitely like over-reliant on Nick Bostrom's super intelligence, you know, as a resource.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
And I think at one point it said that most scientists still think that AGI is a decade or two away, which is obviously like so not the, you know, it's not reading Twitter, let's put it that way. So I think that there's, you know, we're just going to figure out like,
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Basically, research or grabbing a bunch of sources and turning them into a consolidated bucket of knowledge is actually a very diverse use case. It's not one use case. It's about a thousand use cases embedded in one category of use case that it's going to take some time for us to figure out what pieces of it this particular tool is actually good at.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Yeah, it's actually a much bigger deal for businesses than it is for consumers, I think. Consumers have a higher kind of threshold for what they can deal with, especially if it's, you know, so much of the deep research use case isn't it isn't trying to get something that's production ready. It's trying to get a kind of a thing that gets to 80 or 90%, right?
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Centralized security workflows complete questionnaires up to 5x faster and proactively manage vendor risk. Vanta can help you start or scale up your security program by connecting you with auditors and experts to conduct your audit and set up your security program quickly. Plus, with automation and AI throughout the platform, Vanta gives you time back so you can focus on building your company.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
So one of the use cases that I've seen a number of people have success with is basically background market descriptions and sizing for their startups, right? So they're trying to communicate and understand like how big the total addressable market for the thing that they're building is. And, you know, it's really good at pulling a bunch of different resources in, blah, blah, blah, blah, blah.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
But they're never going to just turn that over to an investor. At least if they're actually a good entrepreneur, they're not going to just turn it over to an investor. But it saves them a huge amount of time. Like I said, it gets them kind of 80% of the way there. And so for them, they're maybe more in a position to actually spot those hallucinations.
The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis
NLW on the Future of AI Agents
Where hallucinations become a real problem is when people are actually, you know, basically replacing a human information source with an AI or an agent information source that really relies on having the right information.