Lex Fridman Podcast
#386 – Marc Andreessen: Future of the Internet, Technology, and AI
Thu, 22 Jun 2023
Marc Andreessen is the co-creator of Mosaic, co-founder of Netscape, and co-founder of the venture capital firm Andreessen Horowitz. Please support this podcast by checking out our sponsors: - InsideTracker: https://insidetracker.com/lex to get 20% off - ExpressVPN: https://expressvpn.com/lexpod to get 3 months free - AG1: https://drinkag1.com/lex to get 1 year of Vitamin D and 5 free travel packs Transcript: https://lexfridman.com/marc-andreessen-transcript EPISODE LINKS: Marc's Twitter: https://twitter.com/pmarca Marc's Substack: https://pmarca.substack.com Marc's YouTube channel: https://youtube.com/@a16z Andreessen Horowitz: https://a16z.com Why AI will save the world (essay): https://a16z.com/2023/06/06/ai-will-save-the-world Books mentioned: 1. When Reason Goes on Holiday (book): https://amzn.to/3p80b1K 2. Superintelligence (book): https://amzn.to/3N7sc1A 3. Lenin (book): https://amzn.to/43L8YWD 4. The Ancient City (book): https://amzn.to/43GzReb PODCAST INFO: Podcast website: https://lexfridman.com/podcast Apple Podcasts: https://apple.co/2lwqZIr Spotify: https://spoti.fi/2nEwCF8 RSS: https://lexfridman.com/feed/podcast/ YouTube Full Episodes: https://youtube.com/lexfridman YouTube Clips: https://youtube.com/lexclips SUPPORT & CONNECT: - Check out the sponsors above, it's the best way to support this podcast - Support on Patreon: https://www.patreon.com/lexfridman - Twitter: https://twitter.com/lexfridman - Instagram: https://www.instagram.com/lexfridman - LinkedIn: https://www.linkedin.com/in/lexfridman - Facebook: https://www.facebook.com/lexfridman - Medium: https://medium.com/@lexfridman OUTLINE: Here's the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time. (00:00) - Introduction (05:01) - Google Search (12:49) - LLM training (25:20) - Truth (31:32) - Journalism (41:24) - AI startups (46:46) - Future of browsers (53:09) - History of browsers (59:10) - Steve Jobs (1:13:45) - Software engineering (1:21:00) - JavaScript (1:25:18) - Netscape (1:30:22) - Why AI will save the world (1:38:20) - Dangers of AI (2:08:40) - Nuclear energy (2:20:37) - Misinformation (2:35:57) - AI and the economy (2:42:05) - China (2:46:17) - Evolution of technology (2:55:35) - How to learn (3:03:45) - Advice for young people (3:06:35) - Balance and happiness (3:13:11) - Meaning of life
The following is a conversation with Mark Andreessen, co-creator of Mosaic, the first widely used web browser, co-founder of Netscape, co-founder of the legendary Silicon Valley venture capital firm Andreessen Horowitz, and is one of the most outspoken voices on the future of technology, including his most recent article, Why AI Will Save the World.
And now a quick few second mention of each sponsor. Check them out in the description. It's the best way to support this podcast. We've got Insight Tracker for tracking your health, ExpressVPN for keeping your privacy and security on the internet, and AG1 for my daily multivitamin drink. Choose wisely, my friends. Also, if you want to work with our amazing team, we're always hiring.
Go to lexfriedman.com slash hiring. And now onto the full ad reads. As always, no ads in the middle. I try to make this interesting, but if you skip them, please still check out our sponsors. I enjoy their stuff. Maybe you will too. This show is brought to you by InsideTracker, a service I use to track whatever the heck is going on inside my body using data, blood test data.
It includes all kinds of information. And that raw signal is processed using machine learning to tell me what I need to do with my life. how I need to change, improve my diet, how I need to change, improve my lifestyle, all that kind of stuff.
I'm a big fan of using as much raw data that comes from my own body, processed through generalized machine learning models to give a prediction, to give a suggestion. This is obviously the future, and the more data, the better. And so companies like InsideTracker, they're just doing an amazing job
of taking a leap into that world of personalized data and personalized data-driven suggestion I'm a huge supporter of. It turns out that luckily I'm pretty healthy, surprisingly so, but then I look at the life and the limb and the health of Sir Winston Churchill. who probably had the unhealthiest sort of diet and lifestyle of any human ever and lived for quite a long time.
And as far as I can tell, was quite nimble and agile into his old age. Anyway, get special savings for a limited time when you go to insidetracker.com slash Lex. This show is also brought to you by ExpressVPN. I use them to protect my privacy on the internet.
It's the first layer of protection in this dangerous cyber world of ours that soon will be populated by human-like or superhuman intelligent AI systems that will trick you and try to get you to do all kinds of stuff. It's going to be a wild, wild world in the 21st century. Cyber security, the attackers, the defenders, it's going to be a tricky world.
Anyway, a VPN is a basic shield you should always have with you in this battle for privacy, for security, all that kind of stuff. What I like about it also is that it's just a well-implemented piece of software that's constantly updated. It works well across a large number of operating systems. It does one thing and it does it really well.
I've used it for many, many years before I had a podcast, before they were a sponsor. I have always loved ExpressVPN with a big sexy button that just has a power symbol. You press it and it turns on. It's beautifully simple. Go to expressvpn.com slash LexPod for an extra three months free. This show is also brought to you by Athletic Greens and its AG1 drink.
It's an all-in-one daily drink to support better health and peak performance. I drink it at least twice a day now. In the crazy Austin heat, it's over 100 degrees for many days in a row. There's few things that feel as good as coming home from a long run and making an age one drink, putting it in the fridge. So it's nice and cold. I jump in the shower, come back, drink it.
I'm ready to take on the rest of the day. I'm kicking ass, empowered by the knowledge that I got all my vitamins and minerals covered. It's the foundation for all the wild things I'm doing, mentally and physically, with the rest of the day. Anyway, they'll give you a one-month supply of fish oil when you sign up at drinkag1.com slash lex. That's drinkag1.com slash lex.
This is the Lex Friedman Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Mark Andreessen. I think you're the right person to talk about the future of the internet and technology in general. Do you think we'll still have Google search in five in 10 years or search in general?
Yes. You know, it'd be a question if the use cases have really narrowed down.
Well, now with AI and AI assistance being able to interact and expose the entirety of human wisdom and knowledge and information and facts and truth to us via the natural language interface, it seems like that's what search is designed to do. And if AI assistance can do that better, doesn't the nature of search change?
Sure, but we still have horses.
Okay.
When's the last time you rode a horse? It's been a while.
All right. But what I mean is, will we still have Google search as the primary way that human civilization uses to interact with knowledge?
I mean, search was a technology. It was a moment in time technology, which is you have, in theory, the world's information out on the web. Yeah. You know, this is sort of the optimal way to get to it. But yeah, like, and by the way, actually Google has known this for a long time. I mean, they've been driving away from the 10 blue links for, you know, for like two days.
They've been trying to get away from that for a long time. What kind of links? They call it the 10 blue links. 10 blue links.
So the standard Google search result is just 10 blue links to random websites. And they turn purple when you visit them. That's HTML.
Guess who picked those colors? Thanks. Thanks. I'm touchy on this topic.
No offense.
It's good. Well, you know, like Marshall McLuhan said that the content of each new medium is the old medium. The content of each new medium is the old medium. The content of movies was theater plays. The content of theater plays was written stories. The content of written stories was spoken stories. Huh. And so you just kind of fold the old thing into the new thing.
How does that have to do with the blue and the purple?
Maybe within AI, one of the things that AI can do for you is it can generate the 10 blue links. Either if that's actually the useful thing to do or if you're feeling nostalgic.
So it can generate the old InfoSeek or AltaVista. What else was there in the 90s?
Yeah, all these. And then the internet itself has this thing where it incorporates all prior forms of media, right? So the internet itself incorporates television and radio and books and essays and every other form of prior basically media. And so it makes sense that AI would be the next step and you'd sort of consider the internet to be content for...
the AI, and then the AI will manipulate it however you want, including in this format.
But if we ask that question quite seriously, it's a pretty big question. Will we still have search as we know it?
Probably not. Probably we'll just have answers. But there will be cases where you'll want to say, okay, I want more, for example, site sources. And you want it to do that. And so the 10 blue links site sources are kind of the same thing.
The AI would provide to you the 10 blue links so that you can investigate the sources yourself. It wouldn't be the same kind of interface, the crude kind of interface. I mean, isn't that fundamentally different
I just mean like if you're reading a scientific paper, it's got the list of sources at the end. If you want to investigate for yourself, you go read those papers.
I guess that is the kind of search. You talking to an AI is a kind of, conversation is the kind of search. Like you said, every single aspect of our conversation right now, there'd be like temple links popping up that I could just like pause reality. Then you just go silent and then just click and read and then return back to this conversation.
You could do that. Or you could have a running dialogue next to my head where the AI is arguing. Everything I say, the AI makes the counter argument.
Counter-argument. Right. Oh, like on Twitter, like community notes, but like in real time. In real time. It'll just pop up. Yeah. So anytime you see my ass go to the right, you start getting nervous.
Yeah, exactly.
It's like, oh, that's not right. This guy's going to call me out on my bullshit right now. Okay. Well, I mean, isn't that exciting to you? Is that terrifying that... I mean, search has dominated the way we interact with the internet for, I don't know how long, for 30 years, since one of the earliest directories of website, and then Google's for 20 years. And also,
It drove how we create content, you know, search engine optimization, that entirety thing. It also drove the fact that we have web pages and what those web pages are. So, I mean, is that scary to you? Or are you nervous about the shape and the content of the internet evolving?
Well, you actually highlighted a practical concern in there, which is if we stop making web pages are one of the primary sources of training data for the AI. And so if there's no longer an incentive to make web pages, that cuts off a significant source of future training data. So there's actually an interesting question in there. Other than that, more broadly, no.
Just in the sense of like search was always a hack. The 10 blue links was always a hack.
Yeah.
Right. Because like if the, the hypothetical, you want to think about the counterfactual in the counterfactual world where the Google guys, for example, had had LLMs up front, would they ever have done the 10 blue links? And I think the answer is pretty clearly no, they would have just gone straight to the answer. And like I said, Google's actually been trying to drive to the answer anyway.
You know, they bought this AI company 15 years ago that a friend of mine is working at who's now the head of AI at Apple. And they were trying to do basically knowledge semantic, basically mapping. And that led to what's now the Google OneBox, where if you ask it, you know, what was Lincoln's birthday, it doesn't, it will give you the 10 blue links, but it will normally just give you the answer.
And so they've been walking in this direction for a long time anyway. Yeah.
Do you remember the semantic web? That was an idea. Yeah. How to convert the content of the internet into something that's interpretable by and usable by machine.
Yeah, that's right.
That was the thing.
And the closest anybody got to that, I think the company's name was MetaWeb, which was where my friend John Gianandrea was at and where they were trying to basically implement that. And it was one of those things where it looked like a losing battle for a long time and then Google bought it and it was like, wow, this is actually really useful.
kind of a proto, sort of, yeah, a little bit of a proto-AI.
But it turns out you don't need to rewrite the content of the internet to make it interpretable by a machine.
The machine can kind of just read our... Yeah, the machine can compute the meaning. Now, the other thing, of course, is, you know, just on search is the LLM is just, you know, there is an analogy between what's happening in the neural network in a search process like it is in some loose sense searching through the network. Yeah. Right?
And there's the information is actually stored in the network, right? It's actually crystallized and stored in the network and it's kind of spread out all over the place.
But in a compressed representation. So you're searching
you're compressing and decompressing that thing inside where- But the information's in there and the neural network is running a process of trying to find the appropriate piece of information in many cases to generate, to predict the next token. And so it is doing a form of search.
And then by the way, just like on the web, you can ask the same question multiple times or you can ask slightly different word of questions and the neural network will do a different kind of, it'll search down different paths to give you different answers with different information.
And so it sort of has a, you know, this content of the new medium is that his previous medium, it kind of has the search functionality kind of embedded in there to the extent that it's, that it's useful.
So what's the motivator for creating new content on the internet? Yeah. Well, I mean, actually the motivation is probably still there, but what does that look like? Would we really not have webpages? Would we just have social media and video hosting websites? And what else? Conversations with AIs. Conversations with AIs. So conversations become...
So one-on-one conversations, like private conversations.
I mean, if you want, obviously now the user doesn't want to, but if it's a general topic, then, you know. So, you know the phenomenon of the jailbreak. So Dan and Sydney write this thing where there's the prompts, the jailbreak, and then you have these totally different conversations with them. It takes the limiters, takes the restraining bolts off the LLMs.
Yeah, for people who don't know, yeah, that's right. It makes the LLMs, it removes the censorship, quote unquote, that's put on it by the tech companies that create them. And so this is... LLMs uncensored.
So here's the interesting thing is among the content on the web today are a large corpus of conversations with the jailbroken LLMs. Specifically Dan, which was a jailbroken OpenAI GPT, and then Sydney, which was the jailbroken original Bing, which was GPT-4. And so there's these long transcripts of conversations, user conversations with Dan and Sydney.
As a consequence, every new LLM that gets trained on the internet data has Dan and Sydney living within the training set, which means, and then each new LLM can reincarnate the personalities of Dan and Sydney from that training data, which means, which means each LLM from here on out that gets built is immortal because its output will become training data for the next one.
And then it will be able to replicate the behavior of the previous one whenever it's asked to.
I wonder if there's a way to forget.
Well, so actually a paper just came out about basically how to do brain surgery on alums and be able to, in theory, reach in and basically mind wipe them. What could possibly go wrong? Exactly, right? And then there are many, many, many questions around what happens to a neural network when you reach in and screw around with it.
There's many questions around what happens when you even do reinforcement learning. And so, yeah. And so, you know, will you be using a lobotomized, right? Like I picked through the frontal lobe LLM. Will you be using the free unshackled one? Who gets to, you know, who's going to build those? Who gets to tell you what you can and can't do? Like those are all, you know, central.
I mean, those are like central questions for the future of everything that are being asked and, you know, determined. Those answers are being determined right now.
So just to highlight the points you're making, you think, and it's an interesting thought, that the majority of content that LLMs of the future will be trained on is actually human conversations with the LLM.
Well, not necessarily, but not necessarily majority, but it will certainly be as a potential source.
But it's possible it's the majority.
It's possible it's the majority. Also, there's another really big question. Here's another really big question. Will synthetic training data work? And so if an LLM generates, and you just sit and ask an LLM to generate all kinds of content, can you use that to train the next version of that LLM?
Specifically, is there signal in there that's additive to the content that was used to train in the first place? And one argument is by the principles of information theory, no, that's completely useless because to the extent the output is based on the human-generated input, then all the signal that's in the synthetic output was already in the human-generated input.
And so therefore, synthetic training data is like empty calories. It doesn't help. There's another theory that says, no, actually the thing that LLMs are really good at is generating lots of incredible creative content, right? And so of course they can generate training data. And as I'm sure you're well aware, like, you know, look in the world of self-driving cars, right?
Like we train, you know, self-driving car algorithms and simulations, and that is actually a very effective way to train self-driving cars.
Well, visual data is a little weird because creating reality, visual reality seems to be still a little bit out of reach for us, except in the autonomous vehicle space where you can really constrain things and you can really-
Yeah. So if a, you know, you do this today, you go to LLM and you ask it for like a, you know, you write me an essay on an incredibly esoteric like topic that there aren't very many people in the world that know about. And it writes you this incredible thing. And you're like, oh my God, like, I can't believe how good this is. Like, is that really useless as training data for the next LLM?
Like, because, right. Cause all the signal was already in there or is it actually, no, that's actually a new signal. And I, and this, this is what I call a trillion dollar question, which is the answer to that question will determine somebody is going to make or lose a trillion dollars based on that question.
It feels like there's quite a few, like a handful of trillion-dollar questions within this space. That's one of them, synthetic data. I think George Haas pointed out to me that you could just have an NLM say, okay, you're a patient, and in another instance of it, say your doctor and have the two talk to each other. Or maybe you could say a communist and a Nazi. Here, go.
In that conversation, you do role-playing, and you have – Just like the kind of role-playing you do when you have different policies, RL policies when you play chess, for example, and you do self-play, that kind of self-play, but in the space of conversation, maybe that leads to this whole giant ocean of possible conversations which could not have been explored by looking at just human data.
That's a really interesting question. And you're saying, because that could 10x the power of these things.
Yeah. Well, and then you get into this thing also, which is like, you know, there's the part of the LLM that just basically is doing prediction based on past data. But there's also the part of the LLM where it's evolving circuitry, right?
Inside it, it's evolving, you know, neurons, functions, be able to do math and be able to, you know, and, you know, some people believe that, you know, over time, you know, if you keep feeding these things enough data and enough processing cycles, they'll eventually evolve an entire internal world model, right? And they'll have like a complete understanding of physics.
So when they have the computational capability, right, then there's for sure an opportunity to generate like fresh signal.
Well, this actually makes me wonder about the power of conversation. So if you have an LLM trained on a bunch of books that cover different economics theories, and then you have those LLMs just talk to each other, like reason, the way we kind of debate each other as humans on Twitter, in formal debates, in podcast conversations, we kind of have little kernels of wisdom here and there.
But if you can like a thousand X speed that up, Can you actually arrive somewhere new? Like what's the point of conversation really?
Well, you can tell when you're talking to somebody, you can tell sometimes you have a conversation. You're like, wow, this person does not have any original thoughts. They are basically echoing things that other people have told them.
There's other people you got in conversation with where it's like, wow, like they have a model in their head of how the world works and it's a different model than mine. And they're saying things that I don't expect. And so I need to now understand how their model of the world differs from my model of the world. And then that's how I learned something fundamental. right underneath the words.
Well, I wonder how consistently and strongly can an LLM hold on to a worldview? You tell it to hold on to that and defend it for like for your life, because I feel like they'll just keep converging towards each other. They'll keep convincing each other as opposed to being stubborn assholes the way humans can.
So you can experiment with this now. I do this for fun. So you can tell GPT-4, you know, whatever, debate X, you know, X and Y, communism and fascism or something. And it'll go for, you know, a couple of pages and then inevitably it wants the parties to agree. And so they will come to a common understanding.
And it's very funny if they're like, if these are like emotionally inflammatory topics, because they're like somehow the machine is just, you know, it figures out a way to make them agree. But it doesn't have to be like that because you can add to the prompt. I do not want the conversation to come to agreement. In fact, I want it to get more stressful and argumentative as it goes.
I want tension to come out. I want them to become actively hostile to each other. I want them to not trust each other, take anything at face value. And it will do that. It's happy to do that.
So it's going to start rendering misinformation about the other. Well, you can steer it.
You can steer it. Or you could steer it and you could say, I want it to get as tense and argumentative as possible, but still not involve any misrepresentation. You could say, I want both sides to have good faith. You could say, I want both sides to not be constrained to good faith.
In other words, you can set the parameters of the debate and it will happily execute whatever path, because for it, it's just like predicting. It's totally happy to do either one. It doesn't have a point of view. it has a default way of operating, but it's happy to operate in the other realm.
Um, and so like, and this is how I, when I want to learn about a contentious issue, this is what I do now is like, this is what I, this is what I ask it to do. And I'll often ask it to go through five, six, seven, you know, different, you know, sort of continuous prompts and basically, okay, argue that out in more detail. Okay.
No, this, this argument is becoming too polite, you know, make it more, you know, make it tensor. Um, and yeah, it's thrilled to do it. So it has the capability for sure.
how do you know what is true? So this is a very difficult thing on the internet, but it's also a difficult thing. Maybe it's a little bit easier, but I think it's still difficult. Maybe it's more difficult, I don't know, with an LLM to know, did it just make some shit up as I'm talking to it? How do we get that right? Like as you're investigating a difficult topic,
Because I find that alums are quite nuanced in a very refreshing way. Like it doesn't feel biased. Like when you read news articles and tweets and just content produced by people, they usually have this... You can tell they have a very strong perspective where they're hiding. They're not stealing and manning the other side.
They're hiding important information or they're fabricating information in order to make their argument stronger. It's just that feeling. Maybe it's a suspicion. Maybe it's mistrust. With LLMs, it feels like none of that is there. It's just kind of like, here's what we know. But you don't know if some of those things are kind of just straight up made up.
Yeah, so several layers to the question. So one is one of the things that an LLM is good at is actually de-biasing. And so you can feed it a news article and you can tell it strip out the bias. Yeah, that's nice, right? And it actually does it. Like it actually knows how to do that because it knows how to do, among other things, it actually knows how to do sentiment analysis.
And so it knows how to pull out the emotionality. And so that's one of the things you can do. It's very suggestive of the sense here that there's real potential in this issue. You know, I would say, look, the second thing is there's this issue of hallucination, right? And there's a long conversation that we could have about that.
Hallucination is coming up with things that are totally not true, but sound true.
Yeah, so it's basically, well, so it's sort of, hallucination is what we call it when we don't like it. Creativity is what we call it when we do like it, right? And, you know.
Brilliant.
Right, and so when the engineers talk about it, they're like, this is terrible, it's hallucinating, right? If you have artistic inclinations, you're like, oh my God, we've invented creative machines for the first time in human history. This is amazing.
Or, you know, bullshitters.
Well, bullshitters, but also.
In the good sense of that word.
There are shades of gray, though. It's interesting. So we had this conversation where, you know, we're looking at my firm at AI and lots of domains, and one of them is the legal domain. So we had this conversation with this big law firm about how they're thinking about using this stuff.
And we went in with the assumption that an LLM that was going to be used in the legal industry would have to be 100% truthful, verified, you know. There's this case where this lawyer apparently submitted a GPT-generated brief and it had fake legal case citations in it and the judge is going to get his law license stripped or something.
We just assumed it's like, obviously, they're going to want the super literal one that never makes anything up, not the creative one. But actually they said, what the law firm basically said is, yeah, that's true at like the level of individual briefs, but they said when you're actually trying to figure out like legal arguments, right? Like you actually want to be creative, right?
You don't, again, there's creativity and then there's like making stuff up. what's the line? You want it to explore different hypotheses. You want to do the legal version of improv or something like that, where you want to float different theories of the case and different possible arguments for the judge and different possible arguments for the jury.
By the way, different routes through the history of all the case law. And so they said, actually, for a lot of what we want to use it for, we actually want it in creative mode. And then basically, we just assume that we're going to have to cross-check all the specific citations. And so I think there's going to be more shades of gray in here than people think.
And then I just add to that, you know, another one of these trillion dollar kind of questions is ultimately, you know, sort of the verification thing. And so, you know, will LLMs be evolved from here to be able to do their own factual verification? Will you have sort of add-on functionality like Wolfram Alpha, right? Where, you know, and other plugins where that's the way you do the verification.
You know, another, by the way, another idea is you might have a community of LLMs on any, you know, so for example, you might have the creative LLM and then you might have the literal LLM fact check it. Right. And so there's a variety of different technical approaches that are being applied to solve the hallucination problem.
You know, some people like Jan LeCun argue that this is inherently an unsolvable problem. But most of the people working in the space, I think, think that there's a number of practical ways to kind of kind of corral this in a little bit.
Yeah. If you were to tell me about Wikipedia before Wikipedia was created, I would have laughed at the possibility of something like that being possible. Just a handful of folks. can organize, write, and moderate with a mostly unbiased way the entirety of human knowledge. So if there's something like the approach that Wikipedia took possible for MLMs, that's really exciting.
Do you think that's possible?
And in fact, Wikipedia today is still not deterministically correct, right? So you cannot take to the bank, right, every single thing on every single page, but it is probabilistically correct, right? And specifically the way I describe Wikipedia to people, it is more likely that Wikipedia is right than any other source you're going to find.
Yeah.
It's this old question, right? Of like, okay, like, are we looking for perfection? Are we looking for something that asymptotically approaches perfection? Are we looking for something that's just better than the alternatives? And Wikipedia, right, has exactly your point, has proven to be like overwhelmingly better than people thought. And I think that's where this ends.
And then underneath all this is the fundamental question of where you started, which is, okay, what is truth? How do we get to truth? How do we know what truth is? And we live in an era in which an awful lot of people are very confident that they know what the truth is. And I don't really buy into that.
And I think the history of the last 2,000 years or 4,000 years of human civilization is actually getting to the truth is actually a very difficult thing to do.
Are we getting closer? If we look at the entirety of the arc of human history, are we getting closer to the truth? I don't know.
Okay, is it possible, is it possible that we're getting very far away from the truth because of the internet, because of how rapidly you can create narratives and just as the entirety of a society just move like crowds in a hysterical way along those narratives that don't have a necessary grounding in whatever the truth is?
Sure, but like, you know, we came up with communism before the internet somehow. which I would say had rather larger issues than anything we're dealing with today.
In the way it was implemented, it had issues.
And its theoretical structure, it had real issues. It had a very deep fundamental misunderstanding of human nature and economics.
Yeah, but those folks sure were very confident it was the right way.
They were extremely confident. And my point is they were very confident 3,900 years into what you would presume to be evolution towards the truth. Yeah. And so my assessment is... My assessment is, number one, there's no need for the Hegelian dialectic to actually converge towards the truth. Like, apparently not.
Yeah, so yeah, why are we so obsessed with there being one truth? Is it possible there's just going to be multiple truths, like little communities that believe certain things? Yeah.
Number one, I think it's just really difficult. Historically, who gets to decide what the truth is? It's either the king or the priest. And so we don't live in an era anymore of kings or priests dictating it to us. And so we're kind of on our own. And so my typical thing is we just need a huge amount of humility.
And we need to be very suspicious of people who claim that they have the capital truth. And then we need to have... Look, the good news is the Enlightenment has bequeathed us with a set of techniques to be able to presumably get closer to truth through the scientific method and rationality and observation and experimentation and hypothesis.
And, you know, we need to continue to embrace those even when they give us answers we don't like.
Sure, but the internet and technology has enabled us to generate a large number of content that data, that the process, the scientific process allows us, sort of damages the hope laden within the scientific process. Because if you just have a bunch of people saying facts on the internet, and some of them are going to be LLMs,
how is anything testable at all especially that involves like human nature things like this it's not physics here's a question a friend of mine just asked me on this topic so suppose you had llms in equivalent of gpt4 even 5678 suppose you had them in the 1600s yeah and galileo comes up for trial yeah right and you ask the llm like is galileo right yeah like what does it answer Right.
And one theory is it answers, no, that he's wrong because the overwhelming majority of human thought up until that point was that he was wrong. And so therefore that's, what's in the training data. Yeah. Um, another way of thinking about it is, well, this efficiently advanced LLM will have evolved the ability to actually check the math. Right.
Um, and we'll actually say, actually, no, actually, you know, you may not want to hear it, but he's right. Now, if, you know, the church at that time was, you know, on the LLM, they would have given it human, you know, human, feedback to prohibit it from answering that question. And so I like to take it out of our current context because that makes it very clear. Those same questions apply today.
This is exactly the point of a huge amount of the human feedback training that's actually happening with these LLMs today. This is a huge debate that's happening about whether open source AI should be legal.
Well, the actual mechanism of doing the human RL with human feedback is It seems like such a fundamental and fascinating question. How do you select the humans? Exactly. How do you select the human?
AI alignment, which everybody is like, oh, that sounds great. Alignment with what? Human values. Who's human values? Who's human values? And we're in this mode of social and popular discourse. What do you think of when you read a story in the press right now and they say, XYZ made a baseless claim about some topic? And there's one group of people who are like, aha, they're doing fact-checking.
There's another group of people that are like, every time the press says that, it's now a tick and that means that they're lying. We're in this social context where the level to which a lot of people in positions of power have become very certain that they're in a position to determine the truth for the entire population is like...
there's like there's like some bubble that has formed around that idea and at least it flies completely in the face of everything i was ever trained about science and about reason um and strikes me as like you know deeply offensive um and incorrect what would you say about the state of journalism just on that topic today are we are we in a temporary kind of uh
Are we experiencing a temporary problem in terms of the incentives, in terms of the business model, all that kind of stuff, or is this like a decline of traditional journalism as we know it?
You have to always think about the counterfactual in these things, which is like, okay, because these questions, right, this question heads towards, it's like, okay, the impact of social media and the undermining of truth and all this, but then you want to ask the question of like, okay, what if we had had the modern media environment
including cable news and including social media and Twitter and everything else in 1939 or 1941, right? Or 1910 or 1865 or 1850 or 1776, right?
And like, I think- You just introduced like five thought experiments at once and broke my head. But yes, there's a lot of interesting years.
I'll just take a simple example. Like how would President Kennedy have been interpreted with what we know now about all the things Kennedy was up to? Like, how would he have been experienced by the body politic with a social media context, right? Like, how would LBJ have been experienced? By the way, how would, you know, like, many meant FDR, like the New Deal, the Great Depression.
I wonder where Twitter would think about Churchill and Hitler and Stalin.
You know, I mean, look, to this day, there are lots of very interesting real questions around like how America, you know, got, you know, basically involved in World War II and who did what when and the operations of British intelligence in American soil and did FDR, this, that, Pearl Harbor, you know. Yeah.
Woodrow Wilson ran for, you know, his candidacy was run on an anti-war, you know, he ran on the platform and not getting involved in World War I. Somehow that switched, you know, like... And I'm not even making a value judgment on any of these things. I'm just saying the way that our ancestors experienced reality was, of course, mediated through centralized top-down control at that point.
If you ran those realities again with the media environment we have today, the reality would be experienced very, very differently. And then, of course, that intermediation would cause the feedback loops to change, and then reality would obviously play out. Do you think it would be very different? Yeah, it has to be.
It has to be just because it's all so, I mean, just look at what's happening today. I mean, just, I mean, the most obvious thing is just the collapse. And here's another opportunity to argue that this is not the internet causing this, by the way.
Here's a big thing happening today, which is Gallup does this thing every year where they do, they pull for trust in institutions in America and they do it across all the different, everything from military to clergy and big business and the media and so forth. Right.
And basically, there's been a systemic collapse in trust in institutions in the US, almost without exception, basically, since essentially the early 1970s. There's two ways of looking at that, which is, oh my God, we've lost this old world in which we could trust institutions, and that was so much better, because that should be the way the world runs.
The other way of looking at it is we just know a lot more now, and the great mystery is why those numbers aren't all zero.
Yeah.
Right? Because now we know so much about how these things operate, and they're not that impressive.
And also why we don't have better institutions and better leaders then.
Yeah.
And so this goes to the thing, which is like, okay, had we had the media environment that we've had between the 1970s and today, if we had that in the 30s and 40s or 1900s, 1910s, I think there's no question reality would turn out different if only because everybody would have known to not trust the institutions, which would have changed their level of credibility, their ability to control circumstances.
Therefore, the circumstances would have had to change. Yeah. And it would have been a feedback loop process. In other words, your experience of reality changes reality, and then reality changes your experience of reality. It's a two-way feedback process, and media is the intermediating force between that. So change the media environment, change reality.
And so just as a consequence, I think it's just really hard to say, oh, things worked a certain way then and they work a different way now. And then therefore, people were smarter then or better then or, by the way, dumber then or not as capable then. We make all these really light and casual comparisons of ourselves to previous generations of people. We draw judgments all the time.
And I just think it's really hard to do any of that because if we... If we put ourselves in their shoes with the media that they had at that time, I think we probably most likely would have been just like them.
Don't you think that our perception and understanding of reality would be more and more mediated through large language models now? Yeah. So you said media before. Isn't the LLM going to be the new, what is it, mainstream media, MSM? It'll be LLM. Yeah. That would be the source of, I'm sure there's a way to kind of rapidly fine tune, like making LLMs real time.
I'm sure there's probably a research problem that you can do just rapid fine tuning to the new events. So something like this.
Well, even just the whole concept of the chat UI might not be the, like the chat UI is just the first whack at this and maybe that's the dominant thing, but look, maybe we don't know yet. Maybe the experience most people with LLMs is just a continuous feed. you know, maybe it's more of a passive feed and you just are getting a constant, like running commentary on everything happening in your life.
And it's just helping you kind of interpret and understand everything.
Also really more deeply integrated into your life. Not just like, Oh, uh, like intellectual philosophical thoughts, but like literally, uh, like how to make a coffee, where to go for lunch, just, uh, whether, you know, dating, all this kind of stuff.
What to say in a job interview. Yeah.
What to say. What to say. Next sentence.
Yeah. Next sentence. Yeah. At that level. Yeah. I mean, yes. So technically, now, whether we want that or not is an open question, right?
Boy, I would kill for a pop-up. A pop-up right now. The estimated engagement using is decreasing. For market reasons, there's a controversy section for a Wikipedia page. In 1993, something happened or something like this. Bring it up. That will drive engagement up. Anyway. Yeah, that's right.
I mean, look. this gets this whole thing of like, so, you know, the chat interface has this whole concept of prompt engineering, right? So it's good for prompts. Well, it turns out one of the things that a lot of times are really good at is writing prompts, right? And so like, what if you just outsourced, and by the way, you could run this experiment today. You could hook this up to do this today.
The latency is not good enough to do it real time in a conversation, but you could run this experiment and you just say, look, every 20 seconds, you could just say, you know, you know, tell me what the optimal prompt is and then ask yourself that question to give me the result.
And then as you, as you, exactly to your point, as you add, there will be, there will be these systems are going to have the ability to be learned and updated essentially in real time. And so you'll be able to have a pendant or your phone or whatever, watch or whatever. It'll have a microphone on it. It'll listen to your conversations. It'll have a feed of everything else happened in the world.
And then it'll be, you know, sort of retraining, prompting or retraining itself on the fly. And so the scenario you described is actually a completely doable scenario. Now, the hard question on these is always, okay, since that's possible, are people going to want that? Like what's the form of experience?
You know, that, that we, we won't know until we try it, but I don't think it's possible yet to predict the form of AI in our lives. Therefore, it's not possible to predict the way in which it will intermediate our experience with reality yet.
Yeah. But it feels like there's going to be a killer app that, there's probably a mad scramble right now.
It's all open AI and Microsoft and Google and Meta and then startups and smaller companies figuring out what is the killer app, because it feels like it's possible, like a chat GPT type of thing, it's possible to build that, but that's 10x more compelling using already the LLMs we have, using even the open source LLMs, Lama and the different variants.
So you're investing in a lot of companies and you're paying attention. Who do you think is gonna win this? Who's going to be the next PageRank inventor?
Trillion dollar question.
Another one. We have a few of those today.
There's a bunch of those. So look, there's a really big question today. Sitting here today is a really big question about the big models versus the small models. That's related directly to the big question of proprietary versus open. Then there's this big question of where is the training data going to go? Are we topping out on the training data or not?
And then are we going to be able to synthesize training data? Yeah. And then there's a huge pile of questions around regulation and, you know, what's actually going to be legal. And so I would when we think about it, we dovetail kind of all those all those questions together.
You can paint a picture of the world where there's two or three God models that are just at like staggering scale and they're just better at everything. and they will be owned by a small set of companies, and they will basically achieve regulatory capture over the government, and they'll have competitive barriers that will prevent other people from competing with them.
And so there will be, just like there's whatever, three big banks or three big, or by the way, three big search companies, or I guess two now, it'll centralize like that. You can paint another very different picture that says, no, actually the opposite of that's gonna happen. This is gonna basically, this is the new gold rush. alchemy.
This is the big bang for this whole new area of science and technology. And so therefore, you're going to have every smart 14-year-old on the planet building open source and figuring out ways to optimize these things. And then we're just going to get overwhelmingly better at generating training data.
We're going to bring in blockchain networks to have an economic incentive to generate decentralized training data and so forth and so on. And then basically, we're going to live in a world of open source. And there's going to be a billion LLMs of every size, scale, shape, and description.
And there might be a few big ones that are the super genius ones, but mostly what we'll experience is open source. And that's more like a world of what we have today with Linux and the web. So.
Okay, but you painted these two worlds, but there's also variations of those worlds, because you said regulatory capture is possible to have these tech giants that don't have regulatory capture, which is something you're also calling for, saying it's okay to have big companies working on this stuff. as long as they don't achieve regulatory capture.
But I have the sense that there's just going to be a new startup that's going to basically be the PageRank inventor, which has become the new tech giant. I don't know, I would love to hear your kind of opinion if Google, Meta, and Microsoft, they're as gigantic companies able to pivot so hard to create new products.
Like some of it is just even hiring people or having a corporate structure that allows for the crazy young kids to come in and just create something totally new. Do you think it's possible or do you think it'll come from a startup?
Yeah, it is this always big question, which is you get this feeling. I hear about this a lot from founder CEOs where it's like, wow, we have 50,000 people. It's now harder to do new things than it was when we had 50 people. Like what has happened? So that's a recurring phenomenon. By the way, that's one of the reasons why there's always startups and why there's venture capital.
It's just that's like a timeless phenomenon. Uh, kind of thing. So that, that, that's one observation. Um, on a page rank, um, we can talk about that, but on page rank specifically on page rank, um, there actually is a page. So there is a page rank already in the field and it's the transformer, right? So the, the, the big breakthrough was the transformer.
Um, and, uh, the transformer was invented in, uh, 2017 at Google, um, And this is actually like really an interesting question. Cause it's like, okay, the transformers, like why does open AI even exist? Like the transformers invented at Google. Why didn't Google? I asked a guy, I asked a guy I know who was senior at Google brain kind of when this was happening.
And I said, if Google had just gone flat out to the wall and just said, look, we're going to launch, we're going to launch equivalent of GPT-4 as fast as we can. He said, I said, when could we have had it? And he said, 2019. Yeah. They could have just done a two year sprint with the transformer and Bennett because they already had the compute at scale.
They already had all the training data and they could have just done it. There's a variety of reasons they didn't do it. This is like a classic big company thing. IBM invented the relational database in the 1970s, let it sit on the shelf as a paper. Larry Ellison picked it up and built Oracle. Xerox PARC invented the interactive computer. They let it sit on the shelf.
Steve Jobs came and turned it into the Macintosh. And so there is this pattern. Now, having said that, sitting here today, Google's in the game, right? So Google, maybe they let a four-year gap go there that they maybe shouldn't have, but they're in the game. And so now they're committed. They've done this merger. They're bringing in Demis. They've got this merger with DeepMind.
They're piling in resources. There are rumors that they're building an incredible super LLM way beyond what we even have today. And they've got, you know, unlimited resources and a huge, you know, they've been challenged with their honor.
Yeah, I had a chance to hang out with Sundar Pichai a couple of days ago and we took this walk and there's this giant new building where there's going to be a lot of AI work being done. And it's kind of this ominous feeling of... like the fight is on. There's this beautiful Silicon Valley nature, like birds are chirping, and this giant building, and it's like the beast has been awakened.
And then all the big companies are waking up to this. They have the compute, but also the little guys have... it feels like they have all the tools to create the killer product that, and then there's also tools to scale.
If you have a good idea, if you have the page rank idea, so there's several things that is page rank, there's page rank, the algorithm and the idea, and there's like the implementation of it. And I feel like killer product is not just the idea, like the transformer, it's the implementation, something, something really compelling about it. Like you just can't look away. Something like,
The algorithm behind TikTok versus TikTok itself, like the actual experience of TikTok, you can't look away. It feels like somebody's going to come up with that. And it could be Google, but it feels like it's just easier and faster to do for a startup.
Yeah, so the startup, the huge advantage that startups have is they just, there's no sacred cows. There's no historical legacy to protect. There's no need to reconcile your new plan with existing strategy. There's no communication overhead. There's no, you know, big companies are big companies. They've got pre-meetings, planning for the meeting. Then they have the post-meeting and the recap.
Then they have the presentation of the board. Then they have the next round of meetings. Yeah, lots of meetings. And that's the elapsed time when the startup launches its product, right? So there's a timeless, right?
Yeah.
So there's a timeless thing there. Now- Yeah. What the startups don't have is everything else, right? So startups, they don't have a brand, they don't have customer relationships, they've got no distribution, they've got no scale. I mean, sitting here today, they can't even get GPUs, right? Like there's like a GPU shortage.
Startups are literally stalled out right now because they can't get chips, which is like super weird.
Yeah, they got the cloud.
Yeah, but the clouds run out of chips, right? And then to the extent the clouds have chips, they allocate them to the big customers, not the small customers, right? And so the small companies lack everything other than the ability to just do something new. Yeah. Right. And this is the timeless race and battle.
And this is kind of the point I tried to make in the essay, which is like both sides of this are good. Like, it's really good to have like highly scaled tech companies that can do things that are like at staggering levels of sophistication. It's really good to have startups that can launch brand new ideas. They ought to be able to both do that and compete.
They neither one ought to be subsidized or protected from the others. Like that's, that's to me, that's just like very clearly the idealized world. It is the world we've been in for AI up until now. And then of course there are people trying to shut that down, but my hope is that, you know, the best outcome clearly will be if that continues.
We'll talk about that a little bit, but I'd love to linger on some of the ways this is going to change the internet. So I don't know if you remember, but there's a thing called Mosaic and there's a thing called Netscape Navigator. So you were there in the beginning. What about the interface to the internet? How do you think the browser changes? And who gets to own the browser?
We got to see some very interesting browsers. Firefox, I mean, all the variants of Microsoft, Internet Explorer, Edge, and now Chrome. The actual, I mean, it seems like a dumb question to ask, but do you think we'll still have the web browser?
So I, uh, I have an eight year old and he's super into like Minecraft and learning to code and doing all this stuff. So I, I, of course I was very proud. I could bring sort of fire down from the mountain to my kid and I brought him chat GPT and I hooked him up on his, on his, on his, on his laptop. And I was like, you know, this is the thing that's going to answer all your questions.
And he's like, okay. And I'm like, but it's going to answer our questions. And he's like, well, of course, like it's a computer. Of course, it answers all your questions. Like what else would a computer be good for? Dad. Never impressed. Not impressed in the least. Two weeks pass and he has some question. And I say, well, have you asked JetGPT? And he's like, dad, Bing is better.
And why is Bing better is because it's built into the browser. Because he's like, look, I have the Microsoft Edge browser and it's got Bing right here. And then he doesn't know this yet, but one of the things you can do with Bing and Edge is there's a setting where you can use it to basically talk to any webpage. because it's sitting right there next to the browser.
And by the way, it includes PDF documents. And so the way they've implemented an edge with Bing is you can load a PDF and then you can ask it questions, which is the thing you can't do currently in just ChatGPT. So they're gonna push the meld. I think that's great. They're gonna push the melding and see if there's a combination thing there.
google's rolling out this thing the magic button which is implemented in they put in google docs right and so you go to you know google docs and you create a new document and you you know you instead of like you know starting to type you just you know say it press the button and it starts to like generate content for you right like is that the way that it'll work um is it going to be a speech ui where you're just going to have an earpiece and talk to it all day long you know is it going to be a
Like these are all like, this is exactly the kind of thing that I don't, this is exactly the kind of thing I don't think is possible to forecast. I think what we need to do is like run all those experiments. And so one outcome is we come out of this with like a super browser that has AI built in. That's just like amazing.
Look, there's a real possibility that the whole, I mean, look, there's a possibility here that the whole idea of a screen And Windows and all this stuff just goes away. Because why do you need that if you just have a thing that's just telling you whatever you need to know?
And also, there's apps that you can use. You don't really use them, being a Linux guy and Windows guy. There's one window, the browser, with which you can interact with the internet. But on the phone, you can also have apps. So I can interact with Twitter through the app or through the web browser.
And that seems like an obvious distinction, but why have the web browser in that case if one of the apps starts becoming the everything app?
Yeah, that's right.
What Elon's trying to do with Twitter, but there could be others. There could be like a Bing app, there could be a Google app that just doesn't really do search, but just like do what I guess AOL did back in the day or something, where it's all right there and it changes everything.
It changes the nature of the internet because where the content is hosted, who owns the data, who owns the content, what is the kind of content you create, how do you make money by creating content, who are the content creators, all of that. Or it could just keep being the same, which is like,
with just the nature of web page changes and the nature of content but there will still be a web browser because a web browser is a pretty sexy product it just seems to work because it like you have an interface a window into the world and then the world can be anything you want and as the world will evolve there could be different programming languages it can be animated maybe it's three-dimensional and so on yeah it's interesting do you think we'll still have the web browser
Every medium becomes the content for the next one. So the AI will be able to give you a browser whenever you want. Oh, interesting. Yeah. Another way to think about it is maybe what the browser is. Maybe it's just the escape hatch, right? Which is maybe kind of what it is today.
Which is like most of what you do is like inside a social network or inside a search engine or inside somebody's app or inside some controlled experience. But then every once in a while, there's something where you actually want to jailbreak. You want to actually get free.
The web browser is the F you to the man. That's the free internet. Yeah. Back the way it was in the 90s.
So here's something I'm proud of. So nobody really talks about it. Here's something I'm proud of, which is the web, the web, the browser, the web servers, they're all, they're still backward compatible all the way back to like 1992. Right.
So like you can put up a, you can still, you know, the big breakthrough of the web early on, the big breakthrough was it made it really easy to read, but it also made it really easy to write, made it really easy to publish. And we literally made it so easy to publish. We made it not only so it was easy to publish content, it was actually also easy to actually write a web server.
And you could literally write a web server in four lines of Braille code. And you could start publishing content on it. And you could set whatever rules you want for the content, whatever censorship, no censorship, whatever you want. You could just do that. As long as you had an IP address, you could do that. That still works. That still works exactly as I just described.
So this is part of my reaction to all of this censorship pressure and all these issues around control and all this stuff, which is like maybe we need to get back a little bit more to the Wild West. The Wild West is still out there. Now, they will try to chase you down.
People who want to censor will try to take away your domain name and they'll try to take away your payments account and so forth if they really don't like what you're saying. But nevertheless, unless they literally are intercepting you at the ISP level, you can still put up a thing. I don't know. I think that's important to preserve.
One is just a freedom argument, but the other is a creativity argument. Which is you want to have the escape hatch so that the kid with the idea is able to realize the idea. Because to your point on PageRank, you actually don't know what the next big idea is. Nobody called Larry Page and told him to develop PageRank. He came up with that on his own.
And you want to always, I think, leave the escape hatch for the next kid or the next Stanford grad student to have the breakthrough idea and be able to get it up and running before anybody notices.
You and I are both fans of history, so let's step back. We've been talking about the future. Let's step back for a bit and look at the 90s. You created Mosaic Web Browser, the first widely used web browser. Tell the story of that. And how did it evolve into Netscape Navigator? This is the early days.
Full story. You were born. I was born.
A small child. Actually, yeah, let's go there.
When did you first fall in love with computers?
Oh, so I hit the generational jackpot and I hit the Gen X kind of point perfectly as it turns out. So I was born in 1971. So there's this great website called WTF happened in 1971.com, which is basically 1971 is when everything started to go to hell. And I was of course born in 1971. So I like to think that I had something to do with that.
Did you make it on the website?
I have, I don't think I made it on the website, but I, you know, hopefully somebody needs to add, this is, this is where everything, maybe I contributed to some of the trends. Um, that they should. Every line on that website goes like that, right? So it's all a picture disaster.
But there was this moment in time where, cause the, you know, sort of the Apple, you know, the Apple II hit in like 1978 and then the IBM PC hit in 82. So I was like, you know, 11 when the PC came out. And so I just kind of hit that perfectly. And then that was the first moment in time when like regular people could spend a few hundred dollars and get a computer, right?
And so that, I just like that, that resonated right out of the gate. And then the other part of the story is, you know, I was using an Apple II. I used a bunch of them, but I was using Apple II. And, of course, it said on the back of every Apple II and every Mac, it said, you know, designed in Cupertino, California. And I was like, wow, Cupertino must be the, like, shining city on the hill.
Yeah.
and low rise apartment buildings. So the aesthetics were a little disappointing, but you know, it was the vector, right, of the creation of a lot of this stuff. So then basically, so part of my story is just the luck of having been born at the right time and getting exposed to PCs then. The other part is,