Lex Fridman Podcast
#434 – Aravind Srinivas: Perplexity CEO on Future of AI, Search & the Internet
Aravind Srinivas
There's a lot of JavaScript rendering. You have to decide what's the real thing you want from a page. And obviously, people have robots.txt file. And that's a politeness policy where you should respect the delay time so that you don't overload their service by continually crawling them. And then there's like stuff that they say is not supposed to be crawl and stuff that they allowed to be crawl.
0
💬
0
Comments
Log in to comment.
There are no comments yet.