Menu
Sign In Pricing Add Podcast

Lex Fridman Podcast

#434 – Aravind Srinivas: Perplexity CEO on Future of AI, Search & the Internet

7867.507 - 7891.675 Aravind Srinivas

There's a lot of JavaScript rendering. You have to decide what's the real thing you want from a page. And obviously, people have robots.txt file. And that's a politeness policy where you should respect the delay time so that you don't overload their service by continually crawling them. And then there's like stuff that they say is not supposed to be crawl and stuff that they allowed to be crawl.

0
💬 0

Comments

There are no comments yet.

Log in to comment.