Lex Fridman Podcast
#447 – Cursor Team: Future of Programming with AI
Aman Sanger
You're bottlenecked by how quickly, for long context with large batch sizes, by how quickly you can read those cache keys and values. That's memory bandwidth, and how can we make this faster? We can try to compress the size of these keys and values. Multi-query attention is the most aggressive of these.
0
💬
0
Comments
Log in to comment.
There are no comments yet.