Panos Panay
Appearances
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
To invoke all of that in one place is I don't I think it's beyond it's an incredible like it's what Alexa can do. I don't see that anywhere else. It's quite powerful.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It's a simple example, right?
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
But you're like a slideshow. What's the big deal, Pete? I'm like, well, I'll be clear. Like on that screen, it's emotional. It's ambient. It was natural. Like, yeah, but it is somewhat simple in the way you talk about it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Correct. Well, no, Spotify would have worked there too, but yes. Sure. You need to have a music service, but I would like it to be Amazon music.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah. I happen to be responsible for photo service. So I got that. That's a blessing. Yeah.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
First off, all of Amazon's rallying around Alexa. It's crazy. It's so cool. It comes down to a few things.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, I think Andy's been a huge part of it. I have a role. I mean, I came in with a vision that I think Alexa is the thing that we can anchor and change the world with. Yeah.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I don't know if it was Alexa Plus. I'm not going to say that.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It was the advent of where we can take AI. You can see the turning point. I was there. I was in the middle of it. And it's just awesome moments. what Amazon brings relative to just even what I'm responsible for and how they can all connect magically through AI. You saw Fire TV and Ring yesterday.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I fully believe like this transformation's happening and Amazon's the leader in ambient AI, period, end of story, and in the home. And like, if we can connect all these things, that was, now we're talking whatever a year and a half ago when I was talking to Andy about joining Amazon. And he was just so ambitious about it. He's like, look, come in and do it. Let's do it. That is the tipping point.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
There's a lot of nuance in that, but that was the tipping point. Like, we can change the world. You can think the scale, the relative level of investment, the ambition, the patience that Amazon brings, we're happy to talk about. Yeah, the answer to the first question is, sure, I come in, lay down a vision, re-architect the team a little bit.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Get the explicit focus on first thing we got to do is get Alexa right. Once we do that, we'll bring the hardware together. To get Alexa right, it takes music, photos, shopping. And these are photos, of course, under me, but you have across the company, you have music, video, shopping. We'll just use those three as huge tenants to the product. And, you know, those leaders are exceptional.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
There's no, we're not gonna work together. It's the opposite. You know, and at Amazon, we set goals. They are cross company goals. And so the goals are set out, you know, from Amazon Nova, which is one of the anchoring points of the product to what music needs to be on the product.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Sure, you know, the expert is kind of a joint thing, the music expert, but ultimately like that music service has to be perfect and music team is killing it right now. Shopping, you know, all in. How to make it great. We didn't do a lot of shopping yesterday just because it would have been like a meme. I mean, of course, shopping. Like, you know, oh, yeah, it's going to be amazing.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And then video, same. And there's other areas. But we align and we go. But it does start with a commitment from me for sure. I'm in. I'm all in. I'm ready to re-architect it. It's not going to be easy. It's going to take time. Andy's patience, I would say, the company's patience to get it right for the customer is extraordinary, like extraordinary.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I mean, Andy was pushing me like, okay, you know, don't, you know, he wants urgency, of course, like you would expect from an Andy Jassy, but he also wants the right thing for the customer. And I mean, when you talk about customer obsession, like, let's get it right. Let's do it right and get it right. And we didn't move slow, even though, you know, you asked what's taken so long.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I don't see that, you know what I mean, from where I'm sitting. I know it feels late because there's been a lot of announcements, but I think we're here at the right time.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
We refocused on Alexa, like we really did. It was in a lot of different places. And so we just made it super clear. I had an Alexa platform team and Alexa product team. It's not a platform team. I guess it's not the right way to say it, but just an engineering going across and a product team vertically is the way I look at it. And, you know, that AI stack going across.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And so once you get that focus and that clear ownership, leadership, you quickly see speed change. That was the biggest, I think, shift Also made some shifts as a team where a lot of the core horizontal functions, if you think about the lowest level of the OS or the stack as a horizontal or hardware or supply chain, were kind of intermixed with the product verticals.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
So I've shifted that around too just to get more product focus. One of the number one tenets is we're going to make great products. Let's just start there.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I don't know exactly what was said, but at the end of the day, I immediately started pushing the team to have amazing pride in their products.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Like we have to because that pride shows up for our customers. And, yeah, we want to push for it. That is a little bit of a, you know, it's just let's be super clear. These products have to be great. We're not making tradeoffs if they're not.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I think you've got to focus the roadmap. I think there's no doubt. What you need is products that people want in their home, but also need. So I don't think that history is broken. Obviously, the more endpoints, the better. But they got to be the right ones and they got to be the ones that people want to use.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Here's what I will say. The go forward is focus on making great products and the right ones. I don't think you're going to see thousands of products a year coming out. That's not the goal at all.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
What I want is, you know, some attention to detail, making sure the right products for the customers are there, the things that fit into your home, the things that fit on your eyes, things that fit in your ears, so you can take Alexa with you. And just narrow the experiences that are great that way. And I have to tell you, the focal point, yeah, it is a screen on an Echo device in the home.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Like that can run your home. You don't need it. With Alexa Plus, you actually don't need it. It's just a better experience. And so when I'm asked, because there is a little bit of, I mean, I'm treading a little bit here on some hallowed ground. There's a little bit of, look, we're going to light up all your Echo devices, but it's just going to be awesome if you have a screen.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And when somebody says, so do you recommend a screen? My answer is, yeah. Do you have to have a screen? No. You're still going to have a great experience. Remember, you have a screen in your pocket. It's called a phone. That phone has an incredible new Alexa Plus app on it. And so you have a screen.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You don't need it to operate it, but if you, let's say you start a conversation with your voice and you just want to remember what that conversation was, You're going to go to your phone to just capture it or you can send something to your phone. We didn't do it in the demo. Did we demo it yesterday? I think we cut it just for time.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
But anything you're doing, you can send a phone because it's like longer form. I wanted my phone. We're also launching Alexa.com. So you're going to use it on your PC. So it'll be in the right places. But at the end of the day, if the focal point is to control your home, which, by the way, hundreds of millions of customers, that's really the focal point today. You put a screen there.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It's emotional. It's informative. It's useful. And it'll make a difference. It'll make a difference.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah. I often get accused for making the final decision only when I have to. It doesn't mean I'm not making decisions. When I was studying up to come to Amazon and kind of making that decision, that was a life decision for me. It was a big one. And I was so inspired talking to Jeff, talking to Andy, like just inspired, no doubt. I also love Microsoft.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
So, you know, I'm inspired where I was sitting. So there's just all these conflicts. Those are personal to me, you know, like, oh my gosh. But when I started reading, let's go to decision-making. And then I just watched a few stories that Jeff had told, talked to Andy about it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It basically, from a leadership principle standpoint, and from some of the things you hear about, like on decision-making principles, like one-way, two-way doors, it's hard to explain this, but it's so aligned to the way I was running my team. That's how I'd operate. It was weird. You know, you're just reading the LPs, and I'm like... I used to have a culture box.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Oh yeah. Right. So leadership principles at Amazon, you should check them out. You can go to Amazon, find them. They're rad. I mean, they're, they're inspiring. They're almost, sometimes they're just obvious. Not all, not all of them, you know? And they're hard to believe like big bets. Is that real? I'm like, yeah, it's pretty damn real. You know, it's pretty incredible.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You know, leaders, they do, they dive deep. Yeah, they do. They get into everything. And I, I think those are, Those are real, but in the spirit of when I started reading them and then the way I made decisions, Nilay, they were aligned. Yeah. I mean, I'm not, no BS. Like they were just, it was like, it felt like right. I had a culture box when I was running my team at Surface and Windows.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And that culture box, you know, had five cultural principles. They were basically five of the LPs. But that's how I ran it. It was so connected. And when I got to Amazon, it was almost, what a team. I found this team that was not only hungry, but unbelievably talented, massive, and capable. Knows how to ship. knows how to invent, and it's just a little bit of direction.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It's all, I mean, I don't, my job's to give that direction. And so making sure I lay out the vision, making sure everyone knows where we're going, what are the highest priorities? But when it came to decision-making, to answer your question is, I fully operate in the values of, all right, let's make this call today, but no,
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And I think one of the strongest points of a leader, without any doubt, and I learned this from one of my colleagues in the past, you know, who I worked for, he used to teach me. He'd go, hey, Pete, when you've made a decision, the best leaders in the world are willing to be wrong. Now, you got to be right a lot, but you're willing to be wrong.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Like, this is very, very, it's like simple to say, but it's a powerful concept. What does willing to be wrong mean? It means you got to put your ego aside. You got to be vulnerable. Like, do you know how hard that is? Like in front of a team of thousands of people, like just, yep, I was wrong. And what does being wrong mean? It's not like this dramatic, I'm wrong, I'm sorry. That's not it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
What being wrong, it's not necessarily the wrong statement. It is the, you got new information a week later, then use the information. And if it was a two-way door decision, guess what? Make the right decision.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
But if you're not a great leader, you don't change that decision because you're like, I already made the call, sorry, but you knew it wasn't right for the customer or for the business or whatever the reason. It's just a fail. And this was very early in my career. It's very similar to the two-way door, one-way door. Once you've made the hard call and you're past the point of return, that's it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You made the call and you have to make decisions sometimes. Man, and those are hard. Like you lose sleep over it. Like when I made the decision to have the event, Yeah. We're doing it. And we're like, well, product's not 100% done. I go, doesn't matter. I'm at 90% usage. We're going. And everyone's like, you realize that, and that was a two-way door decision, until I send out the invites.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And so we checked the information the day before we sent out the invites. They're like, we're going. The minute you send the invites, that's a one-way door decision. There's no pulling back. It didn't matter how sick I was. You know, it doesn't matter who couldn't make it. It didn't, none of it mattered.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And, you know, then now we're lining it up, lining, you know, we had the venue booked and we're like, okay, that wasn't a one-way door decision. You can always cancel the venue. Not cool. But if you had to, you know, and you kind of go through it and then you go, you get to that point. Yep. That's it. There's no new, there's no new information that was going to change it. And so great leaders are,
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
They'll make those decisions, but they'll always be willing. They'll always be willing to check themselves and not just check themselves, but be willing then when they have new information, if the right decision is in front of them, you gotta change it. And I always live by it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And so when you come to this world of, when you say, you know, this culture, like Nilay, the Amazon culture is incredible. You have no idea how empowering that is. Hey, it's a two-way door decision. All right, let's make the call. If we're wrong, let's deal with it. Yeah. But then we move and we move. And I get accused a lot of, you know, like to make a call and like, do you have all the info?
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Probably not, but. Yeah, we got to try something. We're moving. Yeah. That's been pretty fun that way.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Credit to Andy. That's Andy. That wasn't me.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, I think the exact words were $19.99 per month, but free with Prime.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I want customers to understand that the service is better with Prime. Yeah. At the end of the day, if you have Amazon, sorry, Prime Video, Prime Shopping, Amazon Prime, you fundamentally get the best music experience. You get photos, unlimited photos. That just makes the Alexa experience better. You don't need to have it. It's a great experience without it. But it's just better. Yeah.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And so we talked about it. Like, we want people on Prime. You know, if you're on all those services, it comes together as a collection on your product. It just makes the personalization so much stronger. It makes the invocation of services so much easier.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Just a series of events. I think back to two-way door decisions, I don't think it was the first decision. There were different ways to think about it. It costs more to run the service. That's all there is to it. You're going to invoke an LM. You have many models working. There's a lot of inference. That's true.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Then you heard Andy talk about how much cost is coming down with Tranium 2, and you just see... the efficiencies, if you will, that are coming through, those are plumbed through the plan. We have an incredible opportunity in front of us. And so it wasn't about how much you're spending, how much you're making. It's about making a great product.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And once we were like, we want to make sure people have the best product possible. That is the anchor. And so we're like, all right, it's got to be with Prime. That's the best way to get customers there. The other piece of that that I see- I think people want it to be more complicated because I've been asked this question a bunch of times. I generally haven't answered it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I'd be like, oh, you have a choice. You can pay 15 or 20. It's your choice. Just choose. But not to be a- I'm not trying to be pompous or whatever. Just like, I think- if you're on Prime, you're going to love it. So, you know, I kind of inverse the equation.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I'd have to think about it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
No, they're not connecting. I don't know.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, hard to do.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Well, it's not gone, I think, when HP, right?
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I don't see it that way.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I think it's more of a compliment now than it's ever been. You need the phone. We send things to the phone. We want you on it as well. Want you on the Alexa app on your phone. It's an awesome experience. We can play with it if you want after. But I think it's a compliment to the phone. I think it does replace a lot of things. I'll tell this. I say it to my team all the time.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Look, our customers are going to find the easiest path to something. They just will. It's innate. It saves time. It's about speed. It's about efficiency. The only time that's not true is when you're getting more joy. And a lot of times joy comes from speed, you know, or happiness comes from being able to complete a task quicker. And so let me go back to the point of ambient.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Like we have one of the core tenants when we started Alexa Plus and the vision for it was we have the largest install base in homes on the planet. I think that's a pretty definitive statement. I think it's true. Probably have to check with the lawyers to say something like that. So maybe I'm wrong. So let me qualify it. We might have the largest install base on the planet.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
The way Alexa Plus is designed is it's meant to be ambient. It's meant to be a conversation and it will replace tasks you do on your phone. It's gonna happen. And so does it replace the phone? Absolutely not. But does it replace certain things? I think I told you this story before. Let me tell you again.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
When I was building laptops 12 years ago, when I first started on Surface, people came to me and said, there were a few people that were like, you've lost the plot, P. You're going after this thing and the laptop is dead. Like, why? Because phones are replacing the laptop. And I mean, you're using a laptop 12 years later, and it's pretty damn important to you.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Probably more important now than it was 12 years ago. So what had happened was jobs moved to the phone that were really important, shopping, social media, your photos, I don't know, communication. But what happened was the things that didn't move to the phone only got stronger on the PC over that time. And so they essentially became compliments to one another.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
If you're going to sit down and write a long story, you're going to do it with a keyboard. You want to be snackable information, you're going to pick up your phone. And then one got better at one of them and the other got better at the other. And incredibly so. Actually strengthened them both. I see this very similar.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I think as Alexa Plus comes into market, I think it's going to be better at a lot of things and it's going to move jobs to it. I believe that. I think there'll be more emotion to be pulled out of something that's conversational, knows you well, is personal to you. You can have a conversation, knows your calendar, can get some stuff done in a simple way. You might not always do it on it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I don't know. It doesn't matter to me where you do it. I just want to give you the shot. And if it's the easiest way to do it, can I give you just a fun example? I was sitting on the couch last week with Costas, my son. He's 24. We were hanging out and we were talking about the Clippers. And he had asked me a few questions. And, you know, I'm a fan of the Clippers and growing up.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And then, of course, you know, since Steve bought them, I just love the team. I asked, Costas, did the Clippers win last night? He goes, it's quite even playing. This is, I think, a week and a half ago. I remember the day. And now we have Alexa Plus in the house everywhere. And my son works on AI now. He's blown away by it. You know, he had to sign an NDA. He can't talk about what he sees.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And I realized right at that moment, Eli, I was going to lose him because you know what happens? You pick up your phone. You open it. Now you look, you see your notifications. Okay. You know that feeling. And you're like, oh, I'm going to check my notifications or I'm going to jump on TikTok or whatever it is that you love about your phone. He's going to go get the information, answer it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And I'm going to lose my kid to my phone, to his phone. And now all of a sudden we went from this moment hanging out to him on the phone. It happens all the time. And it blew my mind. He goes, I don't know. Alexa? Alexa? did the Clippers win last night? And Alex goes, the Clippers did win last night. And then his score, and blah, blah, blah, Kawhi Leonard scored so many.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And he's like, is Kawhi playing? Yeah, Kawhi's been back for several weeks. And he now started having a conversation. The three of us are having a conversation. The job moved. Yeah. He would have never done that.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Good to see you, man. I love being here.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It was a great ad, by the way. It was a great ad. Oh, my gosh. What a great ad.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
They're durable, but they're going to continue to evolve at a rapid pace, and they have to. Yeah, they are. But you have to be smart about how you build on top of it. I mean, obviously, everyone's doing a great job. I mean, I'm sure. But yeah, I think the promise is there. I'm not going to understate it. I won't overstate it. I can't. I just, I believe the promise is there.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I'm here at Amazon because I believe it's going to change the world on how people engage AI. And it's going to be easier because your device is there and ready for you. And we're going to make beautiful devices. And so all this will come together in a way where there's a team that's going to connect all these experiences. You saw a little bit of Fire TV and Ring.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
That all of a sudden, these natural moments are going to happen. And you're not going to have to guess. You're not going to wonder.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Correct. Exactly right. And so hopefully everyone understands that concept. But since it's not deterministic and now you're going to ask a question, even if Alexa doesn't do it, she's going to talk about what you're trying to solve and you're going to actually get to an answer. Yeah. As opposed to, I don't know.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
That's exactly right. And some we'll do on the edge, too. You don't have to do it all. If it's point and shoot, old command, we'll do it in a simpler way.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
The orchestrator picks the model that's right for the job. The how, I won't get into the details, but there's some awesomeness here. One of the things that inspired most people is that we're using a multi-model approach, which I think is a little bit novel. But at the end of the day, it depends on what the task is. It depends on what's being asked for.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Think right now, you're seeing 70% of the utterances running through Amazon Nova, 30% running through...
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
uh anthropic something at that rate it changes just depends on how you use the product and what you're using it for um it is also non-deterministic basically there's a model that's like what's the best model to pick and then you're looking for accuracy and speed understand you know just first understanding then accuracy then speed and you target um then you move it you pick the right model and then you fire to the expert and there's a small model in the expert if you will sometimes and then those all orchestrate together and that's how it works
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Inside of that is the way that you talk to your partners. Mm-hmm.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Uber's been awesome.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And that is— Uber, OpenTable, Grubhub, like these things that you use every day, like they're just in-depth connected.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
He was going to choose last minute depending on how the demos went. I think he did – did he do a Miele dishwasher?
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
That's what I knew. By the way, that's what he said. That's pretty funny. Yeah.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, I mean, then the partner just has to do the work.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, you don't want to do the work, no problem. There's just a couple of different ways to engage it. From an SDK perspective, this is just basically permissions.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And we have to work on, you know, kind of authorization and payment at the end of that, which is the trickiest part. And so, I'm not going to get into how, but that's the trickiest part. Yeah. And so, completing the task is the trick. Getting almost there... It's not that hard, but completing it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And so that's where you need the partner to be like, yeah, sure, we want this traffic, and we're going to go create the service and send it through. Great. If you don't, no problem. But the answer on why not do the API is just these relationships are different. Partners want to work in different ways. One of the things we are trying to do, and I'm really reengaging...
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Alexa is, we want to open SDKs. Basically, we want to open the product up for developers to come in and do what they want. Like, come make it great. And if somebody asks to fix something in their house, we got it. We have a way to get you there.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I don't think resetting Amazon. Like, Amazon's incredibly ambitious in so many ways. Always learning, changing. I mean, it's pretty powerful. The devices team, a little bit. First off, we hadn't really had a large-scale event, I don't think, as I understand it. Obviously, I wasn't there since pre-pandemic.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I think the partnerships are unique for sure. I think it's quite different. Remember, you always go back to your phone. The information is there. It's in the app. It's not like we're doing something on the side and doing it anonymously. And, you know, you don't have the customer info, I think, is one thing. Second thing is, like, when you have those challenging, like, let's use a Thumbtack example.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Let's stick there for a minute. If you don't have a Thumbtack account, the first time you do it, it'll just pop a QR code and say, here, connect. Authorize go. And then forever then you're going to fix things and Thumbtack's going to push you through it. There's just some simple things that you can do that makes the customer journey simple and get those connection points.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And once you do that, which is everything, you understand this setup is everything. removing that barrier to entry. Yeah. Like, to make Alexa Plus great, like, you've got to share your contacts, you're going to want to add your photos, you're going to want to connect your service providers. You know, it's a one-time kind of low barrier to entry go, and then you're all in.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And, you know, the partners... We don't talk about the deals with the partners or anything like that, but there's benefit on both sides. But at the end of the day, it's the right thing for the customer. I think there's a lot of partners out there that believe in that same philosophy, like let's get our customer to the end game.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, but you have preferences. Customers have preferences. They know what to say.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
What intermediary brings you the sandwich? You can't speak for the customer. But I would say they just have a choice. There will be simple ways to make it clear to the customer what they want.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Just think it's right. Yeah. It's their business. And so we're seeing a lot of participation. People are all, like, there's a lot of partners. Yeah. And they're excited from what I see. Not all, you know, I mean, I can't speak for all of them, so it's hard to, I'm not trying to talk in absolutes. But you have this moment where you're like, the promise of Alexa's here, Ambient's here, right?
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Forever. They've all made skills in the past or they've done something that, you know, they didn't get invoked. And it's hard because the customer had to point, shoot, as opposed to just speak natural language, know exactly what they were asking for. But at the end of the day, now you have like a truth in just speak and something comes up.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And now a partner's like, well, if they're looking for something from me, in, you know. But I think it's only, I think it's right to be partnering and not, You know, not doing it another way. Yeah. The other big— Which I'm pumped about. We have a great biz dev team that just—it's what they do.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
We do it all before then. This one's different. We already have partners that work with Alexa. If you already work with Alexa, you get the magic. That's it. It's awesome. You saw it yesterday. There was no new code written on the partner side.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Nothing. I have my Govee lights at home right now that I put on the house. I mean, I'm just talking to them to change the color.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
That's it. I would have never opened the app to change the color on my lights.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You don't know how badass, you have no idea how badass my team is. Okay. You have no idea. Yeah. Like, and you haven't asked me, so I have to tell you. Yeah. Like, this team.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
This team. I'm now, I'm now talking Eero, Ring, Blink, Fire TV. I mean this team, including Alexa, Kuiper. They're incredible, man. They're so damn capable. I've not seen invention like this.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Now, how we get it to the customer, you know, we'll refine a little bit of that, but I got to tell you, and this is a great example, because this works with Alexa program and the thousands and thousands and thousands, dare I say hundreds of thousands, things that work with Alexa, that is one of the largest, like, that is one of the largest connectivity tissues on the planet. It's crazy.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And they've set it up so well that now when Alexa Plus shows up, Your routines are by voice done. Like 100%, Eli. It's so damn cool. Like the other day, Mary was so frustrated with me. And I don't have smart home at my house in the Seattle area, but I use it in another area. And she was so frustrated with me. She's like, the lights are on all the time. And I just grabbed my app.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
What we did yesterday as a team, it was a little bit of a reset. Like the team was pumped to do it, excited. We were never going to announce hardware. Wasn't a goal. We need reset Alexa for the world and bring Alexa Plus forward. That is a bit of a cultural shift. Like we're just going to focus on the service and what it's going to be. Great products are coming.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I'm like, Alexa, every night just turn off the lights outside at 10 p.m. and don't turn them on again until 7 p.m. the next day.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
What story are you going to tell? I've got plenty of customers.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Team has a roadmap.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
in theory, but we also have the cloud to arbitrate. Like we have so many different methods in. You can use Matter, you can use Bluetooth LE sometimes, you can use Zigbee, but you can also- Rain famously runs on Z-Wave. You can use Z-Wave, you can fundamentally use works with Alexa, just plumb right in.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And so there's no limitation for us to connect these things because basically we can orchestrate to it. The team has thought through it from every way to Sunday, but they've also been working on it for 10 years. Yeah. It's phenomenal. It's probably one of the things I'm most excited about because you basically democratize the smart home. 100%.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yes, it won't work unless you gave someone a button on their phone today. But you know, we just talked about this. You know where the job's better? Just say what you want. It's a much better job to be done. I tried to do it with the music demo yesterday. I'm not sure it landed this point of it, which is like, just plugged them in. The speakers were there. I'm going to move music to the speakers.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I'm going to do it nuanced. I think one time I said, move, I want to move the music. I want to hear the music. I want you to bring the music here. I use different language. So it wasn't continuous. That was all real working. Probably those little nuances get lost on the natural language as if I had a direct command. I didn't, it could have been any of those or play, which I try to stay away from.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And so it's the same concept. Like you just think it and say it, think it and say it. It's very powerful. Yeah. And on smart home, it comes to life amazingly. And this is credit to an incredible team. Yeah. Yeah.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I think so. I think this is the tipping point of it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I think so. Tipping point because you don't have to be an expert. Yeah. You just don't have to be an expert. It doesn't, there's no such thing. Just plug it in. That's it. And then say something.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
We already have great products in market. We launched stuff at the holiday. And the team, they rallied. The company rallied. It's pretty awesome. You know, having Andy there is fantastic. You can feel a vibe in that room for sure. I hope you did. Yeah, I mean, you made your snarky comment about the music when you got in there. Man, we check every detail.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I don't care if you believe me or not at this point.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
When you get after it, man.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I'm not going to just say this crap.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Like, it's pretty fascinating because, you know, this is what an LM is great at. And then the expert that we have to go rationalize. And so it doesn't have to be deterministic. And so it's pretty interesting. By the way, it has to learn as well. So if you go turn on that light, which light? That one over there. Oh, you mean the one in the living room? Yeah, okay.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Now, that's not a good example because you're up against a switch, which takes is just go touch the switch. But how fast the system learns, that'll never happen. It'll never happen again. It'd be like, oh, I know what he needs. He's asking on this device and he's, I got it. I know I'm turning on the light.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You know, I've shown you everything, but I'll tell you like, if I can touch back to my Mary example, like I want these moments to connect not only the home, but the family. And it's got some pretty amazing attributes. The idea that I can leave the house and leave a message. And walk out the door. And then when Anastasia shows up downstairs, she gets the message.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And it's a lovely, like, note from her dad with maybe a direction of what to do. The fact that it's this total natural language moment feels magical. Alexa's being proactive on your command. Not intrusive, but, you know, you were asking her to be. When you start seeing those things, that's the thing. That's the thing I want it to be because...
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You're just going to connect deeper into people's lives in a way that makes it better. That you know me well enough. Like, I want you to use these products and tell me your life is better. Yeah. Like, that's it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Look, I have a vision for where this thing goes. I can't take you there. Like we've already revealed everything and we're shipping, you know, we're going to preview in a month and it's like, I'm sure we tipped over a few carts yesterday and like, you know, like, and so you're just got to be careful how far I take it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
There's so much for the future, but I showed you a few of my favorites and that's what we did. We narrowed it down. There's thousands of things it does now. Yeah. Try to narrow it down to the ones that were both told the story, but I also most emotional to me. Because that matters, you know, when I'm presenting and I think sharing.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And the biggest thing, you know, you want the team to have pride in the best stuff they've created. And those moments are pride moments for the team.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You know. There's eight voices with Alexa Plus. I don't think we talked about it yesterday. It's in the blog post that we wrote. Not the blog post, the about Amazon post. But I'm told I'm a dork when I say blog.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It just depends what you're using. Pick your voice and... Yeah. But the default.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I think I missed, I may have missed, I don't know. The chiptune rave music?
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I use a default voice. I love the new voice. You can use the old voice. I love the new voice. It's the default. And then you can pick a male voice or another voice, and you can call it what you want.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
For a decade, we've gendered this robot pretty relentlessly. This voice, the one we're using, I called her she yesterday. I understood that. I have a couple of people ask me. I'm like, well, that's kind of how I was thinking about it. Yeah. I don't think it's more complicated than that. Yeah. Makes sense. Is it fair? I mean, is it a fair question? I don't know.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, yeah, yeah. That was a funny one.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, I think, look, it is. But it is getting more personal. It's going to be more meaningful in your life.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You don't want to go all the way there. But yeah, I think it's okay that you think you have another set of ears when you want them, another set of thinking if you need it. I think it's quite powerful. Yeah.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Actually, the 8, 10, 15, and 21. 8, 10, so the screens. Yeah. It's rolling out next month starting with those devices, and it'll be a gradual rollout, and then it'll roll out to all devices. If you want to be in first, you know, my push is I want people using screen devices for sure. We're rolling it out there first because it's a great experience.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Every single part of that show after the moment the mic starts has been, you know, very, very well thought through. Yesterday's event was the highest risk event I've ever done. Bar none. Really? And that was, yeah, bar none.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You go get a device and you're on the list, you'll be first to get it. That's basically it. If you already have a 10, a 15, a 12—sorry, a 15 or 21, and you subscribe, then we'll get it out to you as well. That's where we're starting.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Do you think you're going to drive— And it'll light up the whole house, by the way.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
So if you have a screen and it comes to your screen— Yeah, let's say you have five Echos at home right now and you just go get a screen and it'll light up your whole house.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I hope so. I mean, I think they should. And not because I want to sell another device, but I want people to have that experience. I think it's a miss not to have it. It's a miss. Yeah. It's just a miss. And so, yeah, I want to drive a cycle in the spirit of not trying to be a sales, not my thing. But I will say, if you want the best experience, go get a screened device.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You know, we're pleased already. I didn't expect, you know, pleased seeing the reaction from yesterday. It's nice to see. But I think in a month, you know, people get in their hands. We'll start the preview. Most features will be done, most. There'll be a few that are coming later for sure. And then we'll, you know, we'll roll it out to everybody when it's the right time.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Not to be too cliche, I think 10 years ago was a magnificent moment when Alexa, 10 years and a couple months, but what a moment. It really was. A reset. I think right now, 10 years later, I actually do think this is that next moment, but this one is, to your point, that promise. I think this is the shift. I think this is that time. I mean, it's going to take years.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
This is not like, don't worry, you're not going to miss out. Somebody's like, well, why are you so late? I'm like, late? Do you know we're just at the beginning? And by the way, our roadmap's awesome. And I believe, I mean, this team, their invention and the company's patience for invention and its ability to make the big bet and stick with it, not only creates an incredible future opportunity,
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
But with that opportunity and bet and invention, you also have, like, the moment right now is just starting. It's literally just starting, dude. It's just starting. Yeah. Like, it's a great future. Like, it's fantastic. And I think the home transforms forever, you know, starting now. But it takes time. I would say patience is one of the strongest qualities of Amazon.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I had once heard infamously, you know, when a great leader, I don't know the quote, but, you know, our best overnight invention took seven years. You know, it takes time. But right now, we're here. Ten years later, here we are. And it's the beginning of that next gen. I think it is a shift. Yeah. I think it is. Right this moment.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Great to see you, Eli.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Doesn't compare.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Because, you know, when you're basically doing hardware, you have fallbacks. You know, the demos aren't, they're not live, but you always can just go to the hardware. When you're reinventing or kind of re-architecting an entire service, there's no backup. It was the product. I think the only product video we had, like actual video, was the kids portion.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Because, you know, honestly, you're not kids in the audience. So sharing a kid's feature without some emotion is a waste of time. It's like, here's a kid's feature. Please write about it. So putting a little bit of emotion and storytelling in it, those were all real demos that all really happened. That was one of the principles of the event.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It wasn't like, let's go make up a fake story and let's see what, you know, we'll just put, you know, film in. That was the one area where it was just a, you know, it was like a, it wasn't a vision piece. It was the product, but it was the only area that wasn't live. And so there was, there was a lot of trepidation.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Like it was not the, it was not this, it was the hardest kind of event we've put together with the risk, risk profile wise.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It's not just an LLM. It seems easy, you know, put a voice to the LLM, let the LLM talk or, you know, TTS, bring it out, bring out the voice, you know, or if it's speech to speech, it doesn't matter which tech. If you want the elements of connecting to thousands of, and you know, I'm speaking for Alexa.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You know, you asked a broader question, but let me just talk about Alexa. You want the element of connecting to thousands and thousands of APIs, partners that have been connected to Alexa forever. You're trying to manage hundreds of millions of customers who already have the product. You want to update as many of those devices as you possibly can. meaning you don't wanna leave a customer behind.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And you know, there will be some devices that are eight, nine years old that won't work, but everything else, most things will, like relative to what's used in the market today. So you gotta carry forward all that history because people still love Alexa. You know, we're still growing. We still have usage that's higher than you would expect. Can't leave those customers behind.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Like that's the worst thing. We focus on not doing that. So there's that element. Sitting on top of an LLM, you're now going, okay, just talking is just not that interesting, although awesome. Like having ambient conversation, I think it's a superpower moving forward for Alexa. It's different today on Alexa. It's like point, shoot, ask the question, hope to get the answer.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
With my team a year ago, we'd be in meetings, in product meetings, and we'd be talking, And people would say, you know, let me show you the new Alexa with a demo and they would Alexa speak to it. And it was like, nope, speak normal. Like go to natural conversation. Don't adjust your speech to, for Alexa, like that's exactly what you don't want if you want natural conversation.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It's hard though, you know, you've been training people for, we've been training ourselves for 10 years. You know, calling a timer is, can you set a timer for eight minutes? Calling a timer on the new Alexa is, I'm making a ramen egg. gotcha, I'll set a timer for eight minutes. Like where she just proactively comes back and sets it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I didn't demo that yesterday because I didn't want the timer headline, but it's really badass experience. You know, it's really cool. And so there's a level of like that transformation where I'm off topic, let me go back. At the end of the day, the LLM needs to be able to now, you know, it's the base layer. Then you got the next layer, which is just a series of different models.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
picking the right model to do the job. And then that model is basically picking the right expert. And so the LLM plays a role, especially in the natural side of it. But as it makes it through the stack, it narrows down for accuracy, it narrows down for speed, it then narrows down for holding memory and personalizing it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And now you just have a series of experts basically sitting on top, and one of them is conversational. That's not just an LLM. That's a series of – and by the way, if you look at any one of these other products, they're not just LLMs. They're basically – they're mainly, I don't know, overstating it, understating it, so not to be rude, but they're chatbots.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And they're pretty good. They're damn good. And then when you start typing, you know, long form and rewriting and dropping in summaries, very powerful. Yeah. creating videos, creating photos, isolated but powerful. But the idea that these experts all sit on top of the stack and basically there's a runtime that orchestrates and says, okay, call these experts.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
These two experts have to work together. Got it. And then it operates. That's just not simple. You know, the first thing I was asked when I got there was, Hey, why don't you just change the brain with an LLM and everything will be fine?
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You might have.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And I go, well, which one? And it won't work. All you'll do is talk and it'll be super verbose and it'll sound like you're talking to the internet. It's just not that. It doesn't work. And then everything else breaks. Which is the hardest thing. I don't think anyone else is doing what we're doing. We've got thousands of APIs now that we're able to call.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
You're able to get these, if you will, experts or agents, whatever you want to call them. It's not a real word. It's just being able to talk to each other at the right time. The invocation is like there's something invoked. And now the LLM at the bottom is arbitrating like, oh, what's she trying to do? What's he trying to do? Got it. Route it to the right model. Route it to the right expert.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Got it. This expert needs to talk to that expert. Give you an example if you want it. There's nothing simple about it. It's why you haven't seen it. It's why it doesn't exist outside of videos. So the biggest thing I needed was to not do a demo, but to use the product live. Meaning you can code a demo just to be a demo. Like, you know, it's code. Like for that moment, make it look like.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
type of quotes, but the principle was very, very clear. And this hasn't changed at Amazon, to be clear. The team's all in, like, we are gonna show the product. And that's what you saw.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Yeah, I think we do. Is that competitive? I think it is. I think it's hugely competitive. It's pretty easy to invoke a single API off an LL. I mean, not easy. I don't want to discount anything. But let's say the expert is a grounding expert. I'm going to ground a local info. We're in New York. I know everything about New York. I'm going to make sure this conversation stays within New York.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
calling one API, make sure you're grounded.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It's just my term.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
It's like if you, as a team, we talk this way. So yeah, I don't want to overstate it. I think some people call them agents. Some people call them APIs. Some people call them, I don't know, grounding to a certain experience maybe. Our challenge was it's not enough. We already have that. Yeah. I mean, it's deterministic today with Alexa, but we already have it.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
And so, meaning you can call a single API at a time, but then you get frustrated because you're like, I needed more than that. Let me give you an example. It's a simple one. Let's call photos agent or photos expert or just photos app. I mean, app's a bad word because you're not opening an app. But let's just say the photos expert and the music expert are both very important to this next example.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
The other day, I'm leaving the house. I have Alexa Plus, obviously. And I go, Alexa, do me a favor. Find all the photos of Mary, start a slideshow, and put music behind it. Okay. I just did a search command. I did a photos expert command. They have to talk to each other. He's looking for Mary. Slideshow. Got it. And then that expert has to call the music expert and basically say, play the music.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
All right. It does a phenomenal job. It does it in under two seconds and I get a slideshow. It's pretty cool. Music's playing. I'm about to leave the house. It automatically chose music in some playlist. And then I just said, you know, change the music to intern without, you know, re-invoking Alexa, which you saw yesterday if you were watching. It's very small.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
I just said, put something on that Mary would like. And then it switched it and I'm perfect. And I just walked out the door. Okay, that's an emotional moment. It's one of my favorite parts of the product. Like if you said, P, what's one of the things? I'm like, that's it. You're like, you're pulling emotion out of your, out of the things that matter most to you.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Mary wakes up, she comes in the kitchen, there's a slideshow playing and it's got music. She texts me, do you know Alexa's like on right now? I don't know what's happening. And I'm like, well, do you like it? You know, she's like, it's fun. I'm not turning it off. I'm like, well, I left, it was a message I left for you.
Decoder with Nilay Patel
Amazon’s Panos Panay on the long road to Alexa’s AI overhaul
Now the next step of that is to, you know, Alexa, leave a message for Mary when you see her. And she will. But these are all like they're multi-turned conversations, but they're also and statements. So when you have these basically these conjunctions coming together, you have the continuation of a statement because I just want to talk in natural language.