Bryan Cantrill
Appearances
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I noticed that you changed up the title. You did not like my... Okay, yeah, go ahead. Go ahead, explain yourself.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I feel like the last time you were on, I was getting a bunch of grief for dropping some dated references. This is an old part, just to be clear.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We're at Chrissy Field with one of these girls, and the other girl comes up.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So this was in 2021. So I assumed you were talking about the, like the, okay, this is, you're not talking about a chip that was made in like 1999. No, no, no, no, no.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Okay, so I've got so many questions about this Wikipedia page. So they were broken up. First of all, like nice use of the passive voice. Like they were, like what broke them up?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, why would someone... Obviously, I love my title. I love your title. Akui, there's a butt coming.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Oh, and you're like, okay, what? And I'm reminded a little bit of Robert on the spot over here. And like Robert, my four-year-old takes the hand of one of the girls, takes the hand of the other girl, and then the three of them all go running off together.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
God, thank you for opening up all of these doors. Did you see this? There's this documentary, The Rise of the Centaur, covering the early history of the company. It's like, okay, that's must-see TV. I mean...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, interesting. And the fact that it's not there at all really does, it is, as you say, it's a separate ISA. Yeah. Is that a good segue to the AVX-512 improvements on Tornax? I mean, I felt like going into this launch, I mean, I felt that's one of the headliners was the improvements to the AVX-512.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, really. There are some... The data path, obviously, is a big part of that, I assume. The fact that it's going from a 256-bit wide data path to a 550-bit wide data path. Or deeper than that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I'm like, all right, you know, go for it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And was that on, was that Sandy Bridge or Haswell, Robert, where it was like, I mean, AVX-512 has always been kind of had this kind of problematic property that if one thread starts using it, it kind of like browns out the rest of the part.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Absolutely, or weddings. I mean, who's to say that this, you know, who's to say that this won't carry into adulthood? Yeah, so I'm raising a bigamist. Anyway, I am, regardless of the title and who it was designed to appease, I am very, we're very excited to be talking about Turin and the Turin launch. This is AMD's latest part. George, thank you very much for joining us. Really appreciate it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Wow. And so you basically run this guest and it would like crater performance for the whole box for no actual approachable gain. Cause no one's actually using it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And it's really hard to have a feature like that, where if you use this feature, it has this, this kind of this adverse effect on the rest of the monkey's paw kind of feature.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's very hard to reason about the performance when you have these kind of problems. So, and the, and AMD is not needed. And I mean, you know, and it sounds like George, you've got the kind of the same question of like, you just come so accustomed to these kinds of intense compromises that come with AVX 512. It's kind of amazing that we can have it all.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and you would expect that to like, I mean, flip a little bit with Sierra Forest, but then you end up with this kind of this e-core business. And I mean, I think there's gonna be, I mean, there's gonna be things that are gonna be interesting over there, but touring is a very hard part to compete with. It's done a pretty good job across the board. Yeah. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so I think, George, I'm not sure you got to all three of the things that you had that were, so the frequency, the high number just in terms of the F cores and getting that up to five gigahertz, especially across all cores.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You are a repeat friend of Oxide. It's good to have you back.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I can tell you that we at Oxide also like that. Folks think about the kind of the rack level, because we're kind of left with the rack level power budget. And yeah, we definitely, it's nice to have a SKU stack that is not all sitting at 500 or 500 watts plus, right? I mean, I think of that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It has, and I think economically, too, it's interesting. These are expensive parts, but they can do so much, especially at that high core count level, when you're not having to sacrifice on what those cores can go do, that you can make it make economic sense, I think. Yeah. it's a big step function over where they've been.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So you had a great blog entry that I was excited to see it at the top of Hacker News over the weekend. Were you surprised by that?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, I think that we, you know, I think like a lot of people, like the Genoa SKU stack was a little less interesting for us.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, it is a good segue, Robert, to the kind of our thinking on Turin, because we, so George, as we're kind of thinking about our, I mean, our next gen sled is obviously a Turin-based sled. We did deliberately elect to kind of bypass Genoa and to intersect with Turin. Maybe describe our thinking a little bit there, Robert, and we've got
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Listen, pal, we've been rejected by enough VCs. I know a breakup male when I see it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But as we were thinking about Cosmo, Cosmo is our codename for our next-gen Turin-based sled, SP5-based sled. What were we thinking in terms of what groups we wanted to target and kind of like the trade-offs there in terms of flexibility?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. Eric, you want to describe some of the kind of the thinking there is as you're, as we were looking at the, what the PDN for this thing was going to look like.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And when you mean route, we're talking the physical layout.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and I feel that, like, also, Eric, I think whatever size Oxide becomes, even if we're selling millions of them, I will help slay the person that comes to your door on the 5. Because I feel like on so many of these parts, I mean, yes, they add up and it's part of the bomb. But, man, look at the cost of these CPUs is so much greater. And getting the flexibility is so much more important.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Oh, and I think, I mean, George, I'm sure your experience has been up there, but I do love a bunch of the reviews online of Turin cautioning people to not do exactly this. Like, by the way, your SP5 motherboard may not be able to take some of these SKUs.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You know, maybe the SEO, it may just be also that you just got a great article on a hot topic.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. And the, and having like, and because the problem too will be that if you push these things to the margin, I mean, you, you can get like misbehavior. It's not, it won't be as simple as like burning the house down.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yes, or if you recall us on the Tales from the Bring Up Lab episode where Eric was regaling us with some of our adventures on Gimlet, where our power was already pretty good, but we could not figure out why this chip would reset itself after 1.25 seconds. So we made our power even better. And Eric, my recollection of this was AMD's like, we have never seen power. This power's amazing.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You've got amazing margin. It can't be that. And sure enough, it was not that. It was firmware, of course.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, okay, I'm glad you brought up the video.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
George is kind of hilarious because we were not, the SVI2 was the protocol this thing uses, that the part uses to talk to the regulator. And the SDLE, which we had used, this great part from AMD that we had used to actually model all this stuff, as it turns out, didn't have a hard dependency on getting the ACK back from the controller when we set the voltage to a specific level.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
The part, as it turns out, wants to hear that ACK, as we learned. We learned that the hard way. We learned that the hardest, most time-consuming, most expensive possible way. But we did learn in the process.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I actually, Eric, I thought it was super interesting to learn that our power margins were really good on that because that was like a first natural line of attack was our power margins aren't like, that's why this thing is resetting because it is in a reset loop because our power's not good enough. But we actually learned in the process of doing that, like, no, no, this power's actually quite good.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
No, no, I'm glad you brought it up. Because those, you, the comments on that video were the nicest YouTube comments I've ever seen in my life.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, Eric, as you're kind of thinking about like, okay, so we need to, you know, there are things we need to do. And were you coming to the conclusion that, okay, I think we can make this all fit? I mean, as you're doing that kind of that trade-off?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And importantly, we're able to use the same regulator that we're, it's not like we're having to swap regulators to accommodate this. We were able to use the same Renaissance parts for this. And then the, and then from a thermal perspective, so we then, okay, so that's kind of like, all right, we've got the insurance there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And then from a thermal perspective, we also needed to do the, because you said we're not water cooling this thing. So, you know, can we, and at 500 Watts, I think we definitely know we will not be talking about how quiet the fans are because you'll be lucky if you can hear us talk over the fans. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We know the fans be cranking, but I think that the, I mean, we've done that and Doug and crew have done the model.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I didn't even know YouTube comments could be nice.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It is in particular, like it's real hard to max out the draw on your DIMM and the draw on your CPU at the same time without being mean spirited. I mean, you have to be really...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Substantially so. You're right. I should not be tempting the gods here.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Interesting is our take. I mean, it's like definitely interesting. I mean, I think that we would love to be able to get some parts. The draw does become an issue back there for us. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, what is going on? This is, this can't be a YouTube video.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's not YouTube. There's something, this thing has fallen into some alternate reality and like, and the comments are all like, you know, thanks for all of your diligent work. And you know, I just, I, I love, I mean, it just, it's great. God, like we talk about lightning in a bottle.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, and I think that, you know, and Ellie in the chat is saying, well, I don't think people realize how, you know, how restrictive the oxide power budget is. And I don't necessarily, it is restrictive. It's more that we are really, we are taking that rack scale approach. And so we're kind of the ones that are like always adding up the visa bill.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And, you know, when you have, you know, 30, 40, 50, 60 watts, 70 watts, 80 watts in your neck, like that adds up in a hurry. And yes, you can offset it elsewhere. But what we're trying to do is try to get you the maximum amount of useful work out of that rack scale power budget.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And, you know, by being the ones that are doing that, we're the ones that are, you know, sometimes having to deliver some tough messages to folks about like, we like this, this is interesting, but it's drawing way, way, way too much power.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We love meeting Oxide. We're very excited about Oxide. Okay. Yeah. Go on.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, I mean, we love, we're huge P4 fans as folks know. And I think we've got... Actually, if folks can see it out there, we've got an exciting announcement in terms of Excite Labs and using their part as our next-gen switch, and it's P4 programmability. So we're really excited. We've been using P4 on the switch. We're going to continue to do that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And using P4 or programmability at the NIC, we're really interested in. But it's got to happen in a way that we can accommodate everything else we need to go do with the rack. So, George, a long answer to your question there, but it's interesting for sure.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. Which is something that's going to be, and that is something that's going to be like, we're definitely interested in.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, I think so. I mean, that's not going to intersect our first cut here of Cosmo. But no, we're really, really interested in it. And again, great to see that P4 programmability.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
There were some moments where it felt like we were a bit of a lonely voice, but I think other folks are beginning to realize, and I think as the hyperscalers themselves have known, that having that network programmability is really essential.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Robert, you'll do the honors. I'm happy to give my P4 spiel.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And having that higher level of abstraction really allows you to kind of express something programmatically that they can then use hardware resources efficiently. Our big challenge has been working with vendors to give us a substrate upon which we can build a true P4 compiler. Honestly, the biggest challenge in that part of the ecosystem has been the proprietariness of the compilers.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, George, I'll tell you that one thing that will be a factor for us as we're looking at kind of a P4-based NIC is what we have been looking for is what is that kind of x86-like substrate going to be? Something that is a documented, committed ISA that we can write our P4 compiler against.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So what we are not looking for, because we are coming out of a bad relationship in this regard, what we are not looking for is kind of a proprietary compiler. We really... uh, want to, and we have written our own P4 compiler for, we use our, we developed our own P4 compiler and have open sourced it, um, X4C, um, for purposes of just doing development, software development and testing and so on.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But we really want to take that and have that, use that to actually, uh, to compile, uh, for these parts for both the switch for sure. Um, and then ultimately the NIC, um, would definitely be our vision for where it's going.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
George, I really appreciate your sensitivity of taking us into the kind of the grieving room and your bedside manner here is really exemplary. I was really feeling you kind of passing the tissues to us as you, as you really felt our loss. I really appreciate that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, are we in charge of AMD now? Because we got lots of ideas.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so you think that this has given the internet some gratitude? You've managed to domesticate the internet.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. And, you know, so yeah, I mean, absolutely. We would be great for them to do that. Although that said, we are, you know, they need to do it in the right way. And the right way from our perspective is really establishing a substrate that people can build an open ecosystem on top of. And this is something that, you know,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And I always, I find vexing is you would think if you make hardware, it's enormously in your best interest to allow many, many software stacks to bloom, um, by having a well-documented committed interface. But, um, They really don't. It's a challenge, I would say. I wouldn't say they don't, it's too reductive. I think that they fight their own instincts on it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so we're very excited with Excite Labs. Again, you can see our announcement today or their announcement today, actually, but it features Oxide for sure. And we definitely see eye to eye with them on their X2. We're looking forward to to moving forward with that part. And we think that there should be, we want to see programmable networking everywhere.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We want to see this open ecosystem everywhere. The, on the note of like the kind of the lowest levels of the platform that can be hard to get at. So George, you may recall that we have no bias in our system. So there is no AGISA, there is no AMI bias. So when we buy us, buy us, am I? Nailed it. Oh, thank you. We have lots of bias. Up and at them, up and at them. Uh, better.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, do you wake up your kids that way? Adam, as, as when you're at castle, we're near wolf castle saying up and at them.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Feels very loaded. I think my kids have definitely gotten sick of that particular Simpsons reference. Rainier Wolf Castle, no longer welcome in our abode. But we have no BIOS. And so Robert, that lowest level platform enablement has fallen to us. What are some of the differences in Turin from, or even from Genoa, but then from Milan?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And look, it was like if there was a YouTube video that we're going to start having nice comments on, that was a good one to start on. That was a great video, went in depth. I love that you kind of had the surprise ending where you set a world record in your hotel room. Let's start with there. What was that benchmark that you were running? And you were running that on Turin, obviously.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, those are effectively hidden cores, right?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
There used to be four. Yeah, interesting. Yeah. And that is presumably, so you are, you're just increasing the parallelism there. And I mean, is that the.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so but but those kind of changes, which if we were at some level of software, that's an implementation detail you don't need to see. But at the level of software we're at, like you actually need to go accommodate those differences.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And could you describe what device training is? Like, why does a link need to be trained? What is that?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Right, when they misbehave. So there's a lot of low-level work that we need to go do. And how do we, in terms of like, we don't yet have, I mean, Eric and Nathaniel and crew are working on Cosmo as we speak, kind of finishing up Cosmo. How do we work on that before we have our own board in hand?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Is it too much to hope that someone has a sense of humor that named that thing volcano?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You had, like, Onyx. Yeah, no sense of humor. I just like the idea that, like, Inferno, you're going with the... So, yeah, so we got the Volcano reference platform from AMD.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And then Nathaniel, do you want to talk about kind of how we use those dev platforms? Because we've got a little, a great little board there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, we should talk about eSpy because this is definitely a difference in Turin.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Okay. Go on. So you love my title. My title was Unshrouding Turin.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That's right. And spy, as my kids would say, spy has no chill. Spy, you need to give it what it needs. You're set on the clock. There's no clock stretching in spy. And so spy interposition becomes a real nightmare because you need to get everything you need to get done, you need to get done in that one clock cycle. It's like, yeah, that's really hard.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and just on the training times, without that, without kind of, and so dim training where you're trying to find the search for the constants that are going to allow you to not have interference when you're talking to these dims, that search can take a long time. And Robert, how long does it take you when you've got the first, that first genoa that you've got-
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, it was. I've put it out of my brain. Yeah, George, I don't know if you've seen some long boot times, but DDR5 takes a long time to train.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Adam apparently really prefers your pronunciation, George, to mine.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
All right, I'll do my Duolingo with George on there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It was pretty great. It was pretty great. And I love that you're like kind of wrapping it up. You're like, no, no, actually, wait a minute. Hold on. I'm not being, this laptop just being handed to me.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It was, yeah, we definitely, and I'm not sure, you know, maybe we're, I guess we're a bit unusual on this, but boy, we were unusual. We need the, yeah, that, So, George, we were using, just as Nathaniel mentioned, we were using one of the UARTs as the IPCC, which is the interprocessor, right? Communication channel.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But this is our protocol for the socket, the host OS, to be able to speak to the SP, which is our replacement for the BMC.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
There's more. Yeah. Okay. So, and obviously that setting, and it's helpful to know this is a very compute intensive workload. Because one of the things that I think that we've heard from a bunch of folks is this thing is so much more compute that now you've got to really ask questions about balance of the system and memory bandwidth and so on. So I want to get into all that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It was very slow. And our friends at AMD, fortunately got us a, a fix to the PSP to operate at three megabod. And that was very, it was life changing for, for, I mean, I know for 30, 30 X makes a big difference for RFK. 30 X means when, when it's 30 minutes, that 30 X is like a real actionable human 30 X. Not all 30 Xs are the same.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And when something takes 30 minutes, taking 30 X off of that is a, is a big deal.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, the go signal still takes several minutes to transmit.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I'm not even joking. This sounds like a dream that I would have that I would describe to Adam.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I'm sure. I would like to believe that original designers and Oxide and Friends listeners would be like, oh God, that was still in use? That was supposed to be for a weekend. That was not supposed to be. That was a temporary fix. Totally.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so Nathaniel, maybe worth elaborating a little bit why the three megabod is so actionable for us beyond just the margining and the Mbis results, because this actually ends up becoming, because this is our conduit for the SP to talk to the host CPU, we use this in the recovery path. So like if you've got a system that can't talk to anything else, it's gonna load its image
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, I guess one thing I would kind of ask you from the top, um, just what is your kind of top takeaway from the Turin launch? And was there anything that surprised you? Was there anything that you either didn't know what was coming or didn't know the kind of magnitude or you're still, yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But I noticed the title here is Benvenuto a Torino.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It takes a minute as the kids say, but it actually takes like, we're actually doing two different things, Nathaniel.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And actually, and so George, in all honesty, like part of the rationale for this is to get us out of those moments of terror when you are flashing a bias and you have gotten often no recourse if that, if that goes sideways. And so this gets us out of that because we know that the system at the, at the absolute lowest layers of the system, we can get the system to, to be able to boot. And we,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
it gives us much more control over the reliability of the system, upgradability of the system, manageability of the system. That's how we're able to get... Oxide rack can arrive, power on, and get going and provision VMs in minutes instead of days, months, weeks, whatever.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, I feel like this is like, we're like the sewer people and you, you know, like you get to, I mean, you've got this glorious palace in terms of the cores that have been built. And meanwhile, it's like the sewer people are happy about not being at three megabot. Like what's going on. No, it's, it's a big deal down here in the sewer.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so we are all in favor. So we have been, and actually it was funny because I actually first heard Turin the code name Turin when it was accidentally blurted out on one of those OpenSeal calls. I'm like, okay, what is Turin? And I remember asking Robert, like, that's a city in Italy, so it must be the next thing, but we hadn't heard of it yet.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And OpenSeal was going to intersect with Turin, which of course, when we were first hearing that, it's like, oh my God, that just feels like Buck Rogers. It's like in the year 2041. But of course Torrent is not here. And that work we are very, very supportive of. We are not actually using any of that because it's a different model.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's kind of going to, it's still a traditional model of a bootloader that's going to effectively make the system look like it's gone backwards or send the system backwards to boot a host operating system. And we've got this staged approach where we are running a single operating system the entire time.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So it doesn't fit our model, but we're extremely supportive of it because we believe that we want these lowest levels of the system to be completely documented. And we want there to be room for many different approaches. And so I think that we're very supportive of OpenSo in that regard.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and it's very nice to be able to go compare nodes, especially when things aren't working. Helpful to have multiple implementations out there. And I also think that the model for AGISA, the programming model, makes it very difficult to reason about the overall system. This is where Robert's eyes are going to start to twitch because...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Robert spent a lot of time in the absence of documentation having to really understand what this code was doing.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, and that's it. I mean, when you have these kind of, this is why it's so healthy to have different software ecosystems on the same hardware, because you don't want things to be working by accident. You want them to be, and it's, you want things to be well-documented and with well-committed abstractions. And failing that, it's good to have the software out there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So it's, no, it's been, George, it's been good. And we're, I think, excited to continue to see that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
um the uh some of the other like lowest level i lowest level differences on turin um we you mentioned the dims for channel and we we kind of had a um a fork in the road in front of us in terms of two dims for channel two dpc versus one one dpc and there's a trade-off there to be made and robert what was the i mean
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That is crazy. I feel like the last time we were really, it's been a minute since we've seen clocks that high from anywhere, I feel. I feel it's been like, I mean, IBM was hitting it with power.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
When I think just in general, when we had a trade-off where we'd have to give up memory latency, we have always felt that memory latency is really important. You want to get maximum. You want to minimize memory latency, and you don't want to take a hit there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
probably could make that you could probably make convince yourself that that is actually worthwhile but you know 5200 4400 that's a that's a that's a lot farther from 6000 it's a big yeah right it's a big big chunk to take out and in terms of mr dims because this is a domain where you know intel is still uh basically the all i mean they because we're pretty standard on mr dims right let me let me let me get on my soapbox for about 30
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, you know, Georgia, what I love about Oxide and Friends is when it comes to the soapbox of a dims being not being per JDEC standard. It's actually a line of a soapbox here at Oxide and Friends. There's actually this is this is the because this is a soapbox. Robert, you know, the soapbox you've been on the soapbox.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And it's, yeah, it's frustrating, but they, but MRDMs, I think when the, when they are JDAX standard, right, because they will be.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, well, I think that, you know, one of our kind of big revelations, and again, this is not due to us, I think the other hyperscalers done this as well, but that the...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
you actually, the way to have maximal density is not necessarily to have maximal physical density, that you want to open up some room for airflow and you can actually get higher density by having, by using a little bit more space and being, you know, where the rack is nine feet tall. And so we, you know, using some of that, trying to use some of that space to get higher density.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, the, uh, actually, can you just talk about bacterial vias for a second, Eric, just because you mentioned in the chat, I don't know, George, do you know about bacterial vias? This is, this is truly amazing stuff.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Oh, put Robert on the spot. Go ahead, Robert. There we go. What an all pro. You know what this reminds me of? This reminds me of when my now 20-year-old was four. We understood from one of his friends in the neighborhood that he and this girl were going to get married. And they were like, okay, that seems like a little bit heavy for four. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I just love the fact that we're going to take a drill to the underside of the board. It does feel like a Adam Leventhal PCB engineer kind of approach to this of like, here, hold on, pass me the drill. I'll take this. Exactly. I need a drill and a running start, and we'll take care of it. Yeah, I'll fix your signal integrity issue with my drill. I'll fix it real good with this here drill.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But it is a total precision, and it is 25 microns. Just amazing, Eric, that this is. And then we've got simulation tools. I mean, how do we kind of figure out where this needs to be done? And this is…
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so let's elaborate on this a little bit because this is, I think, a really interesting point. So we're seeing this from Intel too, right, where once you get the density up to a certain level, you've got to make some compromises. But the compromises that AMD seems to be making are much less than the compromises you're seeing. I mean, the Zen 5C cores are – they're still Zen 5 cores.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And I just love the fact that you got to think of like, you know, who wants that? Like that thing in the SKU stack. It's like, oh, the engineer that actually like.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Exactly. So we got to get that SKU for Eric and Tom and the other folks that are running these simulations.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, we got to get that for you. Not a cheap part, by the way, but still. No. We had the ANSYS folks on. I remember we had the ANSYS along with Tom on talking about our use of simulation, which was another great episode. I really enjoyed talking to those folks. And you just learn about the physicality of the stuff just...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
blows me away and like i feel i mean i don't you feel bad that we end up running like our dumb software on top of this stuff at the end of the day i just feel like we're kind of yeah seriously well i've been taking all these gigahertz for granted too and just like the level of complexity underpinning this is is bananas it's like we back for this thing to run php it's a good look yes sorry yes
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Totally, totally. But it's just amazing. And this part is a great part. You know, I think that we're, you know, I think, George, we were really excited to see, I mean, obviously your in-depth review was terrific. But I mean, I think, George, from your perspective, like this is a part that has really hit the mark in kind of like every dimension it feels like.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And I think that we, we are going to be, I think, you know, another thing that we're looking at is partnering with Murata and getting for those folks that actually do want to go more than 15 KW for the rack, which was our original design target, but which felt very aggressive in 2019. But then, you know, I think it feels like Nvidia is like, that's like two GPUs now for you. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I can get that in for you. Exactly. I can get that to you.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
we are, are, uh, we'll be for folks that can go be up above 15 KW for the rack. We'll be able to go do that. And, uh, it's, it's, it's again, it won't necessarily be quiet. Um, but we, uh, we think we're going to be able to air cool that. Um, and those, that's where you get that, you get that kind of seven X consolidation that AMD was talking about. Um, and I think that there's, yeah. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and I think that we're excited for the AI workloads too, and I think they're going to get a nice pop from AVX 512, certainly the 512-bit data path there, and you're going to see a lot of those. There are nice pops to be had, we think. But you're right, it was not just AI. There are, as it turns out, other... We also need the workloads to simulate the computer for the AI, as it turns out.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, we think so too. And I think that, you know, I think unlike with our first gen where it was kind of every, we only had one SKU, the 7713P, we're going to allow for some flexibility for Oxide customers inside that SKU stack. We're excited to kind of extend that and then do some of the work around dynamic power control.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We got a bunch of ideas on how, you know, we've got the right foundation to go with. actually manage power holistically across the rack and, and use some of the, there's all, we got a lot of stuff to go, a lot of knobs to turn. Um, and I think it's going to yield a pretty great product. I mean, the hats off to AMD for sticking the landing.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, we are definitely wedded to AMD in a lot of ways in terms of our lowest level of platform initialization and so on. So we're always relieved when they execute well.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, we talked about the APU. Yeah, boy, we're hitting all the sympathy cards here. Hold on. Hand me the other sympathy card now.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
When you were asking about the APU, did the oxide people put you up to this?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That's great. We've asked for... Yeah, that'd be great. You will deserve responsibility for an MI300A dev kit. We'd love it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
George, I don't know what we're going to do for you if an MI300A dev kit is for sale on Newegg, but we're going to do something very, very nice for you. I don't know when it's going to be yet. It is going to be... That would be great. We get George a whole wheel of his favorite cheese?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That's right. For sure. But I totally agree. And yeah, we definitely noticed that the... You know, we have brought up APUs so frequently with them. I feel we've kind of overstayed our welcome with respect to that. Sort of like, you know what? We're going to let you guys say the next thing about APUs. We're going to stop telling you how much we love APUs and we'll let you do the next thing.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Or we'll let George do it for us, which is great. Much more effective. Much more effective. Well, we'll say, yeah, again, if we get an MI300A dev kit on Newegg, it's going to be, we're going to have to do, it's going to be something spectacular. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, you know, and I think that you, I think, and just actually honestly with OpenSeal, I mean, I think one of the things that we really like about AMD is it's a company that doesn't like, it listens. I'd like they kind of, they know what the right direction is. It takes them a while to get there sometimes because it's,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's a big vessel. And, but, you know, I mean, I remember when we were getting Naples stood up way back in the day and, you know, there was a lot that was still needed to be done. But you could see that like, okay, this is not a trajectory that's really interesting. And then with Rome, it's like, okay, this has just got a lot more interesting. And it was clear to us in 2019, Robert, that they were,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
they were on a trajectory or they were surpassing Intel effectively. And then obviously with Milan and, and with Genoa now turn, I mean, it's like, we've seen them like continue to execute, execute, execute. And so, yeah, let's go, let's keep Adam on the APU side.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Absolutely. Yeah. No, love it. And I totally agree.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yes. So we, we love the parts and, and, and we're going to be, we want you to be got some ideas for things to be even better, but touring is a great part. We're really excited about it. And George, thank you very much for, for joining us. It's been great to have. Thank you for having me. Oh, yeah, it's great to have the team. I mean, and Nathaniel and Eric here and Aaron as well.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's been, and obviously Robert here in the studio with me. And then Adam, of course, to correct my pronunciations and to inform me that my running start is not quite big enough on the back drilling. Um, the, um, you know, I think it's been, uh, we're really excited about our forthcoming.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So I don't think it's, we're excited about is of course, you're going to be able to take the, taking a touring sled and putting it into a, an unused cubby in an, in a, an oxide rack that has Mulholland sleds and just have the whole thing just work. So we're really excited about that. Um, And onward, great part. And George, thanks again. Really, really appreciate it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Speaking of excitement, we do have one very exciting announcement.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's going to be an unconference. If you were a DTrace user, you want to come hang out at Oxide. We are going to charge you for tickets. It's not going to charge you too much money, but we do have to charge you something. Otherwise, it'll be immediately consumed by teenagers. Teenagers will consume every ticket if we don't charge you anything. Um, very limited supply.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, so, uh, hop in there if you're interested in joining us. Um, yeah, it's going to be out of my, I'm, I'm really excited for this. It's gonna be fun.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And it's going to, yeah, it's going to have a different complexion and flavor this year for sure. It's going to be a lot of fun. So I'm looking forward to it. That's for sure. That's right. I know Robert, you've got to, everyone is just like, okay, what do I need to get done now before I've got until December 11th to get said to get my, but we got a lot of things to talk about. So.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's going to be fun. So join us, detrace.com 2024. And I will not violate any more copyrights by humming.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Exactly. Awesome. All right. Well, George, thanks again. Thank you, everybody. And yes, see you at detrace.com 2024.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And the fact that you've got this kind of SKU stack now that is kind of a uniform SKU stack where you can start to really make some interesting trade-offs as you look at workload. And it just, it feels like, you know, and Robert here is our resident SKU stackologist. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
which definitely, I mean, there were times with Intel where you've got like these three different, where you had like gold, silver, and platinum, and you've got- Also bronze. Bronze, right. And it just, it did require like you to get a postdoc to figure out like which part you want. And Robert, what do you make of this Turin SKU stack?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, it seems like it's a pretty clean SKU stack in terms of making different trade-offs as you go up and down it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. I mean, certainly. And I think that, that, you know, when on the Intel side, when you go from these P cores to E cores, Adam, you're going from the P is for performance and the E is for efficiency. You're like, Oh, as, uh, uh,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, because in particular, you don't have AVX-512 on those.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And we were talking to another parent at the preschool, and she was saying that her daughter and Tobin were going to get married. I'm like, God, this kid's a real, gets around, real gigolo here. Well, as long as, you know, he's got his, I guess he's got, you know, when you're a four-year-old, I guess you have a playmate in every port. And we are at the beach.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, and then it's like, so it's really tough too, because if you are, you know, in our position of like, we are selling a rack scale computer to someone who is making compute available to their kind of customers, right? We're asking, if you have to ask them, well, are you using AVX 512? They'll be like, I don't know. I have to go ask my users that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And it makes it really hard, and you're kind of at this big fork in the road. So to be able to have a...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
to not have to give up i mean just as you said george to not have to compromise on isa and to get the same isa everywhere and yes you may be giving up you know you're making some trade-offs in terms of max frequency and so on but you're not like dropping off your trade-off is an area right right yeah you have to spend that area um but the
Oxide and Friends
Querying Metrics with OxQL
Right, because, I mean, they can't figure out three hours because they got another one of these to do in another half an hour. But they're just kind of rolling them through. But it was also... Anyway, it was weird. It was... And the RFDs, of course, I was feeding it were all the RFDs that were... going to be talking about today.
Oxide and Friends
Querying Metrics with OxQL
As you point out, the amount of work, it's not clear that it's who we're saving work for. by doing something that's like lowest common denominator that I don't know where, I mean, and especially, I also think that like even DSL even, yes, I mean, this is a DSL, but this is a very little language, OXQL. It's not like... I mean, you're not like learning Haskell or something here.
Oxide and Friends
Querying Metrics with OxQL
I mean, this is just like – I mean, I don't know. I feel that – I mean, I get the kind of the – we're obviously careful about that. But I think Benny made an important point that this was not our first conclusion. Our first conclusion was like, hey, we should do a query language. Good thing – because there is nothing else out there, we will invent our own. It was more like –
Oxide and Friends
Querying Metrics with OxQL
So I was feeding it RFD 463, getting the podcast before the podcast, which I dropped into the chat there. So if you want to hear synthetic hosts describe, with also like weird taglines. Did you notice that? I've got like, stay curious until next time. It's like, stay curious. Okay. Stay. Anyway, it was very weird. But Ben, welcome. Thank you.
Oxide and Friends
Querying Metrics with OxQL
Let's try to make everything else work. And coming to the conclusion that we're just having to contort ourselves too much. And it's actually very liberating to be able to do our own DSL.
Oxide and Friends
Querying Metrics with OxQL
And it's not like, I mean, this was not, I mean, not to downplay the amount of work involved, but this is also like you're using a bunch of tooling that makes it really much easier to develop a DSL than maybe it has been historically. Yeah.
Oxide and Friends
Querying Metrics with OxQL
It just feels like also with the, and again, I don't know how much of like, this is just like the Unix having seeped into my own DNA or is Unix kind of an outgrowing of the DNA that exists in all of us, right? I've got no way of actually differentiating those two, but it does feel like the pipe syntax to me feels pretty clear about intent.
Oxide and Friends
Querying Metrics with OxQL
And it feels like we can also then do a lot of things on the back end to optimize that. Because you're being so clear about your intent and you're not having to do unholy things, we can actually make sure that we can optimize those use cases. You can use an entirely different ClickHouse feature. Just some of the things that we were brainstorming about.
Oxide and Friends
Querying Metrics with OxQL
So I feel we've kind of stepped on an exposed nerve ending for the internet here. I got a lot of people like this was, am I wrong? Yeah. To describe it as upsetting seems a bit too strong, but a lot of people are like, a lot of people have issues with this.
Oxide and Friends
Querying Metrics with OxQL
What is the point of you're not actually getting SQL compatibility when you're not doing all these things? All these things have no relevance in this specific domain. There's a reason we have domain-specific languages. I just cannot emphasize this term enough because I think it is a great strength where we can create little languages easily.
Oxide and Friends
Querying Metrics with OxQL
I think we should not be resistant to that because I think the kind of compatibility that you have by doing, let's say, SQL, as you're mentioning, it's like a false compatibility. It gives you the wrong intuition for the system. And it's like, sorry, this is not what the system is actually going to do underneath it. Yeah. you know what we're trying to do.
Oxide and Friends
Querying Metrics with OxQL
And I, the other, another question that kind of came up in the chat is like, wait a minute, it's okay. So you said that like all anyone's going to do with this is like slurp it out. So like, why wouldn't you just use some other protocol that people already know? It's like, well, that's all because that's all a customer might want to do with it. We want to do a lot more than just slurp it out.
Oxide and Friends
Querying Metrics with OxQL
We want to actually go and use it dynamically and, and be able to actually look at, look at the rack and ask questions of it. And so for us, we want something that is much more tightly tailored to that.
Oxide and Friends
Querying Metrics with OxQL
Definitely skeptical. So we're talking about RFD 463, which is OXQL, which our synthetic podcast pronounced Oxquill over and over and over again. Ben, we've never pronounced it Oxquill, I think, right? I think it's OXQL, but now I can't unhear Oxquill, unfortunately.
Oxide and Friends
Querying Metrics with OxQL
Okay, with post-mortem debugging, it's taken me quite literally decades, but I'm willing to acknowledge, okay, I am somehow an outlier with respect to society. This is some sort of software kink that I have with respect to being willing to debug a system from a static state. It's static in memory state. So, okay, fine, but weird. But histograms, really? Are we histogram radicals?
Oxide and Friends
Querying Metrics with OxQL
Yeah, for sure. And I think if we want to actually, now that I know that this is apparently a strange idea around histograms, you chase this through to Bonwick and Lockstat using histograms for looking at lock times, spend times, block times, and looking at that actual distribution of data.
Oxide and Friends
Querying Metrics with OxQL
And so, I mean, honestly, it was when part of the reason we have aggregation as a first-class operation in D-Trace was because of our eye on Lockstat and replacing it. It's like, okay, this is important, this idea of getting a, what the distribution of data looks like.
Oxide and Friends
Querying Metrics with OxQL
And so, you know, I can't really thought about that, that, that, you know, maybe that was, that felt very commonsensical, but maybe I'm not giving Bono much enough credit. Maybe that was very iconoclastic to be thinking in terms of the distribution of data. I mean, he was a stats concentrator. It was a stats grad student.
Oxide and Friends
Querying Metrics with OxQL
I do feel they're right. I think that you're right. That is recent. And I guess we've always just kind of run with sets of people for whom understanding the distribution of data has always been really, really important. And it just feels very natural that that would be a first-class operation.
Oxide and Friends
Querying Metrics with OxQL
It does sound... It sounds like bovine Dayquil. Our development of a DSL, maybe this is in the way I'm phrasing it. Maybe I was too clickbaity with the way I phrased it. Is that a problem? Or something? What do you mean?
Oxide and Friends
Querying Metrics with OxQL
Yeah, when also if you look at the average, you don't actually understand that much about your data. And you think you do. It kind of gives you the sense of like, oh, this is what my data looks like. And it's like, well, you know, maybe.
Oxide and Friends
Querying Metrics with OxQL
But you may want to get just a little more fidelity in what that actual, what the distribution looks like before you conclude that that's what your data looks like.
Oxide and Friends
Querying Metrics with OxQL
And so obviously, ClickHouse was a natural fit in part because of the way they thought about the problem.
Oxide and Friends
Querying Metrics with OxQL
No, you know, it's like, you know, I'm kind of throwing out a little bait for the podcast. So my tweet is, when is a new query language necessary? And clearly there's a decent portion of the internet for whom the answer to that question is actually never.
Oxide and Friends
Querying Metrics with OxQL
And we have not scratched the surface of that kind of stuff that we can go do.
Oxide and Friends
Querying Metrics with OxQL
And by critical wrong, but part of the appeal of a DSL here is the ability to add some functionality that would actually help us express some of that that we can get out of ClickHouse. Yeah, that's right. Through a consumer of that.
Oxide and Friends
Querying Metrics with OxQL
So a question that I think may be very catalyzing for people to answer to is, is it a fair assumption that the main client of raw OXQL is Oxide-provided tooling, dashboards, alerting, et cetera? That's what that question is.
Oxide and Friends
Querying Metrics with OxQL
Exactly. It's like, or it's just like, listen, if you're going to invent a new query language, like that's fine, but you need to get rid of one of these others. Like you need... And in particular, there was a quote tweet from Andy Pavlo, a distributed systems researcher at CMU. The quote tweet starts, Brian is brilliant, comma. You're like, oh, no. No, that's not good. No, that's a bad start.
Oxide and Friends
Querying Metrics with OxQL
And you said the console, also I'd like to point out the CLI also has the ability to... Let's not sleep on the CLI's ability to visualize data.
Oxide and Friends
Querying Metrics with OxQL
Basically, if someone is insulting me, I know they may be warming up to agree with me. It's the old, actually, I agree with Brian, quote. they're kind of like trying to establish their bona fides before they... They don't want to imply that they would agree with anything I say, but they are... Listen, Brian is often... It's Kelsey's line, right? I love Kelsey's line.
Oxide and Friends
Querying Metrics with OxQL
And humility dashboard. I'm a little hurt that humility dashboard is not coming up here. Honestly, I was waiting for someone else to mention it, but apparently no one was going to, you know, it's not going to.
Oxide and Friends
Querying Metrics with OxQL
Listen, I got to plug my own. Around here, you got to... You know what? I'm going to have the AI, my AI overlords, generate a podcast full of praise for Humility Dashboard. I think I need a little pick-me-up from my bots. But Humility Dashboard also is... This allows us to talk to a service processor and graph all of our environmentals, and it's been very... It's been great. I love Ratatouille.
Oxide and Friends
Querying Metrics with OxQL
Yes. Yeah, it is. It has been. It's been great. And it's been great. I also love writing software that the double E's use, because just, you know, they have lived such a tortured existence with respect to software. It's very nice when they can be delighted by software. I feel that the Ws live a hard life.
Oxide and Friends
Querying Metrics with OxQL
And so when software can delight them, when they're not using some vendor-specific Windows goober that they need to program apart or something, you can actually give them something delightful. It's really great. They don't get nice things. It's really true. They don't get nice things. That's what I'm trying to say.
Oxide and Friends
Querying Metrics with OxQL
They don't get nice things, and as a result, their standards are very low, and you can do very little work and give them something nice, and they are just
Oxide and Friends
Querying Metrics with OxQL
filled with praise it's great yeah um so yeah ratatouille's been that's been that's been fun in the end uh folks are asking if the code for that is available that is all open source all that stuff open source i think i mean it's all open source but all of oxql and the dashboard are well the dashboard is in um the oxal oxide cli that adam linked a minute ago um
Oxide and Friends
Querying Metrics with OxQL
Can you speak a little bit to the implementation of the DSL, by the way? Assuming we've gotten people over the hump of like, that we've got the right to implement a DSL here. I'm not sure that we've got everyone on board with that. But you know what? Just bear with us. And can you get into kind of the mechanics of building that?
Oxide and Friends
Querying Metrics with OxQL
Brian is wrong some of the time, but not this time. I love that. But this is like, Brian is brilliant, comma, is like, this is the end of this. This is going straight into the ditch. But it seems misguided for a hardware company to create a custom query language that no other tool supports. If you don't want SQL, you could have used PromQL or PromQL.
Oxide and Friends
Querying Metrics with OxQL
Let's face it, Adam, you and I would, left to our own devices, we would make this thing be an absolute navel-gazer. We'd be like, we are dedicating all of the resources of this rack to gazing at itself. Like Narcissus in the pool, it is enraptured by its own reflection, using all of its storage to store thermal data about itself.
Oxide and Friends
Querying Metrics with OxQL
Yeah, I mean, look, let's just agree that we're glad that Ben is here to point out that they paid for it, so they should be able to use it for the things that they care about. Yeah, and I mean, this is where ClickHouse is just outrageously efficient with the way it stores things. It's mind-bending.
Oxide and Friends
Querying Metrics with OxQL
Yeah, of course. This is an important observation. When Just enters an engineering discussion, yeah, you're right. You're like, okay, cocked eyebrow. Just often does do a lot of heavy lifting.
Oxide and Friends
Querying Metrics with OxQL
Is this gorilla as in like Diane Fossey or gorilla as in like insurgency?
Oxide and Friends
Querying Metrics with OxQL
So to get it back. Um, So we, just to forgive the context before the context, the friend of the pod, Simon Wilson, had this blog entry that was on Hacker News describing this new feature from Google, from their Notebook LM, where they can generate podcasts from arbitrary material. And I've been entertaining myself by sending it RFDs and creating single podcast episodes on RFDs.
Oxide and Friends
Querying Metrics with OxQL
We're talking about the Diane Fossey variant of gorilla, not actually the warfare variant of gorilla.
Oxide and Friends
Querying Metrics with OxQL
I mean, it's kind of, it's kind of an O, not a U. It's not a, not gorilla.
Oxide and Friends
Querying Metrics with OxQL
You do not pronounce those two things separately. I do not. Do you? Okay, thank God.
Oxide and Friends
Querying Metrics with OxQL
Yeah, absolutely. Another question that came by was about getting how we are thinking about notifications when data does become abnormal or some definition of abnormal.
Oxide and Friends
Querying Metrics with OxQL
And for the things, I mean, are we kind of thinking about, because in terms of like the amount of OXQL that's user-facing, I would assume for some of these things, we'll create new endpoints that will basically be distilled into OXQL queries, but will actually be an endpoint that will be its own endpoint.
Oxide and Friends
Querying Metrics with OxQL
But I think... I'm very grateful for this approach. Speaking strictly selfishly, because when Eliza hooked all of those lower-level environmental metrics up, it just meant that it automatically... popped out at the top.
Oxide and Friends
Querying Metrics with OxQL
Do people think we're creating our own instruction set architecture? I was like... I just get confused why this is engendering this kind of reaction, especially for like all of the stuff we've done. We've done like so much of our own. I mean, we've gone our own way so many times over. It's like, this is the one that is a bridge too far. It's like, listen, you guys.
Oxide and Friends
Querying Metrics with OxQL
So this is actually a really important point, I think, is that OXQL kind of gives us the foundation to figure out what you would want to distill into a perhaps even more limited abstraction. And it would be a lot harder to do that with using these other kind of query languages that felt like a much poorer fit for the underlying data model.
Oxide and Friends
Querying Metrics with OxQL
We have our own switch. We developed our own switch. Yeah. Like that, although that's fine. No, that makes sense. You know, you're doing your, yeah, you're an embedded operating system, our own host operating system.
Oxide and Friends
Querying Metrics with OxQL
Yes, it will. As much as you have. When I was saying earlier that ClickHouse likes to eat, that's what... It does.
Oxide and Friends
Querying Metrics with OxQL
never stall for what we made. And then like every operations, a table scan, what have we made table scans really, really, really, really fast.
Oxide and Friends
Querying Metrics with OxQL
It is. And it's like, but query language, no. No, now you've gone too far.
Oxide and Friends
Querying Metrics with OxQL
And then in terms of, I mean, Ben, you've elaborated a bunch of directions that we want to go take this thing. I think that we want to, I mean, at the moment, I'm just like... it's great to actually have a bunch of data in this that we can go mess around with. Yes. And actually go learn the kinds of things that we want to go do. We know that ClickHouse gives us the right foundation.
Oxide and Friends
Querying Metrics with OxQL
We think OXQL gives us the right foundation. And then what are the things that we want to go add either additions to OXQL we want to make, and then especially applications we want to build on top of this to allow one to make better sense of kind of rack-level and ultimately multi-rack-level data.
Oxide and Friends
Querying Metrics with OxQL
I mean, in many ways, our decisions around ClickHouse and Cockroach have almost opposite constraints. Correct. And we have really made two very different decisions there for very different reasons. We would not want to... Certainly, it's hard to see one database ruling them all. These are two extremely different ways of thinking about data, looking at data, reasoning about data.
Oxide and Friends
Querying Metrics with OxQL
Yeah, and also, like, we're also not, like, advocating for the elimination of any other query language. That's the other thing. I feel like we're not actually... People can use whatever query language. I mean, we're not... We're kind of encouraged. It's just a DSL that we developed for our own use, really. So, Ben, do you want to describe kind of the origin of how we got here? And I thought...
Oxide and Friends
Querying Metrics with OxQL
similarly I don't think we want to put like instance information in Klaus so yeah I think we can agree that these are these are very different problems oh and one thing I wanted to when we were talking about just like using peg and so on and these other various rust crates um
Oxide and Friends
Querying Metrics with OxQL
I'm going to bring up Antler. That's exactly where I'm going. I'm going to Antler. So where are you? I mean, you're an Antler lover, just to be clear.
Oxide and Friends
Querying Metrics with OxQL
Maybe we would take half a step back, because we've tried to open up a bunch of the surrounding RFDs that get us to 463 or why 463 is relevant. Do you want to talk a little bit about 125 and how we got to ClickHouse? Because I think that's a big part of why not promQL, promQL.
Oxide and Friends
Querying Metrics with OxQL
I think that DSLs are... When someone is developing a DSL, it is almost always coming out of exhausting the alternatives. I think.
Oxide and Friends
Querying Metrics with OxQL
Totally. And, you know, I think that we... I've always found that these little languages... we've never developed them superfluously, I think. In fact, I think to the contrary, sometimes we think we got something, we want to use something general purpose.
Oxide and Friends
Querying Metrics with OxQL
We try to make something general purpose work and you realize this is actually creating more drag than it's solving a problem for all the reasons you mentioned at the top. Actually, a former colleague of ours, Mike Shapiro, wrote a uh, ACMQ paper years ago on, on purpose-built languages, uh, featuring MDB, um, and are talking about ADB and, and the language in ADB. So, um, um,
Oxide and Friends
Querying Metrics with OxQL
Okay, I'm gathering from the chat that apparently there are some DSLs out there that feel more elective. I realized that now I have turned into like, people are like, you've gone from an okay, fine, like, oh, XQL, we reluctantly acknowledge its right to exist. But now you're just like a DSL apologist? Now it's like any DSL. You're a DSL maximalist.
Oxide and Friends
Querying Metrics with OxQL
Yeah, which I think actually gives us terrific power. I mean, I think that that's a very important layer of abstraction that we've injected. I also think, and I know you mentioned this at the top, but in terms of why not SQL, you also don't want to give people the impression like, oh, this is great, it's SQL, I know that.
Oxide and Friends
Querying Metrics with OxQL
It's like, no, no, no, sorry, did you miss the 16 asterisks that are after SQL? It's like, this is actually not just SQL, sorry. Okay.
Oxide and Friends
Querying Metrics with OxQL
Yeah. Totally. Well, and I mean, I think that I know that we've gone our own way on lots of different things. Again, I think it's kind of, it's still surprising to me that the query language is like the bridge too far for the internet. The mob has shown up with the kind of the pitchforks and torches here. But the, I mean, we also did our own P4 compiler. We're going to talk about that. We did.
Oxide and Friends
Querying Metrics with OxQL
I mean, This isn't even our own... We've done other compilers. We've done multiple operating systems.
Oxide and Friends
Querying Metrics with OxQL
Obviously the database people are like, I don't care. Yeah. Do your own switch. It's fine. It's like, no, no, you're sorry. You don't understand how like that's madness. Like that actually is. Yeah. But the query language, that's just, that's great. Right. Right. So I think, but I would say that just in general, when we make one of these decisions, it's because we have exhausted the alternatives.
Oxide and Friends
Querying Metrics with OxQL
Like we have not done on any of our own Silicon people. Like just, just, you know, just clip that one.
Oxide and Friends
Querying Metrics with OxQL
We certainly, there is all, there are plenty of folks at Oxide who are, who exactly, everyone's like, we haven't done our own silicon yet. Right. And there is plenty, you know, we haven't written our own database yet. Yet. We have not written our own database yet.
Oxide and Friends
Querying Metrics with OxQL
Don't you think that feels more... But we also have already announced that we're supporting our own database with respect to Cockroach. That's right. Yeah, look. But we are not doing our own silicon. And that's basically it. We have that in our own instruction set architecture. I mean, I don't know. There are things we have. But we are doing these things on every one of these decisions.
Oxide and Friends
Querying Metrics with OxQL
It is almost certainly the case that we actually went into, we assumed that we were going to use, you know, we assumed we were going to use talk before we came to Hubris. We assumed that we were, and I think for all of these things, we certainly were using Intel's tooling with respect to P4 before we kind of came to the conclusion we need to do our own P4 compiler. I mean, for all of these things,
Oxide and Friends
Querying Metrics with OxQL
We went in wanting to use something else and then realizing this doesn't fit exactly. And I think, Ben, as you mentioned, we have great apprehension when we go our own way. I know it doesn't feel like it, honestly, that you people have any. Are you sure you have any apprehension? It doesn't really feel like it.
Oxide and Friends
Querying Metrics with OxQL
For people that have apprehension about going your own way, you sure go your own way a lot. It's like, well, yes, I know. I know. I get it. It's a bad look. But we do go our own way. We do have apprehension about it. We do really carefully deliberate on this stuff. And this was, I think, to me, this is a very clear example where going our own way is the right decision.
Oxide and Friends
Querying Metrics with OxQL
I am. I'm also glad that someone has mentioned the AT&T Hobbit in the chat. I am less than like 50 feet away from an AT&T Hobbit manual here in the Oxford office. I think to point out just to our, our, our, there's our return to the office conversation. I like to return to the office just to be close to my AT&T Hobbit manual, but we are, um, Well, Ben, this has been great. Thank you very much.
Oxide and Friends
Querying Metrics with OxQL
In terms of comparing this to your thesis defense, do you feel the questions were, how did these two compare? You're just like, hey, next time I would do something less stressful, like get a PhD.
Oxide and Friends
Querying Metrics with OxQL
I'm glad you brought this up because we now would like you to leave. And now we're going to invite everyone up on stage and we're going to discuss whether the novelty of OXQL and whether you should have just used SQL.
Oxide and Friends
Querying Metrics with OxQL
Oh, OK. Oh, yeah. Thanks. Thanks for bearing the lead there. I mean, I was curious. Right. Yeah. Oh, my God. I got to say, like, if you have news, big news, bad news, good news, do not bury the lead. Just this is like life lesson. Just get that news out there like early.
Oxide and Friends
Querying Metrics with OxQL
You know, like what's going to call from the from the disciplinary end of the high school, the assistant principal for discipline. And clearly this person has dealt with a lot of people because the first thing she said, I'm like, oh, my God, it's a disciplinary from the high school. Like, oh, like. Do I need a lawyer, basically, is my first thought. It's just like, it's good news.
Oxide and Friends
Querying Metrics with OxQL
Like, okay, it's good. That was nice of them. Oh, very nice of them. Very nice of them. And also, like, I mean, the whole thing was nice. It was very nice that they called with good news. Which is good. It's not always good news. So, you know, if it had been bad news, like, let's lead with that. Anyway, so there you go. Well, Ben, I'd like to lead you, lead with the good news.
Oxide and Friends
Querying Metrics with OxQL
Yeah. And we test ran the new AHL bot. Adam has been completely replaced with an AI. That seemed plausible. I don't know. I bought it this whole time. Yeah, exactly.
Oxide and Friends
Querying Metrics with OxQL
That's right. Just somebody mentioned Antler, it'll immediately start generating. Exactly. And, uh, and, and check out the, uh, we, we dropped the link in the chat to the auto podcast. That was very creepy. They were talking about at the top. It's kind of fun to check.
Oxide and Friends
Querying Metrics with OxQL
It's creepy. For, uh, well, Ben, thanks again for Oxquill. Um, and I'm going to go take it so I can go to bed. All right. Stay curious, everybody. Yeah. Thank you. Thanks everyone. Talk to you next time. Bye.
Oxide and Friends
Querying Metrics with OxQL
With the auto-generated podcasts, it's mesmerizing. I can't stop. It's really weird. So Ben, I'm sorry. I was also sending Ben all these, obviously. Uh, you, you have now you've lost your, your gravitas voice. I don't know. I'm taking you a little less seriously right now. You get kind of back to, back to clown college.
Oxide and Friends
Querying Metrics with OxQL
And just to be clear, what data are we storing here? Because I think people have kind of a natural question of what ramifications does this have for the user of the rack? The decision around ClickHouse is really an implementation decision.
Oxide and Friends
Querying Metrics with OxQL
And they're very cheerful and they are very interested in promoting whatever document it is that you've put in front of them. They definitely believe whatever they've just read.
Oxide and Friends
Querying Metrics with OxQL
And we're not precluding anyone from slurping this data out and shoving it into some other system that they might want that has some other query language that they... No.
Oxide and Friends
RFDs: The Backbone of Oxide
Because at some point, and my memory is that Alex Wilson, our colleague at Joyent, was working on a container naming service for what was then Triton. And was really looking for a vessel in which to write it down. And I think that there's been a lot of pressure building in general where we actually need to have a way for people to write things down.
Oxide and Friends
RFDs: The Backbone of Oxide
And it had been enough time since the scarring of Peace Arc. I think actually one of the things I kind of present about Peace Arc is that it took me years to really appreciate the value. And Adam, maybe you're still getting there because you're like, did you just say Peace Arc was valuable? Because that's not my memory at all.
Oxide and Friends
RFDs: The Backbone of Oxide
But it took me years to appreciate the value and be able to separate it out from the things that were really – that could be, frankly, pretty upsetting as an engineer. But once I kind of had not had this for an extended period of time, it's like, yeah, we really need to get – I miss writing. I miss writing ideas down. Robert, is that your – do I have the –
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, and that's what I was trying to remember is like whether – to what degree it was like Alex is like I need a place to write this. But yeah, Alex was definitely – however it was, Alex was the first RFD.
Oxide and Friends
RFDs: The Backbone of Oxide
That's right. And I think that, you know, and this has happened to me a lot in my career where I you know, I'm trying to come up with something that I think is like, wow, this is like totally unique. And like, Oh, you know, this is very Socratic or a kind of classic, what have you. And then, I mean, at some point, as we were thinking about this, Robert, like kind of the light was going on.
Oxide and Friends
RFDs: The Backbone of Oxide
So what you want is something that's like, I want it to be pretty loose. I want to encourage people to write things down more than getting them exactly right. Um, and yeah, as we're kind of thinking about this, we're like, wait a minute, these are RFCs. This is what, this is what the, this is how the internet was built. This is right in front of us.
Oxide and Friends
RFDs: The Backbone of Oxide
I'm loving this. I have managed to offend all of our German listeners. We apologize, Germany. Exactly. A message to our German community from Oxide and Friends. You know, apology videos have kind of gotten... I'm trying to bring apology videos back. I just feel like, guys, I can't believe we have to have this episode. I never thought I'd be saying this, but...
Oxide and Friends
RFDs: The Backbone of Oxide
This is what RFCs and, and, you know, in joints, it joins our fee repo. I, I reference RFC three and the, the IETF has this request for comments and RFC three just absolutely nailed the, what we were after. And it was kind of like, I've managed to reinvent something that's right in front of me for my entire career, more or less. And I felt a little bit silly for not having seen it.
Oxide and Friends
RFDs: The Backbone of Oxide
But then another like, okay, so this is like, we need to take this and extend it. And I just love this note from RFC3 that the content of a note may be any thought, suggestion, et cetera, related to the software or other aspect of the network. Notes are encouraged to be timely rather than polished. I love that line.
Oxide and Friends
RFDs: The Backbone of Oxide
Philosophical positions without examples or other specific suggestions or implementation techniques without introductory or background explication and explicit questions without any attempted answers are all acceptable. The minimum length for a note is one sentence. and these standards or lack of them are stated explicitly for two reasons.
Oxide and Friends
RFDs: The Backbone of Oxide
First, there's a tendency to view a written statement as ipso facto authoritative, and we hope to promote the exchange and discussion of considerably less than authoritative ideas. Second, there is a natural hesitancy to publish something unpolished, and we hope to ease this inhibition. Those paragraphs are from RFC3. That is Steve Crocker, April 1969.
Oxide and Friends
RFDs: The Backbone of Oxide
Adam, doesn't that feel like that is just absolutely true today?
Oxide and Friends
RFDs: The Backbone of Oxide
Totally. And as a result, there is going to be a wide variety of quality here. I mean, there has to be, right? There's going to be, and there are going to be RF and there are RFCs that are extraordinary. And there are many RFCs that are forgettable and that's okay.
Oxide and Friends
RFDs: The Backbone of Oxide
We're not trying to make something that is kind of, and we're in, in particular, what I loved about that is like, this is not something that's kind of up for approval. This is you writing down your own ideas and,
Oxide and Friends
RFDs: The Backbone of Oxide
And then, Robert, as we were kind of thinking about RFCs versus – my recollection is, like, we just wanted to have a different nomenclature because we felt we'd be just confusing ourselves all the time if we were talking about RFCs. And we were kind of looking for a different nomenclature. Is that right?
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, I don't think it was too much beyond that. I think it was just kind of like, you know, I think we would have called it like, yeah.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, which is, we use the term request for discussion in lieu of request for comments to avoid conflation with the IETF construct. and the more formal writing that has come to represent. Because RFCs are now kind of drifted from that RFC3 ethos and zeitgeist and are now quite a bit more formal. So we wanted to draw inspiration from RFCs, but be something slightly different. And RFDs were born.
Oxide and Friends
RFDs: The Backbone of Oxide
And Robert, I feel it was like, I mean, I don't know if you and I spoke about this. I'm sure we did. But I remember thinking very shortly after Alex's RFD won and we start getting a bunch of RFDs, I'm like, oh, man.
Oxide and Friends
RFDs: The Backbone of Oxide
Germany, I'm so sorry for what Brian did. I think I'm making it worse.
Oxide and Friends
RFDs: The Backbone of Oxide
we waited too long to do this i felt like this was so should have done this earlier i um i mean on the one hand it's like it's great like okay this is clearly like i this is a big step in the right direction but it's such a big step in the right direction that uh felt like we should have done this earlier um yeah and i think we had one or two you know like one-off documents
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. And where did those go? I mean, I just don't remember where were those because I mean, they must've existed. I just don't know where they went.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. So this was in, I mean, it's kind of embarrassing that like this is in the first RVs in 2015. And so we did, how did we do engineering for five years? I just don't know.
Oxide and Friends
RFDs: The Backbone of Oxide
fair. So, um, and then, so Robert, what were our experiences with RFDs at joint? Cause I felt like there were some things that worked really well, namely the idea, the, uh, was great and was important. Um, then there were other things that I don't know, what were your, what was your take on things that worked well and things that did not work well in that kind of that first embodiment of us?
Oxide and Friends
RFDs: The Backbone of Oxide
Really long time coming. I feel like this is like the episode we were in all along. I feel that... You know, I was listening, but actually I'm really glad that we are doing it now and not a couple of years ago because we have made so many... very recent changes that are really, really important.
Oxide and Friends
RFDs: The Backbone of Oxide
It is painful, but it is much better than it was at Joyent. At Joyent, we did not. I mean, the grand irony of RFDs at Joyent is the discussion piece was not. It was really hard to discuss them because we would do this with GitHub issues, Adam, and it did not work well. Um, because it was very hard to like comment on particular pieces. You know what I mean?
Oxide and Friends
RFDs: The Backbone of Oxide
Comment on like, Hey, the word choice or whatever. And like, if you walk up to an RFD that is, you know, old, how do you go and be like, you know, this paragraph is no longer correct. Or you just like you filing issues on all these things. And it's like, the discussion was not great. So that I would, I think was a big, uh, needs improvement. Um,
Oxide and Friends
RFDs: The Backbone of Oxide
But I think the idea, like we could see that a bunch of, another thing that was actually really important, Robert, is that it was in a repo. I felt like this is a very important, because folks may look at RFDs and be like, well, wait a minute, why not just like Google Docs or whatever? And I think I'm sure, what were you doing at Delphix and then Transposit, Adam?
Oxide and Friends
RFDs: The Backbone of Oxide
Yes. Yeah, and I think that what the world wants, and what we're kind of leading up to, is you want kind of a Google Doc front end that has an ASCII doc backend going to a canonical repository. That's what you want. That doesn't exactly exist.
Oxide and Friends
RFDs: The Backbone of Oxide
And I also feel that one of the things that we have done, and we'll talk about this a bunch over this episode, but we have open sourced the software that we use to do this. And I think that if anybody's listening to this wondering like, God, I really like a lot of aspects of the system, but I don't know. I don't, I don't want to just like take theirs. It's like, you can definitely take it.
Oxide and Friends
RFDs: The Backbone of Oxide
Well, part of the reason I'm chuckling is because we did Markdown at Oxide, and we supported Markdown and ASCII doc. And Adam, I think it was you who took me aside at some point. It's been like, hey, you say that RFDs are in Markdown or ASCII doc. Yeah, exactly. It's like, you should know that only yours are in Markdown. Everyone else is on ASCII doc.
Oxide and Friends
RFDs: The Backbone of Oxide
So when you say Markdown or ASCII, I'm like, I actually... Adam, I don't think I'm inventing this conversation. I really appreciated it. I had a giant piece of lettuce on my teeth and only you had... All of my oldest and dearest friends were willing to take me aside and tell me what an ass I was making of myself. I'm like, you mean, no. And I kind of didn't believe you.
Oxide and Friends
RFDs: The Backbone of Oxide
I'm like, surely other, no, nope. Nope. It's just me. And that was a quick, I definitely like, I am reworking them right now. They are all going to be at ASCII doc. I'm so sorry. Yes.
Oxide and Friends
RFDs: The Backbone of Oxide
It is my own weakness that I would like to be able to, I don't know, see what this thing looks like before I, I, I just, it feels like this is like how you always want to test your code before you integrate it.
Oxide and Friends
RFDs: The Backbone of Oxide
Um, we are, if they, and certainly if I may at risk of orienting people, what we are talking about. Oh, okay. Oh, now with this again. Okay. You know, it's like, we're only a couple minutes in, I was going to wait for another 10 minutes, but yeah, sure. Go ahead. Fine. Fine. Blow the big reveal. All right. Oh, what are we talking about? Mr. We have to tell people what we're talking about. Go ahead.
Oxide and Friends
RFDs: The Backbone of Oxide
Yes, Markdown is really frustrating in a lot of ways. And especially if you want to do anything sophisticated or complex. you need things like, you know, footnotes and you want to cross reference and all these things. It's really not good.
Oxide and Friends
RFDs: The Backbone of Oxide
I had kind of assumed, I didn't realize that us, that ASCII doc, I thought ASCII doc, well, obviously cause I was the, like the last to get the memo on like, you're the only one doing it in Marktown. So as far as I was concerned, everyone was doing an ASCII doc the world over.
Oxide and Friends
RFDs: The Backbone of Oxide
So I didn't, don't think I realized that ASCII doc and press was saying also that he'd not heard of ASCII doc till coming to oxide. So I guess that's a little more, it's, I mean, the irony of ASCII doc is that the documentation for ASCII doc is not great. Um, I do find that, and Robert, I can't imagine that ChatGPT is very helpful on AsciiDoc. I want to be able to render this.
Oxide and Friends
RFDs: The Backbone of Oxide
I feel like some of the stuff in AsciiDoc is a little... It's been nice to be able to use ChatGPT for AsciiDoc. That's all I'd like to say. No, I'm just being on brand to get LLMs in every episode. You know, there we go. Ring that bell. Exactly. Um, the, but so I didn't realize that Alex had kind of discovered ASCII doc. Um, we, uh, and Robert, do you use ASCII doctor PDF to render locally?
Oxide and Friends
RFDs: The Backbone of Oxide
And so it should be said also, and Robert, now is as good a time as any. So we have, I created RFD 510 yesterday. So we have perhaps more than 510 now. So I might have created one in the meantime. But we've got over 500 RFDs at Oxide. Adam, you had on an earlier episode had guessed that we must have millions of words. Did you do this, by the way? Have you already done this?
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. So we do have, there's 1.4 million words. Um, and I did look at the, I went through everybody who has kind of contributed an RFD and, uh, just looked at the author's field. Um, so didn't, didn't do anything more kind of sophisticated than that. And to see, in other words, like giving everyone full credit for any RFD that they're on the authors for.
Oxide and Friends
RFDs: The Backbone of Oxide
And of the 1.4 million words across all RFDs, uh, Robert, you have contributed, uh, three over 364,000 words. Is that amazing?
Oxide and Friends
RFDs: The Backbone of Oxide
Also terrifying. It is. And, and Robert has been, so part of the reason, Robert, obviously you, you are, you are our most prolific RFD writer.
Oxide and Friends
RFDs: The Backbone of Oxide
Um, and you know, not, not that we want to, you know, just insert our discussion about the parallel of engineering metrics and that it is the cultural idiosyncrasies episode, but the, um, you know, we're not trying to measure RFDs by the pound here, but I did think it was, it was interesting and it was revealing.
Oxide and Friends
RFDs: The Backbone of Oxide
The other thing that Adam was really interesting to me was the, uh, that, that was just eyeopening. It wasn't hugely surprising, I guess. Um, The person at Oxide who has contributed with the most number of collaborators on different RFDs, who's collaborated with the most number of people on RFDs. Do you know what this might be? I'm going to guess myself. You are absolutely right.
Oxide and Friends
RFDs: The Backbone of Oxide
It is like you by far. And this is one of these things like I didn't, you know, I'm not surprised, but I'm also like, I was actually surprised about how much it stood out. And it makes, and actually, it's because you have also like really encouraged a bunch of people to write RFDs or started RFDs that have, I mean, it's just like, I thought it was interesting.
Oxide and Friends
RFDs: The Backbone of Oxide
I mean, it showed one of your kind of great strengths in terms of cross pollinating in the organization.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, exactly. But I thought that was ranging. So, I mean, both of you have, in different aspects, have interacted with the system a lot. And so, you know, Robert, you have written more ASCII doc than any of us. So we've written a lot of ASCII doc. I would say that the entire collection of Shakespeare is about 800,000 words. So, Robert, you're getting there. What...
Oxide and Friends
RFDs: The Backbone of Oxide
Will Shakespeare, keep an eye on your rearview mirror. Robert's coming for you.
Oxide and Friends
RFDs: The Backbone of Oxide
like all my time all my work time to just reading the new contents of rfds absolutely and because we are you know very writing intensive culture we are adding people people are we've added some very prolific writers and yes it is it is absolutely um and in the ends like that shouldn't be the goal right you don't want to actually like have to read every line of every rfd
Oxide and Friends
RFDs: The Backbone of Oxide
Okay, so this gets us to, because I want to get to the very important bits here that we have extended, because this is where we get to Ben's work and Augusta's work, Crespo's work. So this was, you know, we start Oxide. We know, most of us know that ASCII doc is the right answer. One of us feels that they are still doing it in Markdown. So we solved that problem.
Oxide and Friends
RFDs: The Backbone of Oxide
I think it was in the first year, Adam, when we had the intervention, the Markdown intervention.
Oxide and Friends
RFDs: The Backbone of Oxide
Right. Okay. And I think ASCII doc has been good for us, I would say, broadly speaking. Totally. Other than being, I guess, more unique to us than I realized. So one of the things that we started doing early on, I mean, very early on, actually, the earliest days, because we have, it's just like we're on, the name of the podcast now, Oxide Friends, comes from actually our initial friends of Oxide.
Oxide and Friends
RFDs: The Backbone of Oxide
So when we started the company, we had a bunch of folks who had, and Adam, you were one of the initial friends of Oxide, right? Where before you worked here, you were a friend of Oxide. We had just people that we'd worked with, that we'd known, that we wanted to, be transparent about what we're building to get their perspective on.
Oxide and Friends
RFDs: The Backbone of Oxide
So we added them to that GitHub repo so they could see actually all of the RFDs. So the GitHub repo is a private repo that has all of these ASCII docs, all of these documents in it. And
Oxide and Friends
RFDs: The Backbone of Oxide
The important change that we made from Joyent is the way we did discussions, which is it's a little wonky, but it's better than it was at Joyent by a mile, which is we use the fact that GitHub, that PRs are actually... Better than issues for discussions. I mean, the GitHub PRs have got issues with them for sure, or they've got challenges using it.
Oxide and Friends
RFDs: The Backbone of Oxide
But we use PRs to discuss an RFD, which at least allows people to give kind of line-by-line feedback. It's nowhere near as good as Google Doc comments, for example, but it could be worse. Yeah, it doesn't get in the way, I don't think. I don't think it gets in the way.
Oxide and Friends
RFDs: The Backbone of Oxide
I think it is, you know, I think that there's some challenges, but I think it's broadly, again, it was like way better than certainly at joint. We didn't have discussion that we should have had because it was mechanically so difficult. So kind of solve that problem.
Oxide and Friends
RFDs: The Backbone of Oxide
How does it feel for us to be doing this at a time that includes you? I'm so sorry. This is good. Can you hear me? Adam from Italy. Yes. Yes. I mean, don't take this the wrong way, Adam, but I think you, I think you sound better than you do in Alameda. I mean, I think so too. I have a upgraded microphone for, for this.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, I kind of, I do like it. I wish, obviously, I wish those comments were themselves under kind of Git management, but which would be kind of, that'd be asking GitHub to go to be one step too useful. But the, I do like the fact that they're there and they also don't get in the way of reading the doc, which I think is also important because...
Oxide and Friends
RFDs: The Backbone of Oxide
you know, somebody like, you know, not all those, you know, there are comments like, okay, yeah, I got considered this or whatever, and I'm going to close it. I'm resolving it, but I want you to be able to still see that there's a comment here. Maybe someone else can come along later and have the same comment or what have you, but yeah, great.
Oxide and Friends
RFDs: The Backbone of Oxide
Very early on. I mean, it predates Oxide, really. And we'll kind of get into the history there. But it really is the backbone, as you and I were kind of iterating on titles. I think you were kind of wanting me to go bigger on titles. And you're right. I mean, it's like, this is really important. And you said this in our episode on cultural idiosyncrasies as well, that this is idiosyncratic. And
Oxide and Friends
RFDs: The Backbone of Oxide
Um, so we, and we do that with branches and then we, we put some automation around that where, you know, you could, the RFDs have got different states and they can be in kind of, um, ideation was an early, that was a state that we added at Empire Call. We wanted something that was kind of, uh, earlier than kind of discussion state, which is like, I truly do.
Oxide and Friends
RFDs: The Backbone of Oxide
I've got just like no idea what I, this is a dream I had.
Oxide and Friends
RFDs: The Backbone of Oxide
Right, and that's right. I mean, RFDs are the way we kind of are. Maybe you can argue beginning to abuse it a little bit, but the system was flexible enough that it could be extended that way, and adding ideation was pretty easy. So that worked out pretty well. So then it was great to be able to take the repository and add their GitHub ID as a collaborator on the repository.
Oxide and Friends
RFDs: The Backbone of Oxide
And then that became kind of part of what one though, like it was just great to be able to do that, to get people's perspectives on things. But then secondly, we would use that as part of the hiring process. So part of the hiring process was after we did someone's materials and like, okay, we're interested in this person. We would add them to the RFDs so they could see all of the RFDs.
Oxide and Friends
RFDs: The Backbone of Oxide
And this was a really important step. It was really important for us to be able to show people all of our documentation about how we were building this thing to give them a much more concrete idea of what we were doing technically. What I would tell people is when you get into these RFDs and you're going to, you know, some of them are extraordinary. Some of them are not even correct any longer.
Oxide and Friends
RFDs: The Backbone of Oxide
Some of them are, you know, there's this huge, you know, huge spectrum of quality and so on. But as you read these RFDs, you will begin to feel that you want to work for Oxide more or less, but not the same. That's what I would tell people. And I think it was really, really important.
Oxide and Friends
RFDs: The Backbone of Oxide
And we had, obviously, because we're, by the time we're kind of advancing people's after the materials, we talked about this with our, on our episode on hiring with Gary, but this is all happening after we have decided like to, to kind of pull someone into conversations and we want them to kind of understand what we're building at Oxide.
Oxide and Friends
RFDs: The Backbone of Oxide
We had, we did have people in there and I had one person in particular was like, I thought I really wanted to work for Oxide. But, and I had pointed them to, you know, some pretty gnarly RFDs and it's like, I'd read some of these RFDs and like, I am, I'm kind of more interested in the idea of working for Oxide than I am working for Oxide, which I thought was very helpful. Right.
Oxide and Friends
RFDs: The Backbone of Oxide
Absolutely. Absolutely. And Robert, you should take this as praise, but it was absolutely one of your RFDs that actually was the clarifying RFD. Because I think people don't realize the amount of... You're like, oh, my God. There is so much detail here. It's like, yes. Yes, there's a lot of detail. It's very, very detailed to actually go on. So that was great.
Oxide and Friends
RFDs: The Backbone of Oxide
It doesn't feel like it should be idiosyncratic because we did not invent literacy and we did not invent the importance of writing down one's ideas or reading the ideas of others. But I think a bunch of the things that we've done borrowing from ideas elsewhere, have added up to something that is idiosyncratic, but I think it shouldn't be, which is the reason that I kind of bridle with that.
Oxide and Friends
RFDs: The Backbone of Oxide
The problem that we had, or that I had, was there wasn't a good way to... Well, we had a couple of big problems. One, there was no way to search RFDs, which started off not great and was becoming really, really, really problematic. And then there was no way to share individual RFDs. And other than like rendering them as PDFs and sending them, which I did plenty, but it's like, that's not great.
Oxide and Friends
RFDs: The Backbone of Oxide
And this may be a good time to get you and kind of David in here and Augustus in terms of like, this is kind of the RFD system that you kind of come upon is us kind of dealing with sticks and stones and GitHub. And what is the origin of the RFD site? Because I'm trying to remember, how far did that date back, first of all? That dates back, I think, a while.
Oxide and Friends
RFDs: The Backbone of Oxide
Ben, great question. I got to say, the sun is pouring into this room at Oxide. I don't think I quite realized how much absolutely nuclear bomb direct sun exposure this thing gets in the morning. So I'm kind of like hiding a little bit in the shade, but it is glorious. I got to tell you, I got my morning cup of coffee. I'm really excited for this.
Oxide and Friends
RFDs: The Backbone of Oxide
Totally. When I think, so a couple of things like the, so we added, I mean, the inability to search RFTs was extremely painful. It was also, it's made even more painful because the, the great thing about using PRs for discussion is that you get GitHub's features and
Oxide and Friends
RFDs: The Backbone of Oxide
The downside of that is it means that like the actual repo itself, that is to say like the main branch, only contains RFDs that have been published. They don't have the ones that are in discussion. So you have to like search every branch. And so I had this thing, RFDGREP, that you were referring to, Ben, that was taking minutes.
Oxide and Friends
RFDs: The Backbone of Oxide
But it was like, it was minutes, but it was actually like delivering penicillin. So it was really, you know, it was like, oh my God, thank God we got some way. We were using LuceneGREP.
Oxide and Friends
RFDs: The Backbone of Oxide
um and lm graph is what we were using using what we've seen and it was very helpful but it was also like its performance would vary a lot it would take minutes it was really really painful and in particular like it was so needed and because i remember like getting steve tuck set up with rfd grab because he needed it desperately like gotta have a better way to do this so
Oxide and Friends
RFDs: The Backbone of Oxide
But it is idiosyncratic. But regardless of whether it's idiosyncratic or not, it is essential to oxide. And it has been It is really, really, really important. It is really hard to overstate the importance of RFDs, and they're becoming way more important, actually.
Oxide and Friends
RFDs: The Backbone of Oxide
I can't remember when you did the full text search on the RFD site, but that demo Friday was like a religious experience where, I mean, it was like that affected everyone at Oxide to be able to like, oh my God, I can just go there and search all the RFDs, like clean water, thank God.
Oxide and Friends
RFDs: The Backbone of Oxide
So describe that because that was, I totally agree. So now we've got the RFD site and we've got the RFD site, which also like looks great, which is very helpful. You are doing, so tell me how you're rendering the ASCII doc, because that was a big piece of this. You got to go render the ASCII doc.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, sorry. I think I was definitely a recent version of that where I was definitely using block quotes and attributed quotes. Brian, that's basic.
Oxide and Friends
RFDs: The Backbone of Oxide
I feel like the emperor's new clothes, though, because we do use ASCII doc everywhere, I just assumed ASCII doc is ubiquitous everywhere. And it's like, I've never heard of it. Exactly.
Oxide and Friends
RFDs: The Backbone of Oxide
Look, I... Okay, I just assumed. Because, Ben, as you point out, I'm realizing that part of the reason I thought ASCII doc... was ubiquitous is because it is ubiquitous and oxide, but you're, it is ubiquitous and oxide because you made it ubiquitous.
Oxide and Friends
RFDs: The Backbone of Oxide
I think it's great. I think it's great. I think like it's, it's actually, I think it's terrific. I think it is so much better than Markdown. Um, and, um, so I, yeah, I think it's great. And like blogs and so on are all an ASCII doc.
Oxide and Friends
RFDs: The Backbone of Oxide
Yes. And the other thing that has been just extraordinarily important is the ability to make single RFDs public. And that has been... life-changing for us because it now allows us to begin to... And we've done this over time, but we are getting more and more RFDs out there and allows it for us to be that kind of public vector as well.
Oxide and Friends
RFDs: The Backbone of Oxide
Well, and just the fact that we also use this for the documentation is so great. And then we've been able to make the documentation site is anyone that's available to the public. If you want to, I mean, we're, can see all of the Oxide documentation, which is great. And then we're able to kind of cross-link that with the console. There's a bunch of things we can do there.
Oxide and Friends
RFDs: The Backbone of Oxide
So the other, I mean, and then also it looks gorgeous. Can you talk about the GitHub integration a little bit? Because that's been a huge, that's been hugely valuable to get those discussions. That was another demo Friday that I just think just like left people, you know, weeping with joy. The C comments and the RP site.
Oxide and Friends
RFDs: The Backbone of Oxide
A huge improvement. And this is a good segue to the kind of the RFD database that kind of fronts this thing, I think, because I assume that's hitting the database that is fronting it and not actually GitHub directly, but I actually don't know the answer to that. Augustus, it would be a good opportunity to get your work in here.
Oxide and Friends
RFDs: The Backbone of Oxide
And so that has been absolutely essential and that's all built on the terrific work of Ben and Augustus and Dave Crespo who are all with us today. So their work has been... Like, it kind of started off as like, oh, like, this is nice. Like, oh, this is great. Like, this is great.
Oxide and Friends
RFDs: The Backbone of Oxide
The reason I had assumed that those were coming out of a fronted database is because it was so snappy. So that's why it's so snappy. Because you're exactly... Yeah, yeah, yeah.
Oxide and Friends
RFDs: The Backbone of Oxide
It is fake. I get totally faked out by it, though. I'm not looking at the spinner. I'm generally looking at the rendered content. So that's definitely worked. It also should be said that it just. The rendered content looks gorgeous. And obviously, Ben, you've made sure that everything at Oxide looks gorgeous or faces your ire. And it just looks great.
Oxide and Friends
RFDs: The Backbone of Oxide
It also looks great on the phone, which I think is really important. because I think folks are reading more and more RFDs on a mobile device, right? Augustus, the other reason that having that front-end database has been really, really important is for the visibility for RFDs. Can you describe maybe some of the problems we had with adding increasing number of folks to this Friends of Oxide group?
Oxide and Friends
RFDs: The Backbone of Oxide
And then, you know, they, again, we'll get into the whole history of this, but that added things were just like, oh, wow, okay, this is really great. Like, this is profoundly great. And now we're just like, okay, this is... this is now like low bearing. This is, this is, uh, we now cannot envision the system without this. This is extremely important.
Oxide and Friends
RFDs: The Backbone of Oxide
Because this GitHub group was getting larger and larger and larger. And I think it all had to be like outside collaborators with our organization, I think, if I recall correctly.
Oxide and Friends
RFDs: The Backbone of Oxide
And this is extraordinarily important. I think the full text search was game changing. The rendering was game changing. And I feel that the fine-grained permissions are totally game-changing for RFDs.
Oxide and Friends
RFDs: The Backbone of Oxide
Because we really needed to be... The reason I think it's so game-changing is because it allowed us to much more easily... Well, one, we could make our individual RFDs public, which was a real problem. When did we first have the ability to do that?
Oxide and Friends
RFDs: The Backbone of Oxide
And this has been, I think, huge. Because I feel, especially where we are in the past year, having brought this product to market, and now there are more and more aspects of this that we want to share with everybody. We want to share with all of our customers and we want to share with everyone to see.
Oxide and Friends
RFDs: The Backbone of Oxide
And I feel like there are, you know, it's just kind of amazing that we've, that, you know, we've only been able to do this in the last year. And I, cause I feel like I do this all the time. We did this, obviously, I mean, the last,
Oxide and Friends
RFDs: The Backbone of Oxide
episode as was talking about an rfd that was made public right so the in terms of wither cockroach db and 508 but i feel like it was uh really really important to have that fine grain control and to be able to have like groups as well so you can say like okay this because i think we've got a group for our contract manufacturer for example where they want we want them to be able to see everything about a certain number of rfds that are not public um it's extremely helpful
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, that is great. So that was before we met up in October. And then I know that I was out in New York in November talking to someone who had seen the RFD site. And they're like, how can I get a hold of that?
Oxide and Friends
RFDs: The Backbone of Oxide
Because I think a lot of, I mean, I feel like every time we have an RFD, a public RFD that is on Hacker News or is getting discussed, someone is always like, hey, can someone explain to me how this system works and how can I use it? Because there's a lot of interest in it. Do you want to talk about a little bit of the open sourcing of the RFD site and of the RFD API? Sure.
Oxide and Friends
RFDs: The Backbone of Oxide
You don't have any RFDE that implies that you're building a computer. Other than that, you can use it as much as you want. You just can't, you know?
Oxide and Friends
RFDs: The Backbone of Oxide
I think eventually it might be nice to have something a little bit more... Yeah, I mean... Dave, you... Okay, the CSS is specific to us, but someone could take this and get it working for themselves, correct? I mean, it feels like they could. Oh, for sure.
Oxide and Friends
RFDs: The Backbone of Oxide
I will tell you that there's a reason that it's written. I mean, so Jess and I are listed as the co-authors because Steve was not yet working in the garage. So this is going to be in August of, I think, of 2019. Maybe we retroactively made it after September 9th.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, exactly. That's Oxide TM to you. Java technology, please, TM. So in terms of being able to take it, we've got all the elements that someone could do to go. And what would be some of the ways in which someone could use this stuff? I mean, how would one get started if you wanted to kind of see what it would be like to use this?
Oxide and Friends
RFDs: The Backbone of Oxide
I mean, Adam, you're a proud papa. You and Dave on Progenitor and Trap. You just love it.
Oxide and Friends
RFDs: The Backbone of Oxide
Okay, so Augustus, when I'm using the RFD COI, that is a progenitor-generated COI. Absolutely.
Oxide and Friends
RFDs: The Backbone of Oxide
Okay, so feel shame on that. But I use that CLI a lot. I mean, I make great use of the RPCLI. So that's auto-generated. That's amazing.
Oxide and Friends
RFDs: The Backbone of Oxide
I just want to make clear that I'm not accusing you of using an AI-generated photo of Italy. I'm just saying the photo that you posted is a little on the nose for what a generative AI might imagine that Italy would look like. It's just a little too on brand.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, and I feel like that automation has made it, because the branching bit has been great for discussion, but I think the point you're going to get to is it can feel a bit clumsy at times for the authors of RFDs. And I feel like the automation there has really helped a bunch.
Oxide and Friends
RFDs: The Backbone of Oxide
And I think that given all that, I mean, I could see why someone might want to do it a different way, but there's a lot of strength to the model that we've got. We're just kind of getting GitHub. It's using GitHub for its strength without being, while still having a canonical repo that has everything in it.
Oxide and Friends
RFDs: The Backbone of Oxide
First things we did. Absolutely. The first thing we, the first thing we did and it actually felt like, because it felt like something we could do as well. So do you know what I mean? Where, because we didn't have money in the bank, we were totally raising around and there's like, you're kind of limited about how much of this company you can build with zero additional people and no money. And yeah,
Oxide and Friends
RFDs: The Backbone of Oxide
Yes, and I think it gets to... So folks in the chat are asking about Notion. I've not actually used Notion. Preston, it sounds like you have. I would say that my... Concern about Notion. I would say I've got an Evernote concern with Notion or any other.
Oxide and Friends
RFDs: The Backbone of Oxide
And this is part of the challenge here is that using any kind of paid service here is really problematic because this is so important to us that it's really hard for us to accept restrictions on how things are used. And And in particular, a hard constraint on the problem is we have to have a Git repository that has the RFDs in it. That has to be where they live canonically.
Oxide and Friends
RFDs: The Backbone of Oxide
They cannot live canonically anywhere else. And so that's extremely important. I think it's really important, the ASCII doc element of it, now that I understand that only oxide is used in the ASCII doc, that becomes really important. I bet you're mentioning the chat, the versioning, that is extremely important.
Oxide and Friends
RFDs: The Backbone of Oxide
And I just think that this is one of these things where we really need to control every aspect of our fate. We would love to, I think, integrate with other, you know, we're not, I think, you know, if we could find kind of better ways to do pieces of this, but we really need to control our own fate. And I think that like, that is, that is not unusual for us.
Oxide and Friends
RFDs: The Backbone of Oxide
I think engineering organizations need to control their own fate in terms of the actual artifacts they're creating. And this is a really important artifact that we're creating. This is, and as usual,
Oxide and Friends
RFDs: The Backbone of Oxide
saying adam it's like one of our most enduring artifacts are rfd so i mean it's i i mean it's right up there with the source code honestly i mean it's don't make me pick those are they're both very very important a while ago i stumbled onto a phrase of uh owning your strategic weirdness and i i would have said like rfds are a thing that we we could have outsourced
Oxide and Friends
RFDs: The Backbone of Oxide
Well, so I knew that was going to be a problem. It was a problem in joint too. I knew we were going to want a way of communicating with the public. I just had much worse ideas of how it would be done.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, I don't think so either. In fact, quite the contrary. I think with all these things, I now cannot imagine doing... It's been a lot of work. I mean, not to discount that, but it's also like we haven't... We're not licensing any proprietary software here. We're not paying for service. In fact, quite the contrary.
Oxide and Friends
RFDs: The Backbone of Oxide
The work that Augustus was doing was necessary for us to prune our GitHub collaborators because that was costing us money. This actually saved us a bunch of money. by, so the degree we were using a service, we were using too much of it.
Oxide and Friends
RFDs: The Backbone of Oxide
And we were, I mean, in fact, actually, Adam, I think you kind of need to look no further than GitHub where we were, I mean, just for all the things that Augustus was mentioning about the groups and having to add collaborators, but then the collaborate, just like it was really, We were already kind of breaking GitHub. And we've left GitHub as kind of the canonical repository for comments.
Oxide and Friends
RFDs: The Backbone of Oxide
you know, what are the kinds of things you can go do? And like, like you can do a podcast. We did, we were recording on the metal during that time. Right. And, um, But we were established. You look at like RFD2, I think, is principles and values, right? So that we were, RFD3 is our hiring process.
Oxide and Friends
RFDs: The Backbone of Oxide
But I think the direction we'd want is to use less of that. I think you could easily envision... I mean, God, if... I know that Prespo was defending Ben's time by saying, under no condition will we allow you to add comments on the RFD site. But if you were to add that, you could easily... you could see comments going into a Git repo and then us not using GitHub for comments on that. Totally.
Oxide and Friends
RFDs: The Backbone of Oxide
It'd be amazing. And then I think that, like, I also think that the strength of us doing this, by the way, is that we've been able to open source it. So, this is now, like, we have generated something that, and I feel that, like, you know, as we've kind of gone through our careers, like, we've added more and more wisdom into the system. And,
Oxide and Friends
RFDs: The Backbone of Oxide
it now reflects a bunch of wisdom that we can all carry forward. Even folks that are outside of Oxide can actually leverage those. One thing I did want to talk about because it did come up in discussion online where people were curious about how we use RFDs to resolve conflict. And that's not exactly their question. I'm rephrasing it a little bit. But how do you deal with conflict in an RFD?
Oxide and Friends
RFDs: The Backbone of Oxide
How do you deal with where... And the comment was kind of like, hey, we experimented with it, but it just became too much of a lightning rod. And I would say that RFDs do not solve your organizational problems. They don't necessarily solve... They don't resolve those things. And, you know, we have had RFDs that become lightning rods. And it's not good.
Oxide and Friends
RFDs: The Backbone of Oxide
I mean, it's not, I mean, I think it's better than, I would say these are lightning rods that are lightning rods anyway. I think as we alluded to on the cultural idiosyncrasies episode, of course, it's like the chat system, which we've already like run under the bus several times here. It's like chat is what will rip oxide apart. Yeah. Absolutely.
Oxide and Friends
RFDs: The Backbone of Oxide
And this is a good example where, and I think, Adam, you mentioned this at the time we were talking about it, where all of these systems have got drawbacks. It's easy for everyone to have an opinion on it. The opinions get kind of tribal, and it's not great. And in terms of like, it's not great in that there's not an unequivocal answer. And, you know, the RFD on chat got a little hot. Yeah.
Oxide and Friends
RFDs: The Backbone of Oxide
It is not worth checking out the hot comments on chat. And I think that on the one hand, you're not going to resolve that. On the other hand, it is actually helpful to have all the – I think it's like if you're going to have organizational conflict, Let it be in writing. You know, it is very helpful to have everything because you can actually like it.
Oxide and Friends
RFDs: The Backbone of Oxide
I think it forces people to be complete about their arguments. I mean, it's obviously like think of the lawyers.
Oxide and Friends
RFDs: The Backbone of Oxide
So we were getting, like, it's not an accident that these are kind of the first things we're laying down. And RFD1 is the RFD process itself. So, yeah, I mean, it was extremely early. And that's interesting. The first commit message isn't until October, right?
Oxide and Friends
RFDs: The Backbone of Oxide
Think of the lawyers. Will somebody please think of the lawyers?
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. And so I do have, and I actually made it public this morning or yesterday on RFD 113. which I did write, which is kind of how we actually make determinations. And it's giving this kind of loose guidance that you're describing. You can't be overly prescriptive about how decisions are made, kind of obviously.
Oxide and Friends
RFDs: The Backbone of Oxide
But, you know, really talking about, and kind of differentiating decisions from determinations, talking about some specific examples where, you know, how we made determinations in the past. Because it's, on the one hand, there are, I would say there are plenty of RFDs where it's like, no, no, I want to throw this out here. I really want to get someone's discussion.
Oxide and Friends
RFDs: The Backbone of Oxide
In other cases, like, we've got, and especially if you've got an issue where we have to move quickly often like, okay, there's been a lot of process that's outside of RFDs, and now actually the RFD is going to pull all this together and kind of describe what we've already done a little bit.
Oxide and Friends
RFDs: The Backbone of Oxide
But it's actually extremely helpful to go do that, where we can actually go through the different things that we've kind of considered, which is very important. So I think that sometimes the RFD process ends up being a little bit behind the process, but I think oftentimes it is. It's an earnest request for discussion. There's a question about whether an RFD becomes immutable.
Oxide and Friends
RFDs: The Backbone of Oxide
maybe I had this kind of image of, but you know what, that as you say that I am feeling cold in that garage, that unheated garage, that thing got like frosty in October, November, December. Like we really did need to move into a place with heating.
Oxide and Friends
RFDs: The Backbone of Oxide
We do have the published state, which is very valuable. I need to get more of my own things into publish. That RFD 113 is not the published state. Actually, it's funny. So just to say it's an interesting like object lesson. RFD 113 is not in the published state. It's in the discussion state, even though it's obviously 113 and we are now at last updated in December of 2020.
Oxide and Friends
RFDs: The Backbone of Oxide
It's like, okay, why is it still in the discussion state? This should clearly be like published. And so I'm like, I'm going to go mark this as the published state. But there's a comment in there, actually, our colleague, Keith Wazowski, left a very thoughtful comment on there that I had read, you know, in 2020.
Oxide and Friends
RFDs: The Backbone of Oxide
But I actually want to go, there's some things in there, I actually want to modify this RFD a little bit before I publish it based on that single comment. And it's not contentious. He doesn't disagree with anything here, but he's got a perspective that I want to capture in here a little bit before I move it into the published state. So that's why that one is still in the discussion.
Oxide and Friends
RFDs: The Backbone of Oxide
But yeah, I mean, in terms of when, and I think Ilyan is also saying that RFDs can languish in discussion, which I think is okay. I think it's like, I think it's, certainly I'm guilty of this too, but I think when we're moving into the published state, we're saying like, okay, this is basically like, We've kind of accepted this. It doesn't mean that it's immutable.
Oxide and Friends
RFDs: The Backbone of Oxide
And I think importantly, we can go back and change RFDs that have been published. We don't want to become captive to our own process in this regard. And I think by and large, we haven't been captive to the RFD. I don't think it's been incarcerating at them.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, and then I think it's also helped them have the abandoned state. Abandoned, it is. And chat RFD may be in the abandoned state. Just to indicate to someone, especially new to the company, like, oh, okay, I don't need to spend a lot of time looking at this one.
Oxide and Friends
RFDs: The Backbone of Oxide
This was an idea that, and I feel that was something, I mean, Robert, I feel that was something that was kind of a needs improvement from the Joyent era as well, where we ended up with some RFDs that were not what we did And then we need to kind of make clear, no, no, this is not, we didn't go this way.
Oxide and Friends
RFDs: The Backbone of Oxide
We had a thoughtful discussion on it or what have you, but you shouldn't view this as kind of canonical because we've abandoned it. I think the other thing I would say is that when we are getting better at is linking RFDs to one another. So you've already seen this, and especially like when we were referring to Dave's RFDs last week and 110 and 53 and kind of the RFDs that they refer to.
Oxide and Friends
RFDs: The Backbone of Oxide
I think that ASCII doc makes it pretty easy to link to those things and kind of footnotes and so on. But I think that being able to navigate that web becomes really important.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. And it was kind of like, you are here in this RFD, and this is how it relates to the other RFDs. And that was really, really important. And then, I mean, in our next-gen sled, Robert, you want to talk about how you, because the RFD there, we've got an RFD that really is the entire architecture of that kind of next-gen sled. Do you want to talk about how we're using RFDs for Cosmo?
Oxide and Friends
RFDs: The Backbone of Oxide
Okay, you are apparently emphasizing barely in that. I am emphasizing heated. I think it is heated, if barely. No, it's fine. It's actually going to be warm, so it's going to be a roaster in the office today.
Oxide and Friends
RFDs: The Backbone of Oxide
No, I just, you're reminding me of, I think, is it 101 much new about optics?
Oxide and Friends
RFDs: The Backbone of Oxide
So in terms of the history here, and Robert, I'm hoping you can help me out because I'm realizing that my recollection of this, I'm going to be interested to compare notes on the recollection of this. But for the history of this, you actually have to go back before Oxide, certainly, before Joyent, actually, and have to go back to Our Days at Sun.
Oxide and Friends
RFDs: The Backbone of Oxide
we do something different at least you have some of the context of what were the different things that went into it oh it is extraordinarily helpful and it was it was uh 112 as much do about optics robert um i would we'll need to look to see if there's any and if there's no nda material in there i want to make maybe you want to make that one public just because that have you read 112 adam
Oxide and Friends
RFDs: The Backbone of Oxide
Because you're like, I don't know, optics, how complicated can it be? It's like, oh, okay. No, I haven't. Oh my God. I mean, this is one of these and I feel like there are many RFDs that are in this category where this thing will take you to absolute crush depth on optics and how complicated it is. And
Oxide and Friends
RFDs: The Backbone of Oxide
I, I mean, Robert, you've done this many, many, many times where it is so helpful to have the entirety of the depth of our thinking in an RFD. Because as you say, it's like, you know, you may look at that and God, there've been so many RFDs that I read once. I'm like, okay, yeah, that makes sense. I don't know. Makes sense.
Oxide and Friends
RFDs: The Backbone of Oxide
And then later, due to change in circumstances, I'm now reading this very closely. Maybe it's because I'm implementing it. Or maybe it's because we're about to have a call with a partner where this partner is misbehaving. Or maybe we're having a call with an investor or having a call with a user. We're having a conversation where we're like, I need to ramp up on everything we know about optics.
Oxide and Friends
RFDs: The Backbone of Oxide
I need to know in the next 45 minutes. It's like, good news. Um, and you can go actually get to the entire depth of our thinking.
Oxide and Friends
RFDs: The Backbone of Oxide
And I think that this is where the, the written artifacts are so important for scaling an organization because it allows, I mean, Robert, with so much of your thinking written down, people can go read your thinking before they, they need to like ask you a followup question. Right. Um, and that is just extraordinarily important. Um,
Oxide and Friends
RFDs: The Backbone of Oxide
So it's been, I mean, you're talking about, Adam, how this has become more valuable over time. It has become more valuable because we are able to use them more. We have more people using them, to Ben's point about making this accessible. We've got more people creating RFDs. We've got more people reading RFDs in terms of the public and customers and so on.
Oxide and Friends
RFDs: The Backbone of Oxide
And then we've got these kind of fine-grained permissions so we can get the right RFD to the right person, which has been... incredibly important for us. It is the backbone of Oxide. Absolutely. This is, it's been great. Not just the discussion, but boy, this work, all of you have done such terrific work on this stuff. And it's been really, really essential.
Oxide and Friends
RFDs: The Backbone of Oxide
Wild. I'd expect an out loud rating. Adam, thank you for all the collaboration you've done with all of your colleagues on RFTs. Much less work than Robert's words. really great stuff. And again, if you're looking at this thinking like, boy, I would love to adopt that system, please adopt it. We really try to make it so it is adoptable.
Oxide and Friends
RFDs: The Backbone of Oxide
So, you know, take it lock, stock and barrel, take the parts that work and report back. We'd love to hear about it because I think this is something that a lot of engineering organizations really, really need. On that, thanks, everybody. Thanks, Europe.
Oxide and Friends
RFDs: The Backbone of Oxide
Adam, this is kind of where the earliest stuff of this gets laid down. When Sun, I think not unusually, had this architectural review process. And in particular, there was a platform software architectural review committee, PSARC. And PSARC would... would approve architectural changes to the operating system.
Oxide and Friends
RFDs: The Backbone of Oxide
I feel like we'll need to do a European-timed episode at a time when you're not in Europe, just so we can prove that we're like, oh, yeah, they're remote-friendly because the CEO no longer wants to work. He doesn't want to move his family. That's why they're remote-friendly. We'll need to do some European-timed episodes. We're going to do this once a quarter or something like that.
Oxide and Friends
RFDs: The Backbone of Oxide
And it was a, and I'm sure we've mentioned, we probably talked about PSARC in the D-Trace episode, but PSARC was this body that fancied itself a pretty august body, but you would develop materials for a PSARC case.
Oxide and Friends
RFDs: The Backbone of Oxide
And I mean, Adam, I'm sure you had the same experience where the development of materials for your PSAR case, I thought was very valuable where you'd be writing your own materials or, you know, you would have a case and would iterate on it with like Mike and me before you submit it or whomever, right? Roger, whatever. That process, I felt to be very valuable.
Oxide and Friends
RFDs: The Backbone of Oxide
Yes. That's what I'm saying. Okay. Okay. So what I'm saying is it was like the, the, the part of the process that was valuable was the part of the process where you wrote everything down. That was valuable. Right. Right. And I found many issues in my own thing, in my own stuff by, I mean, small issues, but like, okay, oh, right.
Oxide and Friends
RFDs: The Backbone of Oxide
As I'm writing this down, I'm like, I realized I hadn't quite thought about this and I'm going to get ahead of some kind of just ill-conceived discussion about this. So I'm going to clarify this. Right. And that was all valuable. And then everything from that was not valuable. And in fact, was awful. Yeah.
Oxide and Friends
RFDs: The Backbone of Oxide
Because you had spent – if you work on this earnestly, you're like, I've spent a lot of time thinking about this. And other than like – It's not impossible that someone would have a suggestion that would be valuable, but that's not the way the PSAR generally phrased it. PSAR was not phrased as suggestions. It was phrased as approval or rejection.
Oxide and Friends
RFDs: The Backbone of Oxide
You know, parole board is a really good way of phrasing it. Like, I don't know, like, are you a recidivist? Is this going to, I just feels like, I know you've had a couple of character witnesses in here, but I don't know.
Oxide and Friends
RFDs: The Backbone of Oxide
The politics of it was onerous. And I mean, I know we turned the cockroach episode into a Postgres episode, so I don't want to turn the RFD episode into a dark episode. But that said, you're like, why would you be saying this unless that is exactly your intent? But I do think it's worth mentioning that. And I would say that if you are in...
Oxide and Friends
RFDs: The Backbone of Oxide
an engineering cultural milieu that has this kind of August review boards, a trick that worked exceedingly well was to deliberately have an issue. And this, by the way, has a deep history in software engineering. This is the queen's duck. But deliberately have an issue that is a small issue that everyone can wrap their brains around that you yourself are equivocating on.
Oxide and Friends
RFDs: The Backbone of Oxide
You just got some things that are not adding up. I do, I gotta say, I love the European window dimensions. I love the tall, narrow windows. That's nice. It's a very European thing. You know, I think it's like, that's how the AI knew to generate in Europe.
Oxide and Friends
RFDs: The Backbone of Oxide
Like, God, I just don't know what to call this thing. What should be the name for this?
Oxide and Friends
RFDs: The Backbone of Oxide
That's right. And I mentioned this is the Queen's Duck, which goes back to a game called Battle Chess. I don't know if you ever played this game, Adam. This is in like 80s, early 90s. And the VP at the company was making Battle Chess was a famous kind of micromanager. So they would deliberately introduce things that they know that he would be, they would basically occupy him.
Oxide and Friends
RFDs: The Backbone of Oxide
And so he would all, and one of the things is they had the queen in Battletrust holding a duck. And he'd be like, you know, he's like kind of looking over it very thoughtfully. And then like, you know, the queen, queen should not be holding a duck. Like, oh God, yes, thank you. Yes, thank you. So they, this is a trick that works extremely well. We did this and it worked all too well.
Oxide and Friends
RFDs: The Backbone of Oxide
But all that is wasted energy. Like you should not need to, like all of that is, is the organizational politicking and it sucks and it's not uplifting even when it works extremely well. Okay, when it works extremely well, it's a little uplifting, but not very uplifting, and it's a waste of energy. But the thing that was valuable was writing all that stuff down.
Oxide and Friends
RFDs: The Backbone of Oxide
So, okay, we went to Fishworks, where we wrote nothing down, I think. Do you remember our first bug database?
Oxide and Friends
RFDs: The Backbone of Oxide
Oh, that's right. Excuse me. I will thank you not to refer to SCCS as RCS. I will thank you to not refer to our defunct source code management system as a different defunct source code management system.
Oxide and Friends
RFDs: The Backbone of Oxide
Do you know where that bug database is from? That's the Detroit bug database. That's right. Don't you remember that? Yeah, yeah, yeah. I wasn't going to air that dirty laundry. That text file has a deep pedigree, friend. But we, other than writing one line of text for every bug that we fixed, there was basically nothing.
Oxide and Friends
RFDs: The Backbone of Oxide
And we certainly weren't dealing with piece art, and we were kind of a renegade outfit to begin with, and talk about an outfit that should not have been purled. And then I went to Joann, and we kind of fast-forward a couple of years, and And I was really missing writing things down. And it was happening kind of sporadically. And Robert, obviously, this is where I would love to call on your memory.
Oxide and Friends
Reflecting on Founder Mode
Okay. So this is not a hot take. This is a little bit of a take in that we are talking about something that was very hot two weeks ago. That's right. We, we've been on a two week hiatus and you are our last episode was the RFD episode coming in from Italy, which is great. Or so the image that you generated said. I got confirmation on that one.
Oxide and Friends
Reflecting on Founder Mode
It's like, this is great. So we don't need to stop. We're not stopping for gas at all. No, no, no, no, no.
Oxide and Friends
Reflecting on Founder Mode
You're doing that thing. You do again. You're only going to say the half the story. Yeah. So I thought that was, uh, was really interesting. I, the one I, but I got to say, and this is where, uh,
Oxide and Friends
Reflecting on Founder Mode
the thing that was very influential for me, and this is like, I do think the positive of the founder mode piece is his, the second failure of that's not how it's, that's how it's done, where he brings in a bunch of HR professionals that, and the thing that was interesting, and you might find this actually really hard to believe given the kind of the way the piece is written.
Oxide and Friends
Reflecting on Founder Mode
So I have to write a hacker news comment, write a hacker news comment. Like now I'm done. Don't think about it. Still thinking about it. Still thinking about it. Got to write a blog entry. And I've learned that it's just like, I can't fight that at that point. Like I got to go. It's, it's just gonna be faster for me to write the goddamn thing and be done with it. And, um, I, I,
Oxide and Friends
Reflecting on Founder Mode
When I read that, I, I didn't think that was the failure. And I was kind of at a moment where I was beginning to question some of my more idiosyncratic methods, which you might already be like, first of all, just don't believe that. That doesn't sound credible at all. But I really was. I was just like, God, maybe... Because I'm doing things that are really differently than other people.
Oxide and Friends
Reflecting on Founder Mode
You're like, maybe we should be bringing in HR professionals like Tim says? I definitely was wondering on things like performance review, where it's like, I don't know, maybe. And... the, and then he kind of like hit. And so he says, I, you know, I complained, but I eventually gave in. Yeah. I literally, what you, I mean, hand on heart, I am speaking my own truth, whether you believe it or not.
Oxide and Friends
Reflecting on Founder Mode
Maybe I don't even believe it. I'm saying, but when I read that in 2013 or whatever, I'm like, maybe I should give in. And then the next sentence is like, no, no, you idiot. It was a disaster. And I thought it was like, and I think the danger, I think I liked about this is it is very specific. Yeah.
Oxide and Friends
Reflecting on Founder Mode
with respect to some specific policies that were on regular reviews and so on that were very, very specific, and any kind of regrets not having preserved the culture longer, that does not mean you discard all expertise, I think. And I think that's where you get into the false dichotomy that gets dangerous. Yeah, totally.
Oxide and Friends
Reflecting on Founder Mode
Totally. Where you have something that worked well and now is not working at all. And not understanding why this thing worked so well. And it's like, well, maybe that was incidental success. Maybe that thing wasn't working. Or, I think probably more likely, it was implicitly dependent on a bunch of factors that you didn't necessarily see. Totally. That you didn't, yeah. I think it's...
Oxide and Friends
Reflecting on Founder Mode
It's tough and it's complicated. But so that was, that was the bit in that piece that was very influential for me and gave me some of the guts actually to be like, okay, I'm actually going to, it's okay to actually go my own way on this stuff. Yeah.
Oxide and Friends
Reflecting on Founder Mode
But I think you need to be really, I do think you need to be deliberate because I think that the, again, the danger of Graham's piece is that like, oh no, you just go founder mode, which to me, doesn't that, what do you think of when you think, oh, here founder mode?
Oxide and Friends
Reflecting on Founder Mode
It does sound like an episode of Silicon Valley is out of HBO Silicon Valley. I don't think you like Beast Mode and Marshawn Lynch. Am I the only one that has a strong Beast Mode? And I think that Paul Graham would dismiss that as a sports ball, so I don't think he would know who Marshawn Lynch is or what Beast Mode is. But that...
Oxide and Friends
Reflecting on Founder Mode
you kind of have this idea of like founder mode is just not going to accept any sort of negotiation or that's right. Absolutist. Absolutist. Founder mode demands results and demands them now. And I mean, that is, So that's kind of what prompted me to be like, all right, can we undo some of the false dichotomy here? Yeah.
Oxide and Friends
Reflecting on Founder Mode
And which I feel Tim's piece does, but I wanted to, I mean, part of what motivated me to write that, write the oxide piece is,
Oxide and Friends
Reflecting on Founder Mode
Wrote the blog entry. Was kind of debating, is this on a personal blog or did I put this on the Oxide blog? But I'm like, you know, I need to get Steve's fingerprints on this and get it on the Oxide. Because obviously I want to. It actually was helpful getting the Oxide blog. It actually helped me. There's some things I probably. Dialed down a little bit. Dialed down a little bit.
Oxide and Friends
Reflecting on Founder Mode
is I wanted to also I wanted did you click on my link for dramaturgical dyad of course thank you I want to see which specific thank you Simpsons reference like what what what frame of thank you thank you I felt underappreciated on dramaturgical dyad where did you expect to get that appreciation I think right here right now so I think I got it I think I got it actually yeah because you were traveling you were out so it's like why yeah no I just got it actually I don't think anyone um
Oxide and Friends
Reflecting on Founder Mode
No, I appreciate that. I felt like it was an audience of like five. And I don't know who the other four people are. Actually, no, it's you and Dave. That's right. Dave. I mean, if I can think of who would get it, I'm not sure they're going to read the article. You know, they're not going to read the article. Do you know who would get that? It's actually my kids.
Oxide and Friends
Reflecting on Founder Mode
Not Tobin, but my 17-year-old and my 12-year-old are scholars of the early. Yeah. And you're like, hey, kids, new blog post from dad. I knew blog posts from dad. I, you know, I didn't do that. I, but you know, I haven't done everything wrong. This is one of those moments when they, uh, when I did read it, they would get it. They did read it. Yeah. Well, this is where I've done other things wrong.
Oxide and Friends
Reflecting on Founder Mode
They're not going to read the blog entry, but I do think that the, that Steve jobs and like Steve jobs and John Scully, uh, is the dramaturgical diet of Silicon Valley.
Oxide and Friends
Reflecting on Founder Mode
Yeah. Absolutely. I mean, this is like the Tom and Jerry. This is, and I feel like that is really reductive. Totally.
Oxide and Friends
Reflecting on Founder Mode
Absolutely. And I mean, again, the question that I am dying to ask the dead Steve jobs is what was the influence of next? Yeah. on your return to Apple. Absolutely. Because I feel like Steve Jobs, the next big thing was such an, I mean, such an outstanding spotlight on this long period of history, 11 years. And, you know, I know we've talked about it before. I think it's an amazing book.
Oxide and Friends
Reflecting on Founder Mode
I love the fact that that book is written when he's left for dead.
Oxide and Friends
Reflecting on Founder Mode
because it just like, there's no reason to like praise this guy. Right. And I don't, I don't think it's like overly condemning, but it's just like, yeah, I'm definitely not like letting him review it or what. I don't, I don't care what he thinks about it. Like part of like, this guy is down and out and I'm like, I'm, I'm here to, to tell the, the story of,
Oxide and Friends
Reflecting on Founder Mode
a fall right it is not a fallen resurrection story it is a false story and and i think that like the the problem with i mean i was shocked that the isaacson because i you know we talked about this on our yeah i think our first podcast right or one of the early ones went back to the isaacson's biography and it's like next is like six pages really and like man i think it's so much more important than that yeah because this guy did not succeed at next
Oxide and Friends
Reflecting on Founder Mode
And how much did that, and like Scully wasn't totally wrong on a lot of things. Yeah. And Jobs definitely wasn't right in that, in that kind of, that 1985 dramaturgical dyad. Just wedging it in there one more time. Good. For all it's worth. So I felt that we kind of need to break that one apart. And I do feel that, I mean, again, I've said it before, but boy, mandatory reading on jobs.
Oxide and Friends
Reflecting on Founder Mode
Yeah, dialed down a little bit. Not, you know, not hugely, but I think dialed down. I mean, so you're abroad when you read the piece. Yes. And what is your take on, on the piece other than like, thank God we're not recording tomorrow.
Oxide and Friends
Reflecting on Founder Mode
You just can't stop with the Isaacson bio. You really have to dig deeper on that guy.
Oxide and Friends
Reflecting on Founder Mode
Totally. And actually, this is where I got to say, Paul Graham and Steve Jobs, I mean, I'm glad we're talking about them in kind of a single episode because they are, to me, they are both, they're complicated for me. Cause Paul Graham is actually complicated for me. Paul Graham is not, I mean, Elon Musk is not complicated. Sam Altman is not complicated for me.
Oxide and Friends
Reflecting on Founder Mode
Paul Graham is complicated for me because he's not always, he's definitely not always wrong. He's got some, he's got some really important insights. And actually I like one of his, the important insights that again was very personally influential for me and for us at Oxide is, uh, do things that don't scale.
Oxide and Friends
Reflecting on Founder Mode
do things that don't scale is a very good way to say like, this is working for us now. You and I are having a disagreement about whether this can work for us in the indefinite future. We don't actually have to have that argument right now.
Oxide and Friends
Reflecting on Founder Mode
What I think, and this is where it is really important to figure out, like, what are... what are the kind of the hard rules of the road? Yeah.
Oxide and Friends
Reflecting on Founder Mode
And, and I feel like, you know, we have spelled that out very explicitly at Oxide in part because of our experience having not where that's not clearly spelled out and where, where founder mode moves into things that are like pretty shady, you know, where, cause you get things like, look, I understand what you're saying about complying with the law, but we'll, and we'll do that.
Oxide and Friends
Reflecting on Founder Mode
Like when we, it's like, I don't think Paul Graham said do things that are sometimes illegal. I think there was something that's legal.
Oxide and Friends
Reflecting on Founder Mode
um but i have seen that you know i've you get some really kind of gross stuff so like where is that line too right you want to have like queer lines there but still allow for a company that is young to do something that it wouldn't do in the indefinite future yeah absolutely and to be clear like when i was talking about recognizing revenue i'm not talking about like cooking the books but there's a difference between like cooking the books and uh like how you're backdating revenue in a
Oxide and Friends
Reflecting on Founder Mode
People were just concerned about what the future would look like. What the future would look like, right. And also what kind of precedent or setting and so on. It doesn't actually matter. Totally. Let's make them happy. And how can we make these customers happy?
Oxide and Friends
Reflecting on Founder Mode
So another thing you had mentioned is kind of this funky dichotomy is that you, as a startup, you're being encouraged to do things that are iconoclastic.
Oxide and Friends
Reflecting on Founder Mode
And then immediately defer to... LinkedIn wisdom. Wisdom from the resume.
Oxide and Friends
Reflecting on Founder Mode
Right. Forgotten the drive. They don't want to do it again. Kind of bored by it. Also, I didn't do that. I was at that company when it was done. And yes, I was the EVP of sales. But actually, I didn't actually grow revenue myself. I was merely in the building when that happened.
Oxide and Friends
Reflecting on Founder Mode
And by the way, we've talked about this in the past, but definitely one of the things that we lived, and I think a lot of startups live, is... where you take on this expertise from your go-to-market function in particular, and you take on an EVP of sales coming from an established company, and then they need their whole organization to come with them in order to be successful.
Oxide and Friends
Reflecting on Founder Mode
And all of that just adds to the burn. And then you, as a founder, realize too late, like, wait a minute, I've made a huge, huge, huge mistake here.
Oxide and Friends
Reflecting on Founder Mode
Yeah. And I got to say, when people talk about in particular, the things that they've done so that, you know, I, it's on my resume. I was at this company for those years and, And I can kind of make arbitrary claims that aren't really fact-checked. Yeah. And there is, I think, a gross inclination.
Oxide and Friends
Reflecting on Founder Mode
And I think this is where – the reason this is a problem is because this leads to the mistakes that Graham is identifying as people that are – not flim-flam men. What does he call them? Those that are misleading liars, effectively. He has a name for it somewhere. Yeah.
Oxide and Friends
Reflecting on Founder Mode
um they're founders being gaslit but he has there's a specific return of phrase he has about people that are are misleading um and the and the problem is that people exaggerate what they've done um and they you don't in fact we had someone i mean recently who is like, Oh, like, let me advise you because I built this company.
Oxide and Friends
Reflecting on Founder Mode
I, I built this company to from a hundred million dollars in revenue to a billion dollars in revenue, making up numbers. And it's like, Oh, wow, that's wild. And then you look at like when they were there and it's like, and actually notice that they, they, they had claimed that they had built it from earlier than that. So kind of its first kind of $250 million revenue.
Oxide and Friends
Reflecting on Founder Mode
It's like, by the time you joined the company, it was a public company.
Oxide and Friends
Reflecting on Founder Mode
And the revenues were much higher than – are you lying to yourself right now? And I think this happens a lot where people have kind of told themselves their own history, and it's kind of impossible to fact check. And so you take on this expertise as a founder, and then you discover that it's not happening for you, and you wonder –
Oxide and Friends
Reflecting on Founder Mode
And this is where, you know, when Graham talks to talking to founders who had felt gaslit, I think this is where that's coming from, where they brought in that expertise. They're not seeing the results. And then they're being told, no, no, you're not seeing the results because you don't understand the problem.
Oxide and Friends
Reflecting on Founder Mode
That's right. And this is where, okay, I found the turn of phrase in Graham's piece. In practice, judging from the report of founder after founder, what this often turns out to mean is hire professional fakers and let them drive the company into the ground. So that's the professional fakers. This is turn of phrase there. And again,
Oxide and Friends
Reflecting on Founder Mode
The thing that I think Graham does a disservice to is he blames the fakers. Yeah. Fine. He does not blame the founders. He does not blame the investors that force that upon their founders. Totally.
Oxide and Friends
Reflecting on Founder Mode
Right. Now, I'm trying to get all of the many, many examples I'm thinking of. Yeah, fine. Let's go with what you're saying. Absolutely. Absolutely.
Oxide and Friends
Reflecting on Founder Mode
Because the success taught them nothing. Yeah, I think that's right. Because they were there. They were there when the success happened. Why would it not be due to their hard work? It's like, well, it's more complicated than that. Yeah. And then you end up with, I mean, that's what Graham calls the most skillful liars in the world. And again, I think you're right.
Oxide and Friends
Reflecting on Founder Mode
That is like a little bit, on the one hand, that is too condemning of them because it's not necessarily dishonesty. On the other hand, it's maybe a little bit too generous because a lot of these people are lying to themselves. They just don't realize it.
Oxide and Friends
Reflecting on Founder Mode
Just faith and experience is not enough. Faith and experience is not enough. And I do think... I mean, I've said this many times before. I'll say it many times again. Team formation is the most expensive, most important thing you, one does at a company and you need to spend a lot of time on it. I think too many people are like, well, the recruiter handed me these four names.
Oxide and Friends
Reflecting on Founder Mode
So that's, those are the names. Yeah. It's like, what's your rubric for, you know, you are hiring someone in a domain in which you currently have no one. What's your rubric? And one of the things that I've always found, and again, this is the specificity that Graham's piece is totally missing, of how do you add? There are people who are superlative people who can help you out as a founder.
Oxide and Friends
Reflecting on Founder Mode
Yeah, and I think the one thing I will say in Graham's defense is he anticipates this a little bit. Okay. He indicates that, he's like, in one of the footnotes in particular, I have another less optimistic prediction.
Oxide and Friends
Reflecting on Founder Mode
You don't need to go to founder mode for everything. You actually do need people who can add a lot to you. How do you find those people? What chances do you take? Because I think you're right that it's surprising how conservative startups get. Yeah. Because they don't know the role. Actually, I don't know what go-to-market is. I don't know what marketing is. I've never worked in support.
Oxide and Friends
Reflecting on Founder Mode
I don't know these things. So I actually don't know how to evaluate someone. And it's like, that's where you've got to... I think it's incumbent on you as a founder. Yeah, you've got to ramp up. You've got to figure out a way to go... solve that problem.
Oxide and Friends
Reflecting on Founder Mode
And then the internet kind of melted down that next weekend over this Paul Graham piece. Yeah. Founder mode. Yeah.
Oxide and Friends
Reflecting on Founder Mode
And this is something, again, I'm not sure people see it, and it sounds like it's true of the Plaid founders as well, but this is what the Carlson brothers did well. I want to actually, I'm curious to understand, and at one point Patrick reached out to me, he was like, what should I be looking for in a VP of engineering? I'm not trying to hire you.
Oxide and Friends
Reflecting on Founder Mode
I'm trying to come over and have free food, which of course, as you know, pretty much. It's okay. It's time for you to leave, actually. I think that people don't do this nearly often enough is to ask for perspective from other people. Who should I talk to? Who's the best person in this role that you know? I'm not trying to hire them. I just want to know what I should be looking for.
Oxide and Friends
Reflecting on Founder Mode
And I think, and we still do that. I mean, I think it's, it is, you know, we've always done that and still do as we extend into kind of new roles. And it is really, really, really important. I'm not sure we, we, I don't think we've talked about it. I don't think we talk about it broadly as an industry and we should like you like look to other people, senior people to get their perspective.
Oxide and Friends
Reflecting on Founder Mode
This again, the bit that Graham is just like totally missing.
Oxide and Friends
Reflecting on Founder Mode
That's a very good point. Because I do think one of the challenges is, of the idiosyncratic, iconoclastic things you're doing, how much of that is really important versus not? I'll give you a very concrete example. We had an early investor in the company who in an early conversation, an early board meeting, I might've brought up the podcast. Mentioned the podcast. Okay. And this is on the metal.
Oxide and Friends
Reflecting on Founder Mode
Okay. The board member made clear in no uncertain terms that I was never to speak of the podcast again. That is, and I was just like embarrassed for bringing it up. Like, okay, I'm not going to bring up the podcast. And I'm like, what do we do? Like, do we, should we not do the podcast?
Oxide and Friends
Reflecting on Founder Mode
Like, I'm kind of like, it feels like the podcast feels like it's pretty cheap and it feels like it's pretty valuable. Yeah. Do I, and I was, I didn't really know it. And then another board member came up to me, took me aside and be like, Hey, I love the podcast podcast. Podcast is great.
Oxide and Friends
Reflecting on Founder Mode
As soon as the concept of founder mode becomes established, little did you know that this would happen within literal minutes of this thing getting out there, people will start misusing it. Founders who are unable to delegate even things that they should will use founder mode as an excuse. I thought that was a good moment of reflection for Paul Graham.
Oxide and Friends
Reflecting on Founder Mode
I like, I don't bring it, don't bring it up around him, but just, and the, like, I think that like you will get conflicting point is like, you will get conflicting advice. I don't think he was necessarily wrong. Like I understood the perspective that he had and like, we do not talk about the podcast at a, at a board level, but, But the podcast is the right thing for us to go do.
Oxide and Friends
Reflecting on Founder Mode
And I think there's a lot of things like that where, in other words, like that in terms of like, I'm getting conflicting advice. Someone's telling me like, absolutely, do not do it. You're out of your mind. And someone's like, that's a great idea. And you have to pick and choose. And you have to know, you have to kind of... know your own self and have your own sense of judgment.
Oxide and Friends
Reflecting on Founder Mode
But you need to seek out that wisdom, though. Ask the question. I'd rather know the perspective. For sure. And I think that too often we just kind of ignore the perspectives.
Oxide and Friends
Reflecting on Founder Mode
So I think we talk, I can't remember if we talked about this on the hiring podcast or not, but you know, one thing that we do is when we're hiring a new role, we go to the person that we know just in our kind of extended world is like, this is the person who I can't hire.
Oxide and Friends
Reflecting on Founder Mode
But I really look up to in this role and I want them to look at the materials that we're going to, that I want them to look at the portfolio. And we, a bunch of times we've gotten, I mean, just extremely good suggestions about things to add to the portfolio of work. It's been really, really valuable. Yeah, totally. Because that's informed the rubric. Yeah.
Oxide and Friends
Reflecting on Founder Mode
Um, also more than once, one of those people have been like, actually, I might be working for you jerks. I don't know you turkeys. Right. Um, so that, that can happen too. Um, but I think that that is definitely important to seek that out. And again, this is what kind of Graham doesn't get to the other thing that, okay.
Oxide and Friends
Reflecting on Founder Mode
Like, as long as we're, can I talk about some other things that just drove me nuts? I mean, or do you want to know the idea of like the idea that he has created some new element in the lab, um, The idea that he has discovered some heretofore unknown super alloy that it will be incumbent upon future generations to study in detail. Books on founder mode, we don't know what it is.
Oxide and Friends
Reflecting on Founder Mode
Books have never been written on founder mode. It's like, what are you talking about? No one's written history. No one knows what leadership is. I mean, Paul Graham, surely you read more than this, right?
Oxide and Friends
Reflecting on Founder Mode
Exactly. In the future, we will understand founder mode. But now it is a complete mystery. It's a complete mystery to you, pal. The rest of us are out here running companies. I've got some thoughts on this one. It's not dark energy, right? It's really not. It is really not. So I think that was also a little bit ridiculous. But I actually also feel that there wasn't... So I felt that was annoying.
Oxide and Friends
Reflecting on Founder Mode
But I felt there was little in this that was completely wrong. Yeah. Other than the false dichotomy that it leaves you with. I don't know. I mean, I feel like the false dichotomy is fairly central. Yeah, I'm already realizing I am over my skis and I'm defending this piece way too much.
Oxide and Friends
Reflecting on Founder Mode
We're in the mirror neuron empathy preschool over here with Silicon Valley. You got to be like, good job, Paul Graham. Good job. Can everyone give Paul a round of applause for his act of empathy? Okay, no. Sam, stop it. No. Elon. Elon. Get back here. I do think it's interesting that Elon... Did you see the list of people that reviewed this, please? Including Elon Musk.
Oxide and Friends
Reflecting on Founder Mode
and I think actually for that, I do think that, so he references Brian Chesky's talk. Yeah. That was given at Y Combinator, so that one we didn't see. But Chesky interviewed a couple of other places talking about his experience at Airbnb, and I felt it was much more interesting. Did you listen to that?
Oxide and Friends
Reflecting on Founder Mode
Oh, he said he wasn't a micromanager. No, he was just in the details.
Oxide and Friends
Reflecting on Founder Mode
That's right. It's not micromanaging you. You can move that finger, I think. Okay, not a bad example. I can't move that finger. I mean, as long as you don't go the wrong way.
Oxide and Friends
Reflecting on Founder Mode
When I think one of the things he says in that, that I actually like, cause I'm like, what would I do to Paul Graham's piece to make it less divisive, less of a false dichotomy? One of the things that he says, there's like people want clarity. Yeah. And often that is what they, I mean, that to me is what splits the difference is like, you've got terrific people that you've hired.
Oxide and Friends
Reflecting on Founder Mode
You want to empower. They do need clarity. Yeah. And I think that part of what he describes Airbnb operating at cross purposes, because he had stepped back from offering that clarity. Yeah. Yeah, that's a great point. And I think it is really important. Because I think that also if you've got an organization in which you foster trust, that clarity is super helpful, I think, to people. Yeah.
Oxide and Friends
Reflecting on Founder Mode
Where it's like, oh, okay, this sale to this customer is really important for our quarter. Or this feature is being weighted. Oh, I get it. Okay, that's why this is important. And that clarity can allow everybody to go to their own autonomy, find the right way to solve the problem.
Oxide and Friends
Reflecting on Founder Mode
I'm not telling people how to solve a problem necessarily, but giving them the clarity about what's important, I think is... Because if you call this piece clarity mode, it would just feel like, that feels good. I don't know. Clarity mode sounds good.
Oxide and Friends
Reflecting on Founder Mode
reviewed the article before he, before he posted it. Yes. No, I did not see that. Yes. Thanks to Brian Chesky. Fine. Patrick Collison. Fine. Ron Conway. Fine. Jessica Livingston. Makes sense. Elon Musk.
Oxide and Friends
Reflecting on Founder Mode
That's right. And you kind of have to like show them what, that's exactly right. And I do think that like autonomy is, I mean, maybe this is where you get to like my own individual style or our style at Oxide. Autonomy is great. It's just in terms of like if people have that clarity, because I think like how many times
Oxide and Friends
Reflecting on Founder Mode
At Oxide, have we seen something unanticipated that is delightful that happens, that you absolutely want to happen? It's like, I don't think anyone would have known to ask for this. But because... I mean, what I love... Maybe I'm just... easily entertained. But when Augustus did this integration between Salesforce and the ability to provision a silo on the colo rack for a prospective customer,
Oxide and Friends
Reflecting on Founder Mode
So you made it really easy for Travis to go from a prospective customer to like spinning up their own silo. So our sales folks can like get off the call with the customer and create their own demo environment. Right. And then using Dropshot and using all the things that we had built to go do this. I just like, that was great. Totally.
Oxide and Friends
Reflecting on Founder Mode
And I just felt like that is where people were like, no, no, I can see that this needs to be done. And I'm just like, I'm empowered to go do it, so I'm just going to go do it.
Oxide and Friends
Reflecting on Founder Mode
Right, and I do think it's really important that leadership founders offer that clarity on priorities and then are going to have to deal with some tough questions. I'm like, okay, these two can't both be the top priority because, I'm sorry, I'm asking, we actually are going to have to... and find a way to stack these. One of these has to be more important than the other.
Oxide and Friends
Reflecting on Founder Mode
All right. Good. And I like, what are you on Musk's comments? Yeah. I mean, probably just copy editing. Is it just, it's a kind of another type of like, is there any, is there anything substantive that you have any substantive complaints? It's just like, that's a very close read. Um, I think we're thinking of our comments for one another. On any document. Both guilty of it. That's right.
Oxide and Friends
Reflecting on Founder Mode
Sorry, which I think is, but if you don't, and this is, I think where, when you have these startups, like I need to go into founder mode because I feel like the right thing isn't happening. it's just easy to turn into a wrecking ball. And you turn into, actually, we have some colleagues from a company that were describing the management style.
Oxide and Friends
Reflecting on Founder Mode
And I'm like, wow, that is one of the worst cases of seagull management I've ever heard. And she was like, seagull management? I'm like, I don't think I made that up. Have you heard the term seagull management? I think I have, but I'm not sure that I'd be able to define it. This is like a seagull, like a seagull. It swoops in, shits all over everything, and takes off. It steals your french fry.
Oxide and Friends
Reflecting on Founder Mode
It steals your french fry. You've met my sister. Her hair looks like nesting material to birds. It's weird to the point where we just need to acknowledge it. She needs to wear a hat. Some sort of protective headdress because there's something about... I don't know what it is. Obviously, she's my younger sister, so I kind of thought she was exaggerating. But over the years, I've come to accept no...
Oxide and Friends
Reflecting on Founder Mode
This is happening to you more than it's happening to other people. Birds are swooping onto your hair and attempting to attack your skull in a way that doesn't... No, this is not routine. You're right. Does she listen to the podcast? I want to find out and I don't want to ask her. She's so busy. I just don't want to ask her. So I figured this is really the best way to find out.
Oxide and Friends
Reflecting on Founder Mode
It was really just to denigrate her on here and see if she listens to it or not. But no, it's no denigration at all. Quite the contrary. She has got... She's got a condition that needs to be... I honor her condition, her challenges in life. Other people don't have this challenge. And we were at Fisherman's Wharf. I guess she's kind of like, this happens to me more frequently than other people.
Oxide and Friends
Reflecting on Founder Mode
I'm kind of dismissive. And we go to Fisherman's Wharf. This is years ago. And... the seagull that descended upon her was the size of a Cessna. That was like, that was like a plane crash. I think it was gigantic. And like, tried to like, just like she, he thought that she was the French fry. Just like tried. It was, it was really quite dramatic. I don't know if the seagull was a show or not.
Oxide and Friends
Reflecting on Founder Mode
Definitely made her point like very, very dramatically. Yeah. So anyway, I was showing this idea of seagull management, and I think it was life-changing. It's like, I need to send this Wikipedia article to many of my former colleagues, because this is exactly... And it was an environment where senior leadership, very technical...
Oxide and Friends
Reflecting on Founder Mode
and would descend in on... People would be working on a problem for two or three weeks and would descend in and disrupt everything and take off. Great. Exactly. And that is the worst. I really think that is... It's really, I'd rather be in like command and control than that. You know what I mean?
Oxide and Friends
Reflecting on Founder Mode
It's where this kind of like false sense of autonomy where you let me, you let us together as a team develop a solution for this. And now you've descended in and like ripped it up. And now it's like, boy, that's that corrosive.
Oxide and Friends
Reflecting on Founder Mode
That's fair. I was very surprised that Elon Musk, and I would love to know, is Elon Musk like, I don't know, don't you feel it needs more Nazi in here? Just feels like... Just 10%. I'm just like, it's just a joke. I mean, those jokes are funny. Who knows? I don't know. I think that was good that he anticipated that. But I guess we should describe the piece in terms of what...
Oxide and Friends
Reflecting on Founder Mode
She listened to the podcast. I guess we're going to fight. Oh, 100%. Maybe 99.8% now. We'll see. Even the baseball one.
Oxide and Friends
Reflecting on Founder Mode
This is a good segue to Camille's piece. Camille Fournier wrote a piece that was pretty interesting in reaction to this. Founders create managers. And I'll drop a link into that in the chat. Which I think is kind of what you're saying. Is that founders created the chaos that created the situation. Founders are creating the situation effectively. And
Oxide and Friends
Reflecting on Founder Mode
which I think in terms of like, they are, you want to be a details person. Right. And you don't know the right way to let go of those details. Yeah. And, and this is where, you know, so you, cause I was in part of what kind of prompted the, the longer blog entry too, is I was noodling on like, how do you do this? How does one do this? How do we do this? And when you said this, what do you mean?
Oxide and Friends
Reflecting on Founder Mode
I mean, I, How do you keep kind of a true north to an organization as it gets larger? How do you avoid the pathologies that Camille describes? How do you avoid the seagull management? How do you keep, while at the same time, Providing clarity, and I think it is very important to stay detail-oriented. I think that is extremely important, and I think that leadership, stay detail-oriented.
Oxide and Friends
Reflecting on Founder Mode
I think it's a real problem when, and you and I have both seen this, where leadership gets divorced from the details, and all of a sudden, you've got a great blog entry from years ago, I am not a resource anymore.
Oxide and Friends
Reflecting on Founder Mode
And to me, that is indicative of this problem where it's like, these have now become opaque blocks that I can kind of move around and I've lost track of like why things are hard, why they take long. And it's now schedule over everything. I've now over-promised to a customer because I've lost track of the details that make this difficult. And now when my team doesn't deliver, I blame the team.
Oxide and Friends
Reflecting on Founder Mode
It's just like, you're in this loop that I think is a loop that is really, really corrosive. Yeah. And so how do you avoid this? How do you not get into that pathology? How do you stay detail-oriented without being a micromanager? How do you have autonomy while still having clarity?
Oxide and Friends
Reflecting on Founder Mode
And what Jeske points out, I think rightfully, is that in the absence of other things, especially where influence is being used, and especially in an organization where people do value autonomy, things can get political. And they talk about shadow hierarchy and things like that, where And so the thing that I, as I was kind of like reflecting on this, I'm like, you know, how do we do this?
Oxide and Friends
Reflecting on Founder Mode
And I do think like, man, the written word is real. I mean, dovetails for our episode from last time on our piece, but the written word is really, really important in terms of broadcast. I mean, I, I mean, God, I am so thankful and I should like, there's a bell I should ring every time this happens where,
Oxide and Friends
Reflecting on Founder Mode
maybe a prospective investor or a customer, someone asks something about an issue that I'm not up to speed on. And, oh, thank God there's an RFD on it. And I spend 20 minutes reading the RFD. It's like, damn, okay, this is great. I didn't have to waste anyone's time. You don't have to create a fire drill where you say, hey, everybody, I know exactly what our thinking is on this.
Oxide and Friends
Reflecting on Founder Mode
I know exactly what we've thought about. It's like, boy, am I really, really, really grateful for it. And I think that Amazon famously does this too. I've never worked at AWS or Amazon, but famously has a writing intensive culture where you're kind of sitting down and reading for the first half of the meeting. Yeah. I kind of love that.
Oxide and Friends
Reflecting on Founder Mode
What is the dichotomy that Graham sees? Because ultimately, this is reductive, and it generates this galactic dichotomy between founder mode and manager mode, and there is nothing in the middle.
Oxide and Friends
Reflecting on Founder Mode
This is back when they were being innovative, by the way. I don't want to... Sorry to... Yeah. As one of our colleagues who was formerly of AWS, I'm like, pretty sure the last innovative thing they did was Lambda. That was in like 2015. But I think that certainly when the Bezos-led company, they would write the press release before working.
Oxide and Friends
Reflecting on Founder Mode
And I think that's an interesting way to get ground truth and clarity. It's a press release, but in terms of getting that true north and getting that on paper. Yeah, okay.
Oxide and Friends
Reflecting on Founder Mode
It is too late. How do you, how do you possibly, this is like, don't look up, but how do you tell me? People don't even know who Dave hits is founder of NetApp.
Oxide and Friends
Reflecting on Founder Mode
I think we should feel fully comfortable. Oh, All right. I think we should feel fully comfortable. All right. I think we should feel fully comfortable. I think this is a, this is a, this is a great story. You know who I, my sister does not listen to the podcast. I don't think she might. Your wife, you say a hundred percent. I think it's unlikely she goes to the podcast. Dave hits zero percent.
Oxide and Friends
Reflecting on Founder Mode
Let's not listen to the podcast. Okay. Says I. Let's go. Okay. So Dave Hitz writes this book, How to Castrate a Bull. That's right. We were at Fishworks at the time. We were at Fishworks. We were at a NetApp competitor.
Oxide and Friends
Reflecting on Founder Mode
And what we had heard from customers is like, I love the product. I hate the price. And I'm kind of not in love with the company. The company is very self-important. But I do love the product. And someone in the chat says, I have a signed copy of Dave Hitz's book. And you, you are not the only one with a signed copy. You are not alone.
Oxide and Friends
Reflecting on Founder Mode
And you're sure this is just on the point of principle of not wanting to line his pockets by... I mean, it's like... I mean, we've said that an undercurrent of this episode is Success Teaches You Nothing. The subtitle of this book is How Success Taught Me Nothing by Dave Hitt.
Oxide and Friends
Reflecting on Founder Mode
So we have two copies. I did not realize that. It's like, wow, this person is very hot. No, you're saying you're true, but misleading things. Did they involve the fact that you'd be, that you needed a review copy?
Oxide and Friends
Reflecting on Founder Mode
That's kind of my recollection, too, is that I feel that this is a consequence of us reading the big con. Yeah. Sure, I read it. The Big Con, terrific book. The Sting is based on the Big Con. A book written in the 30s about the Argo of the grift. All right. Oh, such a good book. And the Big Con, one of the things it repeats over again is you can only con a greedy man.
Oxide and Friends
Reflecting on Founder Mode
And I think I do recall us thinking like there is a kind of an ego con here that if you praise someone who is already praising themselves, you're not going to get fact checked.
Oxide and Friends
Reflecting on Founder Mode
Dave Lightman. Dave Lightman. I, we will leave it to others to figure out the, the Dave Lightman. I mean, I feel like that's a good homework assignment. If you don't recognize the name, Dave Lightman, not in the notes, not in the notes. That's right.
Oxide and Friends
Reflecting on Founder Mode
I will find the link. Can you find the link now? I can talk about something else while you find the link. Because I just feel that we need to... First of all, it should be said that you wrote the fan blog. So one of the things we figured out... I don't know if we knew he was going to pay attention to it or not. It became clear that he was hanging on every word. Every word.
Oxide and Friends
Reflecting on Founder Mode
Like he was commenting on the fan blog. So at that point, it's like, oh my god. We got them. Oh, God bless you. God bless you, Blogspot. Sign of the times that it's on Blogspot.
Oxide and Friends
Reflecting on Founder Mode
So you've written the first summary about this book is awesome and I cannot wait to read it. I'm really, really excited to read it. That's right.
Oxide and Friends
Reflecting on Founder Mode
And when does our persona's mother enter as a kind of a... Pretty early.
Oxide and Friends
Reflecting on Founder Mode
When he, at some point has like a comment on your blog being like, God, I don't want to like, I don't want to give it away, but don't take that. Cause you made it very clear that like, you've taken that advice very literally and have, have maybe said some things to your mother that you, maybe you should, maybe you might, might live to regret. That's right. So, okay.
Oxide and Friends
Reflecting on Founder Mode
So at this point, so the, then you've got the yes, that Dave hits where he reaches out to you. That's right. Um, and the, uh, from, so he's commenting, we are just having, you should, we were having the time of our lives. I mean, just euphoria.
Oxide and Friends
Reflecting on Founder Mode
Oh, I'd say, hey, let's all go into management mode here. Adam wants us to go into management mode, so everybody give a status update because apparently that's what we need to do. We're in management mode now. I know. It feels very dismissive. Yeah. And it feels dismissive of expertise, which I think... And again, it creates this kind of false dichotomy around expertise.
Oxide and Friends
Reflecting on Founder Mode
The mouse is just nibbling away at it, just chopping. The mouse is just pounding on the door, demanding more cheese. Yeah. It's like, oh God, yes, absolutely. So you've got the net app, you've got trade dress, I believe would be the term. May even be the term from the letter they sent you. They'd print out that letter. So they send you basically a cease and desist. That's right.
Oxide and Friends
Reflecting on Founder Mode
And then you have, which I just think is like, we're like, God, just another gift from these people. So you have your nonstop lawyers blog entry. Yeah. which I mean, it was just, it was, it was delightful.
Oxide and Friends
Reflecting on Founder Mode
Future generations will. Did John Jay write this entry? That's right. Was it Alexander Hamilton? Was it Madison? Was it Madison? We can't know. We can't know. For this entry of nonstophits.blogspot.com, I can't believe this site is still up.
Oxide and Friends
Reflecting on Founder Mode
Okay. Wait a minute. You get the note from the lawyers and then you change the blog entry to be all in that app trade dress. Do you remember this? You change it to be like all like the Dave hits eyes. That's right. It was just like his eyes everywhere. Yeah. Um, Yeah, so you got him to sign the book.
Oxide and Friends
Reflecting on Founder Mode
I just like, I could not stop. It is funny. And anyone should be suspicious of a fan blog. I'm sorry. You should just be like, I would like to believe that we would not fall for this. Yeah. And, but here it's like, people are like, all those people pacing for the podcast. No, you all this or all this. Steve hits. That was Dave hits. Uh, it was great. Yeah. Well, it's a miracle that's up.
Oxide and Friends
Reflecting on Founder Mode
All right. So we, um, yeah, you, what did you learn? So nonstop hits is again, I do think this is the success teaches you nothing. But you're saying that we, that's not... You wrote the future histories, right?
Oxide and Friends
Reflecting on Founder Mode
100% true. I mean, we poured over it. Like a book club. Like Talmudic scholars, we poured over that. Every diphthong, every syllable, we, I mean, we were... No one has read that book. I mean, it's like in the end, Dave hits... when you encounter this podcast and you're upset, should you be upset? No one read that book more closely than we did. No, we were the biggest fans. We are the biggest fans.
Oxide and Friends
Reflecting on Founder Mode
We're Xers. What do you want? Agreed. That's all we have. That's all we have. That's all we have. Leave us with our ironic detachment. Okay. But he, so he advocated the future.
Oxide and Friends
Reflecting on Founder Mode
I did not know this. I had forgotten that you were depoed in that case. Yeah, I was depoed in that case. I mean, they were depoing me for stupid reasons. Why were they depoing you?
Oxide and Friends
Reflecting on Founder Mode
Mr. Leventhal, is the name Dave Lightman familiar to you? It is familiar to me. It is familiar. It is a pop culture reference, as I understand it. Uh, that is great. So you were a little bit worried. Well, you know, did you have a game plan? What were we going to do? Like this goes to nonstop hit stop blogspot.com. Like, uh, like fake a heart attack.
Oxide and Friends
Reflecting on Founder Mode
We're obviously in a safe space. I was glad. No way. I was glad. I was glad. I thought like I was glad because I'm just like, I need to get on with my life. This thing has like this piece hit just hits a lot of nerves. Yeah. And it gets a lot of people talking about it, tons of comments on Hacker News.
Oxide and Friends
Reflecting on Founder Mode
Like, because you know, when you go into a deposition, you're talking to her, I'd like to declare the fifth. It's like, that's not, you can't do that. This is not a criminal proceedings. You can't declare the fifth.
Oxide and Friends
Reflecting on Founder Mode
No. Oh my God. You should have. No. Oh, you should have. They would have, they would have died laughing. Of course. Oh my God. They would have died laughing. They would have, I love working with the lawyers in that case. They would have loved it. Oh, they would have eaten it up. Anyway. I, I just, I was unclear if we had committed any, like, I don't even know what we did wrong. Actually.
Oxide and Friends
Reflecting on Founder Mode
Fake heart attack. You're just like, hey, the ambulance is on the way. It's like, I'm fine.
Oxide and Friends
Reflecting on Founder Mode
I have a lot to tell you. I have a lot to tell you. Right. I have a fake block. You're like, what is going on? These guys, you know, maybe I saw frontiers. Okay. But so he talks about future histories. There you go. Bring it all back. Bring it all back. And is this like this idea of like writing down what we have, what we're going to do? Yeah. It's like, what do you want the press release?
Oxide and Friends
Reflecting on Founder Mode
That is extremely valuable, I think. Yeah, totally. Who knows if they did it, but it's a neat idea. It's a neat idea. And I do think, I mean, I think the written, I do think the written word is really, really, really important. And I think that the Because when you write, you have to get to a ground truth. It does scale.
Oxide and Friends
Reflecting on Founder Mode
And people who've come to Oxford are like, oh, this is great because I can understand why they made a past decision. It's been... So I think that that is a very kind of concrete thing that people could do.
Oxide and Friends
Reflecting on Founder Mode
Yeah. I think it's important. Yeah. Interesting. Yeah. I mean, the other thing I'd say is that like, I mean, God, are you, are you just trying to like, do you have a bet with some other oxygen planes? Like I bet I can get him to write an RFD on performance review.
Oxide and Friends
Reflecting on Founder Mode
No, no, no, no, no. Mayday. Mayday. That's right. Wait a minute. Are you having chest pains right now? What's wrong? My lawyer. Come closer. Um, That's interesting. Yes, in terms of offering that clarity. I think I love it when people are like, I love this company because you've written everything down. And we're like, no. There's a lot that's not written down.
Oxide and Friends
Reflecting on Founder Mode
You can't write, you can't, some things you need to leave unsaid. It's just too much of a lightning rod, but no, I think you're right in terms of like, there are things like that. And actually, you know, I honestly, I think there are some things, and this is where I've really benefited.
Oxide and Friends
Reflecting on Founder Mode
I mean, this is where you kind of benefit from just a sense of collaboration anyway, because I mean, how many RFDs have you written because someone has said you should write an RFD on that? Like most of them. So in other words, you're thinking of something that like, okay, I'm just like ideating and I don't think this needs to be written down.
Oxide and Friends
Reflecting on Founder Mode
You're not necessarily thinking about it like that concretely. But someone else is like, I would actually like to see that written down and you should write an RFD on it. I feel like certainly for me, like most of my RFDs have come from, because people are like, I would like to see an RFD on that. And I was like, oh, yeah, okay. It didn't really occur to me that that's sure. Yeah, absolutely.
Oxide and Friends
Reflecting on Founder Mode
We can do that. Yeah. That's a great idea. But it doesn't necessarily occur to you to write down something that you are already kind of feeling very strongly. That's right. But maybe those are the things you should be writing down.
Oxide and Friends
Reflecting on Founder Mode
Yeah, but I do think that if you don't, the less you write down, you're kind of immunocompromised as an organization.
Oxide and Friends
Reflecting on Founder Mode
That's right. Or whatever. Or people coming in being misleading about it. I mean, certainly, you know, We've seen this in past lives where people are, I mean, God forbid, you have someone who's telling different people different things. Yeah, right. Where if you've got it written down, like you can only tell one story, right? You can only tell one story. And that is, it's really, really important.
Oxide and Friends
Reflecting on Founder Mode
And I think that like, and sometimes they're not even telling different people different things deliberately. They are, or they don't, or people are hearing different things and they don't realize that people are hearing different things. Yeah.
Oxide and Friends
Reflecting on Founder Mode
I mean, so I think that getting, I mean, the question that founders, I don't think anyone should put in front of themselves is like, how do we offer better clarity? Yeah. And then how do we promote trust? I mean, you mentioned it earlier. I do think it's like, man, that is, trust is everything. Trust is everything, for sure. And you've got to figure out like, how do I promote it? How do I build it?
Oxide and Friends
Reflecting on Founder Mode
How do I... When it's fractured, how do I repair it? And I think that part of the problem with founder mode at its worst is you have the opportunity to like... do real, real damage to trust. Yeah.
Oxide and Friends
Reflecting on Founder Mode
It's not a me thing. Some people say caloric budget. Do they not? No. This is like blood over the ear. I assume everyone said blood over the ear, and then I Googled it, and I'm like, no one's using blood over the ear.
Oxide and Friends
Reflecting on Founder Mode
That's just me. How many things are like that? A caloric budget is like that. I think so. People can correct it from that one, I guess. Deafening silence, but there's no real corrections forthcoming from the chat over here. Yeah, you're right. Damn it. Another intervention. Another podcast, another intervention. There you go. Um, so yeah, sorry.
Oxide and Friends
Reflecting on Founder Mode
And the CTO wanted to... To be fair, there's another pejorative that's in the Silicon Valley VC lexicon of the first adult.
Oxide and Friends
Reflecting on Founder Mode
Yeah, interesting. Yeah, and I think if you called it clarity plus trust mode, it's like, I don't know, that feels like work. Yeah, totally. I think I'd rather go into founder mode. Founder mode sounds a lot more fun. Sounds a little more like Prince Adam becoming He-Man. No, that's right.
Oxide and Friends
Reflecting on Founder Mode
It also, I got to tell you, I hear clarity mode and trust mode. I just feel slow and I want to move fast and I want to, I want to move fast and I want to break things. Makes sense. Like, and trust and clarity mode. Take my money. But I think it is worth listening to the Brian Chesky.
Oxide and Friends
Reflecting on Founder Mode
Because I think that it was helpful to get that kind of from the source.
Oxide and Friends
Reflecting on Founder Mode
And then, like, get as much as possible from the source, I would say, is part of, like, even if it's Dave Hitz. As it turns out, Dave Hitz, nugget of wisdom. Right there, right? As Dennis Ritchie said in his anti-forward to the Unix Haters Handbook, It is an undigested nugget of nutrition in what is otherwise a fecal pie. I just felt that was like one of the greatest. Oh, it's absolutely a keeper.
Oxide and Friends
Reflecting on Founder Mode
Dennis Ritchie, rest in peace. this ended up being a good discussion I feel yeah turns out who knew thank you Paul Graham thank you Paul Graham we got to we got to not the pod probably along with that yeah exactly Paul Graham and Dave Hitz listen to this together it's one of their like weekly things that they like to do Their little guilty little pleasures. One of their guilty little pleasures.
Oxide and Friends
Reflecting on Founder Mode
Exactly. Well, I think this is it. I know this was a cold take, but I think this was fun. Yeah.
Oxide and Friends
Reflecting on Founder Mode
And I think it's like, and there is something there because I think that the problem is real. And the question is like, how do you, how do you build, how do you build trust? How do you offer clarity to a team? Like that's the challenge. And it's like the way to do that is not to go into seagull mode. Yeah. Can't find a mode. This has been great. It's great to be off our little hiatus.
Oxide and Friends
Reflecting on Founder Mode
Maybe this is a good little cooling off period we have. Maybe we should do this more often. We'll come back to tweets that are two weeks old. Good. I'll do more international travel. Sounds good to me. Exactly. It feels like we're more likely to read the tweet in this case, too. All right. It was fun, and we will see everyone next time.
Oxide and Friends
Reflecting on Founder Mode
Like someone's got to, someone's got to lick the frosting and someone's got to do something with the rest of this thing. I don't really want it. And so is this like, okay, I think this role is being foisted upon you. I don't think you're totally bought into this role.
Oxide and Friends
Reflecting on Founder Mode
Yeah. Yeah. The, the, where they kind of surprisingly like seeking wisdom out of others, I think is actually an, it's unfortunate that it is so unusual a trade, especially in a company that's enjoyed some early success. Yeah. Because they are being kind of implicitly, and this is the dichotomy that's created.
Oxide and Friends
Reflecting on Founder Mode
What we should be encouraging people to do is seek out that wisdom and then make your own decision. Instead, it's like, find an adult. It's like, all right, I guess that makes me the child. I mean, it's the same dichotomy, but from the other perspective of the founder as a child and the professional management. And so how did the plan thing work out? Did they want to...
Oxide and Friends
Reflecting on Founder Mode
That's a great... That's actually an honest story to tell yourself. Because it's so tempting to be like, God, I talked to those guys early. Like they really, we really wanted to make a connection. I walked away from it. It's a multi-billion dollar company. I would be like a billionaire.
Oxide and Friends
Reflecting on Founder Mode
It's like, well, no, maybe you would be like bored, miserable and would have left after six months or I would have fucked everything up or like actually plans on a multi-billion dollar company because it's run by a guy who like, like the CTO doesn't know how to balance a checkbook. He only knows what that means in the abstract. Right. So, but interesting.
Oxide and Friends
Reflecting on Founder Mode
Normally, when you get that much comments, one of the things I like about Hacker News is when you get a lot of comments on things. When the comments exceed the upvotes, a story gets more or less automatically killed. It gets automatically removed from the front page, which I actually think is a great... A great, very simple algorithm. Yeah, indicator of hot-taken-ness or whatever. Hot-taken-ness.
Oxide and Friends
Reflecting on Founder Mode
So they were, they were actually really actively seeking that wisdom. I think it's a very unusual.
Oxide and Friends
Reflecting on Founder Mode
And I was really impressed. We should be encouraging that a lot more. Cause I think that like the, the big problem, I mean, I had a couple of big problems with the piece, but the non-specificity is like first among them of this is like, too reductive. And it's very important to remember that Paul Graham has started a successful company. It is called Y Combinator.
Oxide and Friends
Reflecting on Founder Mode
Like, via web, I'm sorry, we're just not going down to via web as a model for what companies should aspire to do. And I think that the... understand that most of his perspective comes from starting Y Combinator. And so he is getting this kind of hearsay from founders and then deliberately trying to be reductive about it. And as a result, you're losing a lot of nuance and detail.
Oxide and Friends
Reflecting on Founder Mode
And I think that for folks that are interested, I think, and my first kind of tweet on this, my first attempt to think about this only briefly is, was linking to Tim O'Reilly's piece, How I Failed. And I mean, I think this is a piece that I actually had read and then couldn't find again. Huh.
Oxide and Friends
Reflecting on Founder Mode
And I actually DMed Tim about it because I'm like, you know, you wrote a piece years ago and it had a real influence on me and I can't find it. And it turns out it was a LinkedIn post. That's why I couldn't find it. And then he gave me some terms to search on and I think then O'Reilly actually kind of finally put it up. But I think this piece is terrific.
Oxide and Friends
Reflecting on Founder Mode
First of all, and I think Adam, you and I believe this perhaps to a shared fault, where I do worry that I focus too much on how companies fail. Do you know what I mean? Yeah. Yeah. Where it can inhibit your own action. Like you need to be, there's a degree to which you have to be blind to the odds. Yeah. Certainly in a startup. Totally.
Oxide and Friends
Reflecting on Founder Mode
And you, it's like this walk in this fine line between, I need to know how things fail. But if I look, if I learn too much, it might inhibit what I actually need to go do to make this thing successful. But we definitely, you and I both like to understand like, how did things come unglued? And I feel like I learn a lot more from failure than from success. Totally.
Oxide and Friends
Reflecting on Founder Mode
Success is just, I would say success teaches us all nothing, but success certainly teaches me nothing. That's maybe a much more, that's a more, a more concise statement.
Oxide and Friends
Reflecting on Founder Mode
Totally. And also, you don't want, as someone was saying in the chat, a lot of success is luck. And you, it's hard for people to say like, I was really goddamn lucky. Totally. I was in, God, I was in the right place at the right time with the right idea, the right team, like all that just like happened to come together. And which is almost always the case.
Oxide and Friends
Reflecting on Founder Mode
Like there's always this kind of element that is out of your control. I feel. Yeah. And, but I think that part of the reason that the O'Reilly piece is good is it's like not a reflection of, on unequivocal success. It's a, and obviously titled how I failed, but it's getting into some of the specific things that Tim felt that he did wrong.
Oxide and Friends
Reflecting on Founder Mode
An indicator of we've got a lot of discussion going, but people aren't actually reading the underlying article. I always thought that was super elegant. But in this case, it was getting a lot of comments, a lot of upvotes. And so this thing is just going and going. I mean, I read it. I'm like... Don't think about it. Don't think about it. Don't think about it. Still thinking about it.
Oxide and Friends
Reflecting on Founder Mode
And I know that like, I think this is another thing that's really important that even I think Tim is remarkably candid here. I think he's, he's a terrific writer. He's also like telling a narrative that is kind of his narrative and someone, I mean, someone else at O'Reilly, uh, could very easily be like, that is not like consistent with my understanding.
Oxide and Friends
Reflecting on Founder Mode
It's so easy for different people to have different understandings and for a founder to even not, you know, they'll see some things and not see other things. Um, Because I think it's always important to consider that you're getting kind of one narrative and that there are other narratives.
Oxide and Friends
Reflecting on Founder Mode
Michael Lewis, let's reflect back. So what book was that? What was the... Well, let's go tell him. The hagiography you wrote, hagiography or hagiography? I started knowing that one. I think it's hagiography. I think that's right. All right. The hagiography that you accidentally wrote on Sam Bankman Freed, which I guess readers are telling you that's the book you wrote in that case. Yeah.
Oxide and Friends
Reflecting on Founder Mode
But yeah, that's interesting. I think that's right, that you don't necessarily know
Oxide and Friends
Reflecting on Founder Mode
Absolutely. Absolutely. And I think it's very easy. So I think you've got to kind of keep that in mind when you're looking at Tim's piece. But all that said, I really like this. I thought there was a lot of specificity in here that was...
Oxide and Friends
Reflecting on Founder Mode
really important yeah um and some of that is just like let me do like failure number one of people only hear half the story which i think it was kind of interesting right where he's trying to give tough messages with empathy and people like well i'm gonna like i'll take the empathy and i'll kind of like discard the tough message but that's like his perspective like maybe someone else like the specifics on that one also tickled me where i think it was that he was um