Bryan Cantrill
Appearances
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I noticed that you changed up the title. You did not like my... Okay, yeah, go ahead. Go ahead, explain yourself.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I feel like the last time you were on, I was getting a bunch of grief for dropping some dated references. This is an old part, just to be clear.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We're at Chrissy Field with one of these girls, and the other girl comes up.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So this was in 2021. So I assumed you were talking about the, like the, okay, this is, you're not talking about a chip that was made in like 1999. No, no, no, no, no.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Okay, so I've got so many questions about this Wikipedia page. So they were broken up. First of all, like nice use of the passive voice. Like they were, like what broke them up?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, why would someone... Obviously, I love my title. I love your title. Akui, there's a butt coming.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Oh, and you're like, okay, what? And I'm reminded a little bit of Robert on the spot over here. And like Robert, my four-year-old takes the hand of one of the girls, takes the hand of the other girl, and then the three of them all go running off together.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
God, thank you for opening up all of these doors. Did you see this? There's this documentary, The Rise of the Centaur, covering the early history of the company. It's like, okay, that's must-see TV. I mean...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, interesting. And the fact that it's not there at all really does, it is, as you say, it's a separate ISA. Yeah. Is that a good segue to the AVX-512 improvements on Tornax? I mean, I felt like going into this launch, I mean, I felt that's one of the headliners was the improvements to the AVX-512.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, really. There are some... The data path, obviously, is a big part of that, I assume. The fact that it's going from a 256-bit wide data path to a 550-bit wide data path. Or deeper than that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I'm like, all right, you know, go for it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And was that on, was that Sandy Bridge or Haswell, Robert, where it was like, I mean, AVX-512 has always been kind of had this kind of problematic property that if one thread starts using it, it kind of like browns out the rest of the part.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Absolutely, or weddings. I mean, who's to say that this, you know, who's to say that this won't carry into adulthood? Yeah, so I'm raising a bigamist. Anyway, I am, regardless of the title and who it was designed to appease, I am very, we're very excited to be talking about Turin and the Turin launch. This is AMD's latest part. George, thank you very much for joining us. Really appreciate it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Wow. And so you basically run this guest and it would like crater performance for the whole box for no actual approachable gain. Cause no one's actually using it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And it's really hard to have a feature like that, where if you use this feature, it has this, this kind of this adverse effect on the rest of the monkey's paw kind of feature.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's very hard to reason about the performance when you have these kind of problems. So, and the, and AMD is not needed. And I mean, you know, and it sounds like George, you've got the kind of the same question of like, you just come so accustomed to these kinds of intense compromises that come with AVX 512. It's kind of amazing that we can have it all.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and you would expect that to like, I mean, flip a little bit with Sierra Forest, but then you end up with this kind of this e-core business. And I mean, I think there's gonna be, I mean, there's gonna be things that are gonna be interesting over there, but touring is a very hard part to compete with. It's done a pretty good job across the board. Yeah. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so I think, George, I'm not sure you got to all three of the things that you had that were, so the frequency, the high number just in terms of the F cores and getting that up to five gigahertz, especially across all cores.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You are a repeat friend of Oxide. It's good to have you back.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I can tell you that we at Oxide also like that. Folks think about the kind of the rack level, because we're kind of left with the rack level power budget. And yeah, we definitely, it's nice to have a SKU stack that is not all sitting at 500 or 500 watts plus, right? I mean, I think of that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It has, and I think economically, too, it's interesting. These are expensive parts, but they can do so much, especially at that high core count level, when you're not having to sacrifice on what those cores can go do, that you can make it make economic sense, I think. Yeah. it's a big step function over where they've been.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So you had a great blog entry that I was excited to see it at the top of Hacker News over the weekend. Were you surprised by that?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, I think that we, you know, I think like a lot of people, like the Genoa SKU stack was a little less interesting for us.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, it is a good segue, Robert, to the kind of our thinking on Turin, because we, so George, as we're kind of thinking about our, I mean, our next gen sled is obviously a Turin-based sled. We did deliberately elect to kind of bypass Genoa and to intersect with Turin. Maybe describe our thinking a little bit there, Robert, and we've got
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Listen, pal, we've been rejected by enough VCs. I know a breakup male when I see it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But as we were thinking about Cosmo, Cosmo is our codename for our next-gen Turin-based sled, SP5-based sled. What were we thinking in terms of what groups we wanted to target and kind of like the trade-offs there in terms of flexibility?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. Eric, you want to describe some of the kind of the thinking there is as you're, as we were looking at the, what the PDN for this thing was going to look like.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And when you mean route, we're talking the physical layout.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and I feel that, like, also, Eric, I think whatever size Oxide becomes, even if we're selling millions of them, I will help slay the person that comes to your door on the 5. Because I feel like on so many of these parts, I mean, yes, they add up and it's part of the bomb. But, man, look at the cost of these CPUs is so much greater. And getting the flexibility is so much more important.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Oh, and I think, I mean, George, I'm sure your experience has been up there, but I do love a bunch of the reviews online of Turin cautioning people to not do exactly this. Like, by the way, your SP5 motherboard may not be able to take some of these SKUs.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You know, maybe the SEO, it may just be also that you just got a great article on a hot topic.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. And the, and having like, and because the problem too will be that if you push these things to the margin, I mean, you, you can get like misbehavior. It's not, it won't be as simple as like burning the house down.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yes, or if you recall us on the Tales from the Bring Up Lab episode where Eric was regaling us with some of our adventures on Gimlet, where our power was already pretty good, but we could not figure out why this chip would reset itself after 1.25 seconds. So we made our power even better. And Eric, my recollection of this was AMD's like, we have never seen power. This power's amazing.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You've got amazing margin. It can't be that. And sure enough, it was not that. It was firmware, of course.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, okay, I'm glad you brought up the video.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
George is kind of hilarious because we were not, the SVI2 was the protocol this thing uses, that the part uses to talk to the regulator. And the SDLE, which we had used, this great part from AMD that we had used to actually model all this stuff, as it turns out, didn't have a hard dependency on getting the ACK back from the controller when we set the voltage to a specific level.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
The part, as it turns out, wants to hear that ACK, as we learned. We learned that the hard way. We learned that the hardest, most time-consuming, most expensive possible way. But we did learn in the process.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I actually, Eric, I thought it was super interesting to learn that our power margins were really good on that because that was like a first natural line of attack was our power margins aren't like, that's why this thing is resetting because it is in a reset loop because our power's not good enough. But we actually learned in the process of doing that, like, no, no, this power's actually quite good.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
No, no, I'm glad you brought it up. Because those, you, the comments on that video were the nicest YouTube comments I've ever seen in my life.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, Eric, as you're kind of thinking about like, okay, so we need to, you know, there are things we need to do. And were you coming to the conclusion that, okay, I think we can make this all fit? I mean, as you're doing that kind of that trade-off?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And importantly, we're able to use the same regulator that we're, it's not like we're having to swap regulators to accommodate this. We were able to use the same Renaissance parts for this. And then the, and then from a thermal perspective, so we then, okay, so that's kind of like, all right, we've got the insurance there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And then from a thermal perspective, we also needed to do the, because you said we're not water cooling this thing. So, you know, can we, and at 500 Watts, I think we definitely know we will not be talking about how quiet the fans are because you'll be lucky if you can hear us talk over the fans. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We know the fans be cranking, but I think that the, I mean, we've done that and Doug and crew have done the model.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I didn't even know YouTube comments could be nice.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It is in particular, like it's real hard to max out the draw on your DIMM and the draw on your CPU at the same time without being mean spirited. I mean, you have to be really...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Substantially so. You're right. I should not be tempting the gods here.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Interesting is our take. I mean, it's like definitely interesting. I mean, I think that we would love to be able to get some parts. The draw does become an issue back there for us. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, what is going on? This is, this can't be a YouTube video.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's not YouTube. There's something, this thing has fallen into some alternate reality and like, and the comments are all like, you know, thanks for all of your diligent work. And you know, I just, I, I love, I mean, it just, it's great. God, like we talk about lightning in a bottle.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, and I think that, you know, and Ellie in the chat is saying, well, I don't think people realize how, you know, how restrictive the oxide power budget is. And I don't necessarily, it is restrictive. It's more that we are really, we are taking that rack scale approach. And so we're kind of the ones that are like always adding up the visa bill.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And, you know, when you have, you know, 30, 40, 50, 60 watts, 70 watts, 80 watts in your neck, like that adds up in a hurry. And yes, you can offset it elsewhere. But what we're trying to do is try to get you the maximum amount of useful work out of that rack scale power budget.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And, you know, by being the ones that are doing that, we're the ones that are, you know, sometimes having to deliver some tough messages to folks about like, we like this, this is interesting, but it's drawing way, way, way too much power.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We love meeting Oxide. We're very excited about Oxide. Okay. Yeah. Go on.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, I mean, we love, we're huge P4 fans as folks know. And I think we've got... Actually, if folks can see it out there, we've got an exciting announcement in terms of Excite Labs and using their part as our next-gen switch, and it's P4 programmability. So we're really excited. We've been using P4 on the switch. We're going to continue to do that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And using P4 or programmability at the NIC, we're really interested in. But it's got to happen in a way that we can accommodate everything else we need to go do with the rack. So, George, a long answer to your question there, but it's interesting for sure.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. Which is something that's going to be, and that is something that's going to be like, we're definitely interested in.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, I think so. I mean, that's not going to intersect our first cut here of Cosmo. But no, we're really, really interested in it. And again, great to see that P4 programmability.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
There were some moments where it felt like we were a bit of a lonely voice, but I think other folks are beginning to realize, and I think as the hyperscalers themselves have known, that having that network programmability is really essential.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Robert, you'll do the honors. I'm happy to give my P4 spiel.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And having that higher level of abstraction really allows you to kind of express something programmatically that they can then use hardware resources efficiently. Our big challenge has been working with vendors to give us a substrate upon which we can build a true P4 compiler. Honestly, the biggest challenge in that part of the ecosystem has been the proprietariness of the compilers.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, George, I'll tell you that one thing that will be a factor for us as we're looking at kind of a P4-based NIC is what we have been looking for is what is that kind of x86-like substrate going to be? Something that is a documented, committed ISA that we can write our P4 compiler against.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So what we are not looking for, because we are coming out of a bad relationship in this regard, what we are not looking for is kind of a proprietary compiler. We really... uh, want to, and we have written our own P4 compiler for, we use our, we developed our own P4 compiler and have open sourced it, um, X4C, um, for purposes of just doing development, software development and testing and so on.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But we really want to take that and have that, use that to actually, uh, to compile, uh, for these parts for both the switch for sure. Um, and then ultimately the NIC, um, would definitely be our vision for where it's going.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
George, I really appreciate your sensitivity of taking us into the kind of the grieving room and your bedside manner here is really exemplary. I was really feeling you kind of passing the tissues to us as you, as you really felt our loss. I really appreciate that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, are we in charge of AMD now? Because we got lots of ideas.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so you think that this has given the internet some gratitude? You've managed to domesticate the internet.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. And, you know, so yeah, I mean, absolutely. We would be great for them to do that. Although that said, we are, you know, they need to do it in the right way. And the right way from our perspective is really establishing a substrate that people can build an open ecosystem on top of. And this is something that, you know,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And I always, I find vexing is you would think if you make hardware, it's enormously in your best interest to allow many, many software stacks to bloom, um, by having a well-documented committed interface. But, um, They really don't. It's a challenge, I would say. I wouldn't say they don't, it's too reductive. I think that they fight their own instincts on it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so we're very excited with Excite Labs. Again, you can see our announcement today or their announcement today, actually, but it features Oxide for sure. And we definitely see eye to eye with them on their X2. We're looking forward to to moving forward with that part. And we think that there should be, we want to see programmable networking everywhere.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We want to see this open ecosystem everywhere. The, on the note of like the kind of the lowest levels of the platform that can be hard to get at. So George, you may recall that we have no bias in our system. So there is no AGISA, there is no AMI bias. So when we buy us, buy us, am I? Nailed it. Oh, thank you. We have lots of bias. Up and at them, up and at them. Uh, better.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, do you wake up your kids that way? Adam, as, as when you're at castle, we're near wolf castle saying up and at them.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Feels very loaded. I think my kids have definitely gotten sick of that particular Simpsons reference. Rainier Wolf Castle, no longer welcome in our abode. But we have no BIOS. And so Robert, that lowest level platform enablement has fallen to us. What are some of the differences in Turin from, or even from Genoa, but then from Milan?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And look, it was like if there was a YouTube video that we're going to start having nice comments on, that was a good one to start on. That was a great video, went in depth. I love that you kind of had the surprise ending where you set a world record in your hotel room. Let's start with there. What was that benchmark that you were running? And you were running that on Turin, obviously.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, those are effectively hidden cores, right?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
There used to be four. Yeah, interesting. Yeah. And that is presumably, so you are, you're just increasing the parallelism there. And I mean, is that the.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so but but those kind of changes, which if we were at some level of software, that's an implementation detail you don't need to see. But at the level of software we're at, like you actually need to go accommodate those differences.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And could you describe what device training is? Like, why does a link need to be trained? What is that?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Right, when they misbehave. So there's a lot of low-level work that we need to go do. And how do we, in terms of like, we don't yet have, I mean, Eric and Nathaniel and crew are working on Cosmo as we speak, kind of finishing up Cosmo. How do we work on that before we have our own board in hand?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Is it too much to hope that someone has a sense of humor that named that thing volcano?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You had, like, Onyx. Yeah, no sense of humor. I just like the idea that, like, Inferno, you're going with the... So, yeah, so we got the Volcano reference platform from AMD.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And then Nathaniel, do you want to talk about kind of how we use those dev platforms? Because we've got a little, a great little board there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, we should talk about eSpy because this is definitely a difference in Turin.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Okay. Go on. So you love my title. My title was Unshrouding Turin.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That's right. And spy, as my kids would say, spy has no chill. Spy, you need to give it what it needs. You're set on the clock. There's no clock stretching in spy. And so spy interposition becomes a real nightmare because you need to get everything you need to get done, you need to get done in that one clock cycle. It's like, yeah, that's really hard.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and just on the training times, without that, without kind of, and so dim training where you're trying to find the search for the constants that are going to allow you to not have interference when you're talking to these dims, that search can take a long time. And Robert, how long does it take you when you've got the first, that first genoa that you've got-
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, it was. I've put it out of my brain. Yeah, George, I don't know if you've seen some long boot times, but DDR5 takes a long time to train.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Adam apparently really prefers your pronunciation, George, to mine.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
All right, I'll do my Duolingo with George on there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It was pretty great. It was pretty great. And I love that you're like kind of wrapping it up. You're like, no, no, actually, wait a minute. Hold on. I'm not being, this laptop just being handed to me.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It was, yeah, we definitely, and I'm not sure, you know, maybe we're, I guess we're a bit unusual on this, but boy, we were unusual. We need the, yeah, that, So, George, we were using, just as Nathaniel mentioned, we were using one of the UARTs as the IPCC, which is the interprocessor, right? Communication channel.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But this is our protocol for the socket, the host OS, to be able to speak to the SP, which is our replacement for the BMC.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
There's more. Yeah. Okay. So, and obviously that setting, and it's helpful to know this is a very compute intensive workload. Because one of the things that I think that we've heard from a bunch of folks is this thing is so much more compute that now you've got to really ask questions about balance of the system and memory bandwidth and so on. So I want to get into all that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It was very slow. And our friends at AMD, fortunately got us a, a fix to the PSP to operate at three megabod. And that was very, it was life changing for, for, I mean, I know for 30, 30 X makes a big difference for RFK. 30 X means when, when it's 30 minutes, that 30 X is like a real actionable human 30 X. Not all 30 Xs are the same.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And when something takes 30 minutes, taking 30 X off of that is a, is a big deal.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, the go signal still takes several minutes to transmit.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I'm not even joking. This sounds like a dream that I would have that I would describe to Adam.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I'm sure. I would like to believe that original designers and Oxide and Friends listeners would be like, oh God, that was still in use? That was supposed to be for a weekend. That was not supposed to be. That was a temporary fix. Totally.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so Nathaniel, maybe worth elaborating a little bit why the three megabod is so actionable for us beyond just the margining and the Mbis results, because this actually ends up becoming, because this is our conduit for the SP to talk to the host CPU, we use this in the recovery path. So like if you've got a system that can't talk to anything else, it's gonna load its image
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, I guess one thing I would kind of ask you from the top, um, just what is your kind of top takeaway from the Turin launch? And was there anything that surprised you? Was there anything that you either didn't know what was coming or didn't know the kind of magnitude or you're still, yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But I noticed the title here is Benvenuto a Torino.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It takes a minute as the kids say, but it actually takes like, we're actually doing two different things, Nathaniel.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And actually, and so George, in all honesty, like part of the rationale for this is to get us out of those moments of terror when you are flashing a bias and you have gotten often no recourse if that, if that goes sideways. And so this gets us out of that because we know that the system at the, at the absolute lowest layers of the system, we can get the system to, to be able to boot. And we,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
it gives us much more control over the reliability of the system, upgradability of the system, manageability of the system. That's how we're able to get... Oxide rack can arrive, power on, and get going and provision VMs in minutes instead of days, months, weeks, whatever.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, I feel like this is like, we're like the sewer people and you, you know, like you get to, I mean, you've got this glorious palace in terms of the cores that have been built. And meanwhile, it's like the sewer people are happy about not being at three megabot. Like what's going on. No, it's, it's a big deal down here in the sewer.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so we are all in favor. So we have been, and actually it was funny because I actually first heard Turin the code name Turin when it was accidentally blurted out on one of those OpenSeal calls. I'm like, okay, what is Turin? And I remember asking Robert, like, that's a city in Italy, so it must be the next thing, but we hadn't heard of it yet.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And OpenSeal was going to intersect with Turin, which of course, when we were first hearing that, it's like, oh my God, that just feels like Buck Rogers. It's like in the year 2041. But of course Torrent is not here. And that work we are very, very supportive of. We are not actually using any of that because it's a different model.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's kind of going to, it's still a traditional model of a bootloader that's going to effectively make the system look like it's gone backwards or send the system backwards to boot a host operating system. And we've got this staged approach where we are running a single operating system the entire time.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So it doesn't fit our model, but we're extremely supportive of it because we believe that we want these lowest levels of the system to be completely documented. And we want there to be room for many different approaches. And so I think that we're very supportive of OpenSo in that regard.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and it's very nice to be able to go compare nodes, especially when things aren't working. Helpful to have multiple implementations out there. And I also think that the model for AGISA, the programming model, makes it very difficult to reason about the overall system. This is where Robert's eyes are going to start to twitch because...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Robert spent a lot of time in the absence of documentation having to really understand what this code was doing.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, and that's it. I mean, when you have these kind of, this is why it's so healthy to have different software ecosystems on the same hardware, because you don't want things to be working by accident. You want them to be, and it's, you want things to be well-documented and with well-committed abstractions. And failing that, it's good to have the software out there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So it's, no, it's been, George, it's been good. And we're, I think, excited to continue to see that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
um the uh some of the other like lowest level i lowest level differences on turin um we you mentioned the dims for channel and we we kind of had a um a fork in the road in front of us in terms of two dims for channel two dpc versus one one dpc and there's a trade-off there to be made and robert what was the i mean
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That is crazy. I feel like the last time we were really, it's been a minute since we've seen clocks that high from anywhere, I feel. I feel it's been like, I mean, IBM was hitting it with power.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
When I think just in general, when we had a trade-off where we'd have to give up memory latency, we have always felt that memory latency is really important. You want to get maximum. You want to minimize memory latency, and you don't want to take a hit there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
probably could make that you could probably make convince yourself that that is actually worthwhile but you know 5200 4400 that's a that's a that's a lot farther from 6000 it's a big yeah right it's a big big chunk to take out and in terms of mr dims because this is a domain where you know intel is still uh basically the all i mean they because we're pretty standard on mr dims right let me let me let me get on my soapbox for about 30
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, you know, Georgia, what I love about Oxide and Friends is when it comes to the soapbox of a dims being not being per JDEC standard. It's actually a line of a soapbox here at Oxide and Friends. There's actually this is this is the because this is a soapbox. Robert, you know, the soapbox you've been on the soapbox.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And it's, yeah, it's frustrating, but they, but MRDMs, I think when the, when they are JDAX standard, right, because they will be.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, well, I think that, you know, one of our kind of big revelations, and again, this is not due to us, I think the other hyperscalers done this as well, but that the...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
you actually, the way to have maximal density is not necessarily to have maximal physical density, that you want to open up some room for airflow and you can actually get higher density by having, by using a little bit more space and being, you know, where the rack is nine feet tall. And so we, you know, using some of that, trying to use some of that space to get higher density.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, the, uh, actually, can you just talk about bacterial vias for a second, Eric, just because you mentioned in the chat, I don't know, George, do you know about bacterial vias? This is, this is truly amazing stuff.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Oh, put Robert on the spot. Go ahead, Robert. There we go. What an all pro. You know what this reminds me of? This reminds me of when my now 20-year-old was four. We understood from one of his friends in the neighborhood that he and this girl were going to get married. And they were like, okay, that seems like a little bit heavy for four. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I just love the fact that we're going to take a drill to the underside of the board. It does feel like a Adam Leventhal PCB engineer kind of approach to this of like, here, hold on, pass me the drill. I'll take this. Exactly. I need a drill and a running start, and we'll take care of it. Yeah, I'll fix your signal integrity issue with my drill. I'll fix it real good with this here drill.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But it is a total precision, and it is 25 microns. Just amazing, Eric, that this is. And then we've got simulation tools. I mean, how do we kind of figure out where this needs to be done? And this is…
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so let's elaborate on this a little bit because this is, I think, a really interesting point. So we're seeing this from Intel too, right, where once you get the density up to a certain level, you've got to make some compromises. But the compromises that AMD seems to be making are much less than the compromises you're seeing. I mean, the Zen 5C cores are – they're still Zen 5 cores.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And I just love the fact that you got to think of like, you know, who wants that? Like that thing in the SKU stack. It's like, oh, the engineer that actually like.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Exactly. So we got to get that SKU for Eric and Tom and the other folks that are running these simulations.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, we got to get that for you. Not a cheap part, by the way, but still. No. We had the ANSYS folks on. I remember we had the ANSYS along with Tom on talking about our use of simulation, which was another great episode. I really enjoyed talking to those folks. And you just learn about the physicality of the stuff just...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
blows me away and like i feel i mean i don't you feel bad that we end up running like our dumb software on top of this stuff at the end of the day i just feel like we're kind of yeah seriously well i've been taking all these gigahertz for granted too and just like the level of complexity underpinning this is is bananas it's like we back for this thing to run php it's a good look yes sorry yes
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Totally, totally. But it's just amazing. And this part is a great part. You know, I think that we're, you know, I think, George, we were really excited to see, I mean, obviously your in-depth review was terrific. But I mean, I think, George, from your perspective, like this is a part that has really hit the mark in kind of like every dimension it feels like.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And I think that we, we are going to be, I think, you know, another thing that we're looking at is partnering with Murata and getting for those folks that actually do want to go more than 15 KW for the rack, which was our original design target, but which felt very aggressive in 2019. But then, you know, I think it feels like Nvidia is like, that's like two GPUs now for you. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I can get that in for you. Exactly. I can get that to you.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
we are, are, uh, we'll be for folks that can go be up above 15 KW for the rack. We'll be able to go do that. And, uh, it's, it's, it's again, it won't necessarily be quiet. Um, but we, uh, we think we're going to be able to air cool that. Um, and those, that's where you get that, you get that kind of seven X consolidation that AMD was talking about. Um, and I think that there's, yeah. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and I think that we're excited for the AI workloads too, and I think they're going to get a nice pop from AVX 512, certainly the 512-bit data path there, and you're going to see a lot of those. There are nice pops to be had, we think. But you're right, it was not just AI. There are, as it turns out, other... We also need the workloads to simulate the computer for the AI, as it turns out.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, we think so too. And I think that, you know, I think unlike with our first gen where it was kind of every, we only had one SKU, the 7713P, we're going to allow for some flexibility for Oxide customers inside that SKU stack. We're excited to kind of extend that and then do some of the work around dynamic power control.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
We got a bunch of ideas on how, you know, we've got the right foundation to go with. actually manage power holistically across the rack and, and use some of the, there's all, we got a lot of stuff to go, a lot of knobs to turn. Um, and I think it's going to yield a pretty great product. I mean, the hats off to AMD for sticking the landing.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, we are definitely wedded to AMD in a lot of ways in terms of our lowest level of platform initialization and so on. So we're always relieved when they execute well.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, we talked about the APU. Yeah, boy, we're hitting all the sympathy cards here. Hold on. Hand me the other sympathy card now.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
When you were asking about the APU, did the oxide people put you up to this?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That's great. We've asked for... Yeah, that'd be great. You will deserve responsibility for an MI300A dev kit. We'd love it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
George, I don't know what we're going to do for you if an MI300A dev kit is for sale on Newegg, but we're going to do something very, very nice for you. I don't know when it's going to be yet. It is going to be... That would be great. We get George a whole wheel of his favorite cheese?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That's right. For sure. But I totally agree. And yeah, we definitely noticed that the... You know, we have brought up APUs so frequently with them. I feel we've kind of overstayed our welcome with respect to that. Sort of like, you know what? We're going to let you guys say the next thing about APUs. We're going to stop telling you how much we love APUs and we'll let you do the next thing.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Or we'll let George do it for us, which is great. Much more effective. Much more effective. Well, we'll say, yeah, again, if we get an MI300A dev kit on Newegg, it's going to be, we're going to have to do, it's going to be something spectacular. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, you know, and I think that you, I think, and just actually honestly with OpenSeal, I mean, I think one of the things that we really like about AMD is it's a company that doesn't like, it listens. I'd like they kind of, they know what the right direction is. It takes them a while to get there sometimes because it's,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's a big vessel. And, but, you know, I mean, I remember when we were getting Naples stood up way back in the day and, you know, there was a lot that was still needed to be done. But you could see that like, okay, this is not a trajectory that's really interesting. And then with Rome, it's like, okay, this has just got a lot more interesting. And it was clear to us in 2019, Robert, that they were,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
they were on a trajectory or they were surpassing Intel effectively. And then obviously with Milan and, and with Genoa now turn, I mean, it's like, we've seen them like continue to execute, execute, execute. And so, yeah, let's go, let's keep Adam on the APU side.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Absolutely. Yeah. No, love it. And I totally agree.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yes. So we, we love the parts and, and, and we're going to be, we want you to be got some ideas for things to be even better, but touring is a great part. We're really excited about it. And George, thank you very much for, for joining us. It's been great to have. Thank you for having me. Oh, yeah, it's great to have the team. I mean, and Nathaniel and Eric here and Aaron as well.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's been, and obviously Robert here in the studio with me. And then Adam, of course, to correct my pronunciations and to inform me that my running start is not quite big enough on the back drilling. Um, the, um, you know, I think it's been, uh, we're really excited about our forthcoming.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So I don't think it's, we're excited about is of course, you're going to be able to take the, taking a touring sled and putting it into a, an unused cubby in an, in a, an oxide rack that has Mulholland sleds and just have the whole thing just work. So we're really excited about that. Um, And onward, great part. And George, thanks again. Really, really appreciate it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Speaking of excitement, we do have one very exciting announcement.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's going to be an unconference. If you were a DTrace user, you want to come hang out at Oxide. We are going to charge you for tickets. It's not going to charge you too much money, but we do have to charge you something. Otherwise, it'll be immediately consumed by teenagers. Teenagers will consume every ticket if we don't charge you anything. Um, very limited supply.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, so, uh, hop in there if you're interested in joining us. Um, yeah, it's going to be out of my, I'm, I'm really excited for this. It's gonna be fun.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And it's going to, yeah, it's going to have a different complexion and flavor this year for sure. It's going to be a lot of fun. So I'm looking forward to it. That's for sure. That's right. I know Robert, you've got to, everyone is just like, okay, what do I need to get done now before I've got until December 11th to get said to get my, but we got a lot of things to talk about. So.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's going to be fun. So join us, detrace.com 2024. And I will not violate any more copyrights by humming.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Exactly. Awesome. All right. Well, George, thanks again. Thank you, everybody. And yes, see you at detrace.com 2024.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And the fact that you've got this kind of SKU stack now that is kind of a uniform SKU stack where you can start to really make some interesting trade-offs as you look at workload. And it just, it feels like, you know, and Robert here is our resident SKU stackologist. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
which definitely, I mean, there were times with Intel where you've got like these three different, where you had like gold, silver, and platinum, and you've got- Also bronze. Bronze, right. And it just, it did require like you to get a postdoc to figure out like which part you want. And Robert, what do you make of this Turin SKU stack?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, it seems like it's a pretty clean SKU stack in terms of making different trade-offs as you go up and down it.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. I mean, certainly. And I think that, that, you know, when on the Intel side, when you go from these P cores to E cores, Adam, you're going from the P is for performance and the E is for efficiency. You're like, Oh, as, uh, uh,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, because in particular, you don't have AVX-512 on those.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And we were talking to another parent at the preschool, and she was saying that her daughter and Tobin were going to get married. I'm like, God, this kid's a real, gets around, real gigolo here. Well, as long as, you know, he's got his, I guess he's got, you know, when you're a four-year-old, I guess you have a playmate in every port. And we are at the beach.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, and then it's like, so it's really tough too, because if you are, you know, in our position of like, we are selling a rack scale computer to someone who is making compute available to their kind of customers, right? We're asking, if you have to ask them, well, are you using AVX 512? They'll be like, I don't know. I have to go ask my users that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And it makes it really hard, and you're kind of at this big fork in the road. So to be able to have a...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
to not have to give up i mean just as you said george to not have to compromise on isa and to get the same isa everywhere and yes you may be giving up you know you're making some trade-offs in terms of max frequency and so on but you're not like dropping off your trade-off is an area right right yeah you have to spend that area um but the
Oxide and Friends
Querying Metrics with OxQL
Right, because, I mean, they can't figure out three hours because they got another one of these to do in another half an hour. But they're just kind of rolling them through. But it was also... Anyway, it was weird. It was... And the RFDs, of course, I was feeding it were all the RFDs that were... going to be talking about today.
Oxide and Friends
Querying Metrics with OxQL
As you point out, the amount of work, it's not clear that it's who we're saving work for. by doing something that's like lowest common denominator that I don't know where, I mean, and especially, I also think that like even DSL even, yes, I mean, this is a DSL, but this is a very little language, OXQL. It's not like... I mean, you're not like learning Haskell or something here.
Oxide and Friends
Querying Metrics with OxQL
I mean, this is just like – I mean, I don't know. I feel that – I mean, I get the kind of the – we're obviously careful about that. But I think Benny made an important point that this was not our first conclusion. Our first conclusion was like, hey, we should do a query language. Good thing – because there is nothing else out there, we will invent our own. It was more like –
Oxide and Friends
Querying Metrics with OxQL
So I was feeding it RFD 463, getting the podcast before the podcast, which I dropped into the chat there. So if you want to hear synthetic hosts describe, with also like weird taglines. Did you notice that? I've got like, stay curious until next time. It's like, stay curious. Okay. Stay. Anyway, it was very weird. But Ben, welcome. Thank you.
Oxide and Friends
Querying Metrics with OxQL
Let's try to make everything else work. And coming to the conclusion that we're just having to contort ourselves too much. And it's actually very liberating to be able to do our own DSL.
Oxide and Friends
Querying Metrics with OxQL
And it's not like, I mean, this was not, I mean, not to downplay the amount of work involved, but this is also like you're using a bunch of tooling that makes it really much easier to develop a DSL than maybe it has been historically. Yeah.
Oxide and Friends
Querying Metrics with OxQL
It just feels like also with the, and again, I don't know how much of like, this is just like the Unix having seeped into my own DNA or is Unix kind of an outgrowing of the DNA that exists in all of us, right? I've got no way of actually differentiating those two, but it does feel like the pipe syntax to me feels pretty clear about intent.
Oxide and Friends
Querying Metrics with OxQL
And it feels like we can also then do a lot of things on the back end to optimize that. Because you're being so clear about your intent and you're not having to do unholy things, we can actually make sure that we can optimize those use cases. You can use an entirely different ClickHouse feature. Just some of the things that we were brainstorming about.
Oxide and Friends
Querying Metrics with OxQL
So I feel we've kind of stepped on an exposed nerve ending for the internet here. I got a lot of people like this was, am I wrong? Yeah. To describe it as upsetting seems a bit too strong, but a lot of people are like, a lot of people have issues with this.
Oxide and Friends
Querying Metrics with OxQL
What is the point of you're not actually getting SQL compatibility when you're not doing all these things? All these things have no relevance in this specific domain. There's a reason we have domain-specific languages. I just cannot emphasize this term enough because I think it is a great strength where we can create little languages easily.
Oxide and Friends
Querying Metrics with OxQL
I think we should not be resistant to that because I think the kind of compatibility that you have by doing, let's say, SQL, as you're mentioning, it's like a false compatibility. It gives you the wrong intuition for the system. And it's like, sorry, this is not what the system is actually going to do underneath it. Yeah. you know what we're trying to do.
Oxide and Friends
Querying Metrics with OxQL
And I, the other, another question that kind of came up in the chat is like, wait a minute, it's okay. So you said that like all anyone's going to do with this is like slurp it out. So like, why wouldn't you just use some other protocol that people already know? It's like, well, that's all because that's all a customer might want to do with it. We want to do a lot more than just slurp it out.
Oxide and Friends
Querying Metrics with OxQL
We want to actually go and use it dynamically and, and be able to actually look at, look at the rack and ask questions of it. And so for us, we want something that is much more tightly tailored to that.
Oxide and Friends
Querying Metrics with OxQL
Definitely skeptical. So we're talking about RFD 463, which is OXQL, which our synthetic podcast pronounced Oxquill over and over and over again. Ben, we've never pronounced it Oxquill, I think, right? I think it's OXQL, but now I can't unhear Oxquill, unfortunately.
Oxide and Friends
Querying Metrics with OxQL
Okay, with post-mortem debugging, it's taken me quite literally decades, but I'm willing to acknowledge, okay, I am somehow an outlier with respect to society. This is some sort of software kink that I have with respect to being willing to debug a system from a static state. It's static in memory state. So, okay, fine, but weird. But histograms, really? Are we histogram radicals?
Oxide and Friends
Querying Metrics with OxQL
Yeah, for sure. And I think if we want to actually, now that I know that this is apparently a strange idea around histograms, you chase this through to Bonwick and Lockstat using histograms for looking at lock times, spend times, block times, and looking at that actual distribution of data.
Oxide and Friends
Querying Metrics with OxQL
And so, I mean, honestly, it was when part of the reason we have aggregation as a first-class operation in D-Trace was because of our eye on Lockstat and replacing it. It's like, okay, this is important, this idea of getting a, what the distribution of data looks like.
Oxide and Friends
Querying Metrics with OxQL
And so, you know, I can't really thought about that, that, that, you know, maybe that was, that felt very commonsensical, but maybe I'm not giving Bono much enough credit. Maybe that was very iconoclastic to be thinking in terms of the distribution of data. I mean, he was a stats concentrator. It was a stats grad student.
Oxide and Friends
Querying Metrics with OxQL
I do feel they're right. I think that you're right. That is recent. And I guess we've always just kind of run with sets of people for whom understanding the distribution of data has always been really, really important. And it just feels very natural that that would be a first-class operation.
Oxide and Friends
Querying Metrics with OxQL
It does sound... It sounds like bovine Dayquil. Our development of a DSL, maybe this is in the way I'm phrasing it. Maybe I was too clickbaity with the way I phrased it. Is that a problem? Or something? What do you mean?
Oxide and Friends
Querying Metrics with OxQL
Yeah, when also if you look at the average, you don't actually understand that much about your data. And you think you do. It kind of gives you the sense of like, oh, this is what my data looks like. And it's like, well, you know, maybe.
Oxide and Friends
Querying Metrics with OxQL
But you may want to get just a little more fidelity in what that actual, what the distribution looks like before you conclude that that's what your data looks like.
Oxide and Friends
Querying Metrics with OxQL
And so obviously, ClickHouse was a natural fit in part because of the way they thought about the problem.
Oxide and Friends
Querying Metrics with OxQL
No, you know, it's like, you know, I'm kind of throwing out a little bait for the podcast. So my tweet is, when is a new query language necessary? And clearly there's a decent portion of the internet for whom the answer to that question is actually never.
Oxide and Friends
Querying Metrics with OxQL
And we have not scratched the surface of that kind of stuff that we can go do.
Oxide and Friends
Querying Metrics with OxQL
And by critical wrong, but part of the appeal of a DSL here is the ability to add some functionality that would actually help us express some of that that we can get out of ClickHouse. Yeah, that's right. Through a consumer of that.
Oxide and Friends
Querying Metrics with OxQL
So a question that I think may be very catalyzing for people to answer to is, is it a fair assumption that the main client of raw OXQL is Oxide-provided tooling, dashboards, alerting, et cetera? That's what that question is.
Oxide and Friends
Querying Metrics with OxQL
Exactly. It's like, or it's just like, listen, if you're going to invent a new query language, like that's fine, but you need to get rid of one of these others. Like you need... And in particular, there was a quote tweet from Andy Pavlo, a distributed systems researcher at CMU. The quote tweet starts, Brian is brilliant, comma. You're like, oh, no. No, that's not good. No, that's a bad start.
Oxide and Friends
Querying Metrics with OxQL
And you said the console, also I'd like to point out the CLI also has the ability to... Let's not sleep on the CLI's ability to visualize data.
Oxide and Friends
Querying Metrics with OxQL
Basically, if someone is insulting me, I know they may be warming up to agree with me. It's the old, actually, I agree with Brian, quote. they're kind of like trying to establish their bona fides before they... They don't want to imply that they would agree with anything I say, but they are... Listen, Brian is often... It's Kelsey's line, right? I love Kelsey's line.
Oxide and Friends
Querying Metrics with OxQL
And humility dashboard. I'm a little hurt that humility dashboard is not coming up here. Honestly, I was waiting for someone else to mention it, but apparently no one was going to, you know, it's not going to.
Oxide and Friends
Querying Metrics with OxQL
Listen, I got to plug my own. Around here, you got to... You know what? I'm going to have the AI, my AI overlords, generate a podcast full of praise for Humility Dashboard. I think I need a little pick-me-up from my bots. But Humility Dashboard also is... This allows us to talk to a service processor and graph all of our environmentals, and it's been very... It's been great. I love Ratatouille.
Oxide and Friends
Querying Metrics with OxQL
Yes. Yeah, it is. It has been. It's been great. And it's been great. I also love writing software that the double E's use, because just, you know, they have lived such a tortured existence with respect to software. It's very nice when they can be delighted by software. I feel that the Ws live a hard life.
Oxide and Friends
Querying Metrics with OxQL
And so when software can delight them, when they're not using some vendor-specific Windows goober that they need to program apart or something, you can actually give them something delightful. It's really great. They don't get nice things. It's really true. They don't get nice things. That's what I'm trying to say.
Oxide and Friends
Querying Metrics with OxQL
They don't get nice things, and as a result, their standards are very low, and you can do very little work and give them something nice, and they are just
Oxide and Friends
Querying Metrics with OxQL
filled with praise it's great yeah um so yeah ratatouille's been that's been that's been fun in the end uh folks are asking if the code for that is available that is all open source all that stuff open source i think i mean it's all open source but all of oxql and the dashboard are well the dashboard is in um the oxal oxide cli that adam linked a minute ago um
Oxide and Friends
Querying Metrics with OxQL
Can you speak a little bit to the implementation of the DSL, by the way? Assuming we've gotten people over the hump of like, that we've got the right to implement a DSL here. I'm not sure that we've got everyone on board with that. But you know what? Just bear with us. And can you get into kind of the mechanics of building that?
Oxide and Friends
Querying Metrics with OxQL
Brian is wrong some of the time, but not this time. I love that. But this is like, Brian is brilliant, comma, is like, this is the end of this. This is going straight into the ditch. But it seems misguided for a hardware company to create a custom query language that no other tool supports. If you don't want SQL, you could have used PromQL or PromQL.
Oxide and Friends
Querying Metrics with OxQL
Let's face it, Adam, you and I would, left to our own devices, we would make this thing be an absolute navel-gazer. We'd be like, we are dedicating all of the resources of this rack to gazing at itself. Like Narcissus in the pool, it is enraptured by its own reflection, using all of its storage to store thermal data about itself.
Oxide and Friends
Querying Metrics with OxQL
Yeah, I mean, look, let's just agree that we're glad that Ben is here to point out that they paid for it, so they should be able to use it for the things that they care about. Yeah, and I mean, this is where ClickHouse is just outrageously efficient with the way it stores things. It's mind-bending.
Oxide and Friends
Querying Metrics with OxQL
Yeah, of course. This is an important observation. When Just enters an engineering discussion, yeah, you're right. You're like, okay, cocked eyebrow. Just often does do a lot of heavy lifting.
Oxide and Friends
Querying Metrics with OxQL
Is this gorilla as in like Diane Fossey or gorilla as in like insurgency?
Oxide and Friends
Querying Metrics with OxQL
So to get it back. Um, So we, just to forgive the context before the context, the friend of the pod, Simon Wilson, had this blog entry that was on Hacker News describing this new feature from Google, from their Notebook LM, where they can generate podcasts from arbitrary material. And I've been entertaining myself by sending it RFDs and creating single podcast episodes on RFDs.
Oxide and Friends
Querying Metrics with OxQL
We're talking about the Diane Fossey variant of gorilla, not actually the warfare variant of gorilla.
Oxide and Friends
Querying Metrics with OxQL
I mean, it's kind of, it's kind of an O, not a U. It's not a, not gorilla.
Oxide and Friends
Querying Metrics with OxQL
You do not pronounce those two things separately. I do not. Do you? Okay, thank God.
Oxide and Friends
Querying Metrics with OxQL
Yeah, absolutely. Another question that came by was about getting how we are thinking about notifications when data does become abnormal or some definition of abnormal.
Oxide and Friends
Querying Metrics with OxQL
And for the things, I mean, are we kind of thinking about, because in terms of like the amount of OXQL that's user-facing, I would assume for some of these things, we'll create new endpoints that will basically be distilled into OXQL queries, but will actually be an endpoint that will be its own endpoint.
Oxide and Friends
Querying Metrics with OxQL
But I think... I'm very grateful for this approach. Speaking strictly selfishly, because when Eliza hooked all of those lower-level environmental metrics up, it just meant that it automatically... popped out at the top.
Oxide and Friends
Querying Metrics with OxQL
Do people think we're creating our own instruction set architecture? I was like... I just get confused why this is engendering this kind of reaction, especially for like all of the stuff we've done. We've done like so much of our own. I mean, we've gone our own way so many times over. It's like, this is the one that is a bridge too far. It's like, listen, you guys.
Oxide and Friends
Querying Metrics with OxQL
So this is actually a really important point, I think, is that OXQL kind of gives us the foundation to figure out what you would want to distill into a perhaps even more limited abstraction. And it would be a lot harder to do that with using these other kind of query languages that felt like a much poorer fit for the underlying data model.
Oxide and Friends
Querying Metrics with OxQL
We have our own switch. We developed our own switch. Yeah. Like that, although that's fine. No, that makes sense. You know, you're doing your, yeah, you're an embedded operating system, our own host operating system.
Oxide and Friends
Querying Metrics with OxQL
Yes, it will. As much as you have. When I was saying earlier that ClickHouse likes to eat, that's what... It does.
Oxide and Friends
Querying Metrics with OxQL
never stall for what we made. And then like every operations, a table scan, what have we made table scans really, really, really, really fast.
Oxide and Friends
Querying Metrics with OxQL
It is. And it's like, but query language, no. No, now you've gone too far.
Oxide and Friends
Querying Metrics with OxQL
And then in terms of, I mean, Ben, you've elaborated a bunch of directions that we want to go take this thing. I think that we want to, I mean, at the moment, I'm just like... it's great to actually have a bunch of data in this that we can go mess around with. Yes. And actually go learn the kinds of things that we want to go do. We know that ClickHouse gives us the right foundation.
Oxide and Friends
Querying Metrics with OxQL
We think OXQL gives us the right foundation. And then what are the things that we want to go add either additions to OXQL we want to make, and then especially applications we want to build on top of this to allow one to make better sense of kind of rack-level and ultimately multi-rack-level data.
Oxide and Friends
Querying Metrics with OxQL
I mean, in many ways, our decisions around ClickHouse and Cockroach have almost opposite constraints. Correct. And we have really made two very different decisions there for very different reasons. We would not want to... Certainly, it's hard to see one database ruling them all. These are two extremely different ways of thinking about data, looking at data, reasoning about data.
Oxide and Friends
Querying Metrics with OxQL
Yeah, and also, like, we're also not, like, advocating for the elimination of any other query language. That's the other thing. I feel like we're not actually... People can use whatever query language. I mean, we're not... We're kind of encouraged. It's just a DSL that we developed for our own use, really. So, Ben, do you want to describe kind of the origin of how we got here? And I thought...
Oxide and Friends
Querying Metrics with OxQL
similarly I don't think we want to put like instance information in Klaus so yeah I think we can agree that these are these are very different problems oh and one thing I wanted to when we were talking about just like using peg and so on and these other various rust crates um
Oxide and Friends
Querying Metrics with OxQL
I'm going to bring up Antler. That's exactly where I'm going. I'm going to Antler. So where are you? I mean, you're an Antler lover, just to be clear.
Oxide and Friends
Querying Metrics with OxQL
Maybe we would take half a step back, because we've tried to open up a bunch of the surrounding RFDs that get us to 463 or why 463 is relevant. Do you want to talk a little bit about 125 and how we got to ClickHouse? Because I think that's a big part of why not promQL, promQL.
Oxide and Friends
Querying Metrics with OxQL
I think that DSLs are... When someone is developing a DSL, it is almost always coming out of exhausting the alternatives. I think.
Oxide and Friends
Querying Metrics with OxQL
Totally. And, you know, I think that we... I've always found that these little languages... we've never developed them superfluously, I think. In fact, I think to the contrary, sometimes we think we got something, we want to use something general purpose.
Oxide and Friends
Querying Metrics with OxQL
We try to make something general purpose work and you realize this is actually creating more drag than it's solving a problem for all the reasons you mentioned at the top. Actually, a former colleague of ours, Mike Shapiro, wrote a uh, ACMQ paper years ago on, on purpose-built languages, uh, featuring MDB, um, and are talking about ADB and, and the language in ADB. So, um, um,
Oxide and Friends
Querying Metrics with OxQL
Okay, I'm gathering from the chat that apparently there are some DSLs out there that feel more elective. I realized that now I have turned into like, people are like, you've gone from an okay, fine, like, oh, XQL, we reluctantly acknowledge its right to exist. But now you're just like a DSL apologist? Now it's like any DSL. You're a DSL maximalist.
Oxide and Friends
Querying Metrics with OxQL
Yeah, which I think actually gives us terrific power. I mean, I think that that's a very important layer of abstraction that we've injected. I also think, and I know you mentioned this at the top, but in terms of why not SQL, you also don't want to give people the impression like, oh, this is great, it's SQL, I know that.
Oxide and Friends
Querying Metrics with OxQL
It's like, no, no, no, sorry, did you miss the 16 asterisks that are after SQL? It's like, this is actually not just SQL, sorry. Okay.
Oxide and Friends
Querying Metrics with OxQL
Yeah. Totally. Well, and I mean, I think that I know that we've gone our own way on lots of different things. Again, I think it's kind of, it's still surprising to me that the query language is like the bridge too far for the internet. The mob has shown up with the kind of the pitchforks and torches here. But the, I mean, we also did our own P4 compiler. We're going to talk about that. We did.
Oxide and Friends
Querying Metrics with OxQL
I mean, This isn't even our own... We've done other compilers. We've done multiple operating systems.
Oxide and Friends
Querying Metrics with OxQL
Obviously the database people are like, I don't care. Yeah. Do your own switch. It's fine. It's like, no, no, you're sorry. You don't understand how like that's madness. Like that actually is. Yeah. But the query language, that's just, that's great. Right. Right. So I think, but I would say that just in general, when we make one of these decisions, it's because we have exhausted the alternatives.
Oxide and Friends
Querying Metrics with OxQL
Like we have not done on any of our own Silicon people. Like just, just, you know, just clip that one.
Oxide and Friends
Querying Metrics with OxQL
We certainly, there is all, there are plenty of folks at Oxide who are, who exactly, everyone's like, we haven't done our own silicon yet. Right. And there is plenty, you know, we haven't written our own database yet. Yet. We have not written our own database yet.
Oxide and Friends
Querying Metrics with OxQL
Don't you think that feels more... But we also have already announced that we're supporting our own database with respect to Cockroach. That's right. Yeah, look. But we are not doing our own silicon. And that's basically it. We have that in our own instruction set architecture. I mean, I don't know. There are things we have. But we are doing these things on every one of these decisions.
Oxide and Friends
Querying Metrics with OxQL
It is almost certainly the case that we actually went into, we assumed that we were going to use, you know, we assumed we were going to use talk before we came to Hubris. We assumed that we were, and I think for all of these things, we certainly were using Intel's tooling with respect to P4 before we kind of came to the conclusion we need to do our own P4 compiler. I mean, for all of these things,
Oxide and Friends
Querying Metrics with OxQL
We went in wanting to use something else and then realizing this doesn't fit exactly. And I think, Ben, as you mentioned, we have great apprehension when we go our own way. I know it doesn't feel like it, honestly, that you people have any. Are you sure you have any apprehension? It doesn't really feel like it.
Oxide and Friends
Querying Metrics with OxQL
For people that have apprehension about going your own way, you sure go your own way a lot. It's like, well, yes, I know. I know. I get it. It's a bad look. But we do go our own way. We do have apprehension about it. We do really carefully deliberate on this stuff. And this was, I think, to me, this is a very clear example where going our own way is the right decision.
Oxide and Friends
Querying Metrics with OxQL
I am. I'm also glad that someone has mentioned the AT&T Hobbit in the chat. I am less than like 50 feet away from an AT&T Hobbit manual here in the Oxford office. I think to point out just to our, our, our, there's our return to the office conversation. I like to return to the office just to be close to my AT&T Hobbit manual, but we are, um, Well, Ben, this has been great. Thank you very much.
Oxide and Friends
Querying Metrics with OxQL
In terms of comparing this to your thesis defense, do you feel the questions were, how did these two compare? You're just like, hey, next time I would do something less stressful, like get a PhD.
Oxide and Friends
Querying Metrics with OxQL
I'm glad you brought this up because we now would like you to leave. And now we're going to invite everyone up on stage and we're going to discuss whether the novelty of OXQL and whether you should have just used SQL.
Oxide and Friends
Querying Metrics with OxQL
Oh, OK. Oh, yeah. Thanks. Thanks for bearing the lead there. I mean, I was curious. Right. Yeah. Oh, my God. I got to say, like, if you have news, big news, bad news, good news, do not bury the lead. Just this is like life lesson. Just get that news out there like early.
Oxide and Friends
Querying Metrics with OxQL
You know, like what's going to call from the from the disciplinary end of the high school, the assistant principal for discipline. And clearly this person has dealt with a lot of people because the first thing she said, I'm like, oh, my God, it's a disciplinary from the high school. Like, oh, like. Do I need a lawyer, basically, is my first thought. It's just like, it's good news.
Oxide and Friends
Querying Metrics with OxQL
Like, okay, it's good. That was nice of them. Oh, very nice of them. Very nice of them. And also, like, I mean, the whole thing was nice. It was very nice that they called with good news. Which is good. It's not always good news. So, you know, if it had been bad news, like, let's lead with that. Anyway, so there you go. Well, Ben, I'd like to lead you, lead with the good news.
Oxide and Friends
Querying Metrics with OxQL
Yeah. And we test ran the new AHL bot. Adam has been completely replaced with an AI. That seemed plausible. I don't know. I bought it this whole time. Yeah, exactly.
Oxide and Friends
Querying Metrics with OxQL
That's right. Just somebody mentioned Antler, it'll immediately start generating. Exactly. And, uh, and, and check out the, uh, we, we dropped the link in the chat to the auto podcast. That was very creepy. They were talking about at the top. It's kind of fun to check.
Oxide and Friends
Querying Metrics with OxQL
It's creepy. For, uh, well, Ben, thanks again for Oxquill. Um, and I'm going to go take it so I can go to bed. All right. Stay curious, everybody. Yeah. Thank you. Thanks everyone. Talk to you next time. Bye.
Oxide and Friends
Querying Metrics with OxQL
With the auto-generated podcasts, it's mesmerizing. I can't stop. It's really weird. So Ben, I'm sorry. I was also sending Ben all these, obviously. Uh, you, you have now you've lost your, your gravitas voice. I don't know. I'm taking you a little less seriously right now. You get kind of back to, back to clown college.
Oxide and Friends
Querying Metrics with OxQL
And just to be clear, what data are we storing here? Because I think people have kind of a natural question of what ramifications does this have for the user of the rack? The decision around ClickHouse is really an implementation decision.
Oxide and Friends
Querying Metrics with OxQL
And they're very cheerful and they are very interested in promoting whatever document it is that you've put in front of them. They definitely believe whatever they've just read.
Oxide and Friends
Querying Metrics with OxQL
And we're not precluding anyone from slurping this data out and shoving it into some other system that they might want that has some other query language that they... No.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Is that true? I can join up. I think I've always joined with putting up my hand. Have I always done this? I thought that's how you joined with, but I thought I put up my hand and then I joined. It's like, are you just like core Brian always raises his hand before. It's like that guy in section.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I want to get to a bunch of that, but before, do you mind, can we talk about Scuttlebutt? Is that inbounds?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, and I, I mean, I got to believe, I don't want to editorialize too much, but based on what I know of Twitter at the time, if I think that there are the, the, the fire hose were kind of the crown jewels of Twitter. They did not want to let people on that. You had to pay a lot of money to get onto the fire hose.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I don't think that that was a, I mean, I I'm sure folks will correct me if I'm wrong, but I question whether that was a technical limitation. I think that was a, a business decision. And I think as folks saying that.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I think it was an injection of ads decision. Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I also think they had also made, I mean, and this is another thing that I think that, you know, as kind of reflecting Paul on the terrific work that you all have done, there is an advantage to you coming after having the world having figured out one, how to architect a Twitter that actually performs, which Twitter itself spent half a decade doing.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And then two, the systems are just better that you can go use, right? You don't have to deal with a Ruby VM, for example. I mean, you've got other things that you can go... I mean, even Ruby has improved. I know I'm going to hear it from the Ruby folks. But all of that software has really improved a lot. So you've just got...
Oxide and Friends
Scaling Bluesky with Paul Frazee
And then that fire hose is so extraordinarily valuable to everyone, I feel. I mean, I think that I love what you had said in the, and I want to drop a link into the paper that you all recently published, which is terrific, by the way. I love this ACM paper that you all wrote on Blue Sky and the app protocol.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Because I want to talk about Scuttlebutt and Beaker. People think they're in here to talk about Blue Sky and Blue Sky, but we're actually talking about Scuttlebutt and Beaker. So, Scuttlebutt, could you describe what Scuttlebutt is and how you got involved in Scuttlebutt?
Oxide and Friends
Scaling Bluesky with Paul Frazee
And just in terms of the value to researchers to be able to get this whole thing is extraordinary.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Okay, but I've got to say the level of growth, I mean, and maybe Elon called you up and like, look, I want to help Blue Sky out all I can. Please tell me the things that I should do to my own social network to drive people to you as quickly as possible. I mean, that actually a lot more makes sense, honestly, if I knew that. But I'm not saying for you to confirm or deny that.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I'm just saying it's like kind of consistent with what's happened. I mean, the level of scaling that you've seen, were you expecting that in your kind of wildest dreams? Oh, man. Yes, this is going exactly according to plan. This is what I expected.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Because this is obviously a problem that you have been thinking about, and I think this is true for a lot of folks on the Blue Sky team. People have been coming from different angles thinking about this problem for a long period of time. How did you get into Scuttlebutt?
Oxide and Friends
Scaling Bluesky with Paul Frazee
That's amazing. And so you all, because I mean, I originally, I mean, you launched your private beta in March, April of 2023. When was that? Somewhere in there. Yeah, it was around. Um, the, um, and which was great. We got, I got lucky enough. Fortunately, we've, um, I always felt Steve Klabnik is here.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I always felt I may have put Steve in an uncomfortable position when I was asking him for a blue sky invite. He's like, I mean, I'm like, I guess I'm technically your boss, but like, and I know, I know this sounds like, but you know, I mean, Steve is here either to defend himself or to, to bring up his attorney on stage. Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Right. Well, I really appreciate you sharing an invite with me. And invites were super precious. And then, Paul, somewhere along the line, the decision is made to like, really, okay, we need to like, we've got enough users on here to know what we need to go do. And we're going to more or less shut it off for a little bit. I mean, keep the site up, but we're not actually adding more invites.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Or was Steve lying to me this entire time when he said he was out of invites?
Oxide and Friends
Scaling Bluesky with Paul Frazee
October 2022, what was going on? Yeah, what happened then? What happened then?
Oxide and Friends
Scaling Bluesky with Paul Frazee
So... Welcome to Oxide, friends. You don't have to dig too far to get to the Postgres era for anyone. So this is good. This is very on-brand.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So what I can say, I really admire the discipline of the team because when you, I mean, what would, At some level, and I know Blue Sky is interesting because you're a public benefit corporation. There's a bit of both sides to this in terms of you're seeking to make a profit and to be a financially viable company, but you're also seeking to serve the public good.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I know it's a little bit different, but I think it still takes a lot of self-control around here because everybody wants product market fit. And when you have product market fit, which I think we could not, not to disparage secure scuttlebutt or beaker, but this was, this was like probably things that didn't have as much product market fit. Correct. I see what you're getting at.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You can see we're having this conversation about Beaker, but poor Beaker. You've got this enormous product market fit and to have the discipline to say, if we don't fix this, we can't fake it till we make it, move fast and break things, all these other kind of Silicon Valley mantras that actually you're going to end up with something that's broken at the end of it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
which I got to say is another act of real discipline, because I think that part of what makes this challenging is that you are trying to build blue sky as kind of an exemplar of what's possible on at proto. And it's in any of these things, it's tempting to just be like, ah, just like, okay, can we just get like blue sky working? And then we'll, we'll quit.
Oxide and Friends
Scaling Bluesky with Paul Frazee
We'll figure at proto out later and allow people to do this kind of on their own later. I mean, it must be, I mean, obviously that's not a temptation that you all had because of the strong call to mission, but I just think it takes an admirable amount of discipline to really say like, no, we actually need to do this the right way. And we need to allow the,
Oxide and Friends
Scaling Bluesky with Paul Frazee
We want to be sure that this thing is actually, that at Proto is the actual basis that we're building this on, and that Blue Sky is but an instantiation of it. I'm really impressed about that. Was there... I mean, there must have been some level of tension of like, hey, we've got to, like, we're missing a window of opportunity. Did people feel that way?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Or did you feel that the window of opportunity, did you... know that your agent on the inside of X, Mr. Musk, is going to continue to operate per your explicit wishes?
Oxide and Friends
Scaling Bluesky with Paul Frazee
I got to say, I really... Just speaking as a consumer, I really appreciate that choice because there were very few... One of the reasons that I was still sadly on the fucking bird site was because of DMs. And I actually do need DMs. And this... But DMs are a place, and my DMs have always been open on both platforms. It's been very interesting to watch the supposedly bot-free Twitter.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Just like, I'm the guy that likes to stand while he's presenting a section. Is that wrong? I guess so. I mean, I thought everyone did. I guess. Okay.
Oxide and Friends
Scaling Bluesky with Paul Frazee
either right now i thought it was great when blue sky it's like oh i got some i got like unsolicited crypto and and oh without noise and some porn is here too it's like this is great like this guy's gonna make it this is so this is all warming mods uh this is like three quarters of them is like report a spam report a spam but it's so essential for the times when it's not that though so you know it's good yeah
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, and I also feel it's like DMs are... I think that you all made the right call there in that DMs are a good place where it's like, yeah, I try... This does not need to be a part of that proto for this. I actually just need the Blue Sky app to have this functionality. I guess you all have done such a good job... mimicking and it's Steve did the right thing.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So Steve was a Twitter dead ender until he wasn't. And I thought like, there was a time when Steve's like, yeah, I'm like, I'm just, I'm going down with the ship, man. Like, I just don't care how bad this place gets. And I'm like, I don't know, man. I'm kind of, and then I'm like, you know what? I am. I'm going to go down with the ship too. And then I look over and I'm like, where's Steve?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Steve's gone. And I'm like, Steve is Steve has like deleted his Twitter. Steve is gone. Gone. And I'm like, oh, okay, this is actually, maybe I'm being liberated to get Gone Gone. So, I mean, I desperately wanted to get off of that thing as it was just like absolutely descending into the muck.
Oxide and Friends
Scaling Bluesky with Paul Frazee
But the thing that has happened to me a couple of times, and I don't know if this has happened to you, like, look, look, I would love to tell you that I've been off for a long time, but it's a process, man. One day at a time. Okay. Like every day is a struggle.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Some days I got to call my sponsor and the number of times that I have, you've got, you all have done such a good job on just like the look and feel the number of times I have been like, Hey, you know what? Actually looking at these, these posts are actually surprisingly good for the hellscape that Twitter is. Oh no, wait a minute. I'm blue sky.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Or then also what has often happened is that I will get a... You know who's not on Blue Sky? Only the good VCs are on Blue Sky right now. This will change very shortly. But the VCs that really troll the hell out of me are all like have shackled themselves to the mast of Twitter. And I just don't have the self-control to not actually go and go to the rage bait.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Um, but then I will, the, the, the, the, the inverse also happens where, um, I will accidentally leave a Twitter window up and I'll be like, oh man, blue skies. So you've done an excellent, excellent, excellent job of.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, Mastodon. The problem with Mastodon, I love Mastodon. Do you? I love aspects of Mastodon. No, no, no, I don't love Mastodon. I was trying to find, this is like where I'm sorry for not being sorry. I love the idea of loving Mastodon. That's what I love. Yeah, I think that's right. But Mastodon has got the usability of a BBS. And I love BBSs.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Paul, I'm not sure how much... This is like modem era social networking. But the problem was... And this is part of why it's really important to... And I thought you all did actually a really good job of... Talking about why not ActivityPub and why not Mastodon. And there are some really good reasons for that.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I think that Mastodon should... I mean, I think it's going to be... We're going to continue to be on Mastodon. I think it's great. I think that it serves a great purpose, but... the problem is that the, uh, like I'm a, I'm a, I am a baseball and football fan and I like social networking because I, I like sports Twitter. What was sports Twitter?
Oxide and Friends
Scaling Bluesky with Paul Frazee
And, uh, like you're just never going to end up with that on Mastodon. Um, it's just not going to happen and you need, and also, or, or, or Paul, the, the shake test, are you, I'm not sure if you, uh,
Oxide and Friends
Scaling Bluesky with Paul Frazee
so the the real acid test for social networking is like if you're in the bay area and you feel that you just felt a quake oh yeah and um and i'm sure you know this but um the uh you should know that blue sky has passed this test with flying colors because we didn't just have a quake we had a tsunami warning recently
Oxide and Friends
Scaling Bluesky with Paul Frazee
And my phone, like every other phone in the Bay Area, exploded with what I assumed was an Amber Alert. You're like, tsunami warning on a quake? I didn't feel. My first thought is like, this is like an IT error somewhere. This is like a missile alert in Hawaii. And, uh, but it's actually obviously real.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And it was, what was really interesting was blue sky was just, just exploded with, you know, people talking about it, talking about the, the, uh, tsunami know your zone site, uh, is down. You can't get a database handle to know your tsunami zones and no one can figure out if they're going to die in a tsunami or not. The, and blue sky.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I, and what I also really loved is then my geo sciences feed on blue sky, uh,
Oxide and Friends
Scaling Bluesky with Paul Frazee
was great i mean it was it and the and meanwhile on the other side it was just like they're all it's all garbage so i mean it was it really passed this test with flying colors um could you talk a little bit about the feeds because this is something that again i've really i i think feel i've started to make use of i haven't done enough with but i absolutely love and it's really kind of a first class thought in the way you built the thing
Oxide and Friends
Scaling Bluesky with Paul Frazee
Timestamp ordering of the people you follow. It's like, oh, you're interested in the tsunami that may be coming to your place in an hour? I don't know. You should follow more geologists. Maybe in the next life, you'll follow more geologists.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You should have done that. You should have been carefully following geologists. You're like, okay.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You can't say I, but we know that I actually just love the idea of loving Mastodon.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, in the feeds also, Adam, what you're saying about like, it's one of those ideas that you know is an important idea whose time has come because you're immediately like, why are we doing it some other way before? Like, this is, you know what I mean? It's... Not that it's not at all obvious because it obviously wasn't.
Oxide and Friends
Scaling Bluesky with Paul Frazee
But the fact that you – I mean, it's one of these things where you – Paul, you all made a bunch of good decisions that built on one another. And then out of this comes this, I think, artifact that's going to be, I think, more and more important as time goes on because I think that –
Oxide and Friends
Scaling Bluesky with Paul Frazee
you get to bring your entire self because all of us consist of like different things and different things at different times. And the feeds allow to allow, give us agency over that, which I think is extremely important.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Steve, can you explain what that feed is? Because I love that and you had talked about it in your post about how AdProto works.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I love that. Yeah. And I, I think it's, it's so, and I think we're just going to see a lot more experimentation in that. And again, giving people more agency, you know, Adam perversely what it reminds me of, and sorry, Paul, just to go overly Gen X on you here, but the, the day MCA died, I wanted to just listen to all of his music from beginning to end and,
Oxide and Friends
Scaling Bluesky with Paul Frazee
And Pandora is like, let me give you something that sounds like the Beastie Boys. I'm like, a man is dead, Pandora. I'm not interested in your algorithm right now. I am mourning a dead musician you need to actually give me. And that's when I'm like, I'm going to Spotify. I'm not coming back.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah. He's lived this extraordinary life that you've missed because you just couldn't take it anymore. It's like Mr. Peanut on Twitter. I finally muted Mr. Peanut because I couldn't take the ads. Jesus. So, I mean, Paul, this must have been one of these breakthroughs that you just kind of like...
Oxide and Friends
Scaling Bluesky with Paul Frazee
feeds must have felt like a lot of things were clicking I mean it's such a I mean actually let me ask this did you kind of know immediately like oh of course this makes a ton of sense or was it when you started experimenting with it you saw it
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah. Was that as a kind of a response? Because I mean, this is like Pretty early. I mean, this is before the animals were walking on their hind feet in terms of social networks. Or maybe they were already. Maybe they were born on their hind feet.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I agree with you that there's some UX things that you could probably do to... But I think that the bones are really, really sound on this one.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I also think that you've got... The thing that I also loved about it, just like being able to follow my geosciences feed and my earthquakes feed during the tsunami warning, is that you are able to... I mean, there's so many... Part of what made Twitter great
Oxide and Friends
Scaling Bluesky with Paul Frazee
when it was great was you would stumble into one of these like delightful communities and of like super nerds um and you know suddenly you've got you know top scientists in the field who are i mean just absolutely nerding out and going very deep on a particular subject and it's very nice to be able to like when you find those you feel like you've tripped over something
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I think it's like feeds are going to make it way easier to find those. And I just think that like the, there's so much we can go do to find amazing feeds. I also want, I want feeds for podcasts too. It's like my own, now we're getting into my own like weird agendas, but like podcast search is so bad. And I, I,
Oxide and Friends
Scaling Bluesky with Paul Frazee
I think I want feeds for other things in my life, not just posts, not just skeets is what I got to say. I think it's amazing.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, but you're going to turn 39 when all the other millennials turn 39, Steve. You're not beating the allegation here. Listen, Paul, this is why we're trying to talk about BBSs and the Beastie Boys, just to get Gen X here pulling on it a little bit. I appreciate that. Yeah, yeah. So feeds were obviously, I think, really important. And I think feeds are going to be even more important.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I think we're going to see a lot of creativity. I don't know. I'd be interested to get your take on it. But I just feel like, boy, there's so much stuff that's possible there, especially when you go to look at things like LLMs. I mean, there's a lot you can go do there that hasn't been done yet that would be really exciting. Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, and I think that also part of what I'm really excited about is that I think that we've kind of had this era of maximum engagement. And we are seeing kind of the logical conclusion of that in Twitter. And I think I actually want a world where we're actually able to find the hidden nuggets a lot more easily.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And we're able to find that the delightful bits that are at the edges are, to me, way more interesting and way more delightful. And I just feel that feeds are an important way of getting that stuff. It's just like the quiet feed, Steve. Let's get to the people that don't say very much, but when they say something, it's meaningful.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I do think, by the way, that there are a lot of opportunities. I think people will pay for things that add value, and I think there is opportunity to monetize a bunch of this stuff. Adam, this is where our inner VCs are coming out. What's wrong with us? We're broken human beings. That's right.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Totally. So you mentioned feeds, and then the other thing that you wanted to, that you were rethinking kind of at the same time was moderation. And what was your, obviously, another, I mean, just age-old issue in social networking and now a hotter issue than ever, just with our divisive era. What was your thinking on moderation?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Because when you rolled out the beta, moderation was still pretty primitive in terms of the tools that were offered for folks.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Like feeds, you're saying, look, there's not going to be one way to moderate. Can you talk about that approach? Because I think that's a really interesting approach. We're going to allow people to moderate in different ways and allow people to give people some transparency and some choice among different approaches to moderation. I'm probably being overly reductive there, but...
Oxide and Friends
Scaling Bluesky with Paul Frazee
I have to say it's kind of remarkable. And as you said, it permeates your thinking of like thinking, and you call it the credible exit in your paper in terms of thinking beyond blue sky organizationally, I think is kind of remarkable and unusual, I think, for an organization to think we need to build in the things that
Oxide and Friends
Scaling Bluesky with Paul Frazee
that will survive an arbitrary change in management and arbitrary, I mean, which is kind of, I mean, it reminds me of open source in that regard that where, you know, we are open sourcing this to assure basically that it, that it can endure, it can endure beyond its corporate vessel.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And, you know, not that Adam and I might be speaking from personal experience a couple of times over, but the corporate, corporate vessels don't endure folks. The, except for this one, of course, for whatever one you happen to find yourself in. Right.
Oxide and Friends
Scaling Bluesky with Paul Frazee
exactly um so i i think that that that is uh the guy how do i boot some out of a discord larry ellison buys blue sky who would say such a terrible thing i unfortunately oh my god i'm so sorry paul this is the the demographic that we attract you know these are um um But I think that like being able to kind of think beyond yourselves in that way is, is really pretty interesting.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Can you speak to some of the specific tools that you books? I mean, in particular, like I love the ability to detach a quote tweet, I think is a really, I mean, that's not one that I've seen before. Could you talk about some of the stuff that you've done to just give some tools out there for folks?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Paul, thank you very much for joining us. I am super stoked for this today.
Oxide and Friends
Scaling Bluesky with Paul Frazee
2019 somewhere around there and like you experienced that once and i think you'll realize like oh this is you gotta have something to slow this down um yeah so yes i adam actually once gave me the book so you've been publicly shamed which i would highly recommend not necessarily being publicly shamed but uh adam obviously thought of me immediately when he saw the book so i i think that's uh nobody on this voice call has been publicly shamed at all definitely not never before
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I mean... So are there other things that you all did like that? I mean, I think it's like one of the things that I think is interesting is that mute lists and block lists are public. And I'm sure this is an issue that is controversial. Oh, yeah. Not to have you weighted to every issue of controversy, but I mean, one of the things I really didn't like or I thought was lame about Twitter...
Oxide and Friends
Scaling Bluesky with Paul Frazee
is when you had very powerful people that would block people that they disagreed with. And I was like, that's not kind of not what it's for. Mark Andreessen, the blocking functionality is not so you can block me. Sorry. Not to overly personalize it, but I've always been like... I've always wanted to be like, Hey, so I find Mark Andreessen, you get to block little old me.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I also like, I kind of want to follow everyone that you're blocking. So, cause I think I would like, like them a lot. So I just want to like, have a feed of people that Mark Andreessen blocked and that requires public block list.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Oh, but that's not why we, Yeah, I know. So do I get kicked out? Yeah, you're out. Of course you're out. Can't you read the sign? Get out of here.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Pretty prescient, I have to say. Because I feel like in 2012, honestly, maybe I'm just a dunce, but it felt like it was kind of the good old days for Twitter. I mean, it felt like it was fun and light. And maybe Gamergate changed that. I'm trying to kind of replay that era in my head. But definitely, this was... I mean, this is like, why wouldn't anyone like Facebook?
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I mean, obviously you've seen that, I mean, as the popularity and it's just, it must be still a bit surreal to watch the popularity of blue sky really stumble into like, I mean, not stumble, but really become, I mean, like for whatever reason, Adam, you said ESPN has got a blue sky account now. I mean, I just felt like, wow, that is a blue sky has really arrived. Um,
Oxide and Friends
Scaling Bluesky with Paul Frazee
But presumably with that has become, I mean, you get a lot of people who are trolls in the kind of the oldest sense of really deliberately wanting to get a rise out of people who are now attracted to the social network because of its very success. I mean, I assume, is that fair that you've seen more users, more problems, I assume? Oh, yeah. Oh, yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Is it possible that their training data is only older siblings? God, I know. Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
The older siblings, like, I mean, cause older siblings do. And I look, I say this as an older sibling. I mean, you just like, you optimize the least amount of effort for the most amount of, of reaction. And it's like,
Oxide and Friends
Scaling Bluesky with Paul Frazee
yeah i mean it's they're very very good at it um especially you know when the i when my oldest has like the countdown to when my 17 year old is going to crash out is the new term for like going it's just like it's you know uh so i think this is very deep in the human psyche is what i'm saying it's like you're trying to contend with something that is uh very very deep um
Oxide and Friends
Scaling Bluesky with Paul Frazee
And so, I mean, how do you... I mean, you said that the things that you've deployed seem to be so far holding up. Have there been any surprises there in terms of... Or maybe you've just been surprised by the scope and depth of some of these efforts.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Facebook just feels like a net good for society in 2012, says I. I mean, I just feel like I was, anyway, I was very naive. I mean, I felt like at that era... the, I mean, it was more like Twitter was finally kind of functional. I mean, there was such a long period of time where it was just not very functional. Um, and it could kind of keep up with everything and everything else.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, that's great. And I mean, I think also, is this where I presumably at proto is helpful where you think like, this is not like your fight alone, that there are other folks that can like start implementing some of their own ideas. I mean, if you, in terms of like getting into the, the, like the, the technical details of how those moderation servers actually work.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Because I think one of the interesting things, the observations you had is that the skill set to like run one's own server and the skill set to moderate don't overlap very much. Yeah. Can you speak to some of the innovation that you're kind of, I mean, are you seeing some innovation there as kind of third parties experiment with different ways to moderate?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Oh, I... I'm not convinced that there's only one of you. Especially in those early days of Blue Sky, you were everywhere at once, I just felt. It's been remarkable. I'm not sure how many of you there are, but I appreciate your collective allowing one of you to come speak with us today. Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So I really admire the prescience of people saying like, no, there's like a, there's a, there's a centralization of authority here. That's actually problematic. Um, and, Then I got to ask you, were folks like, you know what, I'm burning my boats on Secure Scuttlebutt and I'm now like, you can't find me on any other social network? Or was...
Oxide and Friends
Scaling Bluesky with Paul Frazee
Totally. So how do these labels work? Where are the labels actually stored?
Oxide and Friends
Scaling Bluesky with Paul Frazee
And then, so I subscribe, though, to these labeling services. So I get to choose, like, if there's a label that I really like, I can subscribe to that. Is that right?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, that UX definitely doesn't clutter anything else in that it is definitely... That's great. I actually did not know this at all. And going to, like, someone in the chat had linked to the U.S. government contributions labeler. Yeah, I think so. And this seems like, I mean, this seems like a really good experiment.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Because then, of course, I want to know, okay, what are the labelers that Adam subscribes to? Because I may want those same labelers, or maybe those are labelers just to avoid, because, you know, that guy, I don't know. I mean, he just kicked me out of the Blocked by P. Marker Club because I'm no longer blocked by P. Marker. So I don't know. I've got kind of some axes to grind over there.
Oxide and Friends
Scaling Bluesky with Paul Frazee
But I mean, it just feels like you're giving people kind of another tool to go experiment with this problem that you did not create called bad humans. But also like good humans that actually want to come in and help offer some clarity on some of this stuff. I don't know. I think it's really... To me, it's really inspiring.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I know it's kind of strange to be inspired by the encroaching 8chan of the internet, but I really do think it's inspiring.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, that's exactly it. I feel like it's the giving it a shot bit, and I feel that learning from so many previous experiences, and then also doing it all in the open, because this is all... I mean, you're all open source, right? In addition to having these open specifications, you've got... all or much of your software is actually out there in the open.
Oxide and Friends
Scaling Bluesky with Paul Frazee
That is awesome, and congratulations on that. That is really terrific. Close to without precedent, I assume? Certainly for a social app, it feels like it's without precedent.
Oxide and Friends
Scaling Bluesky with Paul Frazee
This is the very first time. I was just going to ask you, you've obviously got a lot of challenges still ahead, in part because of the popularity. What have been some of the surprises along the way of things that maybe have worked better than you thought, or worked much worse than you thought?
Oxide and Friends
Scaling Bluesky with Paul Frazee
I don't know, maybe you were putting scenes into that category, at least in terms of nomenclature, but what are What are some of the things that you were or are there things that you've been kind of surprised by along the way?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, it was even better than that. It was when ESPN came on, they were ESPN.bsky.social for like a hot second. And literally their first post was like, no, no, no, we know, we know, we know, we're changing. I mean, everyone is dogpiling them being like, come on, you got to change the domain name. You're ESPN.com. Yeah, yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
That is a very elegant solution to the problem. And I mean, and people are obviously calling out like the fact that you get, because you're getting this verification basically, and you're verifying it the way we verify the internet, which is like, why wouldn't you do it that way?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Big win, I think. Can I ask you just a very pedestrian question? Because this is the only thing that has kept me, so I'm currently bcancelbsky.social, but I definitely want to, I actually, I don't know if I want to bcancel.dtrace.org or bcancel.oxide.computer. So this is my own thing I need to go grapple with.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You can, but the only thing that's hung me up is that I need to go immediately squat on my old name, right?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Oh my gosh, that is great. That is terrific. And I'm sorry to disrupt your comms plan on that. Thank you very much for breaking news here on Oxide and Friends. That's just being handed to me. That's terrific. And one of the things that you all are doing so, so, so well, and I just cannot thank you for enough is
Oxide and Friends
Scaling Bluesky with Paul Frazee
is the degree to which you're listening to people who use the product and you're using the product yourselves. It feels basic, but it's amazing how often this doesn't happen. That's a great example. I know you've got so many... terrific ideas for things that can be done.
Oxide and Friends
Scaling Bluesky with Paul Frazee
But that's just like, that's a really kind of like a basic, simple kind of pain point that is not debilitating, but boy, thank you so much for, for fixing it. I think it's just, it's part of what makes it as a user, a blue sky. It makes me so enthusiastic for the future of this platform and, One thing I do, I won't be mindful of time because I know Adam's going to have to bolt here.
Oxide and Friends
Scaling Bluesky with Paul Frazee
His no longer toddler is now, is that kid in high school yet? First grade? What is he in? Second grade. Second grade. God, yeah, built like a high schooler. But there's one last feature I wanted to ask you about, and that is starter packs. Um, because another thing that was, I know is basic. Uh, it was really important.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I feel, um, could you just describe a little bit like kind of the, the, the thinking behind starter packs?
Oxide and Friends
Scaling Bluesky with Paul Frazee
dirty tactic anyway i think that is so damn dirty i mean i agree yeah well and like it gives an ick vibe that like anytime i get up to a nap now and i think everybody has it you know he's like oh no i'm not gonna give you my contact actually get out of there no don't even ask it's like i don't want to peek like no no peaking gross get out of here yeah
Oxide and Friends
Scaling Bluesky with Paul Frazee
And for whatever it's worth, just my own personal experience with starter packs, I was trying to get... I'm an Oakland A's fan. The hashtag sell the team. And we kind of got this now beleaguered lost tribe that is still very much on Twitter, was very much on Twitter.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And talking to the folks who are kind of like the leaders of that community being like, we've got to get one of the starter pack because the entire community wanted to leave, but you kind of needed everyone to go at once. Yes. And that is what starter packs – starter packs were a lifeboat for people who wanted to – and I saw this in various scientific communities.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I saw it in certainly in Oakland A's Twitter where it's like, okay, this is now the opportunity. And I think actually another thing that I love about just like the dynamic was people would start a starter pack with 60 people. And those people would kind of come on and the person who started the starter pack would always say, hey, like, let me know if I forgot anybody.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And you've got like a bunch of people raising their hand being like, oh, God, I'd love to be included in that starter pack. And they ended up being very inclusive and uplifting and a great way to move communities across. So it doesn't surprise me because in that multi-month gap where you're like, why isn't this working? Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I, and I'm sure other people in various other communities are kind of lobbying people who are still on Twitter to be like, no, like let's get one of these together so we can get everyone over at once. Um, yeah, it's been really great. So thank you very much for that. It's a good, it's been a, it's been a great feature.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And just again, thank you. And thank you, Paul, for taking all the time with us tonight. And we are huge fans. Love what you're doing, what you and the team have done. Love that.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I mean, we feel like very much kindred spirits and that like this is a problem that you all have as a team have been thinking about for many, many, many years and watching all of that wisdom get expressed in what you've done. And you've created something. I know it's challenging. And the more people you get in there, there's going to be challenges that are going to arise with that.
Oxide and Friends
Scaling Bluesky with Paul Frazee
with blue sky yeah we were just like what are we doing to ourselves i mean honestly it explains so much too because i think and i just want to like not lose us say kind of off the jump the blue sky is so remarkably available and remarkably usable in a remarkably short period of time and
Oxide and Friends
Scaling Bluesky with Paul Frazee
But boy, what you're doing is so important and special. And we're really deeply, deeply, deeply appreciative. You've given, I think, a lot of people, certainly me anyway, real hope in what we can have out of social networking. So really deeply appreciated.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Absolutely. Well, and I'm sure again, humans are tough, so more challenges to come, but really appreciate it. And thank you again for taking the time to come here. And we'll be glad that no one has to hit up Klavnik for invites anymore. Everyone can hop on there.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And Adam dropped an Oxide computer starter pack if you want to follow folks who are, we got a lot of folks from Oxide, obviously, who are over there. Paul, thanks again. Thank you very much. Um, welcome back anytime, obviously. Um, and, uh, keep up the great work. We'll be, um, we'll be there and in the thick of it. So thanks again. Sounds great. Thanks for having me. All right. Thanks everybody.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And next time we're going to be doing our, our year end wrap up. So, uh, bring your, your, your, your best and worst from, uh, from 2024. Um, so Adam looking forward to that. It's going to be fun. Should be great. All right. Thanks, everyone. See you next time.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You know, it's not surprising, of course, that you all come with this scar tissue from these other endeavors where you had tried other things. Because, I mean, and I think that, I mean, speaking as a user, I mean, social networking is really important to me personally. And what you all have done is so extraordinary. And, like, there's been no fail whale.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You know, I mean, it's like the closest we get is, like, Health Red, which obviously I love. But the... I'm sure you're like, listen, let's not paper over it too much.
Oxide and Friends
Scaling Bluesky with Paul Frazee
There have been plenty of... Just when you're adding millions of users a day and you are retaining... It's so responsive and it feels quick and lively and you're not getting... It's not at all a surprise that, of course, you're learning a lot from these previous systems and
Oxide and Friends
Scaling Bluesky with Paul Frazee
So you said there were a couple of things that you knew you wanted to, that you all kind of collectively from your experience knew that you wanted to replicate and some things you wanted to improve upon. What were some of the things you wanted to replicate?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Exactly. Paul, there's so much to talk about here. But, I mean, you all have been on just an incredible ride. I mean, it's a hard... I mean, having watched this thing from when it was invite-only for kind of a protracted period of time, and then just going, I mean, non-linear, not in the metaphorical sense, but the actual literal quantitative sense, has just been amazing.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I can say, I mean, I love both of these things I think are extremely important. I think that the, obviously, I mean, the portability, it should go without saying, but one of the ways that I realized how just viscerally important this is to me, and this is a super stupid thing, but just in terms of my own age and social networking, tracking my own life,
Oxide and Friends
Scaling Bluesky with Paul Frazee
my first two kids I announced with a blog entry, but my third kid born in 2012, I announced for the tweet. Um, the, so that tweet, for whatever reason, like the replies on that tweet have been deleted by Twitter and, or I can't find them anyway. And it's like, damn it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
In terms of that portability, I feel like, and of course, what idiot would announce the birth of their children on social media and expect it to be retained anywhere? Look, she doesn't have a baby book. She's a third kid. This was her baby book. There's going to be the replies on this tweet. Okay. If, if you are a third kid or you have three kids, you know what I'm talking about?
Oxide and Friends
Scaling Bluesky with Paul Frazee
She's, she's lucky. She's got a tweet. Okay. She knows that she knows she's lucky. She's great. But the, so, I mean, just like having that portability is, is, is really, really, really important to people. And it is a, in order for us to really engage in social networking, we absolutely have to have that portability. So, I mean, obviously I love that. Hey, I don't know what her name is.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You're saying I do know her name. Hold on just a second. Let me actually, you will add up my nine. I've said this before, but she, the day she was born, we, the CEO of the startup I was at was fired. Yes. And fortunately she's got a very good sense of humor. So she's like, Hey dad, it was 12 years ago today. Your CEO was fired. I'm like, it's, Is this going to go on for your entire life?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Probably is. But it's true. That is how I remember it. It's just easier to remember the firing date and then I get to her birthday in one hop. But Paul, I also love what you've done with the fire hose because when did Twitter turn off fire hose access? I want to say that was like in, when was that? 2013, 2014, maybe earlier. I don't know.
Oxide and Friends
Scaling Bluesky with Paul Frazee
But the ability to sit on the fire hose is just, I think, amazing.
Oxide and Friends
Conferences in Tech
Okay, now you can hear me. Oh, yeah. Boy, can we hear you. We'd like to hear a lot less of you. No. It was not plugged all the way in. I know you're dying to know the answer.
Oxide and Friends
Conferences in Tech
But if no one else shows up, like it doesn't make sense to give the talk just to you. You can come in the next, I'm going to give this talk again a third time. You know, I'll kind of like go to lunch with my mom or whatever. I'll come back and I'll give the talk to you. And he just says, I don't really feel like doing that. I'm like, okay. And then no one else shows up. And so he sits down.
Oxide and Friends
Conferences in Tech
I'm like, all right, I guess I'm going to give the talk to you and then my mom. And so I start giving this talk and this guy like starts spacing out during my talk. And like, you can't do this. Like there's only one person here. Like the social contract is now different. Like if you're in a big room, you can like space out. That's fine. There's only like one person. It's just me and you, man.
Oxide and Friends
Conferences in Tech
Like you can't like look at the ceiling or whatever. So I kept like trying to make eye contact with him. He's kind of, this is like before phones and laptops or whatever. So he's like literally, you know, like doing the crossword. Doing the crossword, absolutely. Counting like the flowers on the wallpaper or whatever. And fortunately, there was one other person there. It was my mom.
Oxide and Friends
Conferences in Tech
And she was just like, just totally locked on. And I'm like, you know what? I'm going to give my talk to my mom. And I felt like just mortified. I'm like, my mom is... This is the... like technologically the smallest audience I can possibly have. And, uh, but you actually was, it was after that, I was just like, I felt bad. And my mom was like, that was, I, I learned so much in that talk.
Oxide and Friends
Conferences in Tech
I'm like, mom, I just, I'm so sorry. You had to like see that. Oh, cause then we get the end of the talk and I'm like, well, I mean like obviously stop me if you have any questions. Cause you're the only person here. And we get to the end of the talk and he just like walks up and leaves. I'm like, okay, goodbye. I'm glad you enjoyed it. Whatever. Anyway, sorry.
Oxide and Friends
Conferences in Tech
So that was starting from a real low there. But then, yeah, it was AADbug and then OzCon. But in those early days, the conferences had a vitality to them because it was the only... I mean, it's not that they don't have vitality now, but it was a special kind of unique vitality, I feel, because you had just no... No other way of consuming a bunch of this information. Am I being nostalgic here?
Oxide and Friends
Conferences in Tech
Steven, without acknowledging how old Redmug is, what are your thoughts?
Oxide and Friends
Conferences in Tech
that is, yeah. And, and we understand if you need to drop dead at any moment. So you, you will not need to apologize. If you feel free to drop dead, feel free to, but you know, I, Hey, look, if you're well enough to skate, you're well enough to podcast as my mother is fond of saying. So I'm, I'm glad to see that.
Oxide and Friends
Conferences in Tech
Steve, let me get this week because I, as a speaker, like I, like I will not give a talk if it's not recorded. And I will not give a talk if it's not recorded. And, and like the recording has to be made publicly available. That's not true. And, but so I kind of like, yeah, Oh, yeah, definitely. Where we couldn't record your talk?
Oxide and Friends
Conferences in Tech
Okay, Theo, you know, I have like a few regrets in life that I didn't record that talk as one of them. I, so this is the, it was the, my talk at surge 2010. So this is, yeah, it was kind of moving on a little bit, but this is, uh, Theo becomes frustrated with velocity, right? Theo, I think I'm telling the story correctly. Yeah.
Oxide and Friends
Conferences in Tech
And you're like, I, and cause velocity had just, I mean, I think that had just not served, was not serving the practitioner all that well. I mean, if the, what was, is that, what was your, what was the exact source of the disgruntlement?
Oxide and Friends
Conferences in Tech
Theo, you also may not drop dead. I'm sorry. Well, I am really stoked to have you all here to talk about a conference, a subject that is near and dear to our hearts, which is conferences in tech. And I was kind of reflecting back on Because, Stephen, I met both you and Theo at conferences. Stephen, I think you and I may have met at Sun Network in 2002. Is that possible? Something like that?
Oxide and Friends
Conferences in Tech
Theo, have I voiced to you my desire to have a Surge 2010 reunion?
Oxide and Friends
Conferences in Tech
I would love to do a Surge reunion, just because that... I mean, do we agree that Archer pays for it? I mean, I think so, right? Yeah, and it's like the guy that hit the jackpot. He pays for it, whether he knows for it or not. No, he doesn't know for it. So I don't know. Did you go to the search conferences, Adam? This is like 2010, 2011, 2012, 13.
Oxide and Friends
Conferences in Tech
Oh man. They were really, really good. They were really good because it was like also feel like it was kind of a jackpot in terms of timing too. You know what I mean?
Oxide and Friends
Conferences in Tech
They did, and it was also at an era when it was kind of like before... When folks were hitting these scale issues that weren't just at the hyperscalers. Hyperscalers were not hyperscale at that point. They were all a lot smaller. I don't know. It just felt like those years felt special. And also, it was in Baltimore, which is great. I just... very fond memories of, of surge.
Oxide and Friends
Conferences in Tech
And now you, cause I think that was also, that's an early conference that was, that was making videos available from talks.
Oxide and Friends
Conferences in Tech
And you think that YouTube starts in 2006, 2007, which is just not that long ago. And then you hit this kind of era of conferences where, I don't know, a bit of a different kind of a golden era. If you had this kind of pre-recorded golden era that we can all live as in our memory, you had these talks that are all recorded. But also, like, really broadly attended.
Oxide and Friends
Conferences in Tech
Because, Stephen, people don't have your disposition today of, like, well, I'm just not even going to go to the talk. I'm only going to watch it online. People are definitely watching it in person. But the fact that the recordings are made available, really. And, I mean, what I started to realize after, like, a year or two of this is, like, wait a minute.
Oxide and Friends
Conferences in Tech
There are way more people consuming this when it's recorded than are in the room. Right.
Oxide and Friends
Conferences in Tech
Yeah, and I think that's a very good point because I think when talks became recorded, it did change kind of the tenor of a talk. It was kind of necessarily more of a performance and less of a dialogue. It was not a boff. To go back to kind of the birds of a feather sessions, it was much more of a...
Oxide and Friends
Conferences in Tech
I mean, not necessarily scripted, but definitely not necessarily having that iteration of an idea was not happening as much for sure when you kind of hit that recorded era. And I decided somewhere along in there, like 2013 or 2014, I decided that I'm never giving the same talk twice. And I'm just not going to do that.
Oxide and Friends
Conferences in Tech
Yeah. I'm sorry, I'm doxing you as old. Listen, don't you listen to our podcast? This is like the Gen X Senior Hour. We talk about attending readings from Ad Rock of Beastie Boys.
Oxide and Friends
Conferences in Tech
Am I the only one? I think that's a little bit of edge to it. I kind of like it. I mean, it keeps the audience on edge. Oh, 100%. I do think that some of those were... It is great when you've got a room of people that are really engaged in what you're saying. It's energy that really does bring something that is greater than the sum of its parts.
Oxide and Friends
Conferences in Tech
Theo, one of my favorite talks of yours is your closing plenary from Surge 2011, I think.
Oxide and Friends
Conferences in Tech
This is where it was the Elfin, the Larry Wall Elfin. Do you recall this?
Oxide and Friends
Conferences in Tech
It was wild. A little bit of this is like, you were complaining to me. So the conference is like, we're kind of in the pre-conference, like the days before the conference. You're like, I've got to give this plenary in like three days and I've got no idea what I'm talking about.
Oxide and Friends
Conferences in Tech
And then like unrelated to the closing plenary, you were relaying to me your like the extreme frustration of your past week, which I found hilarious. I'm like, Theo, that is your closing plenary. Like you just need to go and like describe your week. And I still feel, I got to say, I still absolutely love that talk. Because in part, because you can just like feel your visceral pain.
Oxide and Friends
Conferences in Tech
The thing I also love is Adam, Theo goes down a blind alley because someone has his exact problem and suggests a fix. And then someone replies to him saying, thanks, that fix worked for me. The fix was wrong. And it totally, and there's like, it feels like there is a special place in hell for someone who says a fix that does not work works for them.
Oxide and Friends
Conferences in Tech
You can waste an arbitrary amount of time, but it was, uh, it was right. And I think that was kind of like, that was definitely that era of like, you've got a big room and it was great. I mean, it was, it was, it was, it was a lot of energy, live performance. It was, uh, it was really terrific. Thank you. But then I feel like you guys did Surge for, I think, five years and then folded the 10 up.
Oxide and Friends
Conferences in Tech
I think. But you stopped doing Surge, presumably because the dynamics were shifting and obviously conferences are super expensive. And what would... Maybe this gets to part of what Stephen was saying in terms of the dynamics really changing about conferences.
Oxide and Friends
Conferences in Tech
I'll get you Schlossnagel. You could start a conference in opposition to Velocity. I was the Velocity China chair for fuck's sake, but whatever. Yeah, these people eat their own. And then O'Reilly, I mean, that was an O'Reilly conference. O'Reilly had many, many years of conferences. And then after COVID, O'Reilly totally folded that up and decided to go completely online.
Oxide and Friends
Conferences in Tech
But there was this kind of three-legged stool of you had the attendees. It was kind of in their interest to go. You had the speakers. It was kind of in their interest to go because they are able to get their ideas out there. Things are recorded and so on. And then you also have the sponsors who are there.
Oxide and Friends
Conferences in Tech
Okay. All right. Yeah. Red Monk has got an aging complex or something. Are you getting Botox injections for Red Monk? I noticed there's no crow's feet. I mean, Red Monk looks like, as far as I know, it's the Brian Johnson of companies, of consulting organizations, we often say. So you've got the...
Oxide and Friends
Conferences in Tech
And I know that Theo Steve still remembers fondly Surge 2010 when he realized that he is with all of these practitioners. He's like, I am the only sales guy here. He's like, I am fishing in a stocked pond. I'm on a game preserve and I've just got like... no limit, no bag limit. He was like, this is great. I was like, please do not tell anybody else about Surge. He loved that conference.
Oxide and Friends
Conferences in Tech
But of course, that is not sustainable. You can't actually have a game preserve. You actually need to have this kind of other element, this commercial element of people who are there to sell the products.
Oxide and Friends
Conferences in Tech
Oh, it's amazing. It's better than... Okay, easy. Easy, easy, easy. Let's just settle down here.
Oxide and Friends
Conferences in Tech
Ghent is the third largest city in Europe in like 1550, behind Paris and London.
Oxide and Friends
Conferences in Tech
And then it was like... I feel like I gotta come in here and defend Ghent. Come in, you'll appreciate this. This is a medievalist. Ghent was then the changing wool trade, and wool being... It was no longer a port for wool, which was all moved... I feel like this is like...
Oxide and Friends
Conferences in Tech
yeah that's right uh and so it basically is kind of like economically kind of collapses and as a result like it's not economically interesting and so it doesn't get bombed during world war ii so it's like it's totally it's a completely preserved yeah frozen in time amazing city budapest also an amazing city don't make me choose that's what i'm gonna say theo whatever theo says i i don't think they're both great cities but yes craft conf is great
Oxide and Friends
Conferences in Tech
But it's just far away is the problem with GraphConf. And then, I mean, Stephen, you've obviously been, you know, had a front row seat for all of this kind of changing in terms of conferences. And you started, when you guys started, was MonkeyGraw before Monctoberfest or was Monctoberfest before? What's the origin of Monctoberfest versus MonkeyGraw?
Oxide and Friends
Conferences in Tech
Okay, well, so in the unspecified past, Theo, I don't know if you've got the same hangout, but I mean, you and I met, and I think Theo, when do you think we met?
Oxide and Friends
Conferences in Tech
What's the difference? What's the difference? We were on either side of that one. So the... Obviously, Red Monk is your... Have Monctoberfest be your first conference. Monctoberfest, I think, is still something special. Steve, do you want to describe why Monctoberfest and why that's endured?
Oxide and Friends
Conferences in Tech
We understand we are going to be assembling like Red Monk Voltron here. We're going to be putting together various limbs. Steven is here in some diminished capacity. Is he on stage yet? Have we dragged him on stage yet?
Oxide and Friends
Conferences in Tech
Yeah. I speak Gen Z. So it's low-key chill for real, for real. No cap.
Oxide and Friends
Conferences in Tech
No, it's fine. My kids right now, unfortunately, my kids don't listen to this. They would be just absolutely just, oh God, to be real. Mortify the drop dead. Yeah, the hallway track is the, and maybe this is a dated term, but this is like kind of the meeting outside of the talks when you're talking to people in the hallway.
Oxide and Friends
Conferences in Tech
Pour one out for OSCON. Okay, so let's start actually a little bit with OSCON. OSCON was the O'Reilly Open Source Conference. And, Theo, you had been to OSCONs earlier than that, I think. That was my first OSCON in 2005, but I think you had been maybe a couple years prior.
Oxide and Friends
Conferences in Tech
Okay, so let's talk about size a little bit because, you know, you, Stephen, I mean, it'd be interesting to know your perspective on this, but one of the things that is a challenge with Monctoberfest is you want it to be small. So therefore it's like, it's hard to go. It's like hard to score a ticket.
Oxide and Friends
Conferences in Tech
It's not something that like, you are not able to accommodate everyone who wants to attend Monctoberfest.
Oxide and Friends
Conferences in Tech
Totally. Well, and a few, I think in terms of like a venue, cause Steve, did you go to the, any of the search conferences? I'm not sure if you, yeah.
Oxide and Friends
Conferences in Tech
because that venue in Baltimore was definitely amazing and unique. I feel like Theo, I mean, when you scaled up and you changed the venue, something changed in the conference. I mean, it was still great, but it was different.
Oxide and Friends
Conferences in Tech
The, the, where you held it in Baltimore was like, it was like four hotels that were mashed together. And so you would be on and you wouldn't, didn't really know which hotel you were in. And I just remember you would be on floor three in one hotel and then you'd be on the fourth floor in the next hotel without actually going up or down any stairs.
Oxide and Friends
Conferences in Tech
Or it was just like, it was kind of delightfully confusing that they always like, and there were rooms that were like kind of like abandoned ballrooms. I mean,
Oxide and Friends
Conferences in Tech
I think it's, but it's kind of interesting because I think this gets to like what, what conferences need to do to like, if a conference is going to be an in-person conference, there's gotta be something about it. That is that like there's about the venue or about the experience. I mean, obviously at McToberfest, Steven, you've got, you've got the local beers and the local flavor.
Oxide and Friends
Conferences in Tech
I mean, there's, and it's, I mean, it's obviously always going to be, I would assume in Portland, Maine, a very difficult city to get to from the West coast, but always worth a trip.
Oxide and Friends
Conferences in Tech
But that's what we do around here. Look, this is the, you know, this is... We just kind of show up at places. I've done it many times.
Oxide and Friends
Conferences in Tech
Yes. Yeah, totally. Yeah, Foo Camp. That's where actually I met Bob Lee, the late Bob Lee. I met at Foo Camp.
Oxide and Friends
Conferences in Tech
It is invite only. And I knew when I was invited, I'm like, I think I'm going to be invited once. So I should just enjoy it because I don't think I'll be invited back. And that was right. So I was right about that.
Oxide and Friends
Conferences in Tech
Yeah, the food camp I went to was in 2011, where I saw Quinn Norton presenting on Bitcoin. And I often think of how many... If I had only had to see... This is where Adam... We had this meeting with the proto-Twitter, and I'm like, I'm never going to do that. Those guys are jokers. And Adam's like, those guys are jokers, and I should reserve AHL on Twitter. And it's...
Oxide and Friends
Conferences in Tech
Which I noticed, by the way, the American Hockey League, now you and the American Hockey League have this kind of ongoing bit where you've got AHL at Blue Sky and now they've got the AHL at Blue Sky.
Oxide and Friends
Conferences in Tech
It's really heartwarming, honestly. It's like a buddy movie, you two.
Oxide and Friends
Conferences in Tech
Okay. So, so we've got these kinds of conferences that are for, that really are focused on this kind of like social aspect. And I don't think those are going to go away. Um, I think that they've also, I mean, I think one of the unfortunate bits, I mean, I think both the strength and the minus is that you've got this exclusivity with it.
Oxide and Friends
Conferences in Tech
I mean, I do think like these, you kind of think of these food camps that are, you know, these are kind of extraordinary folks, but it's not like the, it's not open. It's invite all things on open all comers. Um, Just to shift gears a little bit, because I do think that the way we present information and giving talks, I do think is important. What does the future of that look like?
Oxide and Friends
Conferences in Tech
And obviously, I've got my thumb on the scale a little bit because of what we're doing with detrace.conf this week. It should be said, Stephen and Theo, both attendees of the 2008 detrace.conf. I'm not sure. That was so much fun.
Oxide and Friends
Conferences in Tech
Oh, that was so great. And that was a very technical, fun conference with a very niche conference. Nobody is asking us to scale up. This is not something that we have. We were like, boy, this conference would be great, but can we do it at the Santa Clara Convention Center? No one's making that request.
Oxide and Friends
Conferences in Tech
And Stephen, as I'm sure you recall, this is a conversation I recall so vividly because I remember exactly where we were at the end of the conference meeting. when you're telling me this has been a great conference, but, uh, next year is going to be like, you just like next year is going to be worse. I was like, like this conference is not even, like today's not even over yet.
Oxide and Friends
Conferences in Tech
Like I can't even have today. And we're already talking about next year. It's like, no, I don't remember that.
Oxide and Friends
Conferences in Tech
Oh, no, no. It was, no, no. It was extremely helpful. You're like, no, like, look, I, you know, this conference, it's like, this was great. I'll hide it to you. Like, this was really great. This is amazing. You brought it. But like next year, like some people can make it, some people can't make it. And then everyone's just going to invite comparison to this year.
Oxide and Friends
Conferences in Tech
And it's just going to be like, people are going to kind of like reminisce about this year. Like next year, just not going to be quite. And then like the next year, like won't be good as getting it. It'll just kind of fizzle out. And I'm like, can we talk about this tomorrow, maybe?
Oxide and Friends
Conferences in Tech
this is 2008, this is before 2011. Yeah, sure, shut up at your first Monctoberfest and be like, hey, Steve, before you go to sleep for 14 hours, I've got some feedback for you.
Oxide and Friends
Conferences in Tech
No, but I actually really appreciate it. Because this is what, like when, this kind of like, you're like an apparition from the future. And as, because we had this idea like, oh, this will be an annual event. And I'm like, no, no, we need to not make it an annual event. And they're like, no, forget it. We're just never going to do it again. That's it. That's the way to play.
Oxide and Friends
Conferences in Tech
We're not going to do it again. And then it was when we hit 2010, 2011, 2012. And then by 2012, we're like, well, maybe we can't do it again because no one can remember 2008. And I'm like, this is why they only do the Olympics every four years. We can do the Olympiad cadence. Um, and this is, this is 2024. It's a, it's a Detroit Olympiad year.
Oxide and Friends
Conferences in Tech
Um, but we did make the decision to go completely online. Um, because, um, for Detroit.com was great for, you know, in kind of that 2008 era, uh,
Oxide and Friends
Conferences in Tech
um but now it's like why would people travel when they can get i mean and this is this is going to be interesting to see like i i'm so this is on wednesday by the way not that this is a not that this entire episode is just like a cheap promo for our our conference but you know it is free to attend but actually because one thing that that's important is that it's uh it is actually freely available for everybody like i'm not going to charge anything and anyone can go and you
Oxide and Friends
Conferences in Tech
Because I like the socialization you get with the exclusivity, but everything else about the exclusivity I don't like. And I want as many people to be able to participate as want to participate. Now, in Detroit.com's case, I'm not exactly worried about being overwhelmed with attendees. But what do you make of the online conferences, Stephen and Thea?
Oxide and Friends
Conferences in Tech
Yeah, Kelly, you're making a very important point. For those of you contemplating grad school, this is Theo's experience is highly unusual and atypical. This is not what grad school normally is.
Oxide and Friends
Conferences in Tech
Much higher budget. And so was this, what conference, this was ApacheCon? What was the conference?
Oxide and Friends
Conferences in Tech
So that works from like the, I definitely agree with you from like the speaker perspective or even the attendee perspective, the economics though become harder at that small scale. Right. I mean, Steven, I mean, I, you guys, I mean, it's, it must be tough to pull off. I would assume.
Oxide and Friends
Conferences in Tech
I mean, it's obviously a huge amount of work, which is another thing actually it's worth talking about in terms of like the actual burnout folks get from organizing conferences.
Oxide and Friends
Conferences in Tech
But the previous year. I think Monctober Fest 2023. I don't know. Did I go to – and this is where I think – no, because I did do Monctober. I may have done it in person last year that I can't remember. Well, increasingly because the online conferences and the in-person conferences are actually blending together.
Oxide and Friends
Conferences in Tech
I do think that folks are getting some tricks about online conferences to give them more of that. I think P99Conf, I think that those folks do a terrific job of getting a great energetic hallway track by mixing up live and pre-recorded events.
Oxide and Friends
Conferences in Tech
Um, so I, I think that the, um, that's part of the reason that they're actually, um, blurring together, uh, in a, in a delightful way, but yeah, I don't go to very many in-person conferences and I only go, I generally only go if I'm a speaker because just, I mean, it's just harder otherwise.
Oxide and Friends
Conferences in Tech
And, you know, I went to a, one of these kind of traditional, I mean, not to call them out because I think if they do a great job, but I went to go to Chicago in 2023, um, And it was, to attend for three days, it was $3,000. It's like, man, that's a lot. I mean, that's not a hotel, and that's not travel. That is just the cost of attending the conference.
Oxide and Friends
Conferences in Tech
It was like $2,800 or something like that for three days. And, like, that does not economically work, really. I don't think. I mean, I'm like, and then I think that they, and I'm sure they're going to change it up because, like, when I was at that conference, I'm like, this conference is many fewer people are attending.
Oxide and Friends
Conferences in Tech
They were not recovering at all from their pre-COVID numbers because the cost is so high. And as a result, like you're losing that critical mass and they're still at that kind of, Stephen, you said that like conferences need to either be online, very intimate, experience oriented, or very large. And it was absolutely falling in that gap.
Oxide and Friends
Conferences in Tech
And it's like that, that's the valley of death for a conference.
Oxide and Friends
Conferences in Tech
And so I, I kind of want to, I mean, so I think the thing that we're experimenting with, I would say with the just.com for 2024 is can we, can you get an online conference that has a bit of an unconference feel to it where it is a little more loosely structured, can we get to some of that kind of community building and get some more people kind of giving talks?
Oxide and Friends
Conferences in Tech
Because I do think it's helpful for people to give talks. But of course, you're giving a talk to... It's like you're giving a presentation at a Google Meet. It's not the same kind of experience. But I think it's actually important as an industry that we are getting younger technologists an opportunity to present their work. And I'm a little worried that we... If conferences...
Oxide and Friends
Conferences in Tech
don't afford that opportunity. Uh, that's a bit of a problem. I think it's really important that we, we afford that to folks. Um, but I don't know, maybe, maybe, maybe I should be, maybe the youths are just like, no, we're all like YouTube influencers. We don't make like, you know, don't know. I, I, I make, I have like 6 million views on this, like unboxing of a toy that I, it's fine. Uh, uh,
Oxide and Friends
Conferences in Tech
But I do think that that's important. So I think this is going to be a different kind of experiment. I think we're also going to try a bit of the unconferency bit. And we're trying to find, I mean, Adam, do you want to describe what we're doing? I think we're kind of trying to use Discord and Google Meet and YouTube streaming and try to hit a sweet spot in here.
Oxide and Friends
Conferences in Tech
It's going to be a stage dive, potentially onto concrete. You never know around here.
Oxide and Friends
Conferences in Tech
And then we are also doing a, an in-person social here in the evening. So we're, we are really trying to have it both ways and knowing that this is like not something that people will travel for. So we'll be, um, I don't know, you know, this could be, maybe it'll be the worst of all worlds, but, um,
Oxide and Friends
Conferences in Tech
I use. Okay. Yes. That I thought you were, you were saying at the very least, it's going to be cheap because it definitely is cheap.
Oxide and Friends
Conferences in Tech
It's economic. Well, no, because I do think this is an issue that like that, that the, this is, it has become so expensive and yeah, and this is also like remote work.
Oxide and Friends
Conferences in Tech
I mean, and, and I mean, Stephen, you must've also had this thought about like, you know, we, if we are transitioning to a more of a, a truly remote world, what does that mean to kind of come into the world when it's all remote and how people, you know, as young technologists, how they actually like connect with people on, on obviously the world is also more connected than ever.
Oxide and Friends
Conferences in Tech
Okay, interesting. Part of my hunch is that just looking at kind of the history, OSCON being one of the early industry conferences, that this kind of came out of one academic computer sciences conferences. Academic computer science has... the very strange tradition of having conferences as their kind of publishing vehicle rather than journals, which is what every other scientific discipline does.
Oxide and Friends
Conferences in Tech
Yeah, I mean, what a great world that you're describing where all of this corporate real estate that is not going to be used properly can now be – we can take all of that money and we can actually invest it properly in having people that are remote but then spending that money on bringing them together.
Oxide and Friends
Conferences in Tech
Well, I think there are going to be some new models. I think Raphael in the chat is saying, sometimes if things are too produced, it feels too intimidating. Raphael, maybe I'm putting words in your mouth. Maybe this is not what you're saying. What I think you're saying is, I love how underproduced DTrace.com is because it tells me that anyone is welcome to present.
Oxide and Friends
Conferences in Tech
That's right. You can't possibly have imposter syndrome around here because we clearly have got no idea what we're doing. But if you can do that, then you can maybe do more of them. I mean, I still think. like this very podcast has been great for us. I don't know why more folks don't do these things or if they do, I can't find them.
Oxide and Friends
Conferences in Tech
So I would like to say that the shirt, and Adam, you were calling out that the shirt is an absolute banger. Kudos to Ben Leonard, our designer. So I don't know if you saw, Theo, our shirt for Teachers.conf 2024 is taking some of this inspiration from 08 and kind of reimagining it with an actual oxide compute sled. Um, people are asking like, wait a minute, can I just get the shirt?
Oxide and Friends
Conferences in Tech
And that combined with these open source and open source projects that needed to have a way or were looking for a way to get people that were otherwise collaborating to physically meet up because they were not co-located. Is that a fair kind of read for the origin of the kind of those early days of conferences?
Oxide and Friends
Conferences in Tech
Do I have to go to the conference? It's like an online conference. We're not taking attendance. So yes, like you get like, I mean, do you need me to spell it out for you?
Oxide and Friends
Conferences in Tech
I mean, you can just, uh, yeah, yes, you can get the shirt and then like, you can also go to the free online conference, but you can, and then you can just like, you know, work on the conference and get, and you can get an amazing shirt. So yes, anyone get the shirt will be, uh, they will be packed by hand by my children. Um,
Oxide and Friends
Conferences in Tech
They are my own children, so this is not – before anyone busts out the labor law on me, which my kids do all the time, it's like my own kids, all legal.
Oxide and Friends
Conferences in Tech
I know which one did it. I know which one did it. If you get a note – okay, yeah, this is a good point, Adam. I'm glad you're bringing this up. No one is being held against their will, despite the note that my 17-year-old may try to slide in there.
Oxide and Friends
Conferences in Tech
As it turns out, like labor law is pretty much all repealed for your own kids. I mean, God bless it. Otherwise, like our kids would all be like little attorneys. They'd have like, what do you know? I don't know. Why am I having to go through your lawyer to ask you to unload the dishwasher? This is not an ocean of, Oh, come on. This dishwasher is all right. Maybe this one is, but.
Oxide and Friends
Conferences in Tech
It's going to be... The conference is free. The t-shirt is not free, but it's pretty cheap, I think.
Oxide and Friends
Conferences in Tech
I don't know that they will be... Adam, I think these things are going to sell out. Not to be too exclusive about it, but I do think... I don't remember paying for anything, but maybe I did.
Oxide and Friends
Conferences in Tech
Oh, there you are. In the audience, if your lights are on and you are about to be towed, Stephen O'Grady, your lights are on.
Oxide and Friends
Conferences in Tech
Well, yeah, we may have. And we have resisted. I have to say, this is like the first time we've made an Oxide shirt publicly available. We're excited. And it's a great shirt. I think it's going to be fun. It'll be an interesting experiment. I hope other things experiment like this. I would like to go to other experiments like this.
Oxide and Friends
Conferences in Tech
So if other folks, in terms of when will the shirts be shipped out, when will the shirts be shipped out?
Oxide and Friends
Conferences in Tech
I know. I think we are still waiting for them to arrive. I mean, they're en route.
Oxide and Friends
Conferences in Tech
Well played. Yes. We're very mindful of this burnout problem. And therefore, first, my children will be doing the work. So just that's presupposed.
Oxide and Friends
Conferences in Tech
Exactly. So, wow. This is the, it's like, wow, this is the most, thank God this unproduced conference. I don't, not only do I not have imposter syndrome, I'm getting these guys won't get burned out because they don't seem to be doing any work. I found my email.
Oxide and Friends
Conferences in Tech
People wanted that yellow shirt. That is a collector's item, Adam, and yours is in good shape. Mine's a little threadbare.
Oxide and Friends
Conferences in Tech
The shirt has gotten much smaller over the years. That's the only thing that makes sense. Um, but it was, uh, it's a lot of fun and this is going to be, so we're, we're trying to like get it all. Um, we'll see, we're obviously going to drop something on the, we're not going to, you can't get it all. You can't, um, you have to give up something.
Oxide and Friends
Conferences in Tech
I think that's the, and you have to kind of know what you're trying to go do. Um, and I think for this, it's like, we want to talk about detrace, which is gonna be fun. Um, so, um, but, um, if you, if folks do get the opportunity, I mean, I assume, uh, Steven, that you've not talked about conference burnout, um,
Oxide and Friends
Conferences in Tech
Um, that's surely you must be, uh, I mean, I know it's a lot of work to put on Oktoberfest every year and I'm sure every year you're like, why am I doing this again?
Oxide and Friends
Conferences in Tech
Well, so all the more reason to be grateful for... And I think you also have to be like, you can never take these conferences for granted because no conference is going to go on forever. And we should all be grateful for them when we have them. And I certainly was extremely grateful for those years of surge, Theo, and
Oxide and Friends
Conferences in Tech
I'm very, very grateful for Monctoberfest, as I've told you repeatedly, Steve, and I think I've given the talks that are most personally important to me at Monctoberfest, in part because they're talks that you explicitly asked me to give, which I would also really encourage conference organizers to do.
Oxide and Friends
Conferences in Tech
If there's a talk you want to see, go find the person you want to give it and ask them to give it, because they may not even... Certainly for the talks you've asked me to give, it has not occurred to me to give a talk on them. And they've ended up being really, really important.
Oxide and Friends
Conferences in Tech
Yeah. And we, we definitely, in terms of like, I mean, this is not a problem, but when you are oversubscribed, I mean, Adam, like we were with systems we love, where I had, I talk about like a one-time bounty of like the old, like we have unspent marketing dollars, just absolute music to my ears. So we're like, can you spend $40,000 on a conference? I'm like a hundred percent. Absolutely.
Oxide and Friends
Conferences in Tech
And there's something actually really special about conferences of that vintage because you kind of felt like you had a super nerdy interest that you were in a Usenet group with people talking about maybe RARC. And then to meet those people in real life felt very kind of validating at a time when to be in software was kind of counterculture. I mean, it feels so...
Oxide and Friends
Conferences in Tech
That was, that was so much fun. That was so much fun, and it was like one and done. I kind of had an idea of maybe we'd do it, but it did end up being one and done. That one, I think, actually, the thing that I am proudest of there was how we selected talks, actually. We were blind. I thought it was really important to be blind and to have people just submit their proposals.
Oxide and Friends
Conferences in Tech
And then we had a huge number of people who wanted to talk, which is great. And then whittled it down to the point where we still had three times the number of applicants or twice maybe the number of applicants as we had spots. And then we unblinded and looked at other things, including we wanted to get some first-time speakers up on stage. And that was a lot of fun. That was a good day.
Oxide and Friends
Conferences in Tech
That was Systems We Love, and that was in 2016, I think. But we kind of had the idea of taking it on the road. We did Systems We Love in Minneapolis, which was fun. But then the unspent marketing dollars problem, that kind of went away, as it turns out. It turns out if you spend the marketing dollars, you no longer have the unspent marketing dollars.
Oxide and Friends
Conferences in Tech
And then, of course, everyone adjusts their budgets next year to make sure that doesn't happen. So that's kind of what happened there. I will say, actually, a problem with... So was briefly a problem with Systems We Love. The talks, we had lost the videos for a bit.
Oxide and Friends
Conferences in Tech
Um, and the, because when you have, they were on Vimeo and they were on kind of a corporate account and then a bunch of us had left and they, you know, changed to a free plan and then Vimeo started deleting stuff. But we, uh, there were some terrific folks inside of joint. Thank you very much. You know, you are who rescued those videos. Um, but then we did not do that for detrace.com for 2016.
Oxide and Friends
Conferences in Tech
YouTube.com 2016 videos are lost, unfortunately. So I would say it's really important when you please upload those videos to YouTube and make sure that they've got... And please, if you haven't, donate to the Internet Archive, which I have more than once. It has saved my bacon. when somebody has deleted... Can I just say, actually, may I make a quick PSA, Adam?
Oxide and Friends
Conferences in Tech
Because this has happened to me a couple of times. If you record a talk at a conference... you are engaged in a social contract with the person who you recorded. And you, this is Mike.
Oxide and Friends
Conferences in Tech
That's my lawyer. I want you to, well, I have had this happen a couple of times where people have deleted the only recordings of talks that I gave. Because I don't give the same talk twice, and it has been the Internet Archive that has absolutely bailed me out and has allowed me to restore those talks. And in some cases, those talks were really personally important.
Oxide and Friends
Conferences in Tech
Um, and I, you, you just, you have a responsibility to a speaker when you, you, you've got like a part of them that they've given you as part of that talk. And, uh, you just, just, you can't destroy someone's art and you can't just, anyway, that's what I have to say about that. Okay. I feel better. I got that out.
Oxide and Friends
Conferences in Tech
Just, Also, Vimeo. If anyone works for Vimeo and can help us restore to the... But I think these things are lost, unfortunately.
Oxide and Friends
Conferences in Tech
I mean, God, it feels just so ancient. Sorry, Stephen, we're going to date Red Monk here as being founded in the 1880s at this rate. I mean, it feels like such a distant time when this is kind of like before software became overrun with tech pros and ate the economy. Before software ate the world, it felt like I don't know. What was your first conference?
Oxide and Friends
Conferences in Tech
Okay. I think that the, in terms of, of extremes in terms of like, so Theo, you gave a talk to, was it 4,000 people, 2,000 people, Apache con big, a lot of people drinking Mai Tai. Um, And I feel I've given talks in rooms of like a thousand plus, but Adam, I think you take the crown, at least among the two of us, certainly. It's not really fair.
Oxide and Friends
Conferences in Tech
It was a very big room. It's the biggest possible room at Moscone, and it was packed to the rafters. I mean, there were 10,000 people in the room. More, I think. And you chose that moment to have... The longest hair that I've... I mean, it was singular. Yeah. It was amazing. Yeah, it was a choice.
Oxide and Friends
Conferences in Tech
And then you gave an amazing demo, an amazing demo showing a code flow from Java into the operating system library and into the operating system kernel in like one unit. It was just amazing. It's great. Underappreciated by those tens of thousands, perhaps, but I appreciated it. Everyone else fixated on this wild man hair that you had. But I mean, you had such mad scientist vibes. It was great.
Oxide and Friends
Conferences in Tech
That's right. So that's, you know, I think that I'm hoping that you're going to have a new hairstyle. You're going to reveal here. I'm looking forward to it.
Oxide and Friends
Conferences in Tech
Well, um, really appreciate it, Theo and, and Stephen Kelly. And thank you so much for, uh, helping us talk about, about the conferences. And I think that, you know, Stephen, I like your way of phrasing it. The conferences either need to kind of be small and tight and intimate or online or huge.
Oxide and Friends
Conferences in Tech
Um, I think that those are the, um, conferences are changing, but, um, the need for them has not gone away. Uh, and they still, um, we're looking forward to our online conference. You can kind of join us, get a, get a banger of a shirt, um, and watch our, watch our trust fall here, stage dive. Um, but, uh, looking, uh, looking forward to it and Adam can't wait to see the new hairstyle.
Oxide and Friends
Conferences in Tech
A conference that I think we've actually already talked about on Oxide, friends, because this is the conference in which we sought to eat horse. I heard that. Yes. This is the California. Were you in Ghent? Yeah. You were in Ghent.
Oxide and Friends
Conferences in Tech
that is exactly what happened. The, we, we called it run, Brian run after run Lola run popular at the time. I, a film, um, the, uh, no, we were Adam at the, uh, California had just made past a, a populist proposition, making it illegal to consume horse meat.
Oxide and Friends
Conferences in Tech
Um, and Adam never wanted to have his diet dictated by the people was, uh, it was enraged and said that while in Europe, we needed to go find horse meat.
Oxide and Friends
Conferences in Tech
Can we, can we get some, some horse meat from you? I mean, you can just maybe a little under the table. We will, we'll make it interesting for you. Okay.
Oxide and Friends
Conferences in Tech
What if Alexander Hamilton wished to order some horse meat? Would that be, would that be possible?
Oxide and Friends
Conferences in Tech
We've been told there was a German tradition that on your birthday, you buy food for everybody else.
Oxide and Friends
Conferences in Tech
I was like, do I actually have to not tell people it's my birthday? I had people who were German who were like, if it's your birthday and you don't want to bring a plate of sausages for your coworkers, you should definitely not tell anybody because it is definitely expected.
Oxide and Friends
Conferences in Tech
Yeah, I was going to say, have you counted how many conferences you've been to in your life? No. No, I don't want to do that.
Oxide and Friends
Conferences in Tech
Oh, there you go. Okay. Wow. Um, I think, I think mine was, I think like yours, Steven was a prior to a debug Adam, which is obviously very memorable for many reasons. Um, the, um, it was an internal, uh, the sun had a sun tech symposium, um, the STS where one of these large, but it was an internal conference, large internal conference.
Oxide and Friends
Conferences in Tech
Steven, I like the fact that your colleague is somewhat surprised at how unwell you sound. It's like, wow, I think he's actually ill. This is the post about the flu test. This all might actually add up. You actually might be sick. Yeah, that's right. I'm here.
Oxide and Friends
Conferences in Tech
This is the conference where they had had apparently a complaint in previous years that there were too many things for people to pick from. And this is before anything was videoed, which I think is kind of important. Remember when we're remembering that super early conference era, like talks are not recorded.
Oxide and Friends
Conferences in Tech
You go to the talk, you watch the talk and like, it's whatever you can remember from the talk basically. Yeah. Which is such a sea change in the way we consume information. But they had gotten the feedback that there was just too many talks to pick from.
Oxide and Friends
Conferences in Tech
So they came up with what they thought was a brilliant solution where every presenter at STS, this is in 2000, every presenter that was selected to give a talk would give their talk three times in a row, back to back to back. And it is the only time in my life I've done this. This was in Denver. I grew up in Denver.
Oxide and Friends
Conferences in Tech
And my mom said, oh, well, if you're coming to Denver, I want to come watch you talk. Very supportive mother. And so I give the first talk and they're like 15 people there. And like, I'm presenting something that's pretty esoteric. And I'm like, I don't think I'm going to be able to fill three rooms on, this is the real-time support in the Solaris operating system. And I'd make 15, 20 people.
Oxide and Friends
Conferences in Tech
And then in that next middle session, one person shows up. And my mom also shows up for that middle session. And I explained to the guy, I'm like, look, this conference has got this very weird idea that everyone presents back to back to back. You know, my mom is here. Why don't you, let's wait to see if someone else shows up.
Oxide and Friends
RFDs: The Backbone of Oxide
Because at some point, and my memory is that Alex Wilson, our colleague at Joyent, was working on a container naming service for what was then Triton. And was really looking for a vessel in which to write it down. And I think that there's been a lot of pressure building in general where we actually need to have a way for people to write things down.
Oxide and Friends
RFDs: The Backbone of Oxide
And it had been enough time since the scarring of Peace Arc. I think actually one of the things I kind of present about Peace Arc is that it took me years to really appreciate the value. And Adam, maybe you're still getting there because you're like, did you just say Peace Arc was valuable? Because that's not my memory at all.
Oxide and Friends
RFDs: The Backbone of Oxide
But it took me years to appreciate the value and be able to separate it out from the things that were really – that could be, frankly, pretty upsetting as an engineer. But once I kind of had not had this for an extended period of time, it's like, yeah, we really need to get – I miss writing. I miss writing ideas down. Robert, is that your – do I have the –
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, and that's what I was trying to remember is like whether – to what degree it was like Alex is like I need a place to write this. But yeah, Alex was definitely – however it was, Alex was the first RFD.
Oxide and Friends
RFDs: The Backbone of Oxide
That's right. And I think that, you know, and this has happened to me a lot in my career where I you know, I'm trying to come up with something that I think is like, wow, this is like totally unique. And like, Oh, you know, this is very Socratic or a kind of classic, what have you. And then, I mean, at some point, as we were thinking about this, Robert, like kind of the light was going on.
Oxide and Friends
RFDs: The Backbone of Oxide
So what you want is something that's like, I want it to be pretty loose. I want to encourage people to write things down more than getting them exactly right. Um, and yeah, as we're kind of thinking about this, we're like, wait a minute, these are RFCs. This is what, this is what the, this is how the internet was built. This is right in front of us.
Oxide and Friends
RFDs: The Backbone of Oxide
I'm loving this. I have managed to offend all of our German listeners. We apologize, Germany. Exactly. A message to our German community from Oxide and Friends. You know, apology videos have kind of gotten... I'm trying to bring apology videos back. I just feel like, guys, I can't believe we have to have this episode. I never thought I'd be saying this, but...
Oxide and Friends
RFDs: The Backbone of Oxide
This is what RFCs and, and, you know, in joints, it joins our fee repo. I, I reference RFC three and the, the IETF has this request for comments and RFC three just absolutely nailed the, what we were after. And it was kind of like, I've managed to reinvent something that's right in front of me for my entire career, more or less. And I felt a little bit silly for not having seen it.
Oxide and Friends
RFDs: The Backbone of Oxide
But then another like, okay, so this is like, we need to take this and extend it. And I just love this note from RFC3 that the content of a note may be any thought, suggestion, et cetera, related to the software or other aspect of the network. Notes are encouraged to be timely rather than polished. I love that line.
Oxide and Friends
RFDs: The Backbone of Oxide
Philosophical positions without examples or other specific suggestions or implementation techniques without introductory or background explication and explicit questions without any attempted answers are all acceptable. The minimum length for a note is one sentence. and these standards or lack of them are stated explicitly for two reasons.
Oxide and Friends
RFDs: The Backbone of Oxide
First, there's a tendency to view a written statement as ipso facto authoritative, and we hope to promote the exchange and discussion of considerably less than authoritative ideas. Second, there is a natural hesitancy to publish something unpolished, and we hope to ease this inhibition. Those paragraphs are from RFC3. That is Steve Crocker, April 1969.
Oxide and Friends
RFDs: The Backbone of Oxide
Adam, doesn't that feel like that is just absolutely true today?
Oxide and Friends
RFDs: The Backbone of Oxide
Totally. And as a result, there is going to be a wide variety of quality here. I mean, there has to be, right? There's going to be, and there are going to be RF and there are RFCs that are extraordinary. And there are many RFCs that are forgettable and that's okay.
Oxide and Friends
RFDs: The Backbone of Oxide
We're not trying to make something that is kind of, and we're in, in particular, what I loved about that is like, this is not something that's kind of up for approval. This is you writing down your own ideas and,
Oxide and Friends
RFDs: The Backbone of Oxide
And then, Robert, as we were kind of thinking about RFCs versus – my recollection is, like, we just wanted to have a different nomenclature because we felt we'd be just confusing ourselves all the time if we were talking about RFCs. And we were kind of looking for a different nomenclature. Is that right?
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, I don't think it was too much beyond that. I think it was just kind of like, you know, I think we would have called it like, yeah.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, which is, we use the term request for discussion in lieu of request for comments to avoid conflation with the IETF construct. and the more formal writing that has come to represent. Because RFCs are now kind of drifted from that RFC3 ethos and zeitgeist and are now quite a bit more formal. So we wanted to draw inspiration from RFCs, but be something slightly different. And RFDs were born.
Oxide and Friends
RFDs: The Backbone of Oxide
And Robert, I feel it was like, I mean, I don't know if you and I spoke about this. I'm sure we did. But I remember thinking very shortly after Alex's RFD won and we start getting a bunch of RFDs, I'm like, oh, man.
Oxide and Friends
RFDs: The Backbone of Oxide
Germany, I'm so sorry for what Brian did. I think I'm making it worse.
Oxide and Friends
RFDs: The Backbone of Oxide
we waited too long to do this i felt like this was so should have done this earlier i um i mean on the one hand it's like it's great like okay this is clearly like i this is a big step in the right direction but it's such a big step in the right direction that uh felt like we should have done this earlier um yeah and i think we had one or two you know like one-off documents
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. And where did those go? I mean, I just don't remember where were those because I mean, they must've existed. I just don't know where they went.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. So this was in, I mean, it's kind of embarrassing that like this is in the first RVs in 2015. And so we did, how did we do engineering for five years? I just don't know.
Oxide and Friends
RFDs: The Backbone of Oxide
fair. So, um, and then, so Robert, what were our experiences with RFDs at joint? Cause I felt like there were some things that worked really well, namely the idea, the, uh, was great and was important. Um, then there were other things that I don't know, what were your, what was your take on things that worked well and things that did not work well in that kind of that first embodiment of us?
Oxide and Friends
RFDs: The Backbone of Oxide
Really long time coming. I feel like this is like the episode we were in all along. I feel that... You know, I was listening, but actually I'm really glad that we are doing it now and not a couple of years ago because we have made so many... very recent changes that are really, really important.
Oxide and Friends
RFDs: The Backbone of Oxide
It is painful, but it is much better than it was at Joyent. At Joyent, we did not. I mean, the grand irony of RFDs at Joyent is the discussion piece was not. It was really hard to discuss them because we would do this with GitHub issues, Adam, and it did not work well. Um, because it was very hard to like comment on particular pieces. You know what I mean?
Oxide and Friends
RFDs: The Backbone of Oxide
Comment on like, Hey, the word choice or whatever. And like, if you walk up to an RFD that is, you know, old, how do you go and be like, you know, this paragraph is no longer correct. Or you just like you filing issues on all these things. And it's like, the discussion was not great. So that I would, I think was a big, uh, needs improvement. Um,
Oxide and Friends
RFDs: The Backbone of Oxide
But I think the idea, like we could see that a bunch of, another thing that was actually really important, Robert, is that it was in a repo. I felt like this is a very important, because folks may look at RFDs and be like, well, wait a minute, why not just like Google Docs or whatever? And I think I'm sure, what were you doing at Delphix and then Transposit, Adam?
Oxide and Friends
RFDs: The Backbone of Oxide
Yes. Yeah, and I think that what the world wants, and what we're kind of leading up to, is you want kind of a Google Doc front end that has an ASCII doc backend going to a canonical repository. That's what you want. That doesn't exactly exist.
Oxide and Friends
RFDs: The Backbone of Oxide
And I also feel that one of the things that we have done, and we'll talk about this a bunch over this episode, but we have open sourced the software that we use to do this. And I think that if anybody's listening to this wondering like, God, I really like a lot of aspects of the system, but I don't know. I don't, I don't want to just like take theirs. It's like, you can definitely take it.
Oxide and Friends
RFDs: The Backbone of Oxide
Well, part of the reason I'm chuckling is because we did Markdown at Oxide, and we supported Markdown and ASCII doc. And Adam, I think it was you who took me aside at some point. It's been like, hey, you say that RFDs are in Markdown or ASCII doc. Yeah, exactly. It's like, you should know that only yours are in Markdown. Everyone else is on ASCII doc.
Oxide and Friends
RFDs: The Backbone of Oxide
So when you say Markdown or ASCII, I'm like, I actually... Adam, I don't think I'm inventing this conversation. I really appreciated it. I had a giant piece of lettuce on my teeth and only you had... All of my oldest and dearest friends were willing to take me aside and tell me what an ass I was making of myself. I'm like, you mean, no. And I kind of didn't believe you.
Oxide and Friends
RFDs: The Backbone of Oxide
I'm like, surely other, no, nope. Nope. It's just me. And that was a quick, I definitely like, I am reworking them right now. They are all going to be at ASCII doc. I'm so sorry. Yes.
Oxide and Friends
RFDs: The Backbone of Oxide
It is my own weakness that I would like to be able to, I don't know, see what this thing looks like before I, I, I just, it feels like this is like how you always want to test your code before you integrate it.
Oxide and Friends
RFDs: The Backbone of Oxide
Um, we are, if they, and certainly if I may at risk of orienting people, what we are talking about. Oh, okay. Oh, now with this again. Okay. You know, it's like, we're only a couple minutes in, I was going to wait for another 10 minutes, but yeah, sure. Go ahead. Fine. Fine. Blow the big reveal. All right. Oh, what are we talking about? Mr. We have to tell people what we're talking about. Go ahead.
Oxide and Friends
RFDs: The Backbone of Oxide
Yes, Markdown is really frustrating in a lot of ways. And especially if you want to do anything sophisticated or complex. you need things like, you know, footnotes and you want to cross reference and all these things. It's really not good.
Oxide and Friends
RFDs: The Backbone of Oxide
I had kind of assumed, I didn't realize that us, that ASCII doc, I thought ASCII doc, well, obviously cause I was the, like the last to get the memo on like, you're the only one doing it in Marktown. So as far as I was concerned, everyone was doing an ASCII doc the world over.
Oxide and Friends
RFDs: The Backbone of Oxide
So I didn't, don't think I realized that ASCII doc and press was saying also that he'd not heard of ASCII doc till coming to oxide. So I guess that's a little more, it's, I mean, the irony of ASCII doc is that the documentation for ASCII doc is not great. Um, I do find that, and Robert, I can't imagine that ChatGPT is very helpful on AsciiDoc. I want to be able to render this.
Oxide and Friends
RFDs: The Backbone of Oxide
I feel like some of the stuff in AsciiDoc is a little... It's been nice to be able to use ChatGPT for AsciiDoc. That's all I'd like to say. No, I'm just being on brand to get LLMs in every episode. You know, there we go. Ring that bell. Exactly. Um, the, but so I didn't realize that Alex had kind of discovered ASCII doc. Um, we, uh, and Robert, do you use ASCII doctor PDF to render locally?
Oxide and Friends
RFDs: The Backbone of Oxide
And so it should be said also, and Robert, now is as good a time as any. So we have, I created RFD 510 yesterday. So we have perhaps more than 510 now. So I might have created one in the meantime. But we've got over 500 RFDs at Oxide. Adam, you had on an earlier episode had guessed that we must have millions of words. Did you do this, by the way? Have you already done this?
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. So we do have, there's 1.4 million words. Um, and I did look at the, I went through everybody who has kind of contributed an RFD and, uh, just looked at the author's field. Um, so didn't, didn't do anything more kind of sophisticated than that. And to see, in other words, like giving everyone full credit for any RFD that they're on the authors for.
Oxide and Friends
RFDs: The Backbone of Oxide
And of the 1.4 million words across all RFDs, uh, Robert, you have contributed, uh, three over 364,000 words. Is that amazing?
Oxide and Friends
RFDs: The Backbone of Oxide
Also terrifying. It is. And, and Robert has been, so part of the reason, Robert, obviously you, you are, you are our most prolific RFD writer.
Oxide and Friends
RFDs: The Backbone of Oxide
Um, and you know, not, not that we want to, you know, just insert our discussion about the parallel of engineering metrics and that it is the cultural idiosyncrasies episode, but the, um, you know, we're not trying to measure RFDs by the pound here, but I did think it was, it was interesting and it was revealing.
Oxide and Friends
RFDs: The Backbone of Oxide
The other thing that Adam was really interesting to me was the, uh, that, that was just eyeopening. It wasn't hugely surprising, I guess. Um, The person at Oxide who has contributed with the most number of collaborators on different RFDs, who's collaborated with the most number of people on RFDs. Do you know what this might be? I'm going to guess myself. You are absolutely right.
Oxide and Friends
RFDs: The Backbone of Oxide
It is like you by far. And this is one of these things like I didn't, you know, I'm not surprised, but I'm also like, I was actually surprised about how much it stood out. And it makes, and actually, it's because you have also like really encouraged a bunch of people to write RFDs or started RFDs that have, I mean, it's just like, I thought it was interesting.
Oxide and Friends
RFDs: The Backbone of Oxide
I mean, it showed one of your kind of great strengths in terms of cross pollinating in the organization.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, exactly. But I thought that was ranging. So, I mean, both of you have, in different aspects, have interacted with the system a lot. And so, you know, Robert, you have written more ASCII doc than any of us. So we've written a lot of ASCII doc. I would say that the entire collection of Shakespeare is about 800,000 words. So, Robert, you're getting there. What...
Oxide and Friends
RFDs: The Backbone of Oxide
Will Shakespeare, keep an eye on your rearview mirror. Robert's coming for you.
Oxide and Friends
RFDs: The Backbone of Oxide
like all my time all my work time to just reading the new contents of rfds absolutely and because we are you know very writing intensive culture we are adding people people are we've added some very prolific writers and yes it is it is absolutely um and in the ends like that shouldn't be the goal right you don't want to actually like have to read every line of every rfd
Oxide and Friends
RFDs: The Backbone of Oxide
Okay, so this gets us to, because I want to get to the very important bits here that we have extended, because this is where we get to Ben's work and Augusta's work, Crespo's work. So this was, you know, we start Oxide. We know, most of us know that ASCII doc is the right answer. One of us feels that they are still doing it in Markdown. So we solved that problem.
Oxide and Friends
RFDs: The Backbone of Oxide
I think it was in the first year, Adam, when we had the intervention, the Markdown intervention.
Oxide and Friends
RFDs: The Backbone of Oxide
Right. Okay. And I think ASCII doc has been good for us, I would say, broadly speaking. Totally. Other than being, I guess, more unique to us than I realized. So one of the things that we started doing early on, I mean, very early on, actually, the earliest days, because we have, it's just like we're on, the name of the podcast now, Oxide Friends, comes from actually our initial friends of Oxide.
Oxide and Friends
RFDs: The Backbone of Oxide
So when we started the company, we had a bunch of folks who had, and Adam, you were one of the initial friends of Oxide, right? Where before you worked here, you were a friend of Oxide. We had just people that we'd worked with, that we'd known, that we wanted to, be transparent about what we're building to get their perspective on.
Oxide and Friends
RFDs: The Backbone of Oxide
So we added them to that GitHub repo so they could see actually all of the RFDs. So the GitHub repo is a private repo that has all of these ASCII docs, all of these documents in it. And
Oxide and Friends
RFDs: The Backbone of Oxide
The important change that we made from Joyent is the way we did discussions, which is it's a little wonky, but it's better than it was at Joyent by a mile, which is we use the fact that GitHub, that PRs are actually... Better than issues for discussions. I mean, the GitHub PRs have got issues with them for sure, or they've got challenges using it.
Oxide and Friends
RFDs: The Backbone of Oxide
But we use PRs to discuss an RFD, which at least allows people to give kind of line-by-line feedback. It's nowhere near as good as Google Doc comments, for example, but it could be worse. Yeah, it doesn't get in the way, I don't think. I don't think it gets in the way.
Oxide and Friends
RFDs: The Backbone of Oxide
I think it is, you know, I think that there's some challenges, but I think it's broadly, again, it was like way better than certainly at joint. We didn't have discussion that we should have had because it was mechanically so difficult. So kind of solve that problem.
Oxide and Friends
RFDs: The Backbone of Oxide
How does it feel for us to be doing this at a time that includes you? I'm so sorry. This is good. Can you hear me? Adam from Italy. Yes. Yes. I mean, don't take this the wrong way, Adam, but I think you, I think you sound better than you do in Alameda. I mean, I think so too. I have a upgraded microphone for, for this.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, I kind of, I do like it. I wish, obviously, I wish those comments were themselves under kind of Git management, but which would be kind of, that'd be asking GitHub to go to be one step too useful. But the, I do like the fact that they're there and they also don't get in the way of reading the doc, which I think is also important because...
Oxide and Friends
RFDs: The Backbone of Oxide
you know, somebody like, you know, not all those, you know, there are comments like, okay, yeah, I got considered this or whatever, and I'm going to close it. I'm resolving it, but I want you to be able to still see that there's a comment here. Maybe someone else can come along later and have the same comment or what have you, but yeah, great.
Oxide and Friends
RFDs: The Backbone of Oxide
Very early on. I mean, it predates Oxide, really. And we'll kind of get into the history there. But it really is the backbone, as you and I were kind of iterating on titles. I think you were kind of wanting me to go bigger on titles. And you're right. I mean, it's like, this is really important. And you said this in our episode on cultural idiosyncrasies as well, that this is idiosyncratic. And
Oxide and Friends
RFDs: The Backbone of Oxide
Um, so we, and we do that with branches and then we, we put some automation around that where, you know, you could, the RFDs have got different states and they can be in kind of, um, ideation was an early, that was a state that we added at Empire Call. We wanted something that was kind of, uh, earlier than kind of discussion state, which is like, I truly do.
Oxide and Friends
RFDs: The Backbone of Oxide
I've got just like no idea what I, this is a dream I had.
Oxide and Friends
RFDs: The Backbone of Oxide
Right, and that's right. I mean, RFDs are the way we kind of are. Maybe you can argue beginning to abuse it a little bit, but the system was flexible enough that it could be extended that way, and adding ideation was pretty easy. So that worked out pretty well. So then it was great to be able to take the repository and add their GitHub ID as a collaborator on the repository.
Oxide and Friends
RFDs: The Backbone of Oxide
And then that became kind of part of what one though, like it was just great to be able to do that, to get people's perspectives on things. But then secondly, we would use that as part of the hiring process. So part of the hiring process was after we did someone's materials and like, okay, we're interested in this person. We would add them to the RFDs so they could see all of the RFDs.
Oxide and Friends
RFDs: The Backbone of Oxide
And this was a really important step. It was really important for us to be able to show people all of our documentation about how we were building this thing to give them a much more concrete idea of what we were doing technically. What I would tell people is when you get into these RFDs and you're going to, you know, some of them are extraordinary. Some of them are not even correct any longer.
Oxide and Friends
RFDs: The Backbone of Oxide
Some of them are, you know, there's this huge, you know, huge spectrum of quality and so on. But as you read these RFDs, you will begin to feel that you want to work for Oxide more or less, but not the same. That's what I would tell people. And I think it was really, really important.
Oxide and Friends
RFDs: The Backbone of Oxide
And we had, obviously, because we're, by the time we're kind of advancing people's after the materials, we talked about this with our, on our episode on hiring with Gary, but this is all happening after we have decided like to, to kind of pull someone into conversations and we want them to kind of understand what we're building at Oxide.
Oxide and Friends
RFDs: The Backbone of Oxide
We had, we did have people in there and I had one person in particular was like, I thought I really wanted to work for Oxide. But, and I had pointed them to, you know, some pretty gnarly RFDs and it's like, I'd read some of these RFDs and like, I am, I'm kind of more interested in the idea of working for Oxide than I am working for Oxide, which I thought was very helpful. Right.
Oxide and Friends
RFDs: The Backbone of Oxide
Absolutely. Absolutely. And Robert, you should take this as praise, but it was absolutely one of your RFDs that actually was the clarifying RFD. Because I think people don't realize the amount of... You're like, oh, my God. There is so much detail here. It's like, yes. Yes, there's a lot of detail. It's very, very detailed to actually go on. So that was great.
Oxide and Friends
RFDs: The Backbone of Oxide
It doesn't feel like it should be idiosyncratic because we did not invent literacy and we did not invent the importance of writing down one's ideas or reading the ideas of others. But I think a bunch of the things that we've done borrowing from ideas elsewhere, have added up to something that is idiosyncratic, but I think it shouldn't be, which is the reason that I kind of bridle with that.
Oxide and Friends
RFDs: The Backbone of Oxide
The problem that we had, or that I had, was there wasn't a good way to... Well, we had a couple of big problems. One, there was no way to search RFDs, which started off not great and was becoming really, really, really problematic. And then there was no way to share individual RFDs. And other than like rendering them as PDFs and sending them, which I did plenty, but it's like, that's not great.
Oxide and Friends
RFDs: The Backbone of Oxide
And this may be a good time to get you and kind of David in here and Augustus in terms of like, this is kind of the RFD system that you kind of come upon is us kind of dealing with sticks and stones and GitHub. And what is the origin of the RFD site? Because I'm trying to remember, how far did that date back, first of all? That dates back, I think, a while.
Oxide and Friends
RFDs: The Backbone of Oxide
Ben, great question. I got to say, the sun is pouring into this room at Oxide. I don't think I quite realized how much absolutely nuclear bomb direct sun exposure this thing gets in the morning. So I'm kind of like hiding a little bit in the shade, but it is glorious. I got to tell you, I got my morning cup of coffee. I'm really excited for this.
Oxide and Friends
RFDs: The Backbone of Oxide
Totally. When I think, so a couple of things like the, so we added, I mean, the inability to search RFTs was extremely painful. It was also, it's made even more painful because the, the great thing about using PRs for discussion is that you get GitHub's features and
Oxide and Friends
RFDs: The Backbone of Oxide
The downside of that is it means that like the actual repo itself, that is to say like the main branch, only contains RFDs that have been published. They don't have the ones that are in discussion. So you have to like search every branch. And so I had this thing, RFDGREP, that you were referring to, Ben, that was taking minutes.
Oxide and Friends
RFDs: The Backbone of Oxide
But it was like, it was minutes, but it was actually like delivering penicillin. So it was really, you know, it was like, oh my God, thank God we got some way. We were using LuceneGREP.
Oxide and Friends
RFDs: The Backbone of Oxide
um and lm graph is what we were using using what we've seen and it was very helpful but it was also like its performance would vary a lot it would take minutes it was really really painful and in particular like it was so needed and because i remember like getting steve tuck set up with rfd grab because he needed it desperately like gotta have a better way to do this so
Oxide and Friends
RFDs: The Backbone of Oxide
But it is idiosyncratic. But regardless of whether it's idiosyncratic or not, it is essential to oxide. And it has been It is really, really, really important. It is really hard to overstate the importance of RFDs, and they're becoming way more important, actually.
Oxide and Friends
RFDs: The Backbone of Oxide
I can't remember when you did the full text search on the RFD site, but that demo Friday was like a religious experience where, I mean, it was like that affected everyone at Oxide to be able to like, oh my God, I can just go there and search all the RFDs, like clean water, thank God.
Oxide and Friends
RFDs: The Backbone of Oxide
So describe that because that was, I totally agree. So now we've got the RFD site and we've got the RFD site, which also like looks great, which is very helpful. You are doing, so tell me how you're rendering the ASCII doc, because that was a big piece of this. You got to go render the ASCII doc.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, sorry. I think I was definitely a recent version of that where I was definitely using block quotes and attributed quotes. Brian, that's basic.
Oxide and Friends
RFDs: The Backbone of Oxide
I feel like the emperor's new clothes, though, because we do use ASCII doc everywhere, I just assumed ASCII doc is ubiquitous everywhere. And it's like, I've never heard of it. Exactly.
Oxide and Friends
RFDs: The Backbone of Oxide
Look, I... Okay, I just assumed. Because, Ben, as you point out, I'm realizing that part of the reason I thought ASCII doc... was ubiquitous is because it is ubiquitous and oxide, but you're, it is ubiquitous and oxide because you made it ubiquitous.
Oxide and Friends
RFDs: The Backbone of Oxide
I think it's great. I think it's great. I think like it's, it's actually, I think it's terrific. I think it is so much better than Markdown. Um, and, um, so I, yeah, I think it's great. And like blogs and so on are all an ASCII doc.
Oxide and Friends
RFDs: The Backbone of Oxide
Yes. And the other thing that has been just extraordinarily important is the ability to make single RFDs public. And that has been... life-changing for us because it now allows us to begin to... And we've done this over time, but we are getting more and more RFDs out there and allows it for us to be that kind of public vector as well.
Oxide and Friends
RFDs: The Backbone of Oxide
Well, and just the fact that we also use this for the documentation is so great. And then we've been able to make the documentation site is anyone that's available to the public. If you want to, I mean, we're, can see all of the Oxide documentation, which is great. And then we're able to kind of cross-link that with the console. There's a bunch of things we can do there.
Oxide and Friends
RFDs: The Backbone of Oxide
So the other, I mean, and then also it looks gorgeous. Can you talk about the GitHub integration a little bit? Because that's been a huge, that's been hugely valuable to get those discussions. That was another demo Friday that I just think just like left people, you know, weeping with joy. The C comments and the RP site.
Oxide and Friends
RFDs: The Backbone of Oxide
A huge improvement. And this is a good segue to the kind of the RFD database that kind of fronts this thing, I think, because I assume that's hitting the database that is fronting it and not actually GitHub directly, but I actually don't know the answer to that. Augustus, it would be a good opportunity to get your work in here.
Oxide and Friends
RFDs: The Backbone of Oxide
And so that has been absolutely essential and that's all built on the terrific work of Ben and Augustus and Dave Crespo who are all with us today. So their work has been... Like, it kind of started off as like, oh, like, this is nice. Like, oh, this is great. Like, this is great.
Oxide and Friends
RFDs: The Backbone of Oxide
The reason I had assumed that those were coming out of a fronted database is because it was so snappy. So that's why it's so snappy. Because you're exactly... Yeah, yeah, yeah.
Oxide and Friends
RFDs: The Backbone of Oxide
It is fake. I get totally faked out by it, though. I'm not looking at the spinner. I'm generally looking at the rendered content. So that's definitely worked. It also should be said that it just. The rendered content looks gorgeous. And obviously, Ben, you've made sure that everything at Oxide looks gorgeous or faces your ire. And it just looks great.
Oxide and Friends
RFDs: The Backbone of Oxide
It also looks great on the phone, which I think is really important. because I think folks are reading more and more RFDs on a mobile device, right? Augustus, the other reason that having that front-end database has been really, really important is for the visibility for RFDs. Can you describe maybe some of the problems we had with adding increasing number of folks to this Friends of Oxide group?
Oxide and Friends
RFDs: The Backbone of Oxide
And then, you know, they, again, we'll get into the whole history of this, but that added things were just like, oh, wow, okay, this is really great. Like, this is profoundly great. And now we're just like, okay, this is... this is now like low bearing. This is, this is, uh, we now cannot envision the system without this. This is extremely important.
Oxide and Friends
RFDs: The Backbone of Oxide
Because this GitHub group was getting larger and larger and larger. And I think it all had to be like outside collaborators with our organization, I think, if I recall correctly.
Oxide and Friends
RFDs: The Backbone of Oxide
And this is extraordinarily important. I think the full text search was game changing. The rendering was game changing. And I feel that the fine-grained permissions are totally game-changing for RFDs.
Oxide and Friends
RFDs: The Backbone of Oxide
Because we really needed to be... The reason I think it's so game-changing is because it allowed us to much more easily... Well, one, we could make our individual RFDs public, which was a real problem. When did we first have the ability to do that?
Oxide and Friends
RFDs: The Backbone of Oxide
And this has been, I think, huge. Because I feel, especially where we are in the past year, having brought this product to market, and now there are more and more aspects of this that we want to share with everybody. We want to share with all of our customers and we want to share with everyone to see.
Oxide and Friends
RFDs: The Backbone of Oxide
And I feel like there are, you know, it's just kind of amazing that we've, that, you know, we've only been able to do this in the last year. And I, cause I feel like I do this all the time. We did this, obviously, I mean, the last,
Oxide and Friends
RFDs: The Backbone of Oxide
episode as was talking about an rfd that was made public right so the in terms of wither cockroach db and 508 but i feel like it was uh really really important to have that fine grain control and to be able to have like groups as well so you can say like okay this because i think we've got a group for our contract manufacturer for example where they want we want them to be able to see everything about a certain number of rfds that are not public um it's extremely helpful
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, that is great. So that was before we met up in October. And then I know that I was out in New York in November talking to someone who had seen the RFD site. And they're like, how can I get a hold of that?
Oxide and Friends
RFDs: The Backbone of Oxide
Because I think a lot of, I mean, I feel like every time we have an RFD, a public RFD that is on Hacker News or is getting discussed, someone is always like, hey, can someone explain to me how this system works and how can I use it? Because there's a lot of interest in it. Do you want to talk about a little bit of the open sourcing of the RFD site and of the RFD API? Sure.
Oxide and Friends
RFDs: The Backbone of Oxide
You don't have any RFDE that implies that you're building a computer. Other than that, you can use it as much as you want. You just can't, you know?
Oxide and Friends
RFDs: The Backbone of Oxide
I think eventually it might be nice to have something a little bit more... Yeah, I mean... Dave, you... Okay, the CSS is specific to us, but someone could take this and get it working for themselves, correct? I mean, it feels like they could. Oh, for sure.
Oxide and Friends
RFDs: The Backbone of Oxide
I will tell you that there's a reason that it's written. I mean, so Jess and I are listed as the co-authors because Steve was not yet working in the garage. So this is going to be in August of, I think, of 2019. Maybe we retroactively made it after September 9th.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, exactly. That's Oxide TM to you. Java technology, please, TM. So in terms of being able to take it, we've got all the elements that someone could do to go. And what would be some of the ways in which someone could use this stuff? I mean, how would one get started if you wanted to kind of see what it would be like to use this?
Oxide and Friends
RFDs: The Backbone of Oxide
I mean, Adam, you're a proud papa. You and Dave on Progenitor and Trap. You just love it.
Oxide and Friends
RFDs: The Backbone of Oxide
Okay, so Augustus, when I'm using the RFD COI, that is a progenitor-generated COI. Absolutely.
Oxide and Friends
RFDs: The Backbone of Oxide
Okay, so feel shame on that. But I use that CLI a lot. I mean, I make great use of the RPCLI. So that's auto-generated. That's amazing.
Oxide and Friends
RFDs: The Backbone of Oxide
I just want to make clear that I'm not accusing you of using an AI-generated photo of Italy. I'm just saying the photo that you posted is a little on the nose for what a generative AI might imagine that Italy would look like. It's just a little too on brand.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, and I feel like that automation has made it, because the branching bit has been great for discussion, but I think the point you're going to get to is it can feel a bit clumsy at times for the authors of RFDs. And I feel like the automation there has really helped a bunch.
Oxide and Friends
RFDs: The Backbone of Oxide
And I think that given all that, I mean, I could see why someone might want to do it a different way, but there's a lot of strength to the model that we've got. We're just kind of getting GitHub. It's using GitHub for its strength without being, while still having a canonical repo that has everything in it.
Oxide and Friends
RFDs: The Backbone of Oxide
First things we did. Absolutely. The first thing we, the first thing we did and it actually felt like, because it felt like something we could do as well. So do you know what I mean? Where, because we didn't have money in the bank, we were totally raising around and there's like, you're kind of limited about how much of this company you can build with zero additional people and no money. And yeah,
Oxide and Friends
RFDs: The Backbone of Oxide
Yes, and I think it gets to... So folks in the chat are asking about Notion. I've not actually used Notion. Preston, it sounds like you have. I would say that my... Concern about Notion. I would say I've got an Evernote concern with Notion or any other.
Oxide and Friends
RFDs: The Backbone of Oxide
And this is part of the challenge here is that using any kind of paid service here is really problematic because this is so important to us that it's really hard for us to accept restrictions on how things are used. And And in particular, a hard constraint on the problem is we have to have a Git repository that has the RFDs in it. That has to be where they live canonically.
Oxide and Friends
RFDs: The Backbone of Oxide
They cannot live canonically anywhere else. And so that's extremely important. I think it's really important, the ASCII doc element of it, now that I understand that only oxide is used in the ASCII doc, that becomes really important. I bet you're mentioning the chat, the versioning, that is extremely important.
Oxide and Friends
RFDs: The Backbone of Oxide
And I just think that this is one of these things where we really need to control every aspect of our fate. We would love to, I think, integrate with other, you know, we're not, I think, you know, if we could find kind of better ways to do pieces of this, but we really need to control our own fate. And I think that like, that is, that is not unusual for us.
Oxide and Friends
RFDs: The Backbone of Oxide
I think engineering organizations need to control their own fate in terms of the actual artifacts they're creating. And this is a really important artifact that we're creating. This is, and as usual,
Oxide and Friends
RFDs: The Backbone of Oxide
saying adam it's like one of our most enduring artifacts are rfd so i mean it's i i mean it's right up there with the source code honestly i mean it's don't make me pick those are they're both very very important a while ago i stumbled onto a phrase of uh owning your strategic weirdness and i i would have said like rfds are a thing that we we could have outsourced
Oxide and Friends
RFDs: The Backbone of Oxide
Well, so I knew that was going to be a problem. It was a problem in joint too. I knew we were going to want a way of communicating with the public. I just had much worse ideas of how it would be done.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, I don't think so either. In fact, quite the contrary. I think with all these things, I now cannot imagine doing... It's been a lot of work. I mean, not to discount that, but it's also like we haven't... We're not licensing any proprietary software here. We're not paying for service. In fact, quite the contrary.
Oxide and Friends
RFDs: The Backbone of Oxide
The work that Augustus was doing was necessary for us to prune our GitHub collaborators because that was costing us money. This actually saved us a bunch of money. by, so the degree we were using a service, we were using too much of it.
Oxide and Friends
RFDs: The Backbone of Oxide
And we were, I mean, in fact, actually, Adam, I think you kind of need to look no further than GitHub where we were, I mean, just for all the things that Augustus was mentioning about the groups and having to add collaborators, but then the collaborate, just like it was really, We were already kind of breaking GitHub. And we've left GitHub as kind of the canonical repository for comments.
Oxide and Friends
RFDs: The Backbone of Oxide
you know, what are the kinds of things you can go do? And like, like you can do a podcast. We did, we were recording on the metal during that time. Right. And, um, But we were established. You look at like RFD2, I think, is principles and values, right? So that we were, RFD3 is our hiring process.
Oxide and Friends
RFDs: The Backbone of Oxide
But I think the direction we'd want is to use less of that. I think you could easily envision... I mean, God, if... I know that Prespo was defending Ben's time by saying, under no condition will we allow you to add comments on the RFD site. But if you were to add that, you could easily... you could see comments going into a Git repo and then us not using GitHub for comments on that. Totally.
Oxide and Friends
RFDs: The Backbone of Oxide
It'd be amazing. And then I think that, like, I also think that the strength of us doing this, by the way, is that we've been able to open source it. So, this is now, like, we have generated something that, and I feel that, like, you know, as we've kind of gone through our careers, like, we've added more and more wisdom into the system. And,
Oxide and Friends
RFDs: The Backbone of Oxide
it now reflects a bunch of wisdom that we can all carry forward. Even folks that are outside of Oxide can actually leverage those. One thing I did want to talk about because it did come up in discussion online where people were curious about how we use RFDs to resolve conflict. And that's not exactly their question. I'm rephrasing it a little bit. But how do you deal with conflict in an RFD?
Oxide and Friends
RFDs: The Backbone of Oxide
How do you deal with where... And the comment was kind of like, hey, we experimented with it, but it just became too much of a lightning rod. And I would say that RFDs do not solve your organizational problems. They don't necessarily solve... They don't resolve those things. And, you know, we have had RFDs that become lightning rods. And it's not good.
Oxide and Friends
RFDs: The Backbone of Oxide
I mean, it's not, I mean, I think it's better than, I would say these are lightning rods that are lightning rods anyway. I think as we alluded to on the cultural idiosyncrasies episode, of course, it's like the chat system, which we've already like run under the bus several times here. It's like chat is what will rip oxide apart. Yeah. Absolutely.
Oxide and Friends
RFDs: The Backbone of Oxide
And this is a good example where, and I think, Adam, you mentioned this at the time we were talking about it, where all of these systems have got drawbacks. It's easy for everyone to have an opinion on it. The opinions get kind of tribal, and it's not great. And in terms of like, it's not great in that there's not an unequivocal answer. And, you know, the RFD on chat got a little hot. Yeah.
Oxide and Friends
RFDs: The Backbone of Oxide
It is not worth checking out the hot comments on chat. And I think that on the one hand, you're not going to resolve that. On the other hand, it is actually helpful to have all the – I think it's like if you're going to have organizational conflict, Let it be in writing. You know, it is very helpful to have everything because you can actually like it.
Oxide and Friends
RFDs: The Backbone of Oxide
I think it forces people to be complete about their arguments. I mean, it's obviously like think of the lawyers.
Oxide and Friends
RFDs: The Backbone of Oxide
So we were getting, like, it's not an accident that these are kind of the first things we're laying down. And RFD1 is the RFD process itself. So, yeah, I mean, it was extremely early. And that's interesting. The first commit message isn't until October, right?
Oxide and Friends
RFDs: The Backbone of Oxide
Think of the lawyers. Will somebody please think of the lawyers?
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. And so I do have, and I actually made it public this morning or yesterday on RFD 113. which I did write, which is kind of how we actually make determinations. And it's giving this kind of loose guidance that you're describing. You can't be overly prescriptive about how decisions are made, kind of obviously.
Oxide and Friends
RFDs: The Backbone of Oxide
But, you know, really talking about, and kind of differentiating decisions from determinations, talking about some specific examples where, you know, how we made determinations in the past. Because it's, on the one hand, there are, I would say there are plenty of RFDs where it's like, no, no, I want to throw this out here. I really want to get someone's discussion.
Oxide and Friends
RFDs: The Backbone of Oxide
In other cases, like, we've got, and especially if you've got an issue where we have to move quickly often like, okay, there's been a lot of process that's outside of RFDs, and now actually the RFD is going to pull all this together and kind of describe what we've already done a little bit.
Oxide and Friends
RFDs: The Backbone of Oxide
But it's actually extremely helpful to go do that, where we can actually go through the different things that we've kind of considered, which is very important. So I think that sometimes the RFD process ends up being a little bit behind the process, but I think oftentimes it is. It's an earnest request for discussion. There's a question about whether an RFD becomes immutable.
Oxide and Friends
RFDs: The Backbone of Oxide
maybe I had this kind of image of, but you know what, that as you say that I am feeling cold in that garage, that unheated garage, that thing got like frosty in October, November, December. Like we really did need to move into a place with heating.
Oxide and Friends
RFDs: The Backbone of Oxide
We do have the published state, which is very valuable. I need to get more of my own things into publish. That RFD 113 is not the published state. Actually, it's funny. So just to say it's an interesting like object lesson. RFD 113 is not in the published state. It's in the discussion state, even though it's obviously 113 and we are now at last updated in December of 2020.
Oxide and Friends
RFDs: The Backbone of Oxide
It's like, okay, why is it still in the discussion state? This should clearly be like published. And so I'm like, I'm going to go mark this as the published state. But there's a comment in there, actually, our colleague, Keith Wazowski, left a very thoughtful comment on there that I had read, you know, in 2020.
Oxide and Friends
RFDs: The Backbone of Oxide
But I actually want to go, there's some things in there, I actually want to modify this RFD a little bit before I publish it based on that single comment. And it's not contentious. He doesn't disagree with anything here, but he's got a perspective that I want to capture in here a little bit before I move it into the published state. So that's why that one is still in the discussion.
Oxide and Friends
RFDs: The Backbone of Oxide
But yeah, I mean, in terms of when, and I think Ilyan is also saying that RFDs can languish in discussion, which I think is okay. I think it's like, I think it's, certainly I'm guilty of this too, but I think when we're moving into the published state, we're saying like, okay, this is basically like, We've kind of accepted this. It doesn't mean that it's immutable.
Oxide and Friends
RFDs: The Backbone of Oxide
And I think importantly, we can go back and change RFDs that have been published. We don't want to become captive to our own process in this regard. And I think by and large, we haven't been captive to the RFD. I don't think it's been incarcerating at them.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, and then I think it's also helped them have the abandoned state. Abandoned, it is. And chat RFD may be in the abandoned state. Just to indicate to someone, especially new to the company, like, oh, okay, I don't need to spend a lot of time looking at this one.
Oxide and Friends
RFDs: The Backbone of Oxide
This was an idea that, and I feel that was something, I mean, Robert, I feel that was something that was kind of a needs improvement from the Joyent era as well, where we ended up with some RFDs that were not what we did And then we need to kind of make clear, no, no, this is not, we didn't go this way.
Oxide and Friends
RFDs: The Backbone of Oxide
We had a thoughtful discussion on it or what have you, but you shouldn't view this as kind of canonical because we've abandoned it. I think the other thing I would say is that when we are getting better at is linking RFDs to one another. So you've already seen this, and especially like when we were referring to Dave's RFDs last week and 110 and 53 and kind of the RFDs that they refer to.
Oxide and Friends
RFDs: The Backbone of Oxide
I think that ASCII doc makes it pretty easy to link to those things and kind of footnotes and so on. But I think that being able to navigate that web becomes really important.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah. And it was kind of like, you are here in this RFD, and this is how it relates to the other RFDs. And that was really, really important. And then, I mean, in our next-gen sled, Robert, you want to talk about how you, because the RFD there, we've got an RFD that really is the entire architecture of that kind of next-gen sled. Do you want to talk about how we're using RFDs for Cosmo?
Oxide and Friends
RFDs: The Backbone of Oxide
Okay, you are apparently emphasizing barely in that. I am emphasizing heated. I think it is heated, if barely. No, it's fine. It's actually going to be warm, so it's going to be a roaster in the office today.
Oxide and Friends
RFDs: The Backbone of Oxide
No, I just, you're reminding me of, I think, is it 101 much new about optics?
Oxide and Friends
RFDs: The Backbone of Oxide
So in terms of the history here, and Robert, I'm hoping you can help me out because I'm realizing that my recollection of this, I'm going to be interested to compare notes on the recollection of this. But for the history of this, you actually have to go back before Oxide, certainly, before Joyent, actually, and have to go back to Our Days at Sun.
Oxide and Friends
RFDs: The Backbone of Oxide
we do something different at least you have some of the context of what were the different things that went into it oh it is extraordinarily helpful and it was it was uh 112 as much do about optics robert um i would we'll need to look to see if there's any and if there's no nda material in there i want to make maybe you want to make that one public just because that have you read 112 adam
Oxide and Friends
RFDs: The Backbone of Oxide
Because you're like, I don't know, optics, how complicated can it be? It's like, oh, okay. No, I haven't. Oh my God. I mean, this is one of these and I feel like there are many RFDs that are in this category where this thing will take you to absolute crush depth on optics and how complicated it is. And
Oxide and Friends
RFDs: The Backbone of Oxide
I, I mean, Robert, you've done this many, many, many times where it is so helpful to have the entirety of the depth of our thinking in an RFD. Because as you say, it's like, you know, you may look at that and God, there've been so many RFDs that I read once. I'm like, okay, yeah, that makes sense. I don't know. Makes sense.
Oxide and Friends
RFDs: The Backbone of Oxide
And then later, due to change in circumstances, I'm now reading this very closely. Maybe it's because I'm implementing it. Or maybe it's because we're about to have a call with a partner where this partner is misbehaving. Or maybe we're having a call with an investor or having a call with a user. We're having a conversation where we're like, I need to ramp up on everything we know about optics.
Oxide and Friends
RFDs: The Backbone of Oxide
I need to know in the next 45 minutes. It's like, good news. Um, and you can go actually get to the entire depth of our thinking.
Oxide and Friends
RFDs: The Backbone of Oxide
And I think that this is where the, the written artifacts are so important for scaling an organization because it allows, I mean, Robert, with so much of your thinking written down, people can go read your thinking before they, they need to like ask you a followup question. Right. Um, and that is just extraordinarily important. Um,
Oxide and Friends
RFDs: The Backbone of Oxide
So it's been, I mean, you're talking about, Adam, how this has become more valuable over time. It has become more valuable because we are able to use them more. We have more people using them, to Ben's point about making this accessible. We've got more people creating RFDs. We've got more people reading RFDs in terms of the public and customers and so on.
Oxide and Friends
RFDs: The Backbone of Oxide
And then we've got these kind of fine-grained permissions so we can get the right RFD to the right person, which has been... incredibly important for us. It is the backbone of Oxide. Absolutely. This is, it's been great. Not just the discussion, but boy, this work, all of you have done such terrific work on this stuff. And it's been really, really essential.
Oxide and Friends
RFDs: The Backbone of Oxide
Wild. I'd expect an out loud rating. Adam, thank you for all the collaboration you've done with all of your colleagues on RFTs. Much less work than Robert's words. really great stuff. And again, if you're looking at this thinking like, boy, I would love to adopt that system, please adopt it. We really try to make it so it is adoptable.
Oxide and Friends
RFDs: The Backbone of Oxide
So, you know, take it lock, stock and barrel, take the parts that work and report back. We'd love to hear about it because I think this is something that a lot of engineering organizations really, really need. On that, thanks, everybody. Thanks, Europe.
Oxide and Friends
RFDs: The Backbone of Oxide
Adam, this is kind of where the earliest stuff of this gets laid down. When Sun, I think not unusually, had this architectural review process. And in particular, there was a platform software architectural review committee, PSARC. And PSARC would... would approve architectural changes to the operating system.
Oxide and Friends
RFDs: The Backbone of Oxide
I feel like we'll need to do a European-timed episode at a time when you're not in Europe, just so we can prove that we're like, oh, yeah, they're remote-friendly because the CEO no longer wants to work. He doesn't want to move his family. That's why they're remote-friendly. We'll need to do some European-timed episodes. We're going to do this once a quarter or something like that.
Oxide and Friends
RFDs: The Backbone of Oxide
And it was a, and I'm sure we've mentioned, we probably talked about PSARC in the D-Trace episode, but PSARC was this body that fancied itself a pretty august body, but you would develop materials for a PSARC case.
Oxide and Friends
RFDs: The Backbone of Oxide
And I mean, Adam, I'm sure you had the same experience where the development of materials for your PSAR case, I thought was very valuable where you'd be writing your own materials or, you know, you would have a case and would iterate on it with like Mike and me before you submit it or whomever, right? Roger, whatever. That process, I felt to be very valuable.
Oxide and Friends
RFDs: The Backbone of Oxide
Yes. That's what I'm saying. Okay. Okay. So what I'm saying is it was like the, the, the part of the process that was valuable was the part of the process where you wrote everything down. That was valuable. Right. Right. And I found many issues in my own thing, in my own stuff by, I mean, small issues, but like, okay, oh, right.
Oxide and Friends
RFDs: The Backbone of Oxide
As I'm writing this down, I'm like, I realized I hadn't quite thought about this and I'm going to get ahead of some kind of just ill-conceived discussion about this. So I'm going to clarify this. Right. And that was all valuable. And then everything from that was not valuable. And in fact, was awful. Yeah.
Oxide and Friends
RFDs: The Backbone of Oxide
Because you had spent – if you work on this earnestly, you're like, I've spent a lot of time thinking about this. And other than like – It's not impossible that someone would have a suggestion that would be valuable, but that's not the way the PSAR generally phrased it. PSAR was not phrased as suggestions. It was phrased as approval or rejection.
Oxide and Friends
RFDs: The Backbone of Oxide
You know, parole board is a really good way of phrasing it. Like, I don't know, like, are you a recidivist? Is this going to, I just feels like, I know you've had a couple of character witnesses in here, but I don't know.
Oxide and Friends
RFDs: The Backbone of Oxide
The politics of it was onerous. And I mean, I know we turned the cockroach episode into a Postgres episode, so I don't want to turn the RFD episode into a dark episode. But that said, you're like, why would you be saying this unless that is exactly your intent? But I do think it's worth mentioning that. And I would say that if you are in...
Oxide and Friends
RFDs: The Backbone of Oxide
an engineering cultural milieu that has this kind of August review boards, a trick that worked exceedingly well was to deliberately have an issue. And this, by the way, has a deep history in software engineering. This is the queen's duck. But deliberately have an issue that is a small issue that everyone can wrap their brains around that you yourself are equivocating on.
Oxide and Friends
RFDs: The Backbone of Oxide
You just got some things that are not adding up. I do, I gotta say, I love the European window dimensions. I love the tall, narrow windows. That's nice. It's a very European thing. You know, I think it's like, that's how the AI knew to generate in Europe.
Oxide and Friends
RFDs: The Backbone of Oxide
Like, God, I just don't know what to call this thing. What should be the name for this?
Oxide and Friends
RFDs: The Backbone of Oxide
That's right. And I mentioned this is the Queen's Duck, which goes back to a game called Battle Chess. I don't know if you ever played this game, Adam. This is in like 80s, early 90s. And the VP at the company was making Battle Chess was a famous kind of micromanager. So they would deliberately introduce things that they know that he would be, they would basically occupy him.
Oxide and Friends
RFDs: The Backbone of Oxide
And so he would all, and one of the things is they had the queen in Battletrust holding a duck. And he'd be like, you know, he's like kind of looking over it very thoughtfully. And then like, you know, the queen, queen should not be holding a duck. Like, oh God, yes, thank you. Yes, thank you. So they, this is a trick that works extremely well. We did this and it worked all too well.
Oxide and Friends
RFDs: The Backbone of Oxide
But all that is wasted energy. Like you should not need to, like all of that is, is the organizational politicking and it sucks and it's not uplifting even when it works extremely well. Okay, when it works extremely well, it's a little uplifting, but not very uplifting, and it's a waste of energy. But the thing that was valuable was writing all that stuff down.
Oxide and Friends
RFDs: The Backbone of Oxide
So, okay, we went to Fishworks, where we wrote nothing down, I think. Do you remember our first bug database?
Oxide and Friends
RFDs: The Backbone of Oxide
Oh, that's right. Excuse me. I will thank you not to refer to SCCS as RCS. I will thank you to not refer to our defunct source code management system as a different defunct source code management system.
Oxide and Friends
RFDs: The Backbone of Oxide
Do you know where that bug database is from? That's the Detroit bug database. That's right. Don't you remember that? Yeah, yeah, yeah. I wasn't going to air that dirty laundry. That text file has a deep pedigree, friend. But we, other than writing one line of text for every bug that we fixed, there was basically nothing.
Oxide and Friends
RFDs: The Backbone of Oxide
And we certainly weren't dealing with piece art, and we were kind of a renegade outfit to begin with, and talk about an outfit that should not have been purled. And then I went to Joann, and we kind of fast-forward a couple of years, and And I was really missing writing things down. And it was happening kind of sporadically. And Robert, obviously, this is where I would love to call on your memory.
Oxide and Friends
Reflecting on Founder Mode
Okay. So this is not a hot take. This is a little bit of a take in that we are talking about something that was very hot two weeks ago. That's right. We, we've been on a two week hiatus and you are our last episode was the RFD episode coming in from Italy, which is great. Or so the image that you generated said. I got confirmation on that one.
Oxide and Friends
Reflecting on Founder Mode
It's like, this is great. So we don't need to stop. We're not stopping for gas at all. No, no, no, no, no.
Oxide and Friends
Reflecting on Founder Mode
You're doing that thing. You do again. You're only going to say the half the story. Yeah. So I thought that was, uh, was really interesting. I, the one I, but I got to say, and this is where, uh,
Oxide and Friends
Reflecting on Founder Mode
the thing that was very influential for me, and this is like, I do think the positive of the founder mode piece is his, the second failure of that's not how it's, that's how it's done, where he brings in a bunch of HR professionals that, and the thing that was interesting, and you might find this actually really hard to believe given the kind of the way the piece is written.
Oxide and Friends
Reflecting on Founder Mode
So I have to write a hacker news comment, write a hacker news comment. Like now I'm done. Don't think about it. Still thinking about it. Still thinking about it. Got to write a blog entry. And I've learned that it's just like, I can't fight that at that point. Like I got to go. It's, it's just gonna be faster for me to write the goddamn thing and be done with it. And, um, I, I,
Oxide and Friends
Reflecting on Founder Mode
When I read that, I, I didn't think that was the failure. And I was kind of at a moment where I was beginning to question some of my more idiosyncratic methods, which you might already be like, first of all, just don't believe that. That doesn't sound credible at all. But I really was. I was just like, God, maybe... Because I'm doing things that are really differently than other people.
Oxide and Friends
Reflecting on Founder Mode
You're like, maybe we should be bringing in HR professionals like Tim says? I definitely was wondering on things like performance review, where it's like, I don't know, maybe. And... the, and then he kind of like hit. And so he says, I, you know, I complained, but I eventually gave in. Yeah. I literally, what you, I mean, hand on heart, I am speaking my own truth, whether you believe it or not.
Oxide and Friends
Reflecting on Founder Mode
Maybe I don't even believe it. I'm saying, but when I read that in 2013 or whatever, I'm like, maybe I should give in. And then the next sentence is like, no, no, you idiot. It was a disaster. And I thought it was like, and I think the danger, I think I liked about this is it is very specific. Yeah.
Oxide and Friends
Reflecting on Founder Mode
with respect to some specific policies that were on regular reviews and so on that were very, very specific, and any kind of regrets not having preserved the culture longer, that does not mean you discard all expertise, I think. And I think that's where you get into the false dichotomy that gets dangerous. Yeah, totally.
Oxide and Friends
Reflecting on Founder Mode
Totally. Where you have something that worked well and now is not working at all. And not understanding why this thing worked so well. And it's like, well, maybe that was incidental success. Maybe that thing wasn't working. Or, I think probably more likely, it was implicitly dependent on a bunch of factors that you didn't necessarily see. Totally. That you didn't, yeah. I think it's...
Oxide and Friends
Reflecting on Founder Mode
It's tough and it's complicated. But so that was, that was the bit in that piece that was very influential for me and gave me some of the guts actually to be like, okay, I'm actually going to, it's okay to actually go my own way on this stuff. Yeah.
Oxide and Friends
Reflecting on Founder Mode
But I think you need to be really, I do think you need to be deliberate because I think that the, again, the danger of Graham's piece is that like, oh no, you just go founder mode, which to me, doesn't that, what do you think of when you think, oh, here founder mode?
Oxide and Friends
Reflecting on Founder Mode
It does sound like an episode of Silicon Valley is out of HBO Silicon Valley. I don't think you like Beast Mode and Marshawn Lynch. Am I the only one that has a strong Beast Mode? And I think that Paul Graham would dismiss that as a sports ball, so I don't think he would know who Marshawn Lynch is or what Beast Mode is. But that...
Oxide and Friends
Reflecting on Founder Mode
you kind of have this idea of like founder mode is just not going to accept any sort of negotiation or that's right. Absolutist. Absolutist. Founder mode demands results and demands them now. And I mean, that is, So that's kind of what prompted me to be like, all right, can we undo some of the false dichotomy here? Yeah.
Oxide and Friends
Reflecting on Founder Mode
And which I feel Tim's piece does, but I wanted to, I mean, part of what motivated me to write that, write the oxide piece is,
Oxide and Friends
Reflecting on Founder Mode
Wrote the blog entry. Was kind of debating, is this on a personal blog or did I put this on the Oxide blog? But I'm like, you know, I need to get Steve's fingerprints on this and get it on the Oxide. Because obviously I want to. It actually was helpful getting the Oxide blog. It actually helped me. There's some things I probably. Dialed down a little bit. Dialed down a little bit.
Oxide and Friends
Reflecting on Founder Mode
is I wanted to also I wanted did you click on my link for dramaturgical dyad of course thank you I want to see which specific thank you Simpsons reference like what what what frame of thank you thank you I felt underappreciated on dramaturgical dyad where did you expect to get that appreciation I think right here right now so I think I got it I think I got it actually yeah because you were traveling you were out so it's like why yeah no I just got it actually I don't think anyone um
Oxide and Friends
Reflecting on Founder Mode
No, I appreciate that. I felt like it was an audience of like five. And I don't know who the other four people are. Actually, no, it's you and Dave. That's right. Dave. I mean, if I can think of who would get it, I'm not sure they're going to read the article. You know, they're not going to read the article. Do you know who would get that? It's actually my kids.
Oxide and Friends
Reflecting on Founder Mode
Not Tobin, but my 17-year-old and my 12-year-old are scholars of the early. Yeah. And you're like, hey, kids, new blog post from dad. I knew blog posts from dad. I, you know, I didn't do that. I, but you know, I haven't done everything wrong. This is one of those moments when they, uh, when I did read it, they would get it. They did read it. Yeah. Well, this is where I've done other things wrong.
Oxide and Friends
Reflecting on Founder Mode
They're not going to read the blog entry, but I do think that the, that Steve jobs and like Steve jobs and John Scully, uh, is the dramaturgical diet of Silicon Valley.
Oxide and Friends
Reflecting on Founder Mode
Yeah. Absolutely. I mean, this is like the Tom and Jerry. This is, and I feel like that is really reductive. Totally.
Oxide and Friends
Reflecting on Founder Mode
Absolutely. And I mean, again, the question that I am dying to ask the dead Steve jobs is what was the influence of next? Yeah. on your return to Apple. Absolutely. Because I feel like Steve Jobs, the next big thing was such an, I mean, such an outstanding spotlight on this long period of history, 11 years. And, you know, I know we've talked about it before. I think it's an amazing book.
Oxide and Friends
Reflecting on Founder Mode
I love the fact that that book is written when he's left for dead.
Oxide and Friends
Reflecting on Founder Mode
because it just like, there's no reason to like praise this guy. Right. And I don't, I don't think it's like overly condemning, but it's just like, yeah, I'm definitely not like letting him review it or what. I don't, I don't care what he thinks about it. Like part of like, this guy is down and out and I'm like, I'm, I'm here to, to tell the, the story of,
Oxide and Friends
Reflecting on Founder Mode
a fall right it is not a fallen resurrection story it is a false story and and i think that like the the problem with i mean i was shocked that the isaacson because i you know we talked about this on our yeah i think our first podcast right or one of the early ones went back to the isaacson's biography and it's like next is like six pages really and like man i think it's so much more important than that yeah because this guy did not succeed at next
Oxide and Friends
Reflecting on Founder Mode
And how much did that, and like Scully wasn't totally wrong on a lot of things. Yeah. And Jobs definitely wasn't right in that, in that kind of, that 1985 dramaturgical dyad. Just wedging it in there one more time. Good. For all it's worth. So I felt that we kind of need to break that one apart. And I do feel that, I mean, again, I've said it before, but boy, mandatory reading on jobs.
Oxide and Friends
Reflecting on Founder Mode
Yeah, dialed down a little bit. Not, you know, not hugely, but I think dialed down. I mean, so you're abroad when you read the piece. Yes. And what is your take on, on the piece other than like, thank God we're not recording tomorrow.
Oxide and Friends
Reflecting on Founder Mode
You just can't stop with the Isaacson bio. You really have to dig deeper on that guy.
Oxide and Friends
Reflecting on Founder Mode
Totally. And actually, this is where I got to say, Paul Graham and Steve Jobs, I mean, I'm glad we're talking about them in kind of a single episode because they are, to me, they are both, they're complicated for me. Cause Paul Graham is actually complicated for me. Paul Graham is not, I mean, Elon Musk is not complicated. Sam Altman is not complicated for me.
Oxide and Friends
Reflecting on Founder Mode
Paul Graham is complicated for me because he's not always, he's definitely not always wrong. He's got some, he's got some really important insights. And actually I like one of his, the important insights that again was very personally influential for me and for us at Oxide is, uh, do things that don't scale.
Oxide and Friends
Reflecting on Founder Mode
do things that don't scale is a very good way to say like, this is working for us now. You and I are having a disagreement about whether this can work for us in the indefinite future. We don't actually have to have that argument right now.
Oxide and Friends
Reflecting on Founder Mode
What I think, and this is where it is really important to figure out, like, what are... what are the kind of the hard rules of the road? Yeah.
Oxide and Friends
Reflecting on Founder Mode
And, and I feel like, you know, we have spelled that out very explicitly at Oxide in part because of our experience having not where that's not clearly spelled out and where, where founder mode moves into things that are like pretty shady, you know, where, cause you get things like, look, I understand what you're saying about complying with the law, but we'll, and we'll do that.
Oxide and Friends
Reflecting on Founder Mode
Like when we, it's like, I don't think Paul Graham said do things that are sometimes illegal. I think there was something that's legal.
Oxide and Friends
Reflecting on Founder Mode
um but i have seen that you know i've you get some really kind of gross stuff so like where is that line too right you want to have like queer lines there but still allow for a company that is young to do something that it wouldn't do in the indefinite future yeah absolutely and to be clear like when i was talking about recognizing revenue i'm not talking about like cooking the books but there's a difference between like cooking the books and uh like how you're backdating revenue in a
Oxide and Friends
Reflecting on Founder Mode
People were just concerned about what the future would look like. What the future would look like, right. And also what kind of precedent or setting and so on. It doesn't actually matter. Totally. Let's make them happy. And how can we make these customers happy?
Oxide and Friends
Reflecting on Founder Mode
So another thing you had mentioned is kind of this funky dichotomy is that you, as a startup, you're being encouraged to do things that are iconoclastic.
Oxide and Friends
Reflecting on Founder Mode
And then immediately defer to... LinkedIn wisdom. Wisdom from the resume.
Oxide and Friends
Reflecting on Founder Mode
Right. Forgotten the drive. They don't want to do it again. Kind of bored by it. Also, I didn't do that. I was at that company when it was done. And yes, I was the EVP of sales. But actually, I didn't actually grow revenue myself. I was merely in the building when that happened.
Oxide and Friends
Reflecting on Founder Mode
And by the way, we've talked about this in the past, but definitely one of the things that we lived, and I think a lot of startups live, is... where you take on this expertise from your go-to-market function in particular, and you take on an EVP of sales coming from an established company, and then they need their whole organization to come with them in order to be successful.
Oxide and Friends
Reflecting on Founder Mode
And all of that just adds to the burn. And then you, as a founder, realize too late, like, wait a minute, I've made a huge, huge, huge mistake here.
Oxide and Friends
Reflecting on Founder Mode
Yeah. And I got to say, when people talk about in particular, the things that they've done so that, you know, I, it's on my resume. I was at this company for those years and, And I can kind of make arbitrary claims that aren't really fact-checked. Yeah. And there is, I think, a gross inclination.
Oxide and Friends
Reflecting on Founder Mode
And I think this is where – the reason this is a problem is because this leads to the mistakes that Graham is identifying as people that are – not flim-flam men. What does he call them? Those that are misleading liars, effectively. He has a name for it somewhere. Yeah.
Oxide and Friends
Reflecting on Founder Mode
um they're founders being gaslit but he has there's a specific return of phrase he has about people that are are misleading um and the and the problem is that people exaggerate what they've done um and they you don't in fact we had someone i mean recently who is like, Oh, like, let me advise you because I built this company.
Oxide and Friends
Reflecting on Founder Mode
I, I built this company to from a hundred million dollars in revenue to a billion dollars in revenue, making up numbers. And it's like, Oh, wow, that's wild. And then you look at like when they were there and it's like, and actually notice that they, they, they had claimed that they had built it from earlier than that. So kind of its first kind of $250 million revenue.
Oxide and Friends
Reflecting on Founder Mode
It's like, by the time you joined the company, it was a public company.
Oxide and Friends
Reflecting on Founder Mode
And the revenues were much higher than – are you lying to yourself right now? And I think this happens a lot where people have kind of told themselves their own history, and it's kind of impossible to fact check. And so you take on this expertise as a founder, and then you discover that it's not happening for you, and you wonder –
Oxide and Friends
Reflecting on Founder Mode
And this is where, you know, when Graham talks to talking to founders who had felt gaslit, I think this is where that's coming from, where they brought in that expertise. They're not seeing the results. And then they're being told, no, no, you're not seeing the results because you don't understand the problem.
Oxide and Friends
Reflecting on Founder Mode
That's right. And this is where, okay, I found the turn of phrase in Graham's piece. In practice, judging from the report of founder after founder, what this often turns out to mean is hire professional fakers and let them drive the company into the ground. So that's the professional fakers. This is turn of phrase there. And again,
Oxide and Friends
Reflecting on Founder Mode
The thing that I think Graham does a disservice to is he blames the fakers. Yeah. Fine. He does not blame the founders. He does not blame the investors that force that upon their founders. Totally.
Oxide and Friends
Reflecting on Founder Mode
Right. Now, I'm trying to get all of the many, many examples I'm thinking of. Yeah, fine. Let's go with what you're saying. Absolutely. Absolutely.
Oxide and Friends
Reflecting on Founder Mode
Because the success taught them nothing. Yeah, I think that's right. Because they were there. They were there when the success happened. Why would it not be due to their hard work? It's like, well, it's more complicated than that. Yeah. And then you end up with, I mean, that's what Graham calls the most skillful liars in the world. And again, I think you're right.
Oxide and Friends
Reflecting on Founder Mode
That is like a little bit, on the one hand, that is too condemning of them because it's not necessarily dishonesty. On the other hand, it's maybe a little bit too generous because a lot of these people are lying to themselves. They just don't realize it.
Oxide and Friends
Reflecting on Founder Mode
Just faith and experience is not enough. Faith and experience is not enough. And I do think... I mean, I've said this many times before. I'll say it many times again. Team formation is the most expensive, most important thing you, one does at a company and you need to spend a lot of time on it. I think too many people are like, well, the recruiter handed me these four names.
Oxide and Friends
Reflecting on Founder Mode
So that's, those are the names. Yeah. It's like, what's your rubric for, you know, you are hiring someone in a domain in which you currently have no one. What's your rubric? And one of the things that I've always found, and again, this is the specificity that Graham's piece is totally missing, of how do you add? There are people who are superlative people who can help you out as a founder.
Oxide and Friends
Reflecting on Founder Mode
Yeah, and I think the one thing I will say in Graham's defense is he anticipates this a little bit. Okay. He indicates that, he's like, in one of the footnotes in particular, I have another less optimistic prediction.
Oxide and Friends
Reflecting on Founder Mode
You don't need to go to founder mode for everything. You actually do need people who can add a lot to you. How do you find those people? What chances do you take? Because I think you're right that it's surprising how conservative startups get. Yeah. Because they don't know the role. Actually, I don't know what go-to-market is. I don't know what marketing is. I've never worked in support.
Oxide and Friends
Reflecting on Founder Mode
I don't know these things. So I actually don't know how to evaluate someone. And it's like, that's where you've got to... I think it's incumbent on you as a founder. Yeah, you've got to ramp up. You've got to figure out a way to go... solve that problem.
Oxide and Friends
Reflecting on Founder Mode
And then the internet kind of melted down that next weekend over this Paul Graham piece. Yeah. Founder mode. Yeah.
Oxide and Friends
Reflecting on Founder Mode
And this is something, again, I'm not sure people see it, and it sounds like it's true of the Plaid founders as well, but this is what the Carlson brothers did well. I want to actually, I'm curious to understand, and at one point Patrick reached out to me, he was like, what should I be looking for in a VP of engineering? I'm not trying to hire you.
Oxide and Friends
Reflecting on Founder Mode
I'm trying to come over and have free food, which of course, as you know, pretty much. It's okay. It's time for you to leave, actually. I think that people don't do this nearly often enough is to ask for perspective from other people. Who should I talk to? Who's the best person in this role that you know? I'm not trying to hire them. I just want to know what I should be looking for.
Oxide and Friends
Reflecting on Founder Mode
And I think, and we still do that. I mean, I think it's, it is, you know, we've always done that and still do as we extend into kind of new roles. And it is really, really, really important. I'm not sure we, we, I don't think we've talked about it. I don't think we talk about it broadly as an industry and we should like you like look to other people, senior people to get their perspective.
Oxide and Friends
Reflecting on Founder Mode
This again, the bit that Graham is just like totally missing.
Oxide and Friends
Reflecting on Founder Mode
That's a very good point. Because I do think one of the challenges is, of the idiosyncratic, iconoclastic things you're doing, how much of that is really important versus not? I'll give you a very concrete example. We had an early investor in the company who in an early conversation, an early board meeting, I might've brought up the podcast. Mentioned the podcast. Okay. And this is on the metal.
Oxide and Friends
Reflecting on Founder Mode
Okay. The board member made clear in no uncertain terms that I was never to speak of the podcast again. That is, and I was just like embarrassed for bringing it up. Like, okay, I'm not going to bring up the podcast. And I'm like, what do we do? Like, do we, should we not do the podcast?
Oxide and Friends
Reflecting on Founder Mode
Like, I'm kind of like, it feels like the podcast feels like it's pretty cheap and it feels like it's pretty valuable. Yeah. Do I, and I was, I didn't really know it. And then another board member came up to me, took me aside and be like, Hey, I love the podcast podcast. Podcast is great.
Oxide and Friends
Reflecting on Founder Mode
As soon as the concept of founder mode becomes established, little did you know that this would happen within literal minutes of this thing getting out there, people will start misusing it. Founders who are unable to delegate even things that they should will use founder mode as an excuse. I thought that was a good moment of reflection for Paul Graham.
Oxide and Friends
Reflecting on Founder Mode
I like, I don't bring it, don't bring it up around him, but just, and the, like, I think that like you will get conflicting point is like, you will get conflicting advice. I don't think he was necessarily wrong. Like I understood the perspective that he had and like, we do not talk about the podcast at a, at a board level, but, But the podcast is the right thing for us to go do.
Oxide and Friends
Reflecting on Founder Mode
And I think there's a lot of things like that where, in other words, like that in terms of like, I'm getting conflicting advice. Someone's telling me like, absolutely, do not do it. You're out of your mind. And someone's like, that's a great idea. And you have to pick and choose. And you have to know, you have to kind of... know your own self and have your own sense of judgment.
Oxide and Friends
Reflecting on Founder Mode
But you need to seek out that wisdom, though. Ask the question. I'd rather know the perspective. For sure. And I think that too often we just kind of ignore the perspectives.
Oxide and Friends
Reflecting on Founder Mode
So I think we talk, I can't remember if we talked about this on the hiring podcast or not, but you know, one thing that we do is when we're hiring a new role, we go to the person that we know just in our kind of extended world is like, this is the person who I can't hire.
Oxide and Friends
Reflecting on Founder Mode
But I really look up to in this role and I want them to look at the materials that we're going to, that I want them to look at the portfolio. And we, a bunch of times we've gotten, I mean, just extremely good suggestions about things to add to the portfolio of work. It's been really, really valuable. Yeah, totally. Because that's informed the rubric. Yeah.
Oxide and Friends
Reflecting on Founder Mode
Um, also more than once, one of those people have been like, actually, I might be working for you jerks. I don't know you turkeys. Right. Um, so that, that can happen too. Um, but I think that that is definitely important to seek that out. And again, this is what kind of Graham doesn't get to the other thing that, okay.
Oxide and Friends
Reflecting on Founder Mode
Like, as long as we're, can I talk about some other things that just drove me nuts? I mean, or do you want to know the idea of like the idea that he has created some new element in the lab, um, The idea that he has discovered some heretofore unknown super alloy that it will be incumbent upon future generations to study in detail. Books on founder mode, we don't know what it is.
Oxide and Friends
Reflecting on Founder Mode
Books have never been written on founder mode. It's like, what are you talking about? No one's written history. No one knows what leadership is. I mean, Paul Graham, surely you read more than this, right?
Oxide and Friends
Reflecting on Founder Mode
Exactly. In the future, we will understand founder mode. But now it is a complete mystery. It's a complete mystery to you, pal. The rest of us are out here running companies. I've got some thoughts on this one. It's not dark energy, right? It's really not. It is really not. So I think that was also a little bit ridiculous. But I actually also feel that there wasn't... So I felt that was annoying.
Oxide and Friends
Reflecting on Founder Mode
But I felt there was little in this that was completely wrong. Yeah. Other than the false dichotomy that it leaves you with. I don't know. I mean, I feel like the false dichotomy is fairly central. Yeah, I'm already realizing I am over my skis and I'm defending this piece way too much.
Oxide and Friends
Reflecting on Founder Mode
We're in the mirror neuron empathy preschool over here with Silicon Valley. You got to be like, good job, Paul Graham. Good job. Can everyone give Paul a round of applause for his act of empathy? Okay, no. Sam, stop it. No. Elon. Elon. Get back here. I do think it's interesting that Elon... Did you see the list of people that reviewed this, please? Including Elon Musk.
Oxide and Friends
Reflecting on Founder Mode
and I think actually for that, I do think that, so he references Brian Chesky's talk. Yeah. That was given at Y Combinator, so that one we didn't see. But Chesky interviewed a couple of other places talking about his experience at Airbnb, and I felt it was much more interesting. Did you listen to that?
Oxide and Friends
Reflecting on Founder Mode
Oh, he said he wasn't a micromanager. No, he was just in the details.
Oxide and Friends
Reflecting on Founder Mode
That's right. It's not micromanaging you. You can move that finger, I think. Okay, not a bad example. I can't move that finger. I mean, as long as you don't go the wrong way.
Oxide and Friends
Reflecting on Founder Mode
When I think one of the things he says in that, that I actually like, cause I'm like, what would I do to Paul Graham's piece to make it less divisive, less of a false dichotomy? One of the things that he says, there's like people want clarity. Yeah. And often that is what they, I mean, that to me is what splits the difference is like, you've got terrific people that you've hired.
Oxide and Friends
Reflecting on Founder Mode
You want to empower. They do need clarity. Yeah. And I think that part of what he describes Airbnb operating at cross purposes, because he had stepped back from offering that clarity. Yeah. Yeah, that's a great point. And I think it is really important. Because I think that also if you've got an organization in which you foster trust, that clarity is super helpful, I think, to people. Yeah.
Oxide and Friends
Reflecting on Founder Mode
Where it's like, oh, okay, this sale to this customer is really important for our quarter. Or this feature is being weighted. Oh, I get it. Okay, that's why this is important. And that clarity can allow everybody to go to their own autonomy, find the right way to solve the problem.
Oxide and Friends
Reflecting on Founder Mode
I'm not telling people how to solve a problem necessarily, but giving them the clarity about what's important, I think is... Because if you call this piece clarity mode, it would just feel like, that feels good. I don't know. Clarity mode sounds good.
Oxide and Friends
Reflecting on Founder Mode
reviewed the article before he, before he posted it. Yes. No, I did not see that. Yes. Thanks to Brian Chesky. Fine. Patrick Collison. Fine. Ron Conway. Fine. Jessica Livingston. Makes sense. Elon Musk.
Oxide and Friends
Reflecting on Founder Mode
That's right. And you kind of have to like show them what, that's exactly right. And I do think that like autonomy is, I mean, maybe this is where you get to like my own individual style or our style at Oxide. Autonomy is great. It's just in terms of like if people have that clarity, because I think like how many times
Oxide and Friends
Reflecting on Founder Mode
At Oxide, have we seen something unanticipated that is delightful that happens, that you absolutely want to happen? It's like, I don't think anyone would have known to ask for this. But because... I mean, what I love... Maybe I'm just... easily entertained. But when Augustus did this integration between Salesforce and the ability to provision a silo on the colo rack for a prospective customer,
Oxide and Friends
Reflecting on Founder Mode
So you made it really easy for Travis to go from a prospective customer to like spinning up their own silo. So our sales folks can like get off the call with the customer and create their own demo environment. Right. And then using Dropshot and using all the things that we had built to go do this. I just like, that was great. Totally.
Oxide and Friends
Reflecting on Founder Mode
And I just felt like that is where people were like, no, no, I can see that this needs to be done. And I'm just like, I'm empowered to go do it, so I'm just going to go do it.
Oxide and Friends
Reflecting on Founder Mode
Right, and I do think it's really important that leadership founders offer that clarity on priorities and then are going to have to deal with some tough questions. I'm like, okay, these two can't both be the top priority because, I'm sorry, I'm asking, we actually are going to have to... and find a way to stack these. One of these has to be more important than the other.
Oxide and Friends
Reflecting on Founder Mode
All right. Good. And I like, what are you on Musk's comments? Yeah. I mean, probably just copy editing. Is it just, it's a kind of another type of like, is there any, is there anything substantive that you have any substantive complaints? It's just like, that's a very close read. Um, I think we're thinking of our comments for one another. On any document. Both guilty of it. That's right.
Oxide and Friends
Reflecting on Founder Mode
Sorry, which I think is, but if you don't, and this is, I think where, when you have these startups, like I need to go into founder mode because I feel like the right thing isn't happening. it's just easy to turn into a wrecking ball. And you turn into, actually, we have some colleagues from a company that were describing the management style.
Oxide and Friends
Reflecting on Founder Mode
And I'm like, wow, that is one of the worst cases of seagull management I've ever heard. And she was like, seagull management? I'm like, I don't think I made that up. Have you heard the term seagull management? I think I have, but I'm not sure that I'd be able to define it. This is like a seagull, like a seagull. It swoops in, shits all over everything, and takes off. It steals your french fry.
Oxide and Friends
Reflecting on Founder Mode
It steals your french fry. You've met my sister. Her hair looks like nesting material to birds. It's weird to the point where we just need to acknowledge it. She needs to wear a hat. Some sort of protective headdress because there's something about... I don't know what it is. Obviously, she's my younger sister, so I kind of thought she was exaggerating. But over the years, I've come to accept no...
Oxide and Friends
Reflecting on Founder Mode
This is happening to you more than it's happening to other people. Birds are swooping onto your hair and attempting to attack your skull in a way that doesn't... No, this is not routine. You're right. Does she listen to the podcast? I want to find out and I don't want to ask her. She's so busy. I just don't want to ask her. So I figured this is really the best way to find out.
Oxide and Friends
Reflecting on Founder Mode
It was really just to denigrate her on here and see if she listens to it or not. But no, it's no denigration at all. Quite the contrary. She has got... She's got a condition that needs to be... I honor her condition, her challenges in life. Other people don't have this challenge. And we were at Fisherman's Wharf. I guess she's kind of like, this happens to me more frequently than other people.
Oxide and Friends
Reflecting on Founder Mode
I'm kind of dismissive. And we go to Fisherman's Wharf. This is years ago. And... the seagull that descended upon her was the size of a Cessna. That was like, that was like a plane crash. I think it was gigantic. And like, tried to like, just like she, he thought that she was the French fry. Just like tried. It was, it was really quite dramatic. I don't know if the seagull was a show or not.
Oxide and Friends
Reflecting on Founder Mode
Definitely made her point like very, very dramatically. Yeah. So anyway, I was showing this idea of seagull management, and I think it was life-changing. It's like, I need to send this Wikipedia article to many of my former colleagues, because this is exactly... And it was an environment where senior leadership, very technical...
Oxide and Friends
Reflecting on Founder Mode
and would descend in on... People would be working on a problem for two or three weeks and would descend in and disrupt everything and take off. Great. Exactly. And that is the worst. I really think that is... It's really, I'd rather be in like command and control than that. You know what I mean?
Oxide and Friends
Reflecting on Founder Mode
It's where this kind of like false sense of autonomy where you let me, you let us together as a team develop a solution for this. And now you've descended in and like ripped it up. And now it's like, boy, that's that corrosive.
Oxide and Friends
Reflecting on Founder Mode
That's fair. I was very surprised that Elon Musk, and I would love to know, is Elon Musk like, I don't know, don't you feel it needs more Nazi in here? Just feels like... Just 10%. I'm just like, it's just a joke. I mean, those jokes are funny. Who knows? I don't know. I think that was good that he anticipated that. But I guess we should describe the piece in terms of what...
Oxide and Friends
Reflecting on Founder Mode
She listened to the podcast. I guess we're going to fight. Oh, 100%. Maybe 99.8% now. We'll see. Even the baseball one.
Oxide and Friends
Reflecting on Founder Mode
This is a good segue to Camille's piece. Camille Fournier wrote a piece that was pretty interesting in reaction to this. Founders create managers. And I'll drop a link into that in the chat. Which I think is kind of what you're saying. Is that founders created the chaos that created the situation. Founders are creating the situation effectively. And
Oxide and Friends
Reflecting on Founder Mode
which I think in terms of like, they are, you want to be a details person. Right. And you don't know the right way to let go of those details. Yeah. And, and this is where, you know, so you, cause I was in part of what kind of prompted the, the longer blog entry too, is I was noodling on like, how do you do this? How does one do this? How do we do this? And when you said this, what do you mean?
Oxide and Friends
Reflecting on Founder Mode
I mean, I, How do you keep kind of a true north to an organization as it gets larger? How do you avoid the pathologies that Camille describes? How do you avoid the seagull management? How do you keep, while at the same time, Providing clarity, and I think it is very important to stay detail-oriented. I think that is extremely important, and I think that leadership, stay detail-oriented.
Oxide and Friends
Reflecting on Founder Mode
I think it's a real problem when, and you and I have both seen this, where leadership gets divorced from the details, and all of a sudden, you've got a great blog entry from years ago, I am not a resource anymore.
Oxide and Friends
Reflecting on Founder Mode
And to me, that is indicative of this problem where it's like, these have now become opaque blocks that I can kind of move around and I've lost track of like why things are hard, why they take long. And it's now schedule over everything. I've now over-promised to a customer because I've lost track of the details that make this difficult. And now when my team doesn't deliver, I blame the team.
Oxide and Friends
Reflecting on Founder Mode
It's just like, you're in this loop that I think is a loop that is really, really corrosive. Yeah. And so how do you avoid this? How do you not get into that pathology? How do you stay detail-oriented without being a micromanager? How do you have autonomy while still having clarity?
Oxide and Friends
Reflecting on Founder Mode
And what Jeske points out, I think rightfully, is that in the absence of other things, especially where influence is being used, and especially in an organization where people do value autonomy, things can get political. And they talk about shadow hierarchy and things like that, where And so the thing that I, as I was kind of like reflecting on this, I'm like, you know, how do we do this?
Oxide and Friends
Reflecting on Founder Mode
And I do think like, man, the written word is real. I mean, dovetails for our episode from last time on our piece, but the written word is really, really important in terms of broadcast. I mean, I, I mean, God, I am so thankful and I should like, there's a bell I should ring every time this happens where,
Oxide and Friends
Reflecting on Founder Mode
maybe a prospective investor or a customer, someone asks something about an issue that I'm not up to speed on. And, oh, thank God there's an RFD on it. And I spend 20 minutes reading the RFD. It's like, damn, okay, this is great. I didn't have to waste anyone's time. You don't have to create a fire drill where you say, hey, everybody, I know exactly what our thinking is on this.
Oxide and Friends
Reflecting on Founder Mode
I know exactly what we've thought about. It's like, boy, am I really, really, really grateful for it. And I think that Amazon famously does this too. I've never worked at AWS or Amazon, but famously has a writing intensive culture where you're kind of sitting down and reading for the first half of the meeting. Yeah. I kind of love that.
Oxide and Friends
Reflecting on Founder Mode
What is the dichotomy that Graham sees? Because ultimately, this is reductive, and it generates this galactic dichotomy between founder mode and manager mode, and there is nothing in the middle.
Oxide and Friends
Reflecting on Founder Mode
This is back when they were being innovative, by the way. I don't want to... Sorry to... Yeah. As one of our colleagues who was formerly of AWS, I'm like, pretty sure the last innovative thing they did was Lambda. That was in like 2015. But I think that certainly when the Bezos-led company, they would write the press release before working.
Oxide and Friends
Reflecting on Founder Mode
And I think that's an interesting way to get ground truth and clarity. It's a press release, but in terms of getting that true north and getting that on paper. Yeah, okay.
Oxide and Friends
Reflecting on Founder Mode
It is too late. How do you, how do you possibly, this is like, don't look up, but how do you tell me? People don't even know who Dave hits is founder of NetApp.
Oxide and Friends
Reflecting on Founder Mode
I think we should feel fully comfortable. Oh, All right. I think we should feel fully comfortable. All right. I think we should feel fully comfortable. I think this is a, this is a, this is a great story. You know who I, my sister does not listen to the podcast. I don't think she might. Your wife, you say a hundred percent. I think it's unlikely she goes to the podcast. Dave hits zero percent.
Oxide and Friends
Reflecting on Founder Mode
Let's not listen to the podcast. Okay. Says I. Let's go. Okay. So Dave Hitz writes this book, How to Castrate a Bull. That's right. We were at Fishworks at the time. We were at Fishworks. We were at a NetApp competitor.
Oxide and Friends
Reflecting on Founder Mode
And what we had heard from customers is like, I love the product. I hate the price. And I'm kind of not in love with the company. The company is very self-important. But I do love the product. And someone in the chat says, I have a signed copy of Dave Hitz's book. And you, you are not the only one with a signed copy. You are not alone.
Oxide and Friends
Reflecting on Founder Mode
And you're sure this is just on the point of principle of not wanting to line his pockets by... I mean, it's like... I mean, we've said that an undercurrent of this episode is Success Teaches You Nothing. The subtitle of this book is How Success Taught Me Nothing by Dave Hitt.
Oxide and Friends
Reflecting on Founder Mode
So we have two copies. I did not realize that. It's like, wow, this person is very hot. No, you're saying you're true, but misleading things. Did they involve the fact that you'd be, that you needed a review copy?
Oxide and Friends
Reflecting on Founder Mode
That's kind of my recollection, too, is that I feel that this is a consequence of us reading the big con. Yeah. Sure, I read it. The Big Con, terrific book. The Sting is based on the Big Con. A book written in the 30s about the Argo of the grift. All right. Oh, such a good book. And the Big Con, one of the things it repeats over again is you can only con a greedy man.
Oxide and Friends
Reflecting on Founder Mode
And I think I do recall us thinking like there is a kind of an ego con here that if you praise someone who is already praising themselves, you're not going to get fact checked.
Oxide and Friends
Reflecting on Founder Mode
Dave Lightman. Dave Lightman. I, we will leave it to others to figure out the, the Dave Lightman. I mean, I feel like that's a good homework assignment. If you don't recognize the name, Dave Lightman, not in the notes, not in the notes. That's right.
Oxide and Friends
Reflecting on Founder Mode
I will find the link. Can you find the link now? I can talk about something else while you find the link. Because I just feel that we need to... First of all, it should be said that you wrote the fan blog. So one of the things we figured out... I don't know if we knew he was going to pay attention to it or not. It became clear that he was hanging on every word. Every word.
Oxide and Friends
Reflecting on Founder Mode
Like he was commenting on the fan blog. So at that point, it's like, oh my god. We got them. Oh, God bless you. God bless you, Blogspot. Sign of the times that it's on Blogspot.
Oxide and Friends
Reflecting on Founder Mode
So you've written the first summary about this book is awesome and I cannot wait to read it. I'm really, really excited to read it. That's right.
Oxide and Friends
Reflecting on Founder Mode
And when does our persona's mother enter as a kind of a... Pretty early.
Oxide and Friends
Reflecting on Founder Mode
When he, at some point has like a comment on your blog being like, God, I don't want to like, I don't want to give it away, but don't take that. Cause you made it very clear that like, you've taken that advice very literally and have, have maybe said some things to your mother that you, maybe you should, maybe you might, might live to regret. That's right. So, okay.
Oxide and Friends
Reflecting on Founder Mode
So at this point, so the, then you've got the yes, that Dave hits where he reaches out to you. That's right. Um, and the, uh, from, so he's commenting, we are just having, you should, we were having the time of our lives. I mean, just euphoria.
Oxide and Friends
Reflecting on Founder Mode
Oh, I'd say, hey, let's all go into management mode here. Adam wants us to go into management mode, so everybody give a status update because apparently that's what we need to do. We're in management mode now. I know. It feels very dismissive. Yeah. And it feels dismissive of expertise, which I think... And again, it creates this kind of false dichotomy around expertise.
Oxide and Friends
Reflecting on Founder Mode
The mouse is just nibbling away at it, just chopping. The mouse is just pounding on the door, demanding more cheese. Yeah. It's like, oh God, yes, absolutely. So you've got the net app, you've got trade dress, I believe would be the term. May even be the term from the letter they sent you. They'd print out that letter. So they send you basically a cease and desist. That's right.
Oxide and Friends
Reflecting on Founder Mode
And then you have, which I just think is like, we're like, God, just another gift from these people. So you have your nonstop lawyers blog entry. Yeah. which I mean, it was just, it was, it was delightful.
Oxide and Friends
Reflecting on Founder Mode
Future generations will. Did John Jay write this entry? That's right. Was it Alexander Hamilton? Was it Madison? Was it Madison? We can't know. We can't know. For this entry of nonstophits.blogspot.com, I can't believe this site is still up.
Oxide and Friends
Reflecting on Founder Mode
Okay. Wait a minute. You get the note from the lawyers and then you change the blog entry to be all in that app trade dress. Do you remember this? You change it to be like all like the Dave hits eyes. That's right. It was just like his eyes everywhere. Yeah. Um, Yeah, so you got him to sign the book.
Oxide and Friends
Reflecting on Founder Mode
I just like, I could not stop. It is funny. And anyone should be suspicious of a fan blog. I'm sorry. You should just be like, I would like to believe that we would not fall for this. Yeah. And, but here it's like, people are like, all those people pacing for the podcast. No, you all this or all this. Steve hits. That was Dave hits. Uh, it was great. Yeah. Well, it's a miracle that's up.
Oxide and Friends
Reflecting on Founder Mode
All right. So we, um, yeah, you, what did you learn? So nonstop hits is again, I do think this is the success teaches you nothing. But you're saying that we, that's not... You wrote the future histories, right?
Oxide and Friends
Reflecting on Founder Mode
100% true. I mean, we poured over it. Like a book club. Like Talmudic scholars, we poured over that. Every diphthong, every syllable, we, I mean, we were... No one has read that book. I mean, it's like in the end, Dave hits... when you encounter this podcast and you're upset, should you be upset? No one read that book more closely than we did. No, we were the biggest fans. We are the biggest fans.
Oxide and Friends
Reflecting on Founder Mode
We're Xers. What do you want? Agreed. That's all we have. That's all we have. That's all we have. Leave us with our ironic detachment. Okay. But he, so he advocated the future.
Oxide and Friends
Reflecting on Founder Mode
I did not know this. I had forgotten that you were depoed in that case. Yeah, I was depoed in that case. I mean, they were depoing me for stupid reasons. Why were they depoing you?
Oxide and Friends
Reflecting on Founder Mode
Mr. Leventhal, is the name Dave Lightman familiar to you? It is familiar to me. It is familiar. It is a pop culture reference, as I understand it. Uh, that is great. So you were a little bit worried. Well, you know, did you have a game plan? What were we going to do? Like this goes to nonstop hit stop blogspot.com. Like, uh, like fake a heart attack.
Oxide and Friends
Reflecting on Founder Mode
We're obviously in a safe space. I was glad. No way. I was glad. I was glad. I thought like I was glad because I'm just like, I need to get on with my life. This thing has like this piece hit just hits a lot of nerves. Yeah. And it gets a lot of people talking about it, tons of comments on Hacker News.
Oxide and Friends
Reflecting on Founder Mode
Like, because you know, when you go into a deposition, you're talking to her, I'd like to declare the fifth. It's like, that's not, you can't do that. This is not a criminal proceedings. You can't declare the fifth.
Oxide and Friends
Reflecting on Founder Mode
No. Oh my God. You should have. No. Oh, you should have. They would have, they would have died laughing. Of course. Oh my God. They would have died laughing. They would have, I love working with the lawyers in that case. They would have loved it. Oh, they would have eaten it up. Anyway. I, I just, I was unclear if we had committed any, like, I don't even know what we did wrong. Actually.
Oxide and Friends
Reflecting on Founder Mode
Fake heart attack. You're just like, hey, the ambulance is on the way. It's like, I'm fine.
Oxide and Friends
Reflecting on Founder Mode
I have a lot to tell you. I have a lot to tell you. Right. I have a fake block. You're like, what is going on? These guys, you know, maybe I saw frontiers. Okay. But so he talks about future histories. There you go. Bring it all back. Bring it all back. And is this like this idea of like writing down what we have, what we're going to do? Yeah. It's like, what do you want the press release?
Oxide and Friends
Reflecting on Founder Mode
That is extremely valuable, I think. Yeah, totally. Who knows if they did it, but it's a neat idea. It's a neat idea. And I do think, I mean, I think the written, I do think the written word is really, really, really important. And I think that the Because when you write, you have to get to a ground truth. It does scale.
Oxide and Friends
Reflecting on Founder Mode
And people who've come to Oxford are like, oh, this is great because I can understand why they made a past decision. It's been... So I think that that is a very kind of concrete thing that people could do.
Oxide and Friends
Reflecting on Founder Mode
Yeah. I think it's important. Yeah. Interesting. Yeah. I mean, the other thing I'd say is that like, I mean, God, are you, are you just trying to like, do you have a bet with some other oxygen planes? Like I bet I can get him to write an RFD on performance review.
Oxide and Friends
Reflecting on Founder Mode
No, no, no, no, no. Mayday. Mayday. That's right. Wait a minute. Are you having chest pains right now? What's wrong? My lawyer. Come closer. Um, That's interesting. Yes, in terms of offering that clarity. I think I love it when people are like, I love this company because you've written everything down. And we're like, no. There's a lot that's not written down.
Oxide and Friends
Reflecting on Founder Mode
You can't write, you can't, some things you need to leave unsaid. It's just too much of a lightning rod, but no, I think you're right in terms of like, there are things like that. And actually, you know, I honestly, I think there are some things, and this is where I've really benefited.
Oxide and Friends
Reflecting on Founder Mode
I mean, this is where you kind of benefit from just a sense of collaboration anyway, because I mean, how many RFDs have you written because someone has said you should write an RFD on that? Like most of them. So in other words, you're thinking of something that like, okay, I'm just like ideating and I don't think this needs to be written down.
Oxide and Friends
Reflecting on Founder Mode
You're not necessarily thinking about it like that concretely. But someone else is like, I would actually like to see that written down and you should write an RFD on it. I feel like certainly for me, like most of my RFDs have come from, because people are like, I would like to see an RFD on that. And I was like, oh, yeah, okay. It didn't really occur to me that that's sure. Yeah, absolutely.
Oxide and Friends
Reflecting on Founder Mode
We can do that. Yeah. That's a great idea. But it doesn't necessarily occur to you to write down something that you are already kind of feeling very strongly. That's right. But maybe those are the things you should be writing down.
Oxide and Friends
Reflecting on Founder Mode
Yeah, but I do think that if you don't, the less you write down, you're kind of immunocompromised as an organization.
Oxide and Friends
Reflecting on Founder Mode
That's right. Or whatever. Or people coming in being misleading about it. I mean, certainly, you know, We've seen this in past lives where people are, I mean, God forbid, you have someone who's telling different people different things. Yeah, right. Where if you've got it written down, like you can only tell one story, right? You can only tell one story. And that is, it's really, really important.
Oxide and Friends
Reflecting on Founder Mode
And I think that like, and sometimes they're not even telling different people different things deliberately. They are, or they don't, or people are hearing different things and they don't realize that people are hearing different things. Yeah.
Oxide and Friends
Reflecting on Founder Mode
I mean, so I think that getting, I mean, the question that founders, I don't think anyone should put in front of themselves is like, how do we offer better clarity? Yeah. And then how do we promote trust? I mean, you mentioned it earlier. I do think it's like, man, that is, trust is everything. Trust is everything, for sure. And you've got to figure out like, how do I promote it? How do I build it?
Oxide and Friends
Reflecting on Founder Mode
How do I... When it's fractured, how do I repair it? And I think that part of the problem with founder mode at its worst is you have the opportunity to like... do real, real damage to trust. Yeah.
Oxide and Friends
Reflecting on Founder Mode
It's not a me thing. Some people say caloric budget. Do they not? No. This is like blood over the ear. I assume everyone said blood over the ear, and then I Googled it, and I'm like, no one's using blood over the ear.
Oxide and Friends
Reflecting on Founder Mode
That's just me. How many things are like that? A caloric budget is like that. I think so. People can correct it from that one, I guess. Deafening silence, but there's no real corrections forthcoming from the chat over here. Yeah, you're right. Damn it. Another intervention. Another podcast, another intervention. There you go. Um, so yeah, sorry.
Oxide and Friends
Reflecting on Founder Mode
And the CTO wanted to... To be fair, there's another pejorative that's in the Silicon Valley VC lexicon of the first adult.
Oxide and Friends
Reflecting on Founder Mode
Yeah, interesting. Yeah, and I think if you called it clarity plus trust mode, it's like, I don't know, that feels like work. Yeah, totally. I think I'd rather go into founder mode. Founder mode sounds a lot more fun. Sounds a little more like Prince Adam becoming He-Man. No, that's right.
Oxide and Friends
Reflecting on Founder Mode
It also, I got to tell you, I hear clarity mode and trust mode. I just feel slow and I want to move fast and I want to, I want to move fast and I want to break things. Makes sense. Like, and trust and clarity mode. Take my money. But I think it is worth listening to the Brian Chesky.
Oxide and Friends
Reflecting on Founder Mode
Because I think that it was helpful to get that kind of from the source.
Oxide and Friends
Reflecting on Founder Mode
And then, like, get as much as possible from the source, I would say, is part of, like, even if it's Dave Hitz. As it turns out, Dave Hitz, nugget of wisdom. Right there, right? As Dennis Ritchie said in his anti-forward to the Unix Haters Handbook, It is an undigested nugget of nutrition in what is otherwise a fecal pie. I just felt that was like one of the greatest. Oh, it's absolutely a keeper.
Oxide and Friends
Reflecting on Founder Mode
Dennis Ritchie, rest in peace. this ended up being a good discussion I feel yeah turns out who knew thank you Paul Graham thank you Paul Graham we got to we got to not the pod probably along with that yeah exactly Paul Graham and Dave Hitz listen to this together it's one of their like weekly things that they like to do Their little guilty little pleasures. One of their guilty little pleasures.
Oxide and Friends
Reflecting on Founder Mode
Exactly. Well, I think this is it. I know this was a cold take, but I think this was fun. Yeah.
Oxide and Friends
Reflecting on Founder Mode
And I think it's like, and there is something there because I think that the problem is real. And the question is like, how do you, how do you build, how do you build trust? How do you offer clarity to a team? Like that's the challenge. And it's like the way to do that is not to go into seagull mode. Yeah. Can't find a mode. This has been great. It's great to be off our little hiatus.
Oxide and Friends
Reflecting on Founder Mode
Maybe this is a good little cooling off period we have. Maybe we should do this more often. We'll come back to tweets that are two weeks old. Good. I'll do more international travel. Sounds good to me. Exactly. It feels like we're more likely to read the tweet in this case, too. All right. It was fun, and we will see everyone next time.
Oxide and Friends
Reflecting on Founder Mode
Like someone's got to, someone's got to lick the frosting and someone's got to do something with the rest of this thing. I don't really want it. And so is this like, okay, I think this role is being foisted upon you. I don't think you're totally bought into this role.
Oxide and Friends
Reflecting on Founder Mode
Yeah. Yeah. The, the, where they kind of surprisingly like seeking wisdom out of others, I think is actually an, it's unfortunate that it is so unusual a trade, especially in a company that's enjoyed some early success. Yeah. Because they are being kind of implicitly, and this is the dichotomy that's created.
Oxide and Friends
Reflecting on Founder Mode
What we should be encouraging people to do is seek out that wisdom and then make your own decision. Instead, it's like, find an adult. It's like, all right, I guess that makes me the child. I mean, it's the same dichotomy, but from the other perspective of the founder as a child and the professional management. And so how did the plan thing work out? Did they want to...
Oxide and Friends
Reflecting on Founder Mode
That's a great... That's actually an honest story to tell yourself. Because it's so tempting to be like, God, I talked to those guys early. Like they really, we really wanted to make a connection. I walked away from it. It's a multi-billion dollar company. I would be like a billionaire.
Oxide and Friends
Reflecting on Founder Mode
It's like, well, no, maybe you would be like bored, miserable and would have left after six months or I would have fucked everything up or like actually plans on a multi-billion dollar company because it's run by a guy who like, like the CTO doesn't know how to balance a checkbook. He only knows what that means in the abstract. Right. So, but interesting.
Oxide and Friends
Reflecting on Founder Mode
Normally, when you get that much comments, one of the things I like about Hacker News is when you get a lot of comments on things. When the comments exceed the upvotes, a story gets more or less automatically killed. It gets automatically removed from the front page, which I actually think is a great... A great, very simple algorithm. Yeah, indicator of hot-taken-ness or whatever. Hot-taken-ness.
Oxide and Friends
Reflecting on Founder Mode
So they were, they were actually really actively seeking that wisdom. I think it's a very unusual.
Oxide and Friends
Reflecting on Founder Mode
And I was really impressed. We should be encouraging that a lot more. Cause I think that like the, the big problem, I mean, I had a couple of big problems with the piece, but the non-specificity is like first among them of this is like, too reductive. And it's very important to remember that Paul Graham has started a successful company. It is called Y Combinator.
Oxide and Friends
Reflecting on Founder Mode
Like, via web, I'm sorry, we're just not going down to via web as a model for what companies should aspire to do. And I think that the... understand that most of his perspective comes from starting Y Combinator. And so he is getting this kind of hearsay from founders and then deliberately trying to be reductive about it. And as a result, you're losing a lot of nuance and detail.
Oxide and Friends
Reflecting on Founder Mode
And I think that for folks that are interested, I think, and my first kind of tweet on this, my first attempt to think about this only briefly is, was linking to Tim O'Reilly's piece, How I Failed. And I mean, I think this is a piece that I actually had read and then couldn't find again. Huh.
Oxide and Friends
Reflecting on Founder Mode
And I actually DMed Tim about it because I'm like, you know, you wrote a piece years ago and it had a real influence on me and I can't find it. And it turns out it was a LinkedIn post. That's why I couldn't find it. And then he gave me some terms to search on and I think then O'Reilly actually kind of finally put it up. But I think this piece is terrific.
Oxide and Friends
Reflecting on Founder Mode
First of all, and I think Adam, you and I believe this perhaps to a shared fault, where I do worry that I focus too much on how companies fail. Do you know what I mean? Yeah. Yeah. Where it can inhibit your own action. Like you need to be, there's a degree to which you have to be blind to the odds. Yeah. Certainly in a startup. Totally.
Oxide and Friends
Reflecting on Founder Mode
And you, it's like this walk in this fine line between, I need to know how things fail. But if I look, if I learn too much, it might inhibit what I actually need to go do to make this thing successful. But we definitely, you and I both like to understand like, how did things come unglued? And I feel like I learn a lot more from failure than from success. Totally.
Oxide and Friends
Reflecting on Founder Mode
Success is just, I would say success teaches us all nothing, but success certainly teaches me nothing. That's maybe a much more, that's a more, a more concise statement.
Oxide and Friends
Reflecting on Founder Mode
Totally. And also, you don't want, as someone was saying in the chat, a lot of success is luck. And you, it's hard for people to say like, I was really goddamn lucky. Totally. I was in, God, I was in the right place at the right time with the right idea, the right team, like all that just like happened to come together. And which is almost always the case.
Oxide and Friends
Reflecting on Founder Mode
Like there's always this kind of element that is out of your control. I feel. Yeah. And, but I think that part of the reason that the O'Reilly piece is good is it's like not a reflection of, on unequivocal success. It's a, and obviously titled how I failed, but it's getting into some of the specific things that Tim felt that he did wrong.
Oxide and Friends
Reflecting on Founder Mode
An indicator of we've got a lot of discussion going, but people aren't actually reading the underlying article. I always thought that was super elegant. But in this case, it was getting a lot of comments, a lot of upvotes. And so this thing is just going and going. I mean, I read it. I'm like... Don't think about it. Don't think about it. Don't think about it. Still thinking about it.
Oxide and Friends
Reflecting on Founder Mode
And I know that like, I think this is another thing that's really important that even I think Tim is remarkably candid here. I think he's, he's a terrific writer. He's also like telling a narrative that is kind of his narrative and someone, I mean, someone else at O'Reilly, uh, could very easily be like, that is not like consistent with my understanding.
Oxide and Friends
Reflecting on Founder Mode
It's so easy for different people to have different understandings and for a founder to even not, you know, they'll see some things and not see other things. Um, Because I think it's always important to consider that you're getting kind of one narrative and that there are other narratives.
Oxide and Friends
Reflecting on Founder Mode
Michael Lewis, let's reflect back. So what book was that? What was the... Well, let's go tell him. The hagiography you wrote, hagiography or hagiography? I started knowing that one. I think it's hagiography. I think that's right. All right. The hagiography that you accidentally wrote on Sam Bankman Freed, which I guess readers are telling you that's the book you wrote in that case. Yeah.
Oxide and Friends
Reflecting on Founder Mode
But yeah, that's interesting. I think that's right, that you don't necessarily know
Oxide and Friends
Reflecting on Founder Mode
Absolutely. Absolutely. And I think it's very easy. So I think you've got to kind of keep that in mind when you're looking at Tim's piece. But all that said, I really like this. I thought there was a lot of specificity in here that was...
Oxide and Friends
Reflecting on Founder Mode
really important yeah um and some of that is just like let me do like failure number one of people only hear half the story which i think it was kind of interesting right where he's trying to give tough messages with empathy and people like well i'm gonna like i'll take the empathy and i'll kind of like discard the tough message but that's like his perspective like maybe someone else like the specifics on that one also tickled me where i think it was that he was um