Adam Leventhal
Appearances
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Skylake is what did that. But you also had this problem on Broadwell with AVX2 and the others. And it's actually worse than just running an instruction. If you actually just leave the AVX save state bit such that Intel thinks it's modified in the register state, that's enough to trigger this slowdown sometimes. Oh, man. You might not remember this. We had a nasty bug back at Joyent.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
where we had a guest in windows and we just weren't properly clearing one part of the save state in the initial state. So the initial state basically was like, you know, like the two 56 bit op masks are the two physics, but register state YMM state is valid. It's like, okay, I'm going to no longer boost.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. I mean, basically just that one for, it was just for it, but it was just, that's, I think that to me is kind of the even more telling, but even if you just leave the state in the save state, then you're, you're toast.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Sure, yeah, we can go a bunch of different places. I think the starting place is really actually going back to Milan.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Because actually like 64 core Milan, like the 7713P, or even if you go up a little bit, getting that in 225 watts or 240 watts, that was actually really nice. Really nice, yeah. That was really nice. Performance per watt on Milan is really pretty great.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Uh, yeah, I mean, I definitely, I mean, when you get, you know, it's hard to compare to the, you know, 192 Zen five C cores, uh, in, in that range. But, um,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Once you kind of get to Zen 5, I mean, I still miss that there's no 225-watt, 64-core part, but I think this is where you're going to see this from our end, trying to think about how do we get a little more flexibility, leverage the fact that you have base DRAM has increased in capacity without going to 3D-stacked RDMs, just because that part of the balance and price equation starts getting...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
really thorny. So the fact that you have 128 gig RDIMS, um, is useful, especially when you start looking at the fact that two DPC stops making, gets challenging, uh, fast, uh, there. So I think there's a bunch of different skews you can kind of start to look at. You know, I think one thing I've been keeping my eye on is actually the 160 core, um, Zen five C, uh, as one thing to look at.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Cause like that kind of keeps you below 400 Watts, uh, So I think it's still like a group E CPU as opposed to a group G in the IRM. So do you want to describe a little bit of those terms? When you say the Group G versus Group E, what are those things?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, when AMD creates a new socket, they put out what they call an infrastructure roadmap or IRM, and then they basically are predefining different TDP ranges into these groups. So, for example, Group E probably has some range off the top of my head from like 320 to 400 watts. Um, these new 500 watt, uh, CPUs, I sometimes joke are group G guzzlers, uh, just cause they, they definitely take a lot.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, but you can design your platform to different TDP kind of thresholds, uh, these kinds of different infrastructure ranges, and then you'll get different kind of CPU and core counts. So like, I think if we look at, um, there's like three or four different 64 core CPUs. I think there's like the 95, 35, uh, which is kind of like the, uh, you know, almost 300 watt, uh, 64 core.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's like, that'll be like a group a CPU. Yeah. And that kind of gives you what the tells you kind of what the TDP range and what you as the platform designer can kind of tweak, uh, from what the min to max is on there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, whereas like the others will come to group E or some of these even smaller ones, um, like some of the 32 and 16 cores, if they're not getting cranked up for frequency might even be in like group B, uh, And related.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, that's a good point. So I think a lot of what we do is a bunch of work that we do with actually Eric, who's also on here. We're trying to figure out, hey, what's the right balance of how many stages, how many components do we need to kind of reach what kind of what power group?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, for example, if we designed when we did SP three, we designed what they called Group X, which was the group they added later for the 3D V cache and like max frequency skews. Maybe it's like a 240 or 280 watt max. But then we ran kind of a 225 watt CPU in there the entire time, giving us plenty of margin, plenty of headroom, which meant that, you know, our power subsystem was very clean.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So here we kind of are saying, hey, let's let's you know, we said, hey, we're going to start with group B as our target. We're going to see what does it cost us to fit group G? You know, does it actually cost us more stages, more inductors, more? more other parts. And, you know, then the first question of, can we cool, can you air cool 500 plus Watts, which is a different question entirely.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, right now we've done all of our, our worst case studies, which is basically saying, assume the CPU is going 500 Watts, right? All the dims are going at their maximum. You've got every SSD going at its maximum. and the NIC, and some amount of loss, you're paying some amount of loss for all the stages, we still think we can cool that.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And then practically speaking, even though the CPUs with turbo boosting have a good way to eat up the rest of your power, you're usually not getting all of those devices maxed out all at the same time.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
They've also gotten a lot more clever about how they do all the hashing across stims.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, I mean, the P4 programmable nature of it for us is something that's actually really powerful. We leverage that in our switching silicon a lot and have been looking for something to get that into the NIC. The big challenge is just, I think where we're a little different is a lot of the DPUs have been designed to basically be like, we're the compute, the DPU is the computer in charge.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And Hey, you big, big CPU. That's like running guests over there. Like, uh, you're subordinate to me. So like, yeah, you don't, you don't, you know, you exist, but like only at my pleasure. Uh, and we were not quite as, uh, split brain, uh, there slash we're not trying to sell the entire server.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So like, you know, it just gets thorny when it's like, okay, that, that device also needs its own DDR five.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Uh, some questions around like, Oh, but yeah,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Totally. Yeah. So when we end up designing for kind of not absolute density, but trying to get the best density in a fixed power budget, which because, you know, unlike the hyperscalers, we're not basically building a power plant next to every new DC. Yeah. that that's where it gets a little more challenging.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so we're trying to work with folks to figure out, you know, Hey, if I don't need, say all of the arm cores that show up there, or let's say I didn't run with DDR five, you know, where can I, what can I get? What can I, can I still get out of there? You know, how can we kind of change this from, you know, some of these parts are, and I don't remember what this one was, you know, or 50, 75 Watts.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And, you know, that's that, or, you know, maybe I, I play games and I say, um, you know, I've got a lot of SSDs, but maybe I don't need all of the IOPS, all those SSDs. So I can double up, you know, capacity instead.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And that gets me back some of the power and I can send that to the neck, but it's definitely, uh, we're not in a, you know, even with just increasing power for the CPU, I'm already trying to think about like, well, what do I do for folks who don't have all that power? If I've got 32 sleds, how do I, uh,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But yeah, overall, really good, really nice to see, excited to see that kind of P4 continue there and hoping someday we can find a way to make it make sense for us. But I think there's a lot of other folks who it does make a lot of sense for.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Sure, I'll see if I can do it justice. Effectively, the way I think about P4 is it basically is a programming language that you can use to compile a program that operates on the Nix data plane. And I think this is an important part because for a lot of these things, the value is to actually run at line rate.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So you've got 100 gig, 200 gig, 400 gig, especially with all these 112 gig 30s coming along. You basically can't necessarily treat that as a general purpose program that's coming in, DMAing everything back to normal, you know, to a normal core's memory and processing it and sending it back out. But instead, this kind of lets it process the packets kind of in line in that hardware receive path.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Actually, George, this is the pitch I've tried to make to them. Just because to me, it's like if you actually look at NVIDIA and what they've done with NVLink and basically buying Mellanox, at the end of the day, to really be able to you know, deal with that, what they're doing with ultra ethernet, it feels like you have a P4 engine. Yes. It's going to be a big change.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Take it from, you know, the Nick kind of two cert, you know, two port form factor to, you know, a switch ASIC and kind of dealing with power. But I think that if you really want to do well in that space, you can't rely on just like, hey, I'm going to convince Broadcom to let me pass through, you know, XGMI, you know, through my switch.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Or whatever they're calling that Infinity Fabric transport these days.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Right. So yeah. So, so yeah, I think it's like, it's good that you have that consortium and you'll be able to push some stuff there. But I also feel like at the end of the day, you know, where you see a lot of value from Nvidia is that they are building, you know, where they've been successful because they have vertically integrated a whole lot of that stuff. Yes.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, I think actually we've been talking about PCIe a bit. So I actually think one of the things that I find has been both fun and sometimes a little vexing, but is ultimately good for the platform, not always as fun for us in how the register numbers sort themselves out, is that they've actually increased the number of IOMS entries in there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So basically in the past where you had a group of 32 PCIe lanes, which are basically two X16 cores, They were consolidated into one connection to the data fabric. Actually, one of the more interesting things is that we've seen that in Turin, each X16 group is connected to the data fabric independently through its own kind of IOMS slash IOHC, which are all, I guess, internal data items.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Or those are core-ish? I mean... Yeah, I don't know how much. I'm sure there's a Z80 hidden in everything, or an 8051. So I'm sure everything's a core at the end of the day. But actually, if you just kind of look at it, this part is less hidden. Because if you just look at, hey, show me the PCI devices on Turin, you'll see, hey, there's eight AMD root complexes where there used to be four.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, that would be my theory is that basically it's getting you more because there's more data fabric ports that you can have just more transactions in flight to different groups of devices.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, yeah, that's definitely it. Otherwise, it's Milan to Genoa was more. There are more changes than Genoa to Turin. Interesting. In kind of some of the lower level stuff. Some of these kind of bits like how do you do PCIe initialization, hot plug have stayed more the same. From Genoa. From kind of Genoa to Turin. Yeah. They have some different firmware blobs that you talk to. So like...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
The SMU interfaces stay the same across these, but they moved to a new what they call MPIO framework, which is what goes and programs the DXIO crossbar. PCIe device training is kind of a collaborative effort between that core and X86 cores.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah. So there's, there's two different pieces here. So if you see AMD, first off, when they sell, you know, in their, all their makers say, Hey, we've got 128 PCIe lanes, which is great. Uh, But the first thing you have to figure out is, well, actually, how do those work on the board? I've got, are these X16 slots? Are they X4 slots that are actually connected to an SSD?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
What's their size and width, and how do they actually fit across the board? So one of the first things that everyone has to do is they kind of will tell the AMD's firmware, hey, here's how this is actually connected. you know, these logical, these physical fives, you know, I've got an X 16 slot. I've got, you know, in our case, we've got 10, uh, X four slots for basically every front facing you.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Dot two, right. You know, an X 16 slot for a Nick. Um, you'll have other things for other folks. Or if you have a kind of like a board, like showed up in the chat, you know, you've got some number of X 16 slots that map to things, some, some probably M dot two slot. So you have to tell it what is all, you know, what all is there.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So it can basically go and reprogram the internal crossbar to say, okay, these lanes should be PCIe. You know, George mentioned earlier that when you have a two-processor configuration, you know, some of those lanes are being used for that. So that's part of it. If you use SATA, which I, you know... getting less and less common.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Uh, you know, some of those lanes are come from those same PCIe lanes. So, um, that's the first step is you're kind of doing that. Then the next phase is after you've done that is, uh, if you open up the PCI, uh, base specification and, in like chapter two is a very long state machine. It has a lot of different states and a lot of different phases.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Obviously, what does it mean to basically have a PCIe device end up at the other end and have both sides be able to talk? Um, and so device training is basically going through that process, discovering, is there even a device there? Right. And from there trying to say, okay, let's start talking to one another, figure out how we can talk, then what speed we can talk at.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Once we're good at, you know, a certain speed, then they'll increase, um, to additional speeds, sending these things called ordered sets and training sets and lots of different acronyms that you hope generally just work and you don't need to think about. And then unfortunately, sometimes you do need to think about them.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so the main way we do this is that there are often reference platforms. So, you know, if you look at George's article, all of his testing was done on a volcano platform, which is the name of a platform that AMD developed that was specific for Turin. There's a couple older generations.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, you had Volcano, you had Puroko. I'm trying to remember what the other two were. The ones before that were all metals.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so we are doing ours mostly on a bunch of Ruby platforms, which were the ones that first came out for Zen 4. And so we have those, which gives us generally most of the schematics and other bits there. You generally get most of the firmware, but not all of it. So you can't always do all the things on the board that you think you should, like a reference platform that you do.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But that gives us a development platform. So we're fortunate that we were able to get some early silicon from AMD, so we could actually start doing development of that ahead of launch. Right.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, and the other nice bit there is that as you get into DDR5, one of the big problems is training time. Yes. So one of the things that AMD has is that you actually, after you train the first time, you actually end up writing back a bunch of this data into that spy flash. Yeah.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And if you have it, you know, without that virtualization, then if you're trying to kind of hash or figure out like, you know, how do I make sure this contents are all what I thought I wrote down and that it just gets gnarlier.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And this kind of just indirection layer, you know, computer sciences, you know, it's one contribution is adding another layer of indirection just to cheat just comes in handy.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, the first thing I had, which admittedly was early, A0 silicon. This is no shade on AMD. But I think it was one dim of GDR5, not that big. It definitely felt like minutes. It was 11 minutes, I believe, was the number. Was it really?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
as a former colleague of mine who retired was fond of saying, why do we have pins for Azalea on SP five? And they couldn't give me two pins to have a second flow controlled. You are.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Um, and we were specifically looking for something that, that we could use that didn't require PCIe training or very much, uh, from the peripheral space that kind of keeps us stuck in the FCH. Uh, even USB obviously requires a lot of smooth bring up and shenanigans. So that was out of the picture. Um, so we ended up with the UART and, uh, actually the AMD UARTs can go up to three megabaud.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
which is more than you know well okay well okay so they actually they can't go up to three megabod by default they didn't we we actually we needed a uh and well the rs232 level translator to go from like the 3.3 volt to i don't know minus 12 plus 12 that could not do Three megabaud.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But we, the three megabaud ended up being very load bearing for us because, because during, well, because the, yes, when we wanted the, when we wanted the PS, when we were doing dim training and are doing dim margining and we were, the PSP is spewing output. It was just happily going at one, one, five, 200. Yes. And there was no token to change it to anything. Yeah. That's for real.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's not to get a three megabaud. So it was very, it's very slow.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
You could practically run a message up there faster.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So we, we first are writing the spine or which actually goes much faster.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
That part is quick. But then we basically, instead of sending the full M dot two image that we would boot from, which would be like a gig and basically be, you know, an eternity in that world. We have a slimmed down, basically kind of phase two image. So unlike a traditional BIOS where you're basically splitting up, you know, the BIOS is in your spy flash.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's, it, it sits there and then kind of goes and pretends to reset the world back into 1970 after waking up and changing everything and, you know, turning on all the CPUs so it can turn them off again. Um, you know, we basically have a continuous operating system image. So basically, but we just kind of say, Hey, you find it half your like Ram disk, half your modules somewhere else.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So, um, we end up when we end up doing the recovery, we end up sending kind of a like slim down, just, just, you know, a measly a hundred megabytes over this, this small link. Um,
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yep. Yeah, so we're excited to see how all that kind of starts to change. I mean, there's a recent... I dropped a link in there. I think they did this at OSFC. They talked about how they're going to have OpenSeal kind of be the mainstay more so for Venice. Yes. And so I think, you know, from our perspective, this is all good. It kind of gets us out there. We can start to point to things that...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
you know, are in open cell and ends up being a, a win for, uh, uh, for everyone. So I think it's, it's basically, it's excited. We're, we're excited to see it. Parts of it. Um, you know, we may be able to leverage directly, but if not, you know, we can be inspired by it. They can be inspired by us and vice versa.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
I mean, I still think the best bit for me is in SP3 where the SMU to do hot plug, it speaks over I2C. And I'm pretty sure it, the smooth itself does not reset the I squared C peripheral to run at a hundred kilohertz.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
When the, when the I squared C peripheral restarts, it starts in basically fast mode plus, which is this weird push pull mode at like faster than 101 megahertz, which basically means it don't work. Yeah. And the only there was definitely no explicit initialization.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
It's just that, hey, this Dixie module in a, you know, dependent on this Dixie, which probably just did a generic blanket I squared C initialization, which changed, you know, which reset everything to 100 kilohertz.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, so the big... So I think to help understand making the trade-off for DDR5, you kind of have to go back to DDR4. So when you have two DIMMs per channel, the way it works is that kind of in the channel, or you go back even further in time, you'll actually find platforms with three DIMMs per channel. you basically are daisy chaining the channel.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So the traces will literally go up to the first dim, then continue on to the second dim or to the third dim in those platforms. So just the presence of that second, of having two dims on there sometimes changes the SI. In DDR4, it often didn't. So if you only had one dim populate, you could still get the maximum memory speed. possible.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
However, in DDR5, just the presence of two DIMMs per channel drops dramatically what maximum speed you can hit. And then if you actually make the mistake of populating it, then that drops the speed.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, then for us, the other big change is that from SP3 to SP5, you went from 8 channels to 12 channels.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
And so just for us, since we kind of have this half-width system, that's, I guess that's, what about, I don't know, one of the other, I don't know, Eric, do you remember what the width is on that?
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So basically, we were in a place where you could fit 16 DIMM slots, but you weren't going to make 24 DIMM slots magically appear in the space of 16, not unless you got very creative. So we ended up saying, okay, between that and the fact that you now had 96 gig and 128 gig RDIMMs without going to 3DS, which means you can actually purchase them and pay for them without a lot of blood money, then...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
or really you're not basically fighting against the GPUs and HBM, which means you can actually get them. Then that kind of, kind of put us down to, okay, well that, that, you know, you want the memory bandwidth. That's definitely one of the big values here. And the memory latency for a number of applications can definitely matter and you can still get to capacity in other ways.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
So that, that's kind of, I ended up kind of going at a kind of 12 channel, one DPC kind of configuration and, Because we looked at saying, okay, was that better? The other option was, hey, eight channel, two DPC. And that just kind of seemed kind of the worst of all worlds.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Well, in the DDR4 world, where you went from 3200 to 2933, that was a very easy cost to pay. If you were telling me to go from 6400 to 6000...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, the MR part of it is there. I think you'll, then once the questions, you know, as that slowly enters the market and memory controller support and, you know, seeing the costs, you know, assuming you can get the cost not to be ridiculous because volume is definitely one of the big parts of the DRAM business.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
But, you know, for us, because we have a platform that's not trying to scrunch everything into a 1U, you know, a higher dim just means a new thermoformed air flow shroud. Right. And that's pretty easy to go fit in. You know, for us, the added height is not a problem. For other platforms and other chassis, you could be kind of SOL.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Yeah, I think AMD's biggest strength is the fact that you're basically only choosing the number of cores, what the frequency is, and cache size. Otherwise, all the other features are the same. And I think that actually ends up being pretty powerful. You're not getting into a case where it's like, oh, do you want to have fast memory? Different SKU. Do you want to have a different...
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
Do you want to have RAS features? Ooh, sorry. That's going to cost you. That's going to cost you.
Oxide and Friends
Unshrouding Turin (or Benvenuto a Torino)
No. That's the most shocking bit to me, is that you basically still never, despite Intel championing it and trying to put all that energy into everything over the years, you still just got nothing.
Oxide and Friends
Querying Metrics with OxQL
And this is someone in the chat has asked about open telemetry in particular. I know we spent some time looking at that. What was your take on open telemetry?
Oxide and Friends
Querying Metrics with OxQL
Only thing you can support is pulling out raw data. Right. We want to actually be able to query those things in RAC effectively. Right.
Oxide and Friends
Querying Metrics with OxQL
But that's something that they should be in control of, I would think, ultimately.
Oxide and Friends
Querying Metrics with OxQL
It's very straightforward. Yeah, I mean, and then what about RAM utilization?
Oxide and Friends
Querying Metrics with OxQL
Yeah, using the same physical device, but you will, but other than that.
Oxide and Friends
Querying Metrics with OxQL
Yeah, interesting. Yeah. Interesting. Yeah. For the same reasons. For the same reasons. Yeah.
Oxide and Friends
Querying Metrics with OxQL
And what is the data that we're storing in this? I'll say that again. What kind of data are we storing in it?
Oxide and Friends
RTO or GTFO
Anyone can get it. Just put your kid in school and see if he brings one home.
Oxide and Friends
RTO or GTFO
So your car had an antenna, and it was like one of these little foam balls that you would stick at the end. God, I feel like I'm telling a story about the Kairos.
Oxide and Friends
RTO or GTFO
If you were sitting in your chair and you could throw it into the glass in the middle of the table, it was like a rectangle effectively, like a kind of oval, like a shaped table. If you were the short side, that was one point. If you were on the longer side, If you're farther away, then that was two points. And if you were in the corner of the conference room, that was like a four-pointer.
Oxide and Friends
RTO or GTFO
There were a bunch of rules about like suddenly it was like deuces wild.
Oxide and Friends
RTO or GTFO
There was like some doubling thing, like suddenly the stakes were higher if certain things happened. And the ball would frequently bounce out of this thing. It should be said. That's right.
Oxide and Friends
RTO or GTFO
So, I mean, the length of the walk we took you all on aside, I do feel like there's something deep in human DNA that causes office games to spawn from nothing, right? Everyone has their in-office game that was conjured from the tchotchkes that happened to be left in a conference room.
Oxide and Friends
RTO or GTFO
So I'm not sure. Like alternative forms where they could have potentially happened.
Oxide and Friends
RTO or GTFO
Yeah. I like that you specified that you were alone in your office, but you didn't specify that for the shower. I just, I'm just going to make note of that. Yeah, listen, I, you know. Listen, I said what I said.
Oxide and Friends
RTO or GTFO
And it didn't require in-person collaboration. I think there's this romantic idea we have of cross-pollination, too, where it's maybe not you struggling to solve a problem at a whiteboard that couldn't have been solved elsewhere, but rather it's the lunchtime conversation or
Oxide and Friends
RTO or GTFO
You walk and get in coffee, or you're doing a drive-by and you overhear something, and that's what spawns this moment of inspiration. I even struggle more to come up with those kinds of examples where
Oxide and Friends
RTO or GTFO
Let's wait until this place clears out and it's on. We're racing these things. I do like this idea of like, instead of fighting return to office mandates, people should instead say, we should return to office and get back to the days of Z-ball and chariot racing. as sort of like a way to undermine the return to office mandate. Like, no, we're on the same page.
Oxide and Friends
RTO or GTFO
Does this cross the species boundary into your MacBook? Maybe. I mean, I think we all went through this experience, and maybe we still are, of our coworkers doing some upgrade and then making some hand gesture and be like, what is going on? Where is that coming from? Yeah.
Oxide and Friends
RTO or GTFO
I don't know. I think I'm, I'm, I'm already like, so skimming over the part of like, just what, what is happening? Where are we going?
Oxide and Friends
RTO or GTFO
Oh, clear founder mode, right? Getting rid of managers? Well, I guess there are two ways to improve any kind of ratio, right? Improve the numerator and improve the denominator.
Oxide and Friends
RTO or GTFO
Got it. You're creating potentially some perverse incentives in empire building to encourage the manager mode you've already embraced. Exactly.
Oxide and Friends
RTO or GTFO
I mean, it would be like the most founder mode end to this email would be, I myself have terminated 15% of my reports.
Oxide and Friends
RTO or GTFO
That was already in place. Yeah. I mean, that was like a tenet of management by the time I got to Sun, I believe.
Oxide and Friends
RTO or GTFO
I think we're strong on leadership and a little bit light on follow-through. Very light on follow-through.
Oxide and Friends
RTO or GTFO
What I appreciate is he is putting his arm around us, walking us right up to the edge of the cliff. which we think he's going to push us off. And then he's like, actually, could you just come to the office more? Like, Oh, thank you. You're not pushing me off the cliff. It's like, no, I'm not pushing you off the cliff. No, it's not like that at all.
Oxide and Friends
RTO or GTFO
No, I mean, you figure like the folks for whom going to the office three days a week was great. They're maybe going four or five days a week. Like if you're actually seeing that benefit, you know, and so much of it comes down to trust.
Oxide and Friends
RTO or GTFO
Like if you trust your folks are doing the right thing for themselves and for the team and for the product and for their teammates and so forth, they're doing what they need to do. They're going in four or five days a week, whatever.
Oxide and Friends
RTO or GTFO
But for the folks who are going in three days a week to sit on Zoom calls, for the other folks who are in three days a week in some other location, it just sounds crazy.
Oxide and Friends
RTO or GTFO
Yeah. I mean, all about seaball and cherry races. I mean, or, or it could be, maybe we'll have a dedicated episode for that, but yeah. But now having read the full context, I'd love to hear what other people are seeing in other organizations and how things are changing if there's a return to office mandate or if there's something more progressive and what's motivating these things.
Oxide and Friends
RTO or GTFO
yeah there's a there's a whole litany of them double thumbs up like gives you some uh some like light show or no pardon me that's if you do like double um i tell my son it's double quiet coyote because like that's the hand signal he knows but like double kind of like rock on double quiet coyote crisscross applesauce yeah exactly yeah
Oxide and Friends
RTO or GTFO
We'll split the difference. Take half as vacation. Take them unpaid. That's fine. It's not vacation.
Oxide and Friends
RTO or GTFO
Amazon is a lot of things, but I don't think they're stupid. And just letting some subset of your employees opt out is pretty stupid. It's an opportunity to make sure that the least productive employees stay and the most productive employees walk out. It is.
Oxide and Friends
RTO or GTFO
Hmm. I think there's something to that. I think there's something to that. I think you're right about the emotional aspect of it and that sort of shock of like, where is everyone? I think there's a lack of trust, which I would also chalk up into like sort of a paranoia. But I think there may also be like a financial aspect to it, like a sunk cost fallacy. Like we're paying for all this.
Oxide and Friends
RTO or GTFO
Yeah, I don't think it's crazy at all. I think, yeah, I think there's something to it. I don't know how to sort of evaluate that. I wonder how we'll kind of shake that out over time, but I can definitely keep it in mind.
Oxide and Friends
RTO or GTFO
Yeah, exactly right. But Matt, I don't want to tee up too much. So just what's on your mind?
Oxide and Friends
RTO or GTFO
I could watch hours of that. The reaction of someone accidentally responding with balloons and then... I mean, it's kind of mortifying, right? Like in a business setting. It's also like, Steve, what did you just do?
Oxide and Friends
RTO or GTFO
And so Matt, what's the reaction been as this becomes more enforced? Like are, I mean, like are there some folks who are receptive to it and think it would be a positive change or are most folks, especially from the VMware culture, is it bugging them?
Oxide and Friends
RTO or GTFO
Yeah, this is an intervention. Right. Oh, thank God there's a dress code. God, they did keep that.
Oxide and Friends
RTO or GTFO
Has anyone reached out to you? Have you gotten any? That's a great question. I was like, wait, hold on. It wasn't the RFD one. Oh, it's like, no, the one where I confessed about my
Oxide and Friends
RTO or GTFO
There's a real clarity, right? Like he's not talking out of both sides of his mouth. It's not a 18 paragraph email. It's like my way or the highway. Let's go.
Oxide and Friends
RTO or GTFO
Actually, you know what? We weren't that remote. We were pretty in-person. And we got an expensive office space in downtown San Francisco, and people went to it. We had some remote time. People would work remotely on Wednesdays or whatever. But by and large, it was in-person. I think I said at the time...
Oxide and Friends
RTO or GTFO
I was looking at what you had done at Joyent, and I thought you had done a great job of having some folks in person and some folks remote. And I don't know. I wasn't sure how to operate in a team like that, especially very early on. Now, I think that's changed for me. That's changed, I think, for the industry broadly. But even in that 2016, eight years ago when I was starting the company...
Oxide and Friends
RTO or GTFO
I felt like that was a choice you could make between whether you were mostly in person or mostly remote and you'd hire different folks or whatever, but it was kind of a viable choice. I'm not so sure how viable this is a choice these days. To be remote or to be in person? Oh, pardon me, to be like, to say we're in person.
Oxide and Friends
RTO or GTFO
Obviously, I think being remote is like, I mean, it feels to me, I'm like very much biased on this one, but it feels like,
Oxide and Friends
RTO or GTFO
Yeah, you just closed your eyes, put my hand on the wheel, and stepped on the accelerator, and off the cliff we went. I did. I did.
Oxide and Friends
RTO or GTFO
I just want to make that clear. No, for sure. You're like, wow, Adam, I loved reading the thing you wrote. The thing I wrote. The thing I wrote.
Oxide and Friends
RTO or GTFO
Chris, I think you're really onto something. Yeah, I think it is. I think it's both a generational thing and kind of one's own experience. But I think it's also the tools that have become commonplace in the way that we work. And yet people sort of having this nostalgia for a time and forgetting the absence of these tools. Brian, you were talking about how...
Oxide and Friends
RTO or GTFO
you know uh talking in chat or on a google meet or uh in lots of these different kind of remote collaboration tools recall like we didn't really have that like when we would when we would work remotely at sun in the like early 2000s or whatever like you would get on a conference call but like there was no uh there was no like video chat or anything i think we did a little bit of like
Oxide and Friends
RTO or GTFO
I think we were on like AOL instant messenger and stuff like randomly.
Oxide and Friends
RTO or GTFO
I think, like, why would you use that for? That doesn't make any sense. I feel like people are going to, you know, If someone comes to my house and like rings the doorbell, like my older son who's now moved out, but like didn't know how to react, like total panic that like somebody would show up at your door. I feel like that's how people are going to regard email. Like I'm receiving an email.
Oxide and Friends
RTO or GTFO
There are things I cannot do for you. Do you know how long our last episode on this topic was?
Oxide and Friends
RTO or GTFO
right and there's like a fatigue thing i think like for for commuting too like i think it just grinds on you uh you know even if you're working you know less hands on keyboard or whatever i think when i personally like have to take public transportation in terms of commuting sorry man what were you saying no i was gonna say i think it's actually interesting where i don't know if it's changed like specifically oh well at this time i walk out but i think it has eroded goodwill
Oxide and Friends
RTO or GTFO
This example you're coming up with, I remember this moment in my career in particular where it was like Sunday night, I was working on getting something done and I got an email from my manager that said, hey, you took a day off on Tuesday and you didn't record it properly. And here's how you record it properly. And here I am Sunday night working my ass off.
Oxide and Friends
RTO or GTFO
And I just remember feeling like just so enraged by that moment. Right. Yeah, and that was at Sun. No, no, it actually was at Delphix. It was at Delphix. That makes a little more sense. Yeah, yeah, yeah. The folks at Sun wouldn't be paying attention. Yeah, exactly.
Oxide and Friends
RTO or GTFO
And those remote access token cards, those one-time password cards, was somehow load-bearing for a long time for how some IT would let us get in remotely. Yes.
Oxide and Friends
RTO or GTFO
I think my critique of remote work at the time was that it was very haphazard. Because the benefit of going to the office was conversing with certain folks or collaborating with certain folks. But then if it turned out like, oh, they didn't go in that day. They didn't go in that day. That's right. Then you sort of feel like, why did I get on the train? What am I doing here?
Oxide and Friends
RTO or GTFO
Yeah. Well, you know, we talk about Oxide. We got, you founded Oxide. as it turns out, sort of moments before the pandemic. Moments before the pandemic, yeah. And I do wonder, if we were starting Oxide today, what would the office situation look like? Or if you were starting it in March 2020 rather than late 2019, Like maybe would have got an office.
Oxide and Friends
RTO or GTFO
Maybe there would have been a place that we needed to like warehouse materials or do some of this bring up or whatever. But it would certainly have looked differently because when you got that office in 2019, sure, you thought we're going to have whatever it is, 50% remote, 20% remote. But you envisioned a building, a room with a bunch of people in it. Like that's what it was for.
Oxide and Friends
RTO or GTFO
That's right. Operations. A bunch of these double E's had to be. Yeah, totally. You're right. It looks very different and I'm not sure we are. Yeah.
Oxide and Friends
RTO or GTFO
And that I don't think he gets. So two and a half years ago when we were talking about this, Matt was on and he said that he hoped remote work wasn't just viewed as a cost savings. I think that at the time people were like shedding their offices and feeling like that was a huge benefit. Not just a cost savings, but instead a way to hire the best people, as you're saying, Brian.
Oxide and Friends
RTO or GTFO
And I think also in that episode, speaking to a question in chat about interns and something, Chris, you had mentioned about, I think, some of the generational differences. One of the questions we had was, what's it like for new employees? What's it like for new college grads coming in? And I think at the time, I was really worried about it.
Oxide and Friends
RTO or GTFO
And I feel like that is maybe my OK Boomer moment, because these folks... Especially if you're a student who's graduating from college and you spent a big chunk of your time remote during the pandemic, you know how to do this. You figured it out. This is how you operate. And so for the person in chat asking about the college intern starting, I think you meet them where they are.
Oxide and Friends
RTO or GTFO
You figure out how they want to work. Because they've already figured out how to do a bunch of this stuff. And yeah, for me coming out of school, it was so important to sit at the lunchroom table. It was so important to stumble into people's office and ask them questions or whatever.
Oxide and Friends
RTO or GTFO
But I think that different generations figure out their own ways that are going to be different than the ways that we were familiar with doing them.
Oxide and Friends
RTO or GTFO
Yeah, I do think it's a trend. I think that you're onto something with regard to this emotionality and justification of one's role. I think the finance folks who have invested in a bunch of real estate, I think it's the confluence of these things is going to keep pushing for these RTO mandates and then
Oxide and Friends
RTO or GTFO
whether it turns out to be like hocked hands my way or the highway or a softer mandate, but don't enforce like Scott McNeely would do if you were still in charge. I think that's a big question for me. What do you think?
Oxide and Friends
RTO or GTFO
No, that was because there were too few people in the office. It was like semi-abandoned.
Oxide and Friends
RTO or GTFO
Yes. In fact, that was the place where I heard about catalytic converter thefts first. Yes.
Oxide and Friends
RTO or GTFO
They were ahead of their time. It was in the building 17 parking lot. Now –
Oxide and Friends
RTO or GTFO
Yeah, yeah. I mean, I think in part this podcast was like an accidental consequence of that missing socialization. I think you are absolutely right.
Oxide and Friends
RTO or GTFO
home to meta you're welcome uh i'm sure they don't have the same problems but like as people like stop you know the the remote work was sort of like self-perpetuating because once you go in a couple times and the people you wanted to see weren't there then you're like well i'm just gonna stay home too people didn't show up they started turning off the lights to save power and people's catalytic converters started getting stolen it was very it was very bleak it was very bleak
Oxide and Friends
RTO or GTFO
Talk about a podcast I would listen to. That'd be great. Well, low bar.
Oxide and Friends
RTO or GTFO
um just uh don't let this get you too far down but um this is um it's it's it's a little bit of a little bit of a downer at the moment yeah but agree i think this too shall pass and i think that companies will distinguish themselves by being remote friendly even as in this trend then hey go work for nvidia Go work for NVIDIA.
Oxide and Friends
RTO or GTFO
Yeah, and it should be said, we're talking about remote work. So at the time, we are now in the future. Oh, you've given it away now. I mean, this feels like at the top already? I mean, are we going to just write into it? A lot of context.
Oxide and Friends
RTO or GTFO
I feel like that is the story. Like I think my greatest point of like socialization with other adults is like kids' birthday parties. Okay, that's just the phase of life I'm in. But like that is like a story at everybody's like, you know, everyone has that story, right? Everyone has the story of like, yeah, I have to go to work.
Oxide and Friends
RTO or GTFO
And it's kind of dumb because when I'm home, I'm on Zoom calls with these folks. And when I'm at work, I'm on noisier Zoom calls with these folks in much less convenient conditions and so forth. And I've wasted or spent, depending, the time to get there. So I think everyone's, whether it's Amazon or kind of you name it,
Oxide and Friends
RTO or GTFO
Everyone feels that kind of like Sisyphean, like I've got this, I'm in the office, the boulders at the top of the hill. Now I'm on a Zoom call with a person who's at a different office. Thanks very much.
Oxide and Friends
RTO or GTFO
The market game was great. Did you play the market game much? Oh, of course. Like, over the light. I mean, we should do an episode on, like, in-office games. I want a call-in show where people do their best to describe their in-office games.
Oxide and Friends
RTO or GTFO
I mean, Unshared insofar as like, what, there's only one long wiki in the public? Oh, you mean on this podcast? Yeah, I mean, I'm sure we'll have a fishpong. At this point, fish pong will be an Olympic sport.
Oxide and Friends
RTO or GTFO
There we go. I look forward to talking about Z-ball. I'm just going to leave that as a teaser.
Oxide and Friends
RTO or GTFO
We had a weekly Detroit meeting. I want to say. That's right. That's right. That conference room is now like a graffitied wall in Facebook. I just want to tell you what happened to Nana's estate. Yeah. It got, it got turned into a big mansion. Sorry. Oh,
Oxide and Friends
RTO or GTFO
Yeah, I mean, or even better, don't, I wouldn't do that. Don't do that. You don't need to do that. I thought it was good, actually. I thought it was a good discussion. I liked it. It was good. Our audio was atrocious. It was Twitter spaces. It was Twitter spaces, and it would have been about half an hour shorter if we weren't discussing the various audio problems we were also experiencing.
Oxide and Friends
RTO or GTFO
No, not a ping pong ball. It was a foam ball. This is really dating it. Do you remember cars used to have antennas? It's really true. Go on. It was a foam ball. Would you put your phone on top of the antenna? Or is this to get Wi-Fi in the car? Oh, God. Where do I start? Yeah. Let's just say yes.
Oxide and Friends
RFDs: The Backbone of Oxide
So when you first load the page, you get the spinner for the comments because it's fetching those in the background. So that's why it feels fast. So it's kind of fake fast.
Oxide and Friends
RFDs: The Backbone of Oxide
Yeah, yeah, yeah. Yeah, for sure. Oh, definitely. And wouldn't they love to have our beautiful styling too? Everyone should be so lucky to have this gorgeous styling.
Oxide and Friends
RFDs: The Backbone of Oxide
I wanted to ask Augusta something. So something I'm not sure if we've really gone into is how it works in the GitHub repo. So the whole thing is when you make a PR, creating an RFD, that is a branch that's named with the RFD number. And so when the RFD site is scanning the repo for all, I mean, when the API is scanning the repo for all the RFDs, it's scanning all the branches.
Oxide and Friends
RFDs: The Backbone of Oxide
You can't just look at main. I'm curious how coupled the RFD API is to that model. We run into some friction with that, even as nice as it is to have PRs as the site of discussion. You could imagine not wanting to do it that way. How coupled is it to that?
Oxide and Friends
RFDs: The Backbone of Oxide
The fact that we're standardized on all these tools means that it's sort of this flywheel. Because we use, for example, like you said, the ASCII doc rendering stuff on the doc site. It's sort of the investments that we make in internal tooling also go into the product itself, which is really a good way to be.
Oxide and Friends
Reflecting on Founder Mode
you've got to fill the car with gas, but a road trip is not a tour of gas stations, which I really like, by the way, it's delightful. And he was like, but people only heard a road trip is not a tour of gas stations. That's right.
Oxide and Friends
Reflecting on Founder Mode
And knowing when, I think one of the dangers of just following expertise is that you're cargo culting without understanding the rationale. You are doing the things that you're being told to do without understanding whether they apply to your business, how they should apply.
Oxide and Friends
Reflecting on Founder Mode
And then I've also seen this in folks doing things at one company, coming to a new company, trying to do it the same way, and not understanding the differences that made them successful that then make them unsuccessful in the new company.
Oxide and Friends
Reflecting on Founder Mode
Yeah, I was, I mean, it is a narrow joke. I mean, I just, I didn't know.
Oxide and Friends
Reflecting on Founder Mode
Especially because as we, as we've talked about years ago on the show, like Steve jobs, I think is not fully understood or appreciated in, in terms of like how he operated when, when he succeeded and when he failed.
Oxide and Friends
Reflecting on Founder Mode
In terms of what you were saying about experience, I just feel like these reductive views of Scully versus jobs or founder mode versus management mode, Like you want founders who are thoughtful and it's sort of like also good at ignoring things. I mean, I think one of the, one of the spoke, I think briefly to this kind of notion of paralysis.
Oxide and Friends
Reflecting on Founder Mode
If you're like, I want to get all of these things right. It's impossible. Yes. So you need to take shorthands for some of them. Some of the shorthands are, I'm going to do it my way arbitrarily. I'm going to do it the way that it's always been done arbitrarily.
Oxide and Friends
Reflecting on Founder Mode
Not that either of those is correct, but then focusing on the ones that are key to your business and they're taking a blend of like what you think is right for your business, but then learning from experience.
Oxide and Friends
Reflecting on Founder Mode
Absolutely. And I think, yeah, I mean the other side of it of like, this won't scale when we're 10 people, a hundred people, a thousand people. They have been in those arguments. I remember one in particular, um, where there was a cranky customer. They weren't getting the performance they wanted out of a thing we had sold them.
Oxide and Friends
Reflecting on Founder Mode
I'm like, well, what if we just give them enough of the thing that we sold them to meet their performance needs? Not at Oxide, by the way. Cherish customers of Oxide. To be clear. And... Also, it was a software product, so giving them twice as much was sort of fine or whatever. And someone started freaking out about revenue recognition when we're a public company.
Oxide and Friends
Reflecting on Founder Mode
This is a company, by the way, that never went public. This is a company that got acquired 10 years after this conversation by private equity. But being able to say, yes, this doesn't scale. Yes, when we are being held to certain standards... You know, by our public investors, we can't do this. Also, it's today. And like, let's just get through the day. Right.
Oxide and Friends
Reflecting on Founder Mode
I mean, first, I mean, first three reactions were, thank God we're not. And they got our brain next week. And you're saying, and you saying like, you didn't want to talk about it. I was like, I'm just going to mute notifications against Brian DMS me or something. I'm just, I'm off the grid. It felt so. So Paul Graham's piece is,
Oxide and Friends
Reflecting on Founder Mode
I remember at Delphix, I joined this company. It was like 20 people at the time. It was the first startup I'd been at. And what I found really interesting startling was the degree to which when we were hiring people, we wanted to see people who had done that exact thing like two or three times. And I sort of thought, oh, we're going to be this scrappy startup.
Oxide and Friends
Reflecting on Founder Mode
We're going to take chances on people. We're going to... But it was clearly not part of the risk budget to find people who were just hungry and we thought could do the thing. And I think there was some, maybe it was just where the risk budget was, and there was some strength to that thought. There are also a bunch of downsides to it.
Oxide and Friends
Reflecting on Founder Mode
We found people who had done it before and maybe weren't hungry to do it again or had forgotten the drive that it took to do that again.
Oxide and Friends
Reflecting on Founder Mode
felt so nonspecific that it was going to be inevitably used as a justification for founders to do whatever they wanted. Yeah. To rationalize whatever sort of like terrible decisions were being made or whatever cultural decisions had been made with very, almost a justification for a lack of introspection. And I mean, that's probably what rankled you so much too.
Oxide and Friends
Reflecting on Founder Mode
Yeah, I mean, it's a very expensive mistake. And it's a big bet. Maybe it's the right bet. But if... that person and that team and that other company match things up really well. But if it doesn't match up, if it's, if it's sort of surface level similarities, then you've just onboarded a big team and you know, it's, it's going to take a lot of money to then undo that.
Oxide and Friends
Reflecting on Founder Mode
I do think there can be, for founders in particular, there's going to be pressure from the board. The pressure from the board is hire someone who is an expert in this domain to de-risk the company, this thing that I've put a bunch of money into. Part of that is the whole hire and retain search firm. They go and find the people and plugging someone in.
Oxide and Friends
Reflecting on Founder Mode
So then after you as a company have spent money on the search and money on an expensive person, then you feel really misled Yes. You told me to just like hold my nose and hire this person who, and with faith in their expertise, and then it didn't pan out.
Oxide and Friends
Reflecting on Founder Mode
To some degree, I feel like the fakers may be the least culpable. Absolutely. Because I don't think fakers, I mean, maybe this is naive, but I think most people are basically honest or try to be. and good people. I think there are very few people who try to represent themselves as one thing to get into a startup, get underpaid, and get some stock compensation, and then try to take the company.
Oxide and Friends
Reflecting on Founder Mode
It's just like it, that that's what grinds so much. The, the, these, these kind of little aphorisms that people say, I'm going to do it my way because conventional wisdom is tautologically busted and therefore founder mode. Therefore, founder mode.
Oxide and Friends
Reflecting on Founder Mode
None of that makes sense. I think by and large, although I can think of some examples.
Oxide and Friends
Reflecting on Founder Mode
I remember in particular, there was one example who was telling me they could sell all this stuff at a previous company and they were going to be amazing. I saw that they had worked with you, and I asked. I was like, clearly this person doesn't tell 100% truth, but I'm not sure it's 100% false either. Oh, yes. And I think you told me it was closer to 99.8% false, but you're right.
Oxide and Friends
Reflecting on Founder Mode
It was like 0.2% true. It is 0.2% true. But this is like a relatively rare example of someone who's actually like... A flim-flam man. I think mostly people show up thinking that they did the thing, that they can do it again, that it's a good fit.
Oxide and Friends
Reflecting on Founder Mode
Absolutely. It is falsehoods that they are promoting. Maybe liars is fine. To some degree, everyone believes their own story. It's up to you, the founders, the board, the folks interviewing these folks to figure out what's true and what's not. That's right. That's true of every hire to some degree. Everyone comes in with some degree of divergence from reality. For a lot of people, it's small.
Oxide and Friends
Reflecting on Founder Mode
For some people, it's bigger, whatever. But especially for a high leverage, for a key role in the company, when you're hiring VP of whatever, you've got to be careful.
Oxide and Friends
Reflecting on Founder Mode
Yeah, I mean, that's what you'd hope that you can get from your board. And sometimes you can, sometimes you can't, but whether it's directly or plugging you into other founders or other leaders to help educate yourself.
Oxide and Friends
Reflecting on Founder Mode
is actually, how do you get that wisdom as opposed to just discarding it? And not get it as the answer. What they say is not the answer, but it's part of what goes into the calculation.
Oxide and Friends
Reflecting on Founder Mode
Especially, as you say, when you're hiring into domains that were previously not needed at the company. When you're hiring your first person in support, when you're hiring your first program manager, your first product manager, whatever.
Oxide and Friends
Reflecting on Founder Mode
I hope. No. The sort of mystical reverence that he speaks for.
Oxide and Friends
Reflecting on Founder Mode
But I think... You know what? I'm glad that we... we had this cooling off period to not just go into the hot take of, of, you know, founder versus management mode. But I think like there, there is, there's like a kernel of wisdom and certainly the kernel of a discussion here about like, when do you value experience and why not?
Oxide and Friends
Reflecting on Founder Mode
Yeah, you sent me a link to that YouTube video. We'll keep that in the show notes. And I watched the first 45 minutes or something of it. I thought it was great. I thought that what I heard from him was... he deferred too much to folks. He absorbed, in particular, the wisdom of hire great people and set them loose and let them do their job and don't mess with them.
Oxide and Friends
Reflecting on Founder Mode
And then found that a lot of the things that he cared about about the business, about the health of the business, were not going well. And then he drew this distinction between
Oxide and Friends
Reflecting on Founder Mode
micromanagement and getting involved in the details yeah in particular you know he kind of made some very significant changes to the way they did product management he got really in the details understanding how these different functions operate um i would be interested to know if if the facts on the ground draw this distinction between micromanagement and being in the details Right.
Oxide and Friends
Reflecting on Founder Mode
I mean, who knows how it plays on the ground. But I thought... Certainly, as a founder, as a leader, there's this fear of losing control. This fear of when I hire someone and I want them to feel responsible. I want them to be able to operate with autonomy. I have trust. I want to build that trust and I want to hand over that trust and operate with trust.
Oxide and Friends
Reflecting on Founder Mode
And that can be scary where you say, and then tell me in three months or six months or 18 months how it went.
Oxide and Friends
Reflecting on Founder Mode
That's a great point because I think there is a... People like to provide autonomy, or at least it sounds nice, right? I trust the person, I'm going to provide autonomy. But if you don't show them what true north is, Yes. Then everyone will try to figure it out and they'll figure it out a little bit differently and you're not going to be happy with the results.
Oxide and Friends
Reflecting on Founder Mode
It was great. I think that's great. I think when there's autonomy without as strong a sense of true north, you can get people unclear about prioritization. With so many things, especially at a startup, When it's like, actually for any given task, there's no owner. It's not that everyone's standing shoulder to shoulder and everyone has their lane. For any given task, there might be no owner.
Oxide and Friends
Reflecting on Founder Mode
So knowing which of the 25 things I could do, should I prioritize? And how does that tie to what the objectives are? And when objectives are changing, as they will, how do I know it's fine that I'm still doing this other thing?
Oxide and Friends
Reflecting on Founder Mode
I can confess. I was a little glad we had a hiatus. That piece came out. You wrote a nice blog post. Other folks were chatting about it. I was glad. For a moment that we'd have to talk about it. Okay.
Oxide and Friends
Reflecting on Founder Mode
Yeah. You know, this is actually almost exactly the pathology at my wife's last job for her that caused her to leave. And part of what led to it.
Oxide and Friends
Reflecting on Founder Mode
But that came about in her organization because of growth. Growth where leaders previously did have the bandwidth to have the level of involvement that they wanted to be able to be comfortable with all these programs. And then as the organization grew, they didn't have the time to do it. They didn't have the bandwidth to do it. There was too much stuff going on.
Oxide and Friends
Reflecting on Founder Mode
And they had not taken the lesson that they needed to like changed the way that they interacted with some of these priorities.
Oxide and Friends
Reflecting on Founder Mode
a buddy of mine said of like, as he was assigning the ranks at a bigger company, he said, you know, the more you get promoted, like the roads tend toward CFO, like all roles, which I thought was a great insight. Cause his point was, you know, you, you start losing the ability to, to be involved with projects at a level of detail that matters.
Oxide and Friends
Reflecting on Founder Mode
And all you really get to do is pull the levers of who gets money, who gets staffing. Those are the kinds of things that you have control over.
Oxide and Friends
Reflecting on Founder Mode
I mean, first, my understanding is that people write a memo of five pages or something. I think that's of a fairly prescribed length. And then people sit down in the meeting, they read it. I guess they raise their pencils when they're done or whatever and discuss it. Slowly, the hands go up. God, I'm the slowest reader in here. It's like, oh, boy. You come 15 minutes early. That's right.
Oxide and Friends
Reflecting on Founder Mode
But I think it's great. I mean, I think it also... Well, they famously write the press release too. Oh, yeah.
Oxide and Friends
Reflecting on Founder Mode
I'm going to go do a deep cut here. Another place where we read about this was one of our favorite books. by Dave hits. Oh, out of cast rateable. He talked about, does he writing the future history, the future history. Now, dear listener, do not purchase this book.
Oxide and Friends
Reflecting on Founder Mode
Anyway, terrible, terrible autobiography. I feel comfortable saying this now. Uh, but do we, do we feel totally comfortable? Not fully comfortable.
Oxide and Friends
Reflecting on Founder Mode
That's right. We were trying to build a product to compete directly with NetApp. Yes.
Oxide and Friends
Reflecting on Founder Mode
Yeah. I mean, the term manager mode feels like so easily pejorative. You know what I mean? Oh, it's super easy pejorative. Yes. Like managers are the enemy. Yes. It's like, do you want to be manager? And as much as founder mode is such like a great mean nothing aphorism that you could use to potentially justify behavior, management mode is such like a Just such a burn, right? It is such a burn.
Oxide and Friends
Reflecting on Founder Mode
I also have a signed copy. So what is the genesis of this? So first of all, we wanted this book. We, the team at Fishworks. Now, Oxide is thrifty. Fishworks was cheap. Yeah. And so we were going to pay like the $18 or whatever those books.
Oxide and Friends
Reflecting on Founder Mode
That's pretty much right. So I contacted the publisher. I used true but misleading statements about how I might want to teach a class. Are you reading from a prepared statement right now? Yeah, I need my lawyer. That's right. I'm not going to go pro se on this one. So we got a copy. And then actually, I don't know if you know this, but... like the FedEx got delayed or something.
Oxide and Friends
Reflecting on Founder Mode
And I sent like a nasty gram to the publisher. And so they sent us two copies.
Oxide and Friends
Reflecting on Founder Mode
Um, I think by this time it was out, but I, I, I just, I, I think I, I needed a copy. You needed a copy, obviously, and you're not going to pay for it. Okay. Not going to pay for it. Right. So, uh, so I read it, I started reading it and it was terrible.
Oxide and Friends
Reflecting on Founder Mode
Right. We would bring it in. So what we started writing a fan blog, I started writing when, where's the germ of this idea? It's such a good idea. I think, first of all, it was like a distraction from our ping pong variant. So we needed something else to do with that other than working on the product. So I think we felt like this personality was someone who was so susceptible to praise.
Oxide and Friends
Reflecting on Founder Mode
No, that's a little, that's a homework assignment. That's a good one. So Dave Lightman wrote a chapter by chapter review, like overly praising concern of embodying like much too much of, do you have the link? Yeah.
Oxide and Friends
Reflecting on Founder Mode
Okay, so at what point do the lawyers... Oh, that's right. So I took the design of the blog from the NetApp blog. Gone, yes. So I made it look exactly like the NetApp blog of the time.
Oxide and Friends
Reflecting on Founder Mode
And then I did get some sort of email from the lawyers saying you can't have a fan. Maybe it was an email from Dave Hitz. I can't remember. Yeah, it looks like it was. So email from Dave Hitz saying love the blog. got to change the way it looks. Our lawyers are pretty sure you can't just rip off the look and feel of the NetApp blog.
Oxide and Friends
Reflecting on Founder Mode
Chapter one, Dave hits as... I mean, spoiler for everyone, but please, again, do not read this book. He starts... I mean, I'll give you my copy. Seriously, just hit me up. He starts the book... With the advice, never listen to your mother. Right. That was like chapter one. I think it might have been the title of chapter one or something. Right.
Oxide and Friends
Reflecting on Founder Mode
It's like if you were saying something in a meeting and someone was like, oh, nice management mode there. Yeah, be like, oh, never mind. I'm leaving.
Oxide and Friends
Reflecting on Founder Mode
Which is just like, God, he's just putting the ball on the tee for you. Just bait. Right. So then my persona here talks about blowing up his mother. It's like, thank God. It's fun at some point getting some good advice from an adult. That's right. So, yeah. And later on, the big reveal at the end was like, sometimes listen to your mother or something.
Oxide and Friends
Reflecting on Founder Mode
I mean, I don't know. I like when you, when, I mean, it's so, it's so soft work too, but just like, it's so, it's so great. We've put this, we've launched this little piece of cheese out into the internet.
Oxide and Friends
Reflecting on Founder Mode
I just, I do want to, uh, yeah, I just want to observe like you, you actually wrote some of these entries and I can't remember which ones, but just to throw you under the bus with me. Oh, I will definitely be under the bus with you. Cause when I, and like the Federalist Papers.
Oxide and Friends
Reflecting on Founder Mode
I can't believe blogspot is still a domain that resolves. Uh, so, uh, anyway, short story long. Uh, so I, I did meet Dave hits. I, I got him to, to sign my copy of the book.
Oxide and Friends
Reflecting on Founder Mode
Got him to sign the book. I said, you know, my buddy can't make it, Dave Lightman. Would you sign it for him? And he said, what's the name of his blog again? And it was, I mean, it speaks volumes that this was sort of like the pinnacle of my pranking. I don't know. Like it was just, it was just, it was so funny to me.
Oxide and Friends
Reflecting on Founder Mode
That one of the practices at NetApp that I remember from a very careful reading of this book was that... There has never been a more careful reading of how to castrate a bull.
Oxide and Friends
Reflecting on Founder Mode
I still have that copy. It's got notes from at least half a dozen people in the margins. Yeah. It's like, yes, it's ironic detachment.
Oxide and Friends
Reflecting on Founder Mode
Spot on. And I think that... I'm sure you've seen this from VCs, like... There's so little weight often put on experience and expertise and the past. I've talked to VCs who say, I only want to work with founders who can conjure things from first principles. And I think there is something valuable to that. There is something valuable to saying, I'm just going to educate myself on a topic.
Oxide and Friends
Reflecting on Founder Mode
There is one, one, one post-it I want to mention on this. So, uh, we were brought in for you and I, uh, or rather, um, NetApp Sue's son, son Sue's NetApp. Yes. Sort of a thing you may recall. Oh my God. Yeah. Huge lawsuit. And you were brought in, you, you, you, you had a delightful deposition. I was brought in for a deposition as well. And I confess that I was worried. I was worried.
Oxide and Friends
Reflecting on Founder Mode
I'd done a bunch of work in CFS, like in Red Sea and stuff like that. That's true. They actually should do a W versus me. There you go. They took my notebooks in Discovery and like, good luck with that. Joke's on you, pal. Literally, page one of the notebook was this whole description about how we were going to do RAIDZ expansion. So I'm like, oh, dang.
Oxide and Friends
Reflecting on Founder Mode
If I just put it on page two, you would have never found it. Right, right. So I'm in this deposition thinking, if they're like, Mr. Leventhal, do you have a blog? I'd be like, yes. Also, I need a break right now. I need a break. I need to talk to my lawyers. Right.
Oxide and Friends
Reflecting on Founder Mode
when you go into a deposition, uh, you know, you're supposed to basically tell your lawyer everything, right? You should, they should like, because they're tell sons lawyers.
Oxide and Friends
Reflecting on Founder Mode
Didn't tell the lawyers. I was waiting until the lawyers needed to know and then I was going to tell them in a fake heart attack.
Oxide and Friends
Reflecting on Founder Mode
What do you want the success to look like? Right. Like sitting here from the beginning, sitting here from the inception of the idea, what do you want it to look like at the end? What's the story you want to be able to tell? And writing all that down. Exactly.
Oxide and Friends
Reflecting on Founder Mode
Here's what I'm going to say. A little Oxford critique, if you'll forgive me. I think there are some things that we don't write down. Oh, 100%. That maybe not only should be written down, but should be open for revision. So I don't want to do performance reviews of folks. I don't think that's something I've had to do in my career and don't want to do in my career.
Oxide and Friends
Reflecting on Founder Mode
There are people at Oxide right now who think that we should do that. And I'll tell you that because I've, I've spoken with them. Right. Yeah. I think even having the discussion, even having the rationale for why we do or don't do things.
Oxide and Friends
Reflecting on Founder Mode
Maybe sometimes it shouldn't be written down because we talked about our discussion of which chat program to use. Obviously, that's way too volatile to write down.
Oxide and Friends
Reflecting on Founder Mode
But it's just so dismissive of everything that came before. Oh, absolutely. And there was a time in my career when I got hit up by recruiters for this one particular role. And the role was usually founder CTO who couldn't manage 10 people. Now the team was 100 people. And they wanted to bring me in.
Oxide and Friends
Reflecting on Founder Mode
Yeah. And it's a good point that you make that that's the way we communicate those kinds of things. That is how our culture gets codified. And that's not true for everyone, right? Other folks are going to have their culture kind of work its way through the system in different ways.
Oxide and Friends
Reflecting on Founder Mode
Interesting. Just because you're too susceptible to things changing unseen or people not knowing what True North is.
Oxide and Friends
Reflecting on Founder Mode
And you know what? Maybe amended trust and clarity or everything because I think absent trust and absent clarity, you can spend a lot of cycles inefficiently. Yes. Worrying about whether you're doing the right things, worrying if you are, how you're being judged by your peers, like working inefficiently. I think it consumes a lot of caloric budget, as you might say, when you're unclear.
Oxide and Friends
Reflecting on Founder Mode
Clark budget, as I might say, as you say, no, I just mean people spend a lot like lack of certainty. I mean, I think it's one thing to say, Hey, don't worry about all this other stuff. Just focus on this thing.
Oxide and Friends
Reflecting on Founder Mode
Like people worry about that other stuff. For sure. If you've told them not to worry about that other stuff, like, yeah, uh, I think, you know, and, um, you know, I mentioned that the RFD episode, um, One of the things I saw at Delphix was everyone had really clear ideas about what we should go do. They were all different. Most of them were terrible.
Oxide and Friends
Reflecting on Founder Mode
But even just being able to get those out of their system, let them ignore them. Let them move on past them and just wash their hands of those ideas and move on to the things that did matter.
Oxide and Friends
Reflecting on Founder Mode
You're right. Founder mode is definitely more like... waving your scepter and zapping folks as opposed to like listening to them and wondering what they're about.
Oxide and Friends
Reflecting on Founder Mode
No, that's totally, that's right. It was kind of first adult mold, but almost to a person, the CTO would kind of describe the role as like, we've got this cake. I enjoy eating cake, but mostly I like eating frosting. So I'm going to lick off all the frosting. And then I thought you as the first adult would enjoy eating the rest of the cake.
Oxide and Friends
Reflecting on Founder Mode
I'm glad we took a little more of a lukewarm take on this one. Because I think there's something there, something worth discussing. And my knee jerk on this was not as helpful.
Oxide and Friends
Reflecting on Founder Mode
And William, that the buying in process was clearly them saying like, you'll still be able to like gorge yourself on frosting. Don't worry. Like it's like, Right. Like you, you only like the process. And I remember this one company in particular plaid where the founders talked to me, young guys. Oh God. No, they were great. They were, they were, they were fantastic.
Oxide and Friends
Reflecting on Founder Mode
And they were like, we want to hire people who can help us see around corners. Like we don't want to have to be like encounter every mistake that uh, the hard way. That is great. Yeah. We want to, we want to learn from the mistakes other folks have made. Uh, and I, I just found it to be so, and like, so terrific. I would say the Collison brothers are similar in that regard.
Oxide and Friends
Reflecting on Founder Mode
Actually, the way it turned out for me, well, Plaid turned out gangbusters. Right, exactly. I mean, put that in, I think I've talked about my poor financial decisions in the past. It turned out, as I thought about it harder, I was just not interested in the problem space. Every engineer I spoke with there was meticulous about personal finances, loved it as a hobby almost.
Oxide and Friends
Reflecting on Founder Mode
That sounds like a great fit for, no, wait a minute, that's not a great fit for me. Well, I was like, I've never balanced my checkbook. I only sort of know what that means to do as an activity. So I just realized that the problem space was not a good match.
Oxide and Friends
Reflecting on Founder Mode
Yeah. Unusual, especially like, I think these guys were in their late twenties at the time. Very unusual. I think. Yeah, exactly.
Oxide and Friends
Reflecting on Founder Mode
Well, it's so hard to know that you're drawing the right lessons from success or like that. Oh, for sure. Like that, you know, um, A couple of companies ago, Delphix, we talked to lots of folks from VMware. We were doing something virtualization-related and VMware was doing something virtualization-related.
Oxide and Friends
Reflecting on Founder Mode
There were lots of folks who were early but not super early at VMware who drew all kinds of weird result inferences from their success.
Oxide and Friends
Reflecting on Founder Mode
Yeah. There's a great line in that piece from Tim where he was talking about an interview with Michael Lewis where Michael Lewis said, you don't know the book you've written until people tell you what they've read or something along those lines. Does this have a Sam Bankman Freed kind of moral to it?
Oxide and Friends
Reflecting on Founder Mode
Yeah. Even the culture you're creating, like even the culture that we're creating here. Oh, for sure. Go figure out how new employees experience it to understand what it actually looks like.
Oxide and Friends
Reflecting on Founder Mode
Because even that has changed and you lose track of it over the years and how it's practiced.