RFDs--Requests for Discussion--are how we at Oxide discuss... just about everything! Technical design, hardware component selection, changes in process, culture, interview systems, (even) chat--we have RFDs for all of these, over 500 in a bit under 5 years. Bryan and Adam were joined by Oxide colleagues instrumental to RFDs, from their most prolific author to those making them more consumable.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Robert Mustacchi, David Crespo, Ben Leonard, and Augustus Mayo.Some of the topics we hit on, in the order that we hit them:We're sorry, GermanyOxide RFD siteRFD 1: Requests for DiscussionA Tool for Discussion (Oxide blog post from Ben)Sun PSARC casesThe Queen's DuckThe Hairy ArmJoyent RFDsRFC-3AsciiDocJoyent RFD 77OxF: Hiring Processes with Gergely OroszOxide RFD API... with it's CLI generated by progenitor... which we talked about some on OxF here and here"Own your strategic weirdness"RFD 113: Engineering Determination, or how we close out RFDsIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
How does it feel to be doing this from the morning?
Ben, great question. I got to say, the sun is pouring into this room at Oxide. I don't think I quite realized how much absolutely nuclear bomb direct sun exposure this thing gets in the morning. So I'm kind of like hiding a little bit in the shade, but it is glorious. I got to tell you, I got my morning cup of coffee. I'm really excited for this.
How does it feel for us to be doing this at a time that includes you? I'm so sorry. This is good. Can you hear me? Adam from Italy. Yes. Yes. I mean, don't take this the wrong way, Adam, but I think you, I think you sound better than you do in Alameda. I mean, I think so too. I have a upgraded microphone for, for this.
I just want to make clear that I'm not accusing you of using an AI-generated photo of Italy. I'm just saying the photo that you posted is a little on the nose for what a generative AI might imagine that Italy would look like. It's just a little too on brand.
If you spent any time in a European Airbnb, regardless of the country, you know that they always have a large canvas of a black and white scene of London with a colorized red bus. So I think that's the first clue that this is not actually Italy.
You just got some things that are not adding up. I do, I gotta say, I love the European window dimensions. I love the tall, narrow windows. That's nice. It's a very European thing. You know, I think it's like, that's how the AI knew to generate in Europe.
Okay. Yeah. I know another very European thing. When I was going through airport security in Germany, they pulled out this microphone and wanted to have a chat about microphones. Are they, are they listeners to the pod? Are they like, now they are. I, I, I'm just one person at a time. We'd be like, even to baseball one. Yeah.
I'm loving this. I have managed to offend all of our German listeners. We apologize, Germany. Exactly. A message to our German community from Oxide and Friends. You know, apology videos have kind of gotten... I'm trying to bring apology videos back. I just feel like, guys, I can't believe we have to have this episode. I never thought I'd be saying this, but...
Germany, I'm so sorry for what Brian did. I think I'm making it worse.
I am really excited for this. Me too. This feels like another long time coming kind of episode.
Really long time coming. I feel like this is like the episode we were in all along. I feel that... You know, I was listening, but actually I'm really glad that we are doing it now and not a couple of years ago because we have made so many... very recent changes that are really, really important.
And I also feel that one of the things that we have done, and we'll talk about this a bunch over this episode, but we have open sourced the software that we use to do this. And I think that if anybody's listening to this wondering like, God, I really like a lot of aspects of the system, but I don't know. I don't, I don't want to just like take theirs. It's like, you can definitely take it.
Um, we are, if they, and certainly if I may at risk of orienting people, what we are talking about. Oh, okay. Oh, now with this again. Okay. You know, it's like, we're only a couple minutes in, I was going to wait for another 10 minutes, but yeah, sure. Go ahead. Fine. Fine. Blow the big reveal. All right. Oh, what are we talking about? Mr. We have to tell people what we're talking about. Go ahead.
Meet the tweet. Yeah.
Mr. I read the comments in social media. we are talking about oxide requests for discussion, which is, you know, we were talking about what to name this episode and you stumbled onto, I think the backbone of oxide. And that is just so spot on.
It is the mechanism by which we make all kinds of decisions, decisions about the product, about the company, the way we articulate ideas, articulate research is really the durable entity that has been with Oxide from very, very early on.
Very early on. I mean, it predates Oxide, really. And we'll kind of get into the history there. But it really is the backbone, as you and I were kind of iterating on titles. I think you were kind of wanting me to go bigger on titles. And you're right. I mean, it's like, this is really important. And you said this in our episode on cultural idiosyncrasies as well, that this is idiosyncratic. And
It doesn't feel like it should be idiosyncratic because we did not invent literacy and we did not invent the importance of writing down one's ideas or reading the ideas of others. But I think a bunch of the things that we've done borrowing from ideas elsewhere, have added up to something that is idiosyncratic, but I think it shouldn't be, which is the reason that I kind of bridle with that.
But it is idiosyncratic. But regardless of whether it's idiosyncratic or not, it is essential to oxide. And it has been It is really, really, really important. It is really hard to overstate the importance of RFDs, and they're becoming way more important, actually.
In fact, the longer they live, I think is what you're getting at. Some of these ones that we wrote ages and ages ago have become really essential to understand where we were, which is not to diminish the value of the ones that we're still writing, but it's really proving its value as we look back and as these things age.
Yes. And the other thing that has been just extraordinarily important is the ability to make single RFDs public. And that has been... life-changing for us because it now allows us to begin to... And we've done this over time, but we are getting more and more RFDs out there and allows it for us to be that kind of public vector as well.
And so that has been absolutely essential and that's all built on the terrific work of Ben and Augustus and Dave Crespo who are all with us today. So their work has been... Like, it kind of started off as like, oh, like, this is nice. Like, oh, this is great. Like, this is great.
And then, you know, they, again, we'll get into the whole history of this, but that added things were just like, oh, wow, okay, this is really great. Like, this is profoundly great. And now we're just like, okay, this is... this is now like low bearing. This is, this is, uh, we now cannot envision the system without this. This is extremely important.
So, um, but if we may, well, if I may, if, well, uh, so I want to get in the history, but I, I, a quick quiz question for you. Cause I was looking back at RFD one, RFD one describes the RFD process. Yes. Uh, do you, do you, do you know when that was written? Because obviously it was very early in the company's history. Would you want to guess on when that was?
I will tell you that there's a reason that it's written. I mean, so Jess and I are listed as the co-authors because Steve was not yet working in the garage. So this is going to be in August of, I think, of 2019. Maybe we retroactively made it after September 9th.
The company started on September 9th, 2019. So the commit message was October 18th, 2019. So maybe you're waiting for a corporation.
I think we're waiting for a corporation.
There you go. Certainly before money was in the bank or lots of other stuff. But it was like, you've got to be one of the first things you did.
First things we did. Absolutely. The first thing we, the first thing we did and it actually felt like, because it felt like something we could do as well. So do you know what I mean? Where, because we didn't have money in the bank, we were totally raising around and there's like, you're kind of limited about how much of this company you can build with zero additional people and no money. And yeah,
you know, what are the kinds of things you can go do? And like, like you can do a podcast. We did, we were recording on the metal during that time. Right. And, um, But we were established. You look at like RFD2, I think, is principles and values, right? So that we were, RFD3 is our hiring process.
So we were getting, like, it's not an accident that these are kind of the first things we're laying down. And RFD1 is the RFD process itself. So, yeah, I mean, it was extremely early. And that's interesting. The first commit message isn't until October, right?
maybe I had this kind of image of, but you know what, that as you say that I am feeling cold in that garage, that unheated garage, that thing got like frosty in October, November, December. Like we really did need to move into a place with heating.
Yeah. And so, yeah, but instead you got the oxide office, which is barely heated, but that's a, that's a subject of a, of a different podcast.
Okay, you are apparently emphasizing barely in that. I am emphasizing heated. I think it is heated, if barely. No, it's fine. It's actually going to be warm, so it's going to be a roaster in the office today.
So in terms of the history here, and Robert, I'm hoping you can help me out because I'm realizing that my recollection of this, I'm going to be interested to compare notes on the recollection of this. But for the history of this, you actually have to go back before Oxide, certainly, before Joyent, actually, and have to go back to Our Days at Sun.
Adam, this is kind of where the earliest stuff of this gets laid down. When Sun, I think not unusually, had this architectural review process. And in particular, there was a platform software architectural review committee, PSARC. And PSARC would... would approve architectural changes to the operating system.
And it was a, and I'm sure we've mentioned, we probably talked about PSARC in the D-Trace episode, but PSARC was this body that fancied itself a pretty august body, but you would develop materials for a PSARC case.
And I mean, Adam, I'm sure you had the same experience where the development of materials for your PSAR case, I thought was very valuable where you'd be writing your own materials or, you know, you would have a case and would iterate on it with like Mike and me before you submit it or whomever, right? Roger, whatever. That process, I felt to be very valuable.
It's so interesting to hear you say that because I felt like at the time, Uh, I'm not sure I remember it the same way. Like I remember there was certainly parts of it that were valuable and parts of it, which felt like a trip to like the DMV or like global entry.
Yes. That's what I'm saying. Okay. Okay. So what I'm saying is it was like the, the, the part of the process that was valuable was the part of the process where you wrote everything down. That was valuable. Right. Right. And I found many issues in my own thing, in my own stuff by, I mean, small issues, but like, okay, oh, right.
As I'm writing this down, I'm like, I realized I hadn't quite thought about this and I'm going to get ahead of some kind of just ill-conceived discussion about this. So I'm going to clarify this. Right. And that was all valuable. And then everything from that was not valuable. And in fact, was awful. Yeah.
Because you had spent – if you work on this earnestly, you're like, I've spent a lot of time thinking about this. And other than like – It's not impossible that someone would have a suggestion that would be valuable, but that's not the way the PSAR generally phrased it. PSAR was not phrased as suggestions. It was phrased as approval or rejection.
It was very much this body of senior engineers, air quotes around that, but folks who had been doing this for a while and felt like they were the parole board of what could be and what could not be. It was.
You know, parole board is a really good way of phrasing it. Like, I don't know, like, are you a recidivist? Is this going to, I just feels like, I know you've had a couple of character witnesses in here, but I don't know.
Oh, and there's, there's a lot of that too, where it was like, I don't know, you would kind of work, work these folks a little bit beforehand, you know, the politics of it was, was onerous.
The politics of it was onerous. And I mean, I know we turned the cockroach episode into a Postgres episode, so I don't want to turn the RFD episode into a dark episode. But that said, you're like, why would you be saying this unless that is exactly your intent? But I do think it's worth mentioning that. And I would say that if you are in...
an engineering cultural milieu that has this kind of August review boards, a trick that worked exceedingly well was to deliberately have an issue. And this, by the way, has a deep history in software engineering. This is the queen's duck. But deliberately have an issue that is a small issue that everyone can wrap their brains around that you yourself are equivocating on.
Like, God, I just don't know what to call this thing. What should be the name for this?
Or even better, even if you've already decided, whether you're equivocating or not, it's something that doesn't matter, but is a delightful shed for them to discuss the color of.
That's right. And I mentioned this is the Queen's Duck, which goes back to a game called Battle Chess. I don't know if you ever played this game, Adam. This is in like 80s, early 90s. And the VP at the company was making Battle Chess was a famous kind of micromanager. So they would deliberately introduce things that they know that he would be, they would basically occupy him.
And so he would all, and one of the things is they had the queen in Battletrust holding a duck. And he'd be like, you know, he's like kind of looking over it very thoughtfully. And then like, you know, the queen, queen should not be holding a duck. Like, oh God, yes, thank you. Yes, thank you. So they, this is a trick that works extremely well. We did this and it worked all too well.
But all that is wasted energy. Like you should not need to, like all of that is, is the organizational politicking and it sucks and it's not uplifting even when it works extremely well. Okay, when it works extremely well, it's a little uplifting, but not very uplifting, and it's a waste of energy. But the thing that was valuable was writing all that stuff down.
So, okay, we went to Fishworks, where we wrote nothing down, I think. Do you remember our first bug database?
It was literally a single text file under RCS control. RCS control, sir.
Oh, that's right. Excuse me. I will thank you not to refer to SCCS as RCS. I will thank you to not refer to our defunct source code management system as a different defunct source code management system.
And every bug had to be summarized on a single line of text, ideally within 80 columns. Yes. Anyway.
Do you know where that bug database is from? That's the Detroit bug database. That's right. Don't you remember that? Yeah, yeah, yeah. I wasn't going to air that dirty laundry. That text file has a deep pedigree, friend. But we, other than writing one line of text for every bug that we fixed, there was basically nothing.
And we certainly weren't dealing with piece art, and we were kind of a renegade outfit to begin with, and talk about an outfit that should not have been purled. And then I went to Joann, and we kind of fast-forward a couple of years, and And I was really missing writing things down. And it was happening kind of sporadically. And Robert, obviously, this is where I would love to call on your memory.
Because at some point, and my memory is that Alex Wilson, our colleague at Joyent, was working on a container naming service for what was then Triton. And was really looking for a vessel in which to write it down. And I think that there's been a lot of pressure building in general where we actually need to have a way for people to write things down.
And it had been enough time since the scarring of Peace Arc. I think actually one of the things I kind of present about Peace Arc is that it took me years to really appreciate the value. And Adam, maybe you're still getting there because you're like, did you just say Peace Arc was valuable? Because that's not my memory at all.
But it took me years to appreciate the value and be able to separate it out from the things that were really – that could be, frankly, pretty upsetting as an engineer. But once I kind of had not had this for an extended period of time, it's like, yeah, we really need to get – I miss writing. I miss writing ideas down. Robert, is that your – do I have the –
Yeah, so I think that's kind of right. And I think we had also reached kind of a software complexity. So when we were talking about this, my recollection is we were kind of going back on what was good about PSARC, what was bad about PSARC. And in particular, it was kind of like the 20 questions part of it.
Not necessarily did you write every manual page down into a case that was there, but more of the what are we trying to do and why. we had kind of gotten to the point where both Triton and Manta had reached a level of complexity as well, where it's just like, how did these things intersect? What actually happens when you're building this?
And to your point, the act of writing it down just forced you to think about it a little bit ahead of time. But yeah, I think that that's kind of where we were. And then I think we kind of used Alex as the first willing
Yeah, and that's what I was trying to remember is like whether – to what degree it was like Alex is like I need a place to write this. But yeah, Alex was definitely – however it was, Alex was the first RFD.
Yeah, I think it was both. I think we kind of both were looking for it and then looking for a way to kind of lead with a couple examples of what we were looking for because it was kind of a hard thing to – if you just like what is the actual shape of this kind of amorphous document? And so you want to have, you know, something that kind of describes like, what's the problem?
You know, what are we actually proposing? How does this intersect? You know, especially for kind of like a bunch of stuff for us was, you know, then, you know, what is actually going to be user visible? Like how does some of these APIs, what are some of these APIs, what are these sketches of them? So we understand how they fit together.
That's right. And I think that, you know, and this has happened to me a lot in my career where I you know, I'm trying to come up with something that I think is like, wow, this is like totally unique. And like, Oh, you know, this is very Socratic or a kind of classic, what have you. And then, I mean, at some point, as we were thinking about this, Robert, like kind of the light was going on.
So what you want is something that's like, I want it to be pretty loose. I want to encourage people to write things down more than getting them exactly right. Um, and yeah, as we're kind of thinking about this, we're like, wait a minute, these are RFCs. This is what, this is what the, this is how the internet was built. This is right in front of us.
This is what RFCs and, and, you know, in joints, it joins our fee repo. I, I reference RFC three and the, the IETF has this request for comments and RFC three just absolutely nailed the, what we were after. And it was kind of like, I've managed to reinvent something that's right in front of me for my entire career, more or less. And I felt a little bit silly for not having seen it.
But then another like, okay, so this is like, we need to take this and extend it. And I just love this note from RFC3 that the content of a note may be any thought, suggestion, et cetera, related to the software or other aspect of the network. Notes are encouraged to be timely rather than polished. I love that line.
Philosophical positions without examples or other specific suggestions or implementation techniques without introductory or background explication and explicit questions without any attempted answers are all acceptable. The minimum length for a note is one sentence. and these standards or lack of them are stated explicitly for two reasons.
First, there's a tendency to view a written statement as ipso facto authoritative, and we hope to promote the exchange and discussion of considerably less than authoritative ideas. Second, there is a natural hesitancy to publish something unpolished, and we hope to ease this inhibition. Those paragraphs are from RFC3. That is Steve Crocker, April 1969.
Adam, doesn't that feel like that is just absolutely true today?
So spot on. In particular, just encouraging people to get over the hump of not letting the perfect be the enemy of the good. Just write the thing down. Doesn't need to be perfect. Better written, better sent. Let other people work on it. Love that sentiment.
Totally. And as a result, there is going to be a wide variety of quality here. I mean, there has to be, right? There's going to be, and there are going to be RF and there are RFCs that are extraordinary. And there are many RFCs that are forgettable and that's okay.
We're not trying to make something that is kind of, and we're in, in particular, what I loved about that is like, this is not something that's kind of up for approval. This is you writing down your own ideas and,
And then, Robert, as we were kind of thinking about RFCs versus – my recollection is, like, we just wanted to have a different nomenclature because we felt we'd be just confusing ourselves all the time if we were talking about RFCs. And we were kind of looking for a different nomenclature. Is that right?
Yeah, that's my recollection, especially, like, in the era of, like – chat bots that will link to things. I think the bot already did RFCX to the actual IETF RFC, so that was going to be doubly confusing. I think it was that, and then also that way we could just distinguish it from the IETF. Yeah, that's mostly why we chose a different name. I don't think there's too much
Yeah, I don't think it was too much beyond that. I think it was just kind of like, you know, I think we would have called it like, yeah.
Yeah, I think the sentence you have, the parenthetical you have at the end of that joint RFD intro is probably about as accurate as it is.
Yeah, which is, we use the term request for discussion in lieu of request for comments to avoid conflation with the IETF construct. and the more formal writing that has come to represent. Because RFCs are now kind of drifted from that RFC3 ethos and zeitgeist and are now quite a bit more formal. So we wanted to draw inspiration from RFCs, but be something slightly different. And RFDs were born.
And Robert, I feel it was like, I mean, I don't know if you and I spoke about this. I'm sure we did. But I remember thinking very shortly after Alex's RFD won and we start getting a bunch of RFDs, I'm like, oh, man.
we waited too long to do this i felt like this was so should have done this earlier i um i mean on the one hand it's like it's great like okay this is clearly like i this is a big step in the right direction but it's such a big step in the right direction that uh felt like we should have done this earlier um yeah and i think we had one or two you know like one-off documents
had come along the way like i don't remember if we had something as formal for manta i think i wrote up a bunch for the fabrics stuff in like 2013 2012 yeah and but they were just kind of like one-off documents that we would just kind of store and toss on you know the like local http server for folks to build to see and then i guess we eventually had manta we tossed it in there but
Yeah. And where did those go? I mean, I just don't remember where were those because I mean, they must've existed. I just don't know where they went.
I think they've been sands of time.
Yeah. So this was in, I mean, it's kind of embarrassing that like this is in the first RVs in 2015. And so we did, how did we do engineering for five years? I just don't know.
Very well. Thank you.
fair. So, um, and then, so Robert, what were our experiences with RFDs at joint? Cause I felt like there were some things that worked really well, namely the idea, the, uh, was great and was important. Um, then there were other things that I don't know, what were your, what was your take on things that worked well and things that did not work well in that kind of that first embodiment of us?
Yeah, I think, you know, But definitely a bunch of things we're trying to figure out is what is the content? How do we section it? I was going back and looking at the prototype we had for the RFD set at Joyent. And it's literally just like, OK, there's a title, and that's it.
And I think sometimes that not even having like, not that there's too many more sections than what we have at Oxide, but I think having a little bit there
helps um and then i think the method of discussion it's also something that we're trying to iterate and figure out you know what's the right way to do it um yeah you know some of it was just kind of like ah there's an issue or you know some folks discuss in chat or sometimes we had an email thread and that worked okay ish um i don't know yeah i mean pros and cons there the github
GitHub PR today is differently painful because of GitHub more so than anything else.
It is painful, but it is much better than it was at Joyent. At Joyent, we did not. I mean, the grand irony of RFDs at Joyent is the discussion piece was not. It was really hard to discuss them because we would do this with GitHub issues, Adam, and it did not work well. Um, because it was very hard to like comment on particular pieces. You know what I mean?
You'd be like, yeah.
Comment on like, Hey, the word choice or whatever. And like, if you walk up to an RFD that is, you know, old, how do you go and be like, you know, this paragraph is no longer correct. Or you just like you filing issues on all these things. And it's like, the discussion was not great. So that I would, I think was a big, uh, needs improvement. Um,
But I think the idea, like we could see that a bunch of, another thing that was actually really important, Robert, is that it was in a repo. I felt like this is a very important, because folks may look at RFDs and be like, well, wait a minute, why not just like Google Docs or whatever? And I think I'm sure, what were you doing at Delphix and then Transposit, Adam?
We were using, at Delphix, we had a sort of similar system. I mean, born of some similar problems. When I joined Delphix, the problem I saw was that like everyone in engineering was very clear on what they thought we should do. And everyone's idea was different and everyone felt ignored, but ignored without having really raised the issue.
So we started a process fairly similar to, to let people express those ideas, express, you know, why they thought we should do them, that kind of stuff. And yeah, kind of have their day in court as part of a group discussion.
So it was somewhere between what you're describing at Joyent and a little bit of the PSAR case without some of the rigmarole and like council of elders, but more focused on product discussions. But yeah, what we did was mostly in Google Docs, which I think has some downsides in particular what you're alluding to about like,
the lack of searchability and there we like layered additional indexing on top of it and like a meta document to keep track of everything. But then some benefits too, like it's much easier to have multiple authors simultaneously. Yes.
Yes. Yeah, and I think that what the world wants, and what we're kind of leading up to, is you want kind of a Google Doc front end that has an ASCII doc backend going to a canonical repository. That's what you want. That doesn't exactly exist.
Yeah, and I think some of the other bits here, you know, locating everything in a repo makes it very easy to see what's changed over time. The discoverability was big, and then We ended up with Markdown initially mostly because, I don't know, it was the thing to do du jour. It was easy to see rendered.
Trent had his Python Markdown 2, which we had used for a lot of docs and other things historically.
Well, part of the reason I'm chuckling is because we did Markdown at Oxide, and we supported Markdown and ASCII doc. And Adam, I think it was you who took me aside at some point. It's been like, hey, you say that RFDs are in Markdown or ASCII doc. Yeah, exactly. It's like, you should know that only yours are in Markdown. Everyone else is on ASCII doc.
So when you say Markdown or ASCII, I'm like, I actually... Adam, I don't think I'm inventing this conversation. I really appreciated it. I had a giant piece of lettuce on my teeth and only you had... All of my oldest and dearest friends were willing to take me aside and tell me what an ass I was making of myself. I'm like, you mean, no. And I kind of didn't believe you.
I'm like, surely other, no, nope. Nope. It's just me. And that was a quick, I definitely like, I am reworking them right now. They are all going to be at ASCII doc. I'm so sorry. Yes.
But I think it's fair to say that we're weird for our obsession with ASCII doc. Is that right? I think so.
I mean, probably. Given how we're weird on everything else, right? Yeah, part of this evolution I think came from a few steps. I mean, once you're in the non-WhizzyWig world, so once you're not in a Word or Google Docs, then there's kind of two challenges. One, and
This may be my own fault, but it is very hard in the markdown days, particularly in 2012, 2015, to see a rendered markdown doc, per se, locally without pushing it somewhere up. And that, for me, was just my own mental foible of wanting to see, hey, does this render correctly before I push it?
It is my own weakness that I would like to be able to, I don't know, see what this thing looks like before I, I, I just, it feels like this is like how you always want to test your code before you integrate it.
Yeah.
It's like tough, tough and fair. Uh, and then I think the other, the other thing that we kind of saw as we were getting down, um, And I think the first Joyent, so actually Alex Wilson, who wrote RFD1, was the first one to use ASCII doc in an RFD at Joyent. Oh, really? Yeah, he was the one who found it and pointed us towards it. And I want to say it was probably for RFD77 there.
And once you kind of got to, you know, I think one of the Markdown strengths was, you know, it was very simple. Uh, one of Markdown's weaknesses is like, if you want to create a table, there's still probably like 20 different ways to create a table. And if you want a table of contents, like, you know, God forbid that, uh, you're, you're really on your own. And so the little bit of additional, um,
syntactic features that we got out of asciidoc was kind of why for complex documents we want to move there you know easy knowing that there's easy ways to link between different sections or other things or kind of a bunch of the other features and you know there's a bunch of features we didn't even use back then that uh now so our colleagues like uh ben knacker is using in the oxql rfd so he could easily do call outs or other things so i i found it to be a you know
Easy to get something basic that looks pretty good. And then if you need to go complex, you had that capability. Whereas with Markdown, you were kind of constrained by the Markdown renderer. And no two renders were going to do it the same way.
Yes, Markdown is really frustrating in a lot of ways. And especially if you want to do anything sophisticated or complex. you need things like, you know, footnotes and you want to cross reference and all these things. It's really not good.
I had kind of assumed, I didn't realize that us, that ASCII doc, I thought ASCII doc, well, obviously cause I was the, like the last to get the memo on like, you're the only one doing it in Marktown. So as far as I was concerned, everyone was doing an ASCII doc the world over.
So I didn't, don't think I realized that ASCII doc and press was saying also that he'd not heard of ASCII doc till coming to oxide. So I guess that's a little more, it's, I mean, the irony of ASCII doc is that the documentation for ASCII doc is not great. Um, I do find that, and Robert, I can't imagine that ChatGPT is very helpful on AsciiDoc. I want to be able to render this.
I feel like some of the stuff in AsciiDoc is a little... It's been nice to be able to use ChatGPT for AsciiDoc. That's all I'd like to say. No, I'm just being on brand to get LLMs in every episode. You know, there we go. Ring that bell. Exactly. Um, the, but so I didn't realize that Alex had kind of discovered ASCII doc. Um, we, uh, and Robert, do you use ASCII doctor PDF to render locally?
How, how do you render locally?
I just always used ASCII doctor and just pulled up the HTML file. Um, and just use that. And it was pretty good. It had default CSS too, which just meant you could use it and toss a document somewhere, and people could see it. And it looked fine on both mobile and not mobile, which was really nice as someone who doesn't want to think about CSS.
And so it should be said also, and Robert, now is as good a time as any. So we have, I created RFD 510 yesterday. So we have perhaps more than 510 now. So I might have created one in the meantime. But we've got over 500 RFDs at Oxide. Adam, you had on an earlier episode had guessed that we must have millions of words. Did you do this, by the way? Have you already done this?
I haven't looked at the numbers for like years, but even then it was overwhelming.
Yeah. So we do have, there's 1.4 million words. Um, and I did look at the, I went through everybody who has kind of contributed an RFD and, uh, just looked at the author's field. Um, so didn't, didn't do anything more kind of sophisticated than that. And to see, in other words, like giving everyone full credit for any RFD that they're on the authors for.
And of the 1.4 million words across all RFDs, uh, Robert, you have contributed, uh, three over 364,000 words. Is that amazing?
That is amazing and profoundly unsurprising.
Also terrifying. It is. And, and Robert has been, so part of the reason, Robert, obviously you, you are, you are our most prolific RFD writer.
Um, and you know, not, not that we want to, you know, just insert our discussion about the parallel of engineering metrics and that it is the cultural idiosyncrasies episode, but the, um, you know, we're not trying to measure RFDs by the pound here, but I did think it was, it was interesting and it was revealing.
The other thing that Adam was really interesting to me was the, uh, that, that was just eyeopening. It wasn't hugely surprising, I guess. Um, The person at Oxide who has contributed with the most number of collaborators on different RFDs, who's collaborated with the most number of people on RFDs. Do you know what this might be? I'm going to guess myself. You are absolutely right.
It is like you by far. And this is one of these things like I didn't, you know, I'm not surprised, but I'm also like, I was actually surprised about how much it stood out. And it makes, and actually, it's because you have also like really encouraged a bunch of people to write RFDs or started RFDs that have, I mean, it's just like, I thought it was interesting.
I mean, it showed one of your kind of great strengths in terms of cross pollinating in the organization.
Well, on that, some of that was learned from what we had done at Delphix, which was that you weren't, you were, maybe you weren't allowed, but you were certainly highly encouraged to have a co-author. Like we kind of didn't want solo authors and things. And part of the reason for that was we thought it made for stronger documents.
We thought that even if it was somebody's deep held idea, having a co-author to walk them through it and convince them of it and get them on board would strengthen the quality of the document. I'm sure many of my collaborators on those RFDs would also hasten to point out that I like to do a half-assed job and then leave the rest to them. But I prefer to think of it as cross-pollination as well.
Yeah, exactly. But I thought that was ranging. So, I mean, both of you have, in different aspects, have interacted with the system a lot. And so, you know, Robert, you have written more ASCII doc than any of us. So we've written a lot of ASCII doc. I would say that the entire collection of Shakespeare is about 800,000 words. So, Robert, you're getting there. What...
Will Shakespeare, keep an eye on your rearview mirror. Robert's coming for you.
So I would also say, folks who are wondering, when someone joins Oxide, then what? Do you just spend the first three years reading? Good question. I think, and Brian, you probably have the data on this, I feel like we're on escape velocity. I feel like it is impossible for me to read every RFD that is being produced, even if I dedicated...
like all my time all my work time to just reading the new contents of rfds absolutely and because we are you know very writing intensive culture we are adding people people are we've added some very prolific writers and yes it is it is absolutely um and in the ends like that shouldn't be the goal right you don't want to actually like have to read every line of every rfd
Okay, so this gets us to, because I want to get to the very important bits here that we have extended, because this is where we get to Ben's work and Augusta's work, Crespo's work. So this was, you know, we start Oxide. We know, most of us know that ASCII doc is the right answer. One of us feels that they are still doing it in Markdown. So we solved that problem.
I think it was in the first year, Adam, when we had the intervention, the Markdown intervention.
Well, for Woodsworth, I had not heard of ASCII doc before joining Oxide. So I had to become a convert in a hurry.
Right. Okay. And I think ASCII doc has been good for us, I would say, broadly speaking. Totally. Other than being, I guess, more unique to us than I realized. So one of the things that we started doing early on, I mean, very early on, actually, the earliest days, because we have, it's just like we're on, the name of the podcast now, Oxide Friends, comes from actually our initial friends of Oxide.
So when we started the company, we had a bunch of folks who had, and Adam, you were one of the initial friends of Oxide, right? Where before you worked here, you were a friend of Oxide. We had just people that we'd worked with, that we'd known, that we wanted to, be transparent about what we're building to get their perspective on.
So we added them to that GitHub repo so they could see actually all of the RFDs. So the GitHub repo is a private repo that has all of these ASCII docs, all of these documents in it. And
The important change that we made from Joyent is the way we did discussions, which is it's a little wonky, but it's better than it was at Joyent by a mile, which is we use the fact that GitHub, that PRs are actually... Better than issues for discussions. I mean, the GitHub PRs have got issues with them for sure, or they've got challenges using it.
But we use PRs to discuss an RFD, which at least allows people to give kind of line-by-line feedback. It's nowhere near as good as Google Doc comments, for example, but it could be worse. Yeah, it doesn't get in the way, I don't think. I don't think it gets in the way.
I think it is, you know, I think that there's some challenges, but I think it's broadly, again, it was like way better than certainly at joint. We didn't have discussion that we should have had because it was mechanically so difficult. So kind of solve that problem.
I feel on Google docs. One of the, I mean, huge downsides is like, you can have a great discussion in a comment or whatever. And someone closes that comment and it's like gone forever. And I think one of the benefits of, of the GitHub history is like, it's not gone forever. It's all there. Just hard to find.
Yeah, I kind of, I do like it. I wish, obviously, I wish those comments were themselves under kind of Git management, but which would be kind of, that'd be asking GitHub to go to be one step too useful. But the, I do like the fact that they're there and they also don't get in the way of reading the doc, which I think is also important because...
you know, somebody like, you know, not all those, you know, there are comments like, okay, yeah, I got considered this or whatever, and I'm going to close it. I'm resolving it, but I want you to be able to still see that there's a comment here. Maybe someone else can come along later and have the same comment or what have you, but yeah, great.
Um, so we, and we do that with branches and then we, we put some automation around that where, you know, you could, the RFDs have got different states and they can be in kind of, um, ideation was an early, that was a state that we added at Empire Call. We wanted something that was kind of, uh, earlier than kind of discussion state, which is like, I truly do.
I've got just like no idea what I, this is a dream I had.
Yeah. I mean, this actually started, I think the first one was the out of box experience stuff, which we kept on stumbling into. Oh yeah. When we first do the installation, we've got to do this. Or when, when we do the install, we got to remember to bring, you know, the tech screwdriver or whatever. And we just wanted somewhere to dump this stuff down.
I mean, in part because RFD is the only durable place we have for this stuff.
Right, and that's right. I mean, RFDs are the way we kind of are. Maybe you can argue beginning to abuse it a little bit, but the system was flexible enough that it could be extended that way, and adding ideation was pretty easy. So that worked out pretty well. So then it was great to be able to take the repository and add their GitHub ID as a collaborator on the repository.
And then that became kind of part of what one though, like it was just great to be able to do that, to get people's perspectives on things. But then secondly, we would use that as part of the hiring process. So part of the hiring process was after we did someone's materials and like, okay, we're interested in this person. We would add them to the RFDs so they could see all of the RFDs.
And this was a really important step. It was really important for us to be able to show people all of our documentation about how we were building this thing to give them a much more concrete idea of what we were doing technically. What I would tell people is when you get into these RFDs and you're going to, you know, some of them are extraordinary. Some of them are not even correct any longer.
Some of them are, you know, there's this huge, you know, huge spectrum of quality and so on. But as you read these RFDs, you will begin to feel that you want to work for Oxide more or less, but not the same. That's what I would tell people. And I think it was really, really important.
And we had, obviously, because we're, by the time we're kind of advancing people's after the materials, we talked about this with our, on our episode on hiring with Gary, but this is all happening after we have decided like to, to kind of pull someone into conversations and we want them to kind of understand what we're building at Oxide.
We had, we did have people in there and I had one person in particular was like, I thought I really wanted to work for Oxide. But, and I had pointed them to, you know, some pretty gnarly RFDs and it's like, I'd read some of these RFDs and like, I am, I'm kind of more interested in the idea of working for Oxide than I am working for Oxide, which I thought was very helpful. Right.
That's like, that's, that's great. I mean, much better for everyone that they figured out before they joined rather than three months in.
Absolutely. Absolutely. And Robert, you should take this as praise, but it was absolutely one of your RFDs that actually was the clarifying RFD. Because I think people don't realize the amount of... You're like, oh, my God. There is so much detail here. It's like, yes. Yes, there's a lot of detail. It's very, very detailed to actually go on. So that was great.
The problem that we had, or that I had, was there wasn't a good way to... Well, we had a couple of big problems. One, there was no way to search RFDs, which started off not great and was becoming really, really, really problematic. And then there was no way to share individual RFDs. And other than like rendering them as PDFs and sending them, which I did plenty, but it's like, that's not great.
And this may be a good time to get you and kind of David in here and Augustus in terms of like, this is kind of the RFD system that you kind of come upon is us kind of dealing with sticks and stones and GitHub. And what is the origin of the RFD site? Because I'm trying to remember, how far did that date back, first of all? That dates back, I think, a while.
There was an initial version that already existed when I joined. And it was just a list of kind of RFDs that were rendered. I think... I think at that point, a lot of people were looking at RFTs directly on GitHub. I guess due to the deficiencies of the site. And then I don't remember exactly when we kind of relaunched sort of the version that we have now. We kind of were built upon it since.
But then we kind of incrementally added things like the full text search. That was a huge pain point for me. I forgot what kind of command it was. There was some kind of grep thing, which would take... would take minutes to search the. It was a long time. It wasn't a great process. And now we have full text search.
We're using some library that indexes all the RFDs and searches, which I think is a good, that's super useful. I can, to your point earlier, we're sort of at an inflection point where there's so much in there that even full-text search doesn't necessarily get you everything that you need. Someone's already mentioned LLM, so I think I'm free to do without kind of...
Yeah, you don't have to feel like you're breaking the schedule. I'm not being accused of a shill, but I think there's, on the roadmap, there's, I think, there's room for something that kind of summarizes all the RFDs and at the very least tells you where to look. And let's say you're joining at Oxide.
There are a bunch of RFDs that everyone should read, but for most people, there's a smaller group which are kind of really pertinent to their work.
And I think that we can use something like that to generate reading lists for people that, that join and they can, they can kind of, they don't feel like they're just losing their mind, like drinking from the fire was trying to read everything to, to, to get up and running.
Totally. When I think, so a couple of things like the, so we added, I mean, the inability to search RFTs was extremely painful. It was also, it's made even more painful because the, the great thing about using PRs for discussion is that you get GitHub's features and
The downside of that is it means that like the actual repo itself, that is to say like the main branch, only contains RFDs that have been published. They don't have the ones that are in discussion. So you have to like search every branch. And so I had this thing, RFDGREP, that you were referring to, Ben, that was taking minutes.
But it was like, it was minutes, but it was actually like delivering penicillin. So it was really, you know, it was like, oh my God, thank God we got some way. We were using LuceneGREP.
um and lm graph is what we were using using what we've seen and it was very helpful but it was also like its performance would vary a lot it would take minutes it was really really painful and in particular like it was so needed and because i remember like getting steve tuck set up with rfd grab because he needed it desperately like gotta have a better way to do this so
I can't remember when you did the full text search on the RFD site, but that demo Friday was like a religious experience where, I mean, it was like that affected everyone at Oxide to be able to like, oh my God, I can just go there and search all the RFDs, like clean water, thank God.
Yeah, and it's such an obvious thing, but then afterwards it felt kind of magical, as if a few of those moments where I kind of added something and it just sort of, I think, transformed at least the way I use the RFD site. Full text search is one of them getting kind of discussion in line on the RFD site. Okay.
So describe that because that was, I totally agree. So now we've got the RFD site and we've got the RFD site, which also like looks great, which is very helpful. You are doing, so tell me how you're rendering the ASCII doc, because that was a big piece of this. You got to go render the ASCII doc.
There's been a few versions of it. And I think like I kind of have a love-hate relationship with ASCII doc. I think it's amazing because it is so versatile and it is so painful because it's so versatile. You can create so much and I'm aware of a small subset of features. The RFD site is the place to check
like a renderer for ASCII doc, because you will see every version of, uh, of, of any way which you can use. That's good. It has been used on the RFD site and that will get kind of an issue now and again. And I'm just baffled. Cause I'm like, I didn't even know this was possible.
Yeah, sorry. I think I was definitely a recent version of that where I was definitely using block quotes and attributed quotes. Brian, that's basic.
You should see the stuff people are doing. But the thing is, it's... It's really rewarding doing this stuff, but there's a balance, right? I don't want to spend all of my time working on like an ASCII doc renderer and kind of the styling for this thing if it's just kind of used in one place. And I think one way in which we have worked around that is we've embraced ASCII doc.
I'm using ASCII doc for everything. So our doc site is built on ASCII doc, our RFD site, and the blog posts on the marketing site are all ASCII doc.
So what we have there is, and that in tandem with our design system, our colors, our typography, all of that stuff is unified across both kind of product and internal sites, which is unusual because usually you have your product design system and then you have some other designers who kind of are off like playing in the sandpit, doing like the website, doing whatever they want. I'm doing both.
So I can have a unified system. And what it means is... when we, when I, when I do the styling for the kind of RFD site it's shared across all these places. So I can justify spending a bit of time on these things because, um, Because it's so useful.
I feel like the emperor's new clothes, though, because we do use ASCII doc everywhere, I just assumed ASCII doc is ubiquitous everywhere. And it's like, I've never heard of it. Exactly.
Yeah.
Element? You know, formerly known as Riot? You know, the chat system that I assume everyone uses. It's like, no, it's just us.
Look, I... Okay, I just assumed. Because, Ben, as you point out, I'm realizing that part of the reason I thought ASCII doc... was ubiquitous is because it is ubiquitous and oxide, but you're, it is ubiquitous and oxide because you made it ubiquitous.
So you, Hey, don't, don't blame me. Don't blame me for that.
I think it's great. I think it's great. I think like it's, it's actually, I think it's terrific. I think it is so much better than Markdown. Um, and, um, so I, yeah, I think it's great. And like blogs and so on are all an ASCII doc.
Really briefly on the stuff that we've made for it. So there is a... My main issue of ASCII Doc is that there is no kind of native... like Rust or JavaScript library for processing it, there is a JavaScript library which is transpiled from Ruby. And so kind of making changes or seeing how it works in another hood is kind of, is sometimes challenging.
And then it kind of, there's a bunch of assumptions on the way that it works. Like it assumes that you're going to process a whole ICO document in one go, top to bottom, right? which makes sense if you're processing it locally. In my case, I kind of worked on a React render of ASCII doc.
And essentially what I do is I take this ASCII doc tree and I work through it and I render kind of each part as kind of React components. If you're not doing that, you have to create a renderer where you're returning a string. So you want to render an image, you return a string, and you drop in.
You're accessing the attributes, and you're swapping stuff in and out, which isn't ideal, especially if you want to do some more interactive things like our images. It gets complicated because we're using signed images that come from GCP. We want a little light box. There's some other kind of features where if you want any kind of interactivity, that model doesn't work.
So we wrote a React renderer for that. But then you kind of, React doesn't run once, top to bottom, and you have issues. The big issue that I spent way too long on was every time you render a piece of content, with a footnote in it, it assumes that it's a new footnote.
So every time it's re-rendering that piece of text with a footnote, your number of footnotes is increasing, increasing, increasing up on the page. So those things are kind of challenging. And we've found ways around them. And I think like I have a, each one of these versions is motivated by kind of a piece of functionality that we want.
So one thing I've been playing with recently is creating this intermediary format for ASCII doc. So I take, I go through the tree, I process it, I turn it into like a JSON object that can be passed easily from the server to the client. And that means that you can kind of pre-process it on the server and then you're not shipping like these big
this like two megabyte, uh, client library just to handle it on the front end. Um, but, and, and, and the reason why that is important is let's say we're in the, we're in the, then the console. And then, and this is a big aspiration of Robert's is if we, if we want documentation that exists within the console, it doesn't feel justified to ship this big ASCII doc JS, um, library in the console.
Cause that kind of is a bit bloated. Um, but I think, uh, kind of working on experiments so that we can have dynamic documentation. Maybe if you're kind of working on, uh,
repairs and kind of swapping components then then we have that without needing to um do yeah kind of do all this stuff so yeah it's it's it's interesting um it's a it's a bit of a challenge at times but uh yeah the versatility is the kind of is the is is the the thing that's great and the thing that kind of trips me up sometimes
Well, and just the fact that we also use this for the documentation is so great. And then we've been able to make the documentation site is anyone that's available to the public. If you want to, I mean, we're, can see all of the Oxide documentation, which is great. And then we're able to kind of cross-link that with the console. There's a bunch of things we can do there.
So the other, I mean, and then also it looks gorgeous. Can you talk about the GitHub integration a little bit? Because that's been a huge, that's been hugely valuable to get those discussions. That was another demo Friday that I just think just like left people, you know, weeping with joy. The C comments and the RP site.
That was a, I think, I guess I think to, to, to begin, I think there's, there's, there's something around like the accessibility of this stuff. Oxide has always been like, um, really kind of engineering driven. Um, and engineers are really familiar with GitHub and that's really easy for them.
But, uh, I think if you want this process to be used outside in kind of like kind of sales and operations and, and, and, and design, then, um, really, you have to make this stuff as accessible as possible. And we're not there. And I think we're working towards it. But getting GitHub discussions directly into the RFD site felt like a big step towards that.
Essentially, what's nice about the GitHub discussions is you're leaving line comments. And one great feature of our ASCII doc is that you can trace you can kind of, given a line in an ASCII doc document, you can get the, no, in the return, in the render document, you can get the associated line number.
So what it meant is we could query the GitHub API, get all of the comments, kind of collect them in,
kind of a format that can be kind of rendered easily which which is which is pretty kind of tricky to begin with um and then we go through sort of line by line and and we we position them alongside the content directly if they're still relevant um and then we have this sidebar which kind of you can see the kind of full canonical discussion and you can kind of jump to the the relevant part um but yeah just that little bit of kind of being able to associate the original
document with the rendered document uh was enough that we could pull like from the api people's avatars and we can kind of have this little this module that people can click on and can and and and view um yeah and i think my kind of first experiment was this super ugly way of getting like a little avatar alongside the text but that yeah that felt kind of a real um kind of a huge improvement
A huge improvement. And this is a good segue to the kind of the RFD database that kind of fronts this thing, I think, because I assume that's hitting the database that is fronting it and not actually GitHub directly, but I actually don't know the answer to that. Augustus, it would be a good opportunity to get your work in here.
Yeah, that does actually talk to GitHub directly at the moment. Oh, wow. Yep. Hopefully not for long. Our goal was to model it in the front end, see how people liked it. And then once we have a good idea, then we port it back down to the back end.
One of the complicated bits, though, is we have this back end processing bit so that we don't have to go to GitHub to pull data, like all the static content from the RFDs and the images and all that. The problem, though, is that database we want to talk about sort of revisions of RFDs. And this is something we did recently. But we currently only expose the currently latest version of RFD.
But we have a database of every commit processed. So eventually, you could go back in time on the RFD site. And then the idea is, how do you tie comments into revisions? And what's that mean? Comments on a specific revision, how do you then pull them all together? So in the process, but yeah, as Chris was saying, we're building GitHub.
Because one of the reasons to do that is the longer the discussion, the longer the response takes. So I think there's one RFD, something to do with time. And yeah, why would that have lots of comments? RFD 34. And it crashed. It crashes every time, I think, because I think it basically timed. Oh, no. Oh, wow, this worked. What a miracle.
But essentially, the more comments, the longer the response takes. And so I think there's some kind of funkiness in the GitHub API, which is, yeah. We do have something where essentially we serve the regular rendered version of And then David might want to talk a little bit about this, but there's a Remix feature.
Remix is the framework that we're using to do all this, where you can stream data from the server later on. So what we do is we give you the main document first, because that's important. And then later on, asynchronously, we give you the comments, just so that's not holding up the ..
The reason I had assumed that those were coming out of a fronted database is because it was so snappy. So that's why it's so snappy. Because you're exactly... Yeah, yeah, yeah.
And then your mileage may vary on the actual GitHub portion.
So when you first load the page, you get the spinner for the comments because it's fetching those in the background. So that's why it feels fast. So it's kind of fake fast.
It is fake. I get totally faked out by it, though. I'm not looking at the spinner. I'm generally looking at the rendered content. So that's definitely worked. It also should be said that it just. The rendered content looks gorgeous. And obviously, Ben, you've made sure that everything at Oxide looks gorgeous or faces your ire. And it just looks great.
It also looks great on the phone, which I think is really important. because I think folks are reading more and more RFDs on a mobile device, right? Augustus, the other reason that having that front-end database has been really, really important is for the visibility for RFDs. Can you describe maybe some of the problems we had with adding increasing number of folks to this Friends of Oxide group?
Because this GitHub group was getting larger and larger and larger. And I think it all had to be like outside collaborators with our organization, I think, if I recall correctly.
That is absolutely correct. So if you don't know, when you have a private repo and you add an outside collaborator, you start paying for it. So you start racking up pretty big GitHub bills around that, as well as it's just a hard list to maintain and understanding who still wants to be on that list. Yeah. And also, it meant that we have to give full access to every RFD to everybody.
We don't have any way to curate that. And we had some workarounds in place, but we have some static config files that list a handful of RFDs and this subset of people get them. We had no real good permissioning system around it. And that was one of the main reasons to rewrite that database and back-end processing was... we needed to model RFD access around actual permission sets.
So being able to grant things all the way down to you have access to a specific RFD, or maybe even have access to a specific RFD and the discussion for that RFD within the RFD site itself.
And this is extraordinarily important. I think the full text search was game changing. The rendering was game changing. And I feel that the fine-grained permissions are totally game-changing for RFDs.
Because we really needed to be... The reason I think it's so game-changing is because it allowed us to much more easily... Well, one, we could make our individual RFDs public, which was a real problem. When did we first have the ability to do that?
So we had this weird... I'd have to look at the actual date. We had a weird workaround prior to the permissions where you could technically mark things as public that we did for one or two. But it wasn't until we actually launched the new RFD API, which was, I don't know, sometime this past year, to actually flag them.
There was a file with an array of RFDs. Yeah, pretty much. And then initially it was semi-public, right? So anyone could log in with any GitHub login, I think. And then I think David really pushed to have it so anyone could access the public RFDs without logging in, which, yeah, was, I think, another big improvement.
Yeah, that's a good thing to point out. There's a difference between public, as in you could sign in and see stuff, versus now you can actually just go to the URL and you can actually just search the things that are public, as in they're actually public, not this fake behind a log in public.
And this has been, I think, huge. Because I feel, especially where we are in the past year, having brought this product to market, and now there are more and more aspects of this that we want to share with everybody. We want to share with all of our customers and we want to share with everyone to see.
And I feel like there are, you know, it's just kind of amazing that we've, that, you know, we've only been able to do this in the last year. And I, cause I feel like I do this all the time. We did this, obviously, I mean, the last,
episode as was talking about an rfd that was made public right so the in terms of wither cockroach db and 508 but i feel like it was uh really really important to have that fine grain control and to be able to have like groups as well so you can say like okay this because i think we've got a group for our contract manufacturer for example where they want we want them to be able to see everything about a certain number of rfds that are not public um it's extremely helpful
Yeah, so I know it was less than a year because it was at last time when we all met up at the office. That's when I demoed the barely working API that was just tied together with string.
And honestly, so all of this permissions and group stuff is, so the RFD site was the first test of this, but that's actually now, that's been factored out of the RFD site, out of the RFD API, and now sort of powers a lot of our internal application permissions. And now the RFD API gets updates now from the stuff we're doing throughout the rest of the company.
Yeah, that is great. So that was before we met up in October. And then I know that I was out in New York in November talking to someone who had seen the RFD site. And they're like, how can I get a hold of that?
Because I think a lot of, I mean, I feel like every time we have an RFD, a public RFD that is on Hacker News or is getting discussed, someone is always like, hey, can someone explain to me how this system works and how can I use it? Because there's a lot of interest in it. Do you want to talk about a little bit of the open sourcing of the RFD site and of the RFD API? Sure.
Yeah, OK.
No, go ahead. I would say part of our goal with the RFD API was planned from the beginning to be open source. So part of it was also just sort of detangling from anything that was like oxide specific and trying to make it a more abstract system. The RFD site, I'll let Ben speak to that one, was I think more of a challenge.
Well, I mean, the RFD site is kind of fake open source in that it's all... A message to our community. We've got the BSL license on it.
You don't have any RFDE that implies that you're building a computer. Other than that, you can use it as much as you want. You just can't, you know?
Yeah, even at Oxide. Right, too real. So, because it's still... The RFDE API is much more... is much broader. I think anyone can use that pretty easily. The RFDE site is... And it's very Oxide at the moment. It has all the styling, all of that. It's kind of directly in the repo.
I think eventually it might be nice to have something a little bit more... Yeah, I mean... Dave, you... Okay, the CSS is specific to us, but someone could take this and get it working for themselves, correct? I mean, it feels like they could. Oh, for sure.
Yeah, yeah, yeah. Yeah, for sure. Oh, definitely. And wouldn't they love to have our beautiful styling too? Everyone should be so lucky to have this gorgeous styling.
Swap the logo. Easy peasy.
Yeah. That's right.
Hey, it's trademark now. Don't touch.
Yeah, exactly. That's Oxide TM to you. Java technology, please, TM. So in terms of being able to take it, we've got all the elements that someone could do to go. And what would be some of the ways in which someone could use this stuff? I mean, how would one get started if you wanted to kind of see what it would be like to use this?
Yeah, in terms of the API, it's probably a decent place to start. It's also built on top of other Oxide stuff, like Dropshot and Progenitor. So there's a built-in CLI that you get out of it for free. which is really nice. But really, you just need a guess.
I mean, Adam, you're a proud papa. You and Dave on Progenitor and Trap. You just love it.
Yeah, so super proud. I would say that Augustus is the only, to my knowledge, other consumer of the Progenitor-generated CLIs, which are pretty gross. So I feel proud and filled with shame.
Okay, so Augustus, when I'm using the RFD COI, that is a progenitor-generated COI. Absolutely.
You should feel no shame, Adam. Oh, it's awesome. There's a function in there whose name is XXX. So there is some reason for shame. I am very proud.
Okay, so feel shame on that. But I use that CLI a lot. I mean, I make great use of the RPCLI. So that's auto-generated. That's amazing.
Yeah, so any shame, right? The end user doesn't see any of that, though. Oh, totally. Perfect technology. Internal shame.
Internal shame. Internal shame.
The only real assumption, it makes a couple assumptions maybe around, if you read RFD1, actually, is what a lot of this is modeled on top of. So it does make some assumptions about how we run RFDs, like what the states are, what they mean. But in theory, you can point this at a repo. I've been working on trying to getting a setup guide to be a little bit cleaner.
Benji ran this himself and sent me a whole bunch of really helpful notes on how painful it was. So there is some work to be done there. But it did make some assumptions around your using ASCII doctor, some of our use cases around mermaid diagrams that require some installation. Realistically, it should be pretty straightforward to run. And what you get out is essentially a nice database.
Technically, it runs on Postgres. So you have a nice Postgres database of all of your RFD contents that you can then really do whatever you want with. We happen to render them as an RFD site then.
I wanted to ask Augusta something. So something I'm not sure if we've really gone into is how it works in the GitHub repo. So the whole thing is when you make a PR, creating an RFD, that is a branch that's named with the RFD number. And so when the RFD site is scanning the repo for all, I mean, when the API is scanning the repo for all the RFDs, it's scanning all the branches.
You can't just look at main. I'm curious how coupled the RFD API is to that model. We run into some friction with that, even as nice as it is to have PRs as the site of discussion. You could imagine not wanting to do it that way. How coupled is it to that?
It's fairly coupled to that right now. Essentially, we have these... If I go into a little bit, so the processor works on this concept of jobs where we scan a repo or we accept webhooks from GitHub to generate jobs that are going to then look at a specific commit and look within that directory structure. So it's less coupled to the branching model and more coupled to the
struct the document structure so we have a director that has rfds then you have a rfd number and then a rfd document uh outside of that you could write a separate scanner that you know looks at branches and does make some other determination to create a new job that says okay here's where my rfd is located i need you to go do some processing on it which and that processing then
pulls that content down into the database. It'll do things like generating PRs if you need them. Like when a RFD goes into discussion, we generate a PR on the GitHub repo for people to start discussing it. Things like closing them out or updating things like idle states.
Yeah, and I feel like that automation has made it, because the branching bit has been great for discussion, but I think the point you're going to get to is it can feel a bit clumsy at times for the authors of RFDs. And I feel like the automation there has really helped a bunch.
And I think that given all that, I mean, I could see why someone might want to do it a different way, but there's a lot of strength to the model that we've got. We're just kind of getting GitHub. It's using GitHub for its strength without being, while still having a canonical repo that has everything in it.
Yeah, I think, and especially from the machine side of it, it makes a lot of sense. And Benji mentioned this earlier of part of our goal is then how do we, as engineers, I think it's fairly comfortable. We use GitHub all the time. But how do we then bring that to everybody and make everybody comfortable with it? And I don't think the plan is to like
force everyone to, yeah, you have to learn GitHub and be super comfortable with it. So understanding what's it look like for the RFD API to make a new RFD for you if you just give it a title or when you want to edit content.
We're doing it kind of incrementally as well. And that's the beauty of working on something on a platform like GitHub is you can use the RFD API package, use GitHub, and you don't even need a front end, at least initially. You don't need to worry about that. And then eventually you can kind of add more as and when. But yeah, the accessibility thing I think is big.
Augustus added an endpoint so that you can create RFDs from the API. And so yeah, I'm kind of working on a way that people can just create a new RFD from the website directly. And then a few months ago, I was working on a little experiment where it was a live ASCII doc editor where you could kind of create and share notes maybe before they're an RFD or notes that aren't going to be an RFD at all.
And within that, because of the API, I had an option where you could then turn, you could turn a note into an RFD and it would kind of hit the API and it would kind of take all of that content and throw it in an RFD. So I think that's what we're working towards eventually. Some basic authoring. There's a line though. It's really tempting to try and recreate GitHub and Google Docs.
Uh, I think we've, yeah, it's, it's, we've got enough on our plate, but it's tempting, tempting to do for sure. And yeah. And this, yeah, we, we, I, I, I found personally that kind of working on internal tooling, uh, is, uh, is, is, uh, really beneficial. Like we, we get a lot of it, especially as we get bigger. Um,
And we really benefit when more people can contribute to RFDs, especially engineers.
The fact that we're standardized on all these tools means that it's sort of this flywheel. Because we use, for example, like you said, the ASCII doc rendering stuff on the doc site. It's sort of the investments that we make in internal tooling also go into the product itself, which is really a good way to be.
Yes, and I think it gets to... So folks in the chat are asking about Notion. I've not actually used Notion. Preston, it sounds like you have. I would say that my... Concern about Notion. I would say I've got an Evernote concern with Notion or any other.
And this is part of the challenge here is that using any kind of paid service here is really problematic because this is so important to us that it's really hard for us to accept restrictions on how things are used. And And in particular, a hard constraint on the problem is we have to have a Git repository that has the RFDs in it. That has to be where they live canonically.
They cannot live canonically anywhere else. And so that's extremely important. I think it's really important, the ASCII doc element of it, now that I understand that only oxide is used in the ASCII doc, that becomes really important. I bet you're mentioning the chat, the versioning, that is extremely important.
And I just think that this is one of these things where we really need to control every aspect of our fate. We would love to, I think, integrate with other, you know, we're not, I think, you know, if we could find kind of better ways to do pieces of this, but we really need to control our own fate. And I think that like, that is, that is not unusual for us.
I think engineering organizations need to control their own fate in terms of the actual artifacts they're creating. And this is a really important artifact that we're creating. This is, and as usual,
saying adam it's like one of our most enduring artifacts are rfd so i mean it's i i mean it's right up there with the source code honestly i mean it's don't make me pick those are they're both very very important a while ago i stumbled onto a phrase of uh owning your strategic weirdness and i i would have said like rfds are a thing that we we could have outsourced
But like RFDs turned out to be our strategic weirdness. Like we didn't even know what they were going to be. Like we didn't envision them as like a way to communicate with the public, for example, as you mentioned that we did around.
Well, so I knew that was going to be a problem. It was a problem in joint too. I knew we were going to want a way of communicating with the public. I just had much worse ideas of how it would be done.
Yeah. I mean, but I just mean like if we had, if we had used some off the shelf tool, it might've been great, but then we wouldn't, we would have struggled to do some of these other things with it. And well, I mean the work that our colleagues who have just been talking about it, like it's not a small amount of work. It's also been hugely valuable.
And I'm not sure we would have made some different decision even knowing how much work was going to be involved.
Yeah, I don't think so either. In fact, quite the contrary. I think with all these things, I now cannot imagine doing... It's been a lot of work. I mean, not to discount that, but it's also like we haven't... We're not licensing any proprietary software here. We're not paying for service. In fact, quite the contrary.
The work that Augustus was doing was necessary for us to prune our GitHub collaborators because that was costing us money. This actually saved us a bunch of money. by, so the degree we were using a service, we were using too much of it.
And we were, I mean, in fact, actually, Adam, I think you kind of need to look no further than GitHub where we were, I mean, just for all the things that Augustus was mentioning about the groups and having to add collaborators, but then the collaborate, just like it was really, We were already kind of breaking GitHub. And we've left GitHub as kind of the canonical repository for comments.
But I think the direction we'd want is to use less of that. I think you could easily envision... I mean, God, if... I know that Prespo was defending Ben's time by saying, under no condition will we allow you to add comments on the RFD site. But if you were to add that, you could easily... you could see comments going into a Git repo and then us not using GitHub for comments on that. Totally.
It'd be amazing. And then I think that, like, I also think that the strength of us doing this, by the way, is that we've been able to open source it. So, this is now, like, we have generated something that, and I feel that, like, you know, as we've kind of gone through our careers, like, we've added more and more wisdom into the system. And,
it now reflects a bunch of wisdom that we can all carry forward. Even folks that are outside of Oxide can actually leverage those. One thing I did want to talk about because it did come up in discussion online where people were curious about how we use RFDs to resolve conflict. And that's not exactly their question. I'm rephrasing it a little bit. But how do you deal with conflict in an RFD?
How do you deal with where... And the comment was kind of like, hey, we experimented with it, but it just became too much of a lightning rod. And I would say that RFDs do not solve your organizational problems. They don't necessarily solve... They don't resolve those things. And, you know, we have had RFDs that become lightning rods. And it's not good.
I mean, it's not, I mean, I think it's better than, I would say these are lightning rods that are lightning rods anyway. I think as we alluded to on the cultural idiosyncrasies episode, of course, it's like the chat system, which we've already like run under the bus several times here. It's like chat is what will rip oxide apart. Yeah. Absolutely.
And this is a good example where, and I think, Adam, you mentioned this at the time we were talking about it, where all of these systems have got drawbacks. It's easy for everyone to have an opinion on it. The opinions get kind of tribal, and it's not great. And in terms of like, it's not great in that there's not an unequivocal answer. And, you know, the RFD on chat got a little hot. Yeah.
Maybe even a lot hot. Maybe I've been right.
I'll tell you, the nice thing about it is you can also tune out for those ones that you know are going to go nowhere, which is how I've viewed the one on chat. So I've yet to read those hot comments, but now I'll go check them out. Please don't. Please don't.
It is not worth checking out the hot comments on chat. And I think that on the one hand, you're not going to resolve that. On the other hand, it is actually helpful to have all the – I think it's like if you're going to have organizational conflict, Let it be in writing. You know, it is very helpful to have everything because you can actually like it.
I think it forces people to be complete about their arguments. I mean, it's obviously like think of the lawyers.
I mean, it's great for depositions. Great for discovery.
Think of the lawyers. Will somebody please think of the lawyers?
Earlier on, Chad had a question about are these really requests for discussion or are they just things that have already been discussed and decided? And very much they are requests for discussion. They're ideas that often have one or a couple of people have seen them. and then we're casting it open to other folks.
And I think, Brian, one of the things you mentioned is often that discussion happens in the GitHub comments, but it doesn't always have to happen that way. As things are getting spicy or as things are not resolving, sometimes people have a meeting, sometimes people have it in a live chat. There are other ways of bringing discussion to determinations.
And there's not like one prescriptive method for doing that or for navigating a conversation or even for knowing the threshold for when, okay, it's time to move on. We've published the thing. We just kind of feel it out and do what feels right for each one.
Yeah. And so I do have, and I actually made it public this morning or yesterday on RFD 113. which I did write, which is kind of how we actually make determinations. And it's giving this kind of loose guidance that you're describing. You can't be overly prescriptive about how decisions are made, kind of obviously.
But, you know, really talking about, and kind of differentiating decisions from determinations, talking about some specific examples where, you know, how we made determinations in the past. Because it's, on the one hand, there are, I would say there are plenty of RFDs where it's like, no, no, I want to throw this out here. I really want to get someone's discussion.
In other cases, like, we've got, and especially if you've got an issue where we have to move quickly often like, okay, there's been a lot of process that's outside of RFDs, and now actually the RFD is going to pull all this together and kind of describe what we've already done a little bit.
But it's actually extremely helpful to go do that, where we can actually go through the different things that we've kind of considered, which is very important. So I think that sometimes the RFD process ends up being a little bit behind the process, but I think oftentimes it is. It's an earnest request for discussion. There's a question about whether an RFD becomes immutable.
We do have the published state, which is very valuable. I need to get more of my own things into publish. That RFD 113 is not the published state. Actually, it's funny. So just to say it's an interesting like object lesson. RFD 113 is not in the published state. It's in the discussion state, even though it's obviously 113 and we are now at last updated in December of 2020.
It's like, okay, why is it still in the discussion state? This should clearly be like published. And so I'm like, I'm going to go mark this as the published state. But there's a comment in there, actually, our colleague, Keith Wazowski, left a very thoughtful comment on there that I had read, you know, in 2020.
But I actually want to go, there's some things in there, I actually want to modify this RFD a little bit before I publish it based on that single comment. And it's not contentious. He doesn't disagree with anything here, but he's got a perspective that I want to capture in here a little bit before I move it into the published state. So that's why that one is still in the discussion.
But yeah, I mean, in terms of when, and I think Ilyan is also saying that RFDs can languish in discussion, which I think is okay. I think it's like, I think it's, certainly I'm guilty of this too, but I think when we're moving into the published state, we're saying like, okay, this is basically like, We've kind of accepted this. It doesn't mean that it's immutable.
And I think importantly, we can go back and change RFDs that have been published. We don't want to become captive to our own process in this regard. And I think by and large, we haven't been captive to the RFD. I don't think it's been incarcerating at them.
Um, no, I agree there. I think there've been very few RFDs that really are trying to draw a very firm line that we then feel a decent tip. Like I think you're, you're spot on there that, that either we go back and we update it or we create a new RFD and say, uh, you know, this, this obsolete, it's this old RFD or our thinking on this has changed or the facts have changed in these ways.
And that's what, what changes these determinations. Um, But no, I think it's been purely valuable in terms of determination process and then hugely valuable for looking back historically and understanding, especially before we change some of these ideas, seeing that full discussion, really embracing that before we decide to go some other path.
Yeah, and then I think it's also helped them have the abandoned state. Abandoned, it is. And chat RFD may be in the abandoned state. Just to indicate to someone, especially new to the company, like, oh, okay, I don't need to spend a lot of time looking at this one.
This was an idea that, and I feel that was something, I mean, Robert, I feel that was something that was kind of a needs improvement from the Joyent era as well, where we ended up with some RFDs that were not what we did And then we need to kind of make clear, no, no, this is not, we didn't go this way.
We had a thoughtful discussion on it or what have you, but you shouldn't view this as kind of canonical because we've abandoned it. I think the other thing I would say is that when we are getting better at is linking RFDs to one another. So you've already seen this, and especially like when we were referring to Dave's RFDs last week and 110 and 53 and kind of the RFDs that they refer to.
I think that ASCII doc makes it pretty easy to link to those things and kind of footnotes and so on. But I think that being able to navigate that web becomes really important.
One of my favorite efforts that Robert undertook was to build a whole graphical representation of the interlock of about a couple dozen RFDs, it felt like. Oh, yeah. Orienting you in space. It felt like a tough thing to maintain, but man, was that helpful.
Yeah, specifically for the Gimlet hardware sled.
Yeah. And it was kind of like, you are here in this RFD, and this is how it relates to the other RFDs. And that was really, really important. And then, I mean, in our next-gen sled, Robert, you want to talk about how you, because the RFD there, we've got an RFD that really is the entire architecture of that kind of next-gen sled. Do you want to talk about how we're using RFDs for Cosmo?
Sure. So, I mean, I think to kind of talk about how we use RFDs for Cosmo, it helps a little bit to talk about RFDs for Gimlet along the way. So when we were developing Gimlet, I'd say, you know, this was a time when the first RFDs were in, there's maybe what, 12 of us, 15 of us. And this is before most of the team has started. I mean, this is mid 2020, maybe early.
So a lot of it was us trying to separate out the specifics of what was important for socket SP3 or specifics for this particular AMD aspect versus what is it we generally want. So a lot of things are split up in those early RFDs in part because of that. We had something about what does PCIe hot plug look like? What are our goals and constraints?
How do we split up responsibility between different subsystems? Just edit broad core strokes. What's even the role of power monitoring? When should we do it? When shouldn't we do it? All these things. So that gives us a lot of initial foundation there. And then from there, with Cosmo, a lot of these things that we have, we're not starting from a clean slate, per se. We have an existing system.
There's a lot of stuff we want to reuse between that. we have a large investment in hubris and tooling and different components. You know, we have vendors that we've worked with. So we're not trying to necessarily reinvent the whole wheel there. So, you know, not everything is like, not everything is starting from a clean shade again. So that kind of helps constrain it.
So in there, we kind of, in the Cosmo RFD, for example, you know, we've broken up that into a bunch of different areas that different folks helped collaborate on. Some of it was Adam and I collaborating on, what is the disk drive interface that we should use? There's U.2, E3S, E1S. There's E1L. There's three different variants of widths and thicknesses of E3S and even longer things.
How many lanes do you want? But that's something there. Sorry.
No, I just, you're reminding me of, I think, is it 101 much new about optics?
There's one of those is up there. But, but that's that. Whereas like we have a very different question, you know, where like Nathaniel, who's kind of lurking, I think on this, you know, was dealing with how do we kind of rethink about how we're doing parts of what the FPGA's role is as we want to add different features around eSpy and other things.
And so that's different places where different folks kind of collaborate with different takes on it. And then we have an appendix, to your point of things that you've abandoned. There was a bunch of early ideas that we kind of ended up not rolling with this time, but we have a note of what they were and then why we didn't, because a lot of them weren't
you know, they're still good ideas, but just didn't make sense based on certain circumstances. So it's helpful there. And I think then the other bit that we've tried to do here is that, you know, to the folks' points, you know, throughout this is like, hey, how do I ramp up when there's you know, a million words, million and a half words of context of this. And I don't know what it is.
And it's like, okay, well, there's, you know, if you're just starting, you kind of have a determination section or even a small overview presentation that someone's given. And the specifics of why don't necessarily matter. But if you do, but later on, if you come back to this, you do have specifics of why. And that's helpful for, you know, that's not helpful when you're trying to ramp up.
But if you're trying to come back to like, why did we make this decision? What were the different trade-offs?
we do something different at least you have some of the context of what were the different things that went into it oh it is extraordinarily helpful and it was it was uh 112 as much do about optics robert um i would we'll need to look to see if there's any and if there's no nda material in there i want to make maybe you want to make that one public just because that have you read 112 adam
Because you're like, I don't know, optics, how complicated can it be? It's like, oh, okay. No, I haven't. Oh my God. I mean, this is one of these and I feel like there are many RFDs that are in this category where this thing will take you to absolute crush depth on optics and how complicated it is. And
I, I mean, Robert, you've done this many, many, many times where it is so helpful to have the entirety of the depth of our thinking in an RFD. Because as you say, it's like, you know, you may look at that and God, there've been so many RFDs that I read once. I'm like, okay, yeah, that makes sense. I don't know. Makes sense.
And then later, due to change in circumstances, I'm now reading this very closely. Maybe it's because I'm implementing it. Or maybe it's because we're about to have a call with a partner where this partner is misbehaving. Or maybe we're having a call with an investor or having a call with a user. We're having a conversation where we're like, I need to ramp up on everything we know about optics.
I need to know in the next 45 minutes. It's like, good news. Um, and you can go actually get to the entire depth of our thinking.
And I think that this is where the, the written artifacts are so important for scaling an organization because it allows, I mean, Robert, with so much of your thinking written down, people can go read your thinking before they, they need to like ask you a followup question. Right. Um, and that is just extraordinarily important. Um,
So it's been, I mean, you're talking about, Adam, how this has become more valuable over time. It has become more valuable because we are able to use them more. We have more people using them, to Ben's point about making this accessible. We've got more people creating RFDs. We've got more people reading RFDs in terms of the public and customers and so on.
And then we've got these kind of fine-grained permissions so we can get the right RFD to the right person, which has been... incredibly important for us. It is the backbone of Oxide. Absolutely. This is, it's been great. Not just the discussion, but boy, this work, all of you have done such terrific work on this stuff. And it's been really, really essential.
Well, thank you. That's all I got to say.
300,000.
Wild. I'd expect an out loud rating. Adam, thank you for all the collaboration you've done with all of your colleagues on RFTs. Much less work than Robert's words. really great stuff. And again, if you're looking at this thinking like, boy, I would love to adopt that system, please adopt it. We really try to make it so it is adoptable.
So, you know, take it lock, stock and barrel, take the parts that work and report back. We'd love to hear about it because I think this is something that a lot of engineering organizations really, really need. On that, thanks, everybody. Thanks, Europe.
I feel like we'll need to do a European-timed episode at a time when you're not in Europe, just so we can prove that we're like, oh, yeah, they're remote-friendly because the CEO no longer wants to work. He doesn't want to move his family. That's why they're remote-friendly. We'll need to do some European-timed episodes. We're going to do this once a quarter or something like that.
Sure, that's the spirit.
There you go. Once a quarter, Europe.
Even you, Germany, and we're sorry again. Exactly. Who could ask for anything more?
All right. Thanks, everybody.