Rain Paharia
Appearances
Oxide and Friends
Crates We Love
Yes. Yes, it is. Right. So as an example, actually, one of the examples is that you can integrate it with SOD JSON. So you can actually get great highlighting for which bit of a SOD JSON thing failed. And I think that's really cool.
Oxide and Friends
Crates We Love
I think in general, though, there is some value in having, you know, the kind of thing I wrote, which is like, you have like a nested tree structure, right? And you are parsing through the nested tree structure.
Oxide and Friends
Crates We Love
And you want to actually not just fail on the first error globally, but you have kind of a notion of like, you want to go through as much as possible and you want to collect as many errors as possible. And I have had to do that a few times, and I've pretty much handwritten something each time.
Oxide and Friends
Crates We Love
So that kind of suggests that maybe, I don't know if the audience has a suggestion for something that kind of does that. Otherwise, that might actually be worth doing and kind of putting out as a separate thing. Yeah, it is kind of a much bigger scope thing, but yeah.
Oxide and Friends
Crates We Love
I have a pair of really, really cool SerD-related crates. So there is a crate called SerD-Ignored, and there is another crate called SerD-PathToError. So I think both of these are really good. And again, kind of coming at it from the you want to produce good error messages kind of thing, right?
Oxide and Friends
Crates We Love
So one of the things that I've noticed when defining, say, a configuration file is that people will often misspell things, right? Surdy has this really cool deny unknown fields feature. Yeah, I love this. Right? So, and deny unknown fields is great, but sometimes you don't want an error, you instead want a warning. And Surdy ignored actually kind of lets you get that warning.
Oxide and Friends
Crates We Love
So it's kind of somewhere in the middle between like the silently accepting the, you know, maybe the typo. or failing, and I really like Serdy Ignored for that because often you want to support some kind of forward compatibility, and if you have that forward compatibility, then you don't just want to choke if you see a new option or whatever, right?
Oxide and Friends
Crates We Love
And so Serdy Ignored does a really, really good job of reporting that. And then kind of paired with that, but kind of solving a slightly different problem, is certipath2error. And what certipath2error does is it will try and report the nearest part of kind of what failed. So it'll kind of maintain some state and like, you know, which keys have you traversed into and so on.
Oxide and Friends
Crates We Love
And it does a pretty good job of that. So, you know, there's one specific asterisk which we don't really want to kind of get into right now because it detracts. But overall, like this pair of crates has just kind of been, I feel like this has like really elevated the, you know, like kind of error handling experience around configuration files for me.
Oxide and Friends
Crates We Love
Oh, boy. Um, uh, that was actually, uh, as far as, uh, I mean, I had a couple of other detail nuggets, but they'll come up there. Um, another crate that I wanted to call out, uh, and kind of, you know, as a cool proc macro crate is, uh, derive where, um, so, um, so one of the things that, you know, kind of people run into sometimes is that, uh, you want to do say like a derived debug, right.
Oxide and Friends
Crates We Love
Um, or like a derived clone or something. And like, uh, if you have a. create which has a generic, like t colon clone, then the implementation that Rust generates is you only derive clone if t also implements clone. You only get to implement clone that way. That is like mostly what you want, but sometimes not. And so, you know, one option is you, if you don't want that clone bound, right?
Oxide and Friends
Crates We Love
Like sometimes you're not actually storing a T in there. You are storing, say, some kind of derivative type of T. then you do that. But the one I really like that kind of automates this is derive where. So what derive where lets you do is it lets you say derive clone where sum bound. So you can say derive t clone, sorry, derive
Oxide and Friends
Crates We Love
my type clone where T is, you know, like some implementation, some implementation of some trait that you've defined, but you can also save like, you know, you don't have any restrictions on T. So you can just do derive where clone. I remember like showing something else at a demo day and then everyone else was like, what's this derive where thing?
Oxide and Friends
Crates We Love
And then that ended up joining into the Driveware demo. It was very funny. But it's a create that I really like, and I end up reaching for it a few times a year. How does this work? As far as I can tell, it just generates a proc macro that iterates over the fields. But it just puts bounds on them. So it's just a proc macro in that sense.
Oxide and Friends
Crates We Love
Yeah. Definitely one of my favorite little proc macros that help out.
Oxide and Friends
Crates We Love
I think what I ended up having was I was storing a phantom data of D or something like that. So I was storing like... So I had this thing which only stored the T as a marker, so it didn't actually store any concrete values of T. And implementing something like clone for that should not require that T implement clone, right? Just logically, that is not a requirement.
Oxide and Friends
Crates We Love
So I ended up reaching for derive where with that. That's a pretty common thing I end up having to do.
Oxide and Friends
Crates We Love
Um, I think, uh, I think what I ended up doing, God, this is telepathic crate powers.
Oxide and Friends
Crates We Love
I think, you know, I was thinking about how I'd do that, and I feel like there's no magic, right? In this case, I ended up Googling for, I think, Rust-derived custom trait bounds or something, and derived where is in the first page of results. But, you know...
Oxide and Friends
Crates We Love
I'll spend a little time looking on Google and Create.io, and I'll also maybe ask some people who are there, like, I know, do you know something? And then sometimes I can't find it. It's just, it's really hard. I think one of the ways, one of the more, I think, structured ways that has helped is like,
Oxide and Friends
Crates We Love
Like if you have a particular code base you like, which you feel like might use something like that, then kind of dig around in that code base source code. I think that is kind of, you know, that, that feels like a good way. And I've discovered a whole bunch of crates that way.
Oxide and Friends
Crates We Love
I think a place where that's kind of organically arose is... is with CLI parsing crates, because there's a whole bunch of CLI parsing crates. So some things like clap, which many, if you've written a Rust CLI tool, you've almost certainly come across clap. But there's a whole bunch of other points in this design space that people have hit with various trade-offs.
Oxide and Friends
Crates We Love
And I was really appreciative of people really put together benchmarks for like, you're considering things like, how long a build takes and like how many bytes get added to the final binary, right? Versus like error handling and so on. And I think, you know, different projects can reasonably make different trade-offs here.
Oxide and Friends
Crates We Love
And one of the things I, this table was like when I saw this table and when I saw like, you know, the amount of work put into it, it was just very, very impressive to me.
Oxide and Friends
Crates We Love
Yeah, and in particular, like, I mean, clap has a couple different ways to use it. You can use it with or without the proc macro, but then there's a bunch of others. So actually another one that I really like that is much lower level than clap is lexopt. So the goal of lexopt is like all it gives you is an iterator over the options, right?
Oxide and Friends
Crates We Love
So you're getting an iterator, and in the iterator, you get a little bit of structure. So you get whether it's a single dash or a double dash. So you get very, very basic things like that. And some, if you really want that low level of control, then lexopt is great.
Oxide and Friends
Crates We Love
But the trade-off there is that you need to write your help yourself, and you need to remember that each time you add a thing, you also need to add the help for that. And maybe the error messages aren't as good and so on. So these are the kinds of things that... that you have to consider. So, you know, I recommend clap as the thing to go to, right, if you want to start.
Oxide and Friends
Crates We Love
But these are all things that are, you know, worth considering for things like embedded binaries and so on.
Oxide and Friends
Crates We Love
Yeah. It's, it's, it's really cool. I have, I've actually used it in combination with clap. So I was like, you know, there were places where I had clap to the first wall and then, um, And then I wanted a second level of parsing for something more detailed. And then I used LexOpt for that. So ultimately, it takes a bunch of strings. It is a thing that takes up strings and produces output.
Oxide and Friends
Crates We Love
I have a great idea. So is BetGraph due mainstream to talk about here?
Oxide and Friends
Crates We Love
So PetGraph is a crate I've had the good fortune to use a few times in my career. And it is a crate that lets you represent graphs, right? So it is a crate that essentially has a bunch of graph data structures, and you can represent your things in there.
Oxide and Friends
Crates We Love
One of the things, you know, and I was thinking about why I like PetGraph so much and like, you know, there's some other places where I will like handwrite my own representations rather than using some framework someone has provided. And like, you know, for this, in this case, PetGraph is like, it is a whole framework, right? You kind of model your data, you put it into their data structures.
Oxide and Friends
Crates We Love
And for me, I think the distinguishing thing is that PetGraphs gives you a lot of value from that thing. So there is a wealth of graph algorithms that are included in PetGraph. So, you know, there's like two different SCC algorithms. There's a bunch of different like, you know, like min, you know, the max cut min flow stuff. Like there's a lot of really careful handling.
Oxide and Friends
Crates We Love
And so, you know, at this point it's like, okay, you know, if I have a graph and one way I could do a graph is like, you know, the simplest way is you can imagine like a,
Oxide and Friends
Crates We Love
a node with like a arc of node of children or something right like a or something like that um and i think um you know what kind of what can you end up having to write your own algorithms uh on top of that uh but petcraft just kind of you know you have to do a little bit of work to fit into it but it just gives you all of these algorithms and like there have been times where i have thought that all i want is like a dfs and you know you could probably write a dfs by yourself
Oxide and Friends
Crates We Love
But then I realized, oh, you know, in some cases the graphs can have cycles. So I need an algorithm to kind of convert the graph into like what is called a condensation graph, which is the same graph but without cycles. And then, you know, so it kind of gives you all of these things. And I don't know, there's something very satisfying about PetGraph in a way that I really like it.
Oxide and Friends
Crates We Love
So PetGraph is really interesting because it actually presents four or five different representations of a graph. So there's the adjacency list graph, which is the default graph, right? If you want a graph, then you probably want to reach for an adjacency list graph, right? There is also an adjacency matrix graph,
Oxide and Friends
Crates We Love
um uh which you know if your graph is uh you know in some cases you want to use the matrix representation of things and you can do fancy things with eigenvectors and so on um um there's um also like this uh other one that lets you kind of uh so the the the first representations only let you use integer keys um if i remember correctly
Oxide and Friends
Crates We Love
But then there's also one that lets you use your own keys, anything that implements copy and I think hash and EQ or whatever. But then it also provides essentially like an abstract interface. So it provides a bunch of traits. And if you have your own graph and so you can bring your own graph and you can implement, you know, those traits for your graph.
Oxide and Friends
Crates We Love
And if you do that, then you get access to like the full set of algorithms to the extent that your create support that to the extent that your graph supports. That's awesome.
Oxide and Friends
Crates We Love
So I got one. So this is a crate that... So I maintained this crate, but I didn't write it. I just happened to have been the one that manages its crate cyber releases. So this is a crate called Camino. So this was originally written by Boats, who was a Rust project alumna, So they were the one who drove like async await, for example, in Rust. And so they've done a work.
Oxide and Friends
Crates We Love
And so one of the things that they did was, if you've done anything around bots in Rust, like file bots and file names and stuff, then it's always been bothersome because in the very, very typical Rust will be like anally correct about everything as far as possible. So that mindset kind of gets reflected in the way the path libraries are designed.
Oxide and Friends
Crates We Love
So they will handle weird things like unpaired surrogates on Windows or like non-UTF-8 paths on Unixes. And so that ends up being like, if you want to write a tool that is as correct as possible and handles as many files as possible, then you probably need to take care of all that. But in reality, most of the time you don't, right? Most of the time, like if imagine you're like an oxide, right?
Oxide and Friends
Crates We Love
And you're writing a simple, like a simple like server or whatever, like you are going to, like the files you're going to get and the files you're going to use are like They're going to be, like, well-structured, right, in some way? Right, right. So Boats wrote a library called Camino, which... essentially replaces OS string as the base with string as the base for things.
Oxide and Friends
Crates We Love
So these are bots that behave like strings. So they don't handle every possible path, but they handle basically every realistic bot that most programs are ever going to see. So this is a crate that I use for pretty much everything that I end up writing, and I think most people should use it. There's a,
Oxide and Friends
Crates We Love
there are actually some, now this sounds like, you know, there sounds like a trade-off in some cases, right? Like you're, you know, you're losing some functionality or whatever. But one of the things I've realized from my time working on this stuff is that actually that trade-off was always false.
Oxide and Friends
Crates We Love
And so as an example, like, you know, if you say, if you have a path buff and that path buff has a path that isn't a valid string, then that path does not get serialized as JSON properly. Right. As an example. or if you get a string, it won't get serialized properly. So if you are ever deserializing bots, you are already putting in a restriction that those bots must be valid strings, right?
Oxide and Friends
Crates We Love
So you are not adding anything new here. And I think Mino kind of is a real improvement to anyone who does that. So I know that at Oxide, we use it a bunch. I've used it a bunch. But yeah, I think if you want... If you want to handle paths and you don't already know that you need to handle every possible path, then you should consider using Camino.
Oxide and Friends
Crates We Love
Sure, yeah. So both of this was just something that actually it was your code. I'm pretty sure it blamed you, Eliza. So I love bytes because it kind of presents this unified interface over. So bytes comes with a type called bytes, which represents this So it uses dynamic dispatcher under the hood, but it is a type that represents a contiguous sequence of bytes.
Oxide and Friends
Crates We Love
Bytes also comes with a trait called buff, and that buff trait does not require the sequence of bytes to be contiguous. So you can imagine a different implementation, which actually is the segmented list or a segmented queue, which ends up being the right data structure for this off byte sequences. So buff list is actually that segmented queue. And I might've talked about it.
Oxide and Friends
Crates We Love
I think I talked about it in the episode where we talked about prop test and verification, but that was where I ended up writing a cursor type over it, one that can essentially navigate this queue and use prop test for that, and ended up finding six different bugs, because like Eliza, I find it very, very hard to reason about these things by myself.
Oxide and Friends
Crates We Love
Yeah, that is great, Eliza. I ended up writing the very incorrect at first, but now fully correct cursor implementation. That was my contribution to it.
Oxide and Friends
Crates We Love
Um, yeah, uh, the last one I actually wanted to mention, uh, because I think it, it deserves a real, real shout out is, uh, Winnow. So, um, maybe this came up in the, in the chat earlier, but, um, so, uh, so I got a, you know, I got a, I got a degree in computer science and like, you know, one of the requirements is a compilers class.
Oxide and Friends
Crates We Love
And like, I hated writing compilers and I hated writing parsers. That was my least favorite class out of the whole thing. Um, and. Since then, I've had to implement parsers a few times, and each and every time I've just, like, it's been miserable. And NOM, so I ended up using NOM for something. And NOM, I think, is a great library.
Oxide and Friends
Crates We Love
There's a whole bunch of trade-offs across all the different libraries. But I ended up using NOM for something, and I thought NOM was okay. Winnow feels like the first time where writing a parser was a joyful experience, which is not something I ever thought I would say about a parser library. So I did want to make a special shout out to Winnow. Ed Page has done a lot of work on this stuff.
Oxide and Friends
Crates We Love
And Winnow is absolutely like, I think if you want to write something parser shaped, then you should probably take, you should either use Winnow or if you want to do your own thing, you should look very heavily at Winnow and see what it does and kind of use that as inspiration.
Oxide and Friends
Crates We Love
Yeah, it's one of those things, right, where it's, like, it says 0.6 or whatever, but, like, it is, like, too high quality to be, like, you know, just kind of treated that way. I think it is, like, it is a very, very mature crate. I've used that. I know a bunch of other Oxide have used it.
Oxide and Friends
Crates We Love
Pretty sure Rai, I think I pointed Rai to it, and he was really excited, and he ended up using it, and he was pretty happy with it. So, yeah, Winnow is my shout-out.
Oxide and Friends
Crates We Love
Yeah. Is there anything like that? So I spent, it's funny because after we talked about it last time, I ended up spending a little while looking at it. And there are a bunch of great libraries and actually some of them that I wanted to talk about here. There wasn't quite anything that I noticed kind of hit that exact spot.
Oxide and Friends
Crates We Love
And partly because I think one of the things that kind of becomes challenging is that you're If you want to do good error handling, and this kind of goes into the cosmic balance thing that you were talking about, Brian, if you want to do good error handling, often you can no longer use good type system things.
Oxide and Friends
Crates We Love
So as an example, one of the ways you might model something in Rust is with a result of the OK value or the error value, right? But if you want to like collect errors, then often something you will do is you'll pass in like an ampersand mute error collector or something like that. And the value that you returned like is an option.
Oxide and Friends
Crates We Love
And now you have to know that if, you know, there's kind of this implicit invariant here that if there is an option, then that means that you had at least one error go in and so on. A crate that I did actually want to call out, though, and something that doesn't quite solve this specific problem, even though I wish it would, is Miette. So Miette is a really, really cool crate.
Oxide and Friends
Crates We Love
It is kind of... So if you're familiar with Rust and, of course, DTolney's crateverse, you'll have come across this error and Anyhow, right? Miat is kind of a combination of this error and Anyhow, and it kind of meets both of those things. But another crate that it actually meets is Codespan. So,
Oxide and Friends
Crates We Love
If you're familiar, one of the things that's really interesting about Rust is that Rust C has great error messages. And I think that's one of the reasons that all of us feel pretty good about Rust, right? Is that fair to say?
Oxide and Friends
Crates We Love
And with the error messages, I think one of the things that's really nice is there's this lovely syntax highlighting where it'll show you the exact things that were wrong and it will give you a suggestion of what to do instead and all of those things. Amazing. Yeah, it's so good. And so there's actually a few crates that do that.
Oxide and Friends
Crates We Love
So the Rust C's own error thing is extracted out into a crate that I don't remember the name of off the top of my head. Then there's another crate called ColdSpan, but Miette also captures all of that. And Miette actually, one of the things it can do is it can store a list of errors.
Oxide and Friends
Crates We Love
So what you can do is, and I have used this pattern in some places, is that you actually store a list of errors and then you have Miette report that with, so you can provide the source code that those errors associated with and the byte offset. And so you kind of provide that source code and then Viet will kind of render that in a nice way.
Oxide and Friends
Crates We Love
So it's, I think that kind of style of like high quality error reporting is actually something that is really, really cool about Rust. And I don't know if there's any other ecosystem that has paid this much attention to like how your error messages look, right? Rather than just reporting like your line number or whatever.