Matthias Endler
Appearances
Rust in Production
Zed with Conrad Irwin
When I looked through the code, I actually saw that you use both Tokyo and Small. I would assume that your own async runtime is based on top of Small, or is that even a separate runtime? You have three in total.
Rust in Production
Zed with Conrad Irwin
Yeah, and I also would assume that if you took your async runtime and published it as a separate project, that would essentially mean that you would have to maintain it for external people. And it could also be a bit of an issue in regard to how you want to use that runtime inside Z. So I'm assuming that you don't want to do that.
Rust in Production
Zed with Conrad Irwin
It's very nice to know that you have your own runtime, because I'm kind of a big proponent of diversity. And we have Tokyo, we had AsyncSTD, we have Small, and there's Glomio and a couple others. What's the state of the AsyncGross ecosystem in your mind? Yeah. I can already see that. I have some stuff to say.
Rust in Production
Zed with Conrad Irwin
Is that, in your opinion, something that will just solve itself at some point once the ecosystem grows a bit? Or is it a systemic issue?
Rust in Production
Zed with Conrad Irwin
where we had multiple iterations and we ended up with, anyhow, this error, snafu, and all of these other abstractions that we didn't really have. All of these dependencies are relatively new in comparison to how old Rust is. that took a different turn because it took some time to mature and we didn't really stabilize on one error handling library.
Rust in Production
Zed with Conrad Irwin
Whereas in asyncrust, pretty much from the beginning, people kind of settled on Tokyo because as you said, it's a much more complicated piece of software and the ecosystem just wasn't as mature yet. But now we find ourselves in a situation where Tokyo is the dominant asyncrunt time. And is it just me or do you also see that problem there? And what would be the revelation here?
Rust in Production
Zed with Conrad Irwin
At the same time, it gives me hope when you say that, because the jQuery example was a good outcome in my book, because we were able to experiment with these alternative JavaScript framework or so. And then browsers caught up and then they added some of the features into their own native implementation.
Rust in Production
Zed with Conrad Irwin
I guess the same could happen in Rust where we take parts of Tokyo and stabilize that, put it into the standard library, but we have to be careful there. For example, there's...
Rust in Production
Zed with Conrad Irwin
async read and async write trade and technically you could already stabilize that but it's a bit debated right now whether this is the right abstraction that we want to settle on going forward exactly and it it takes time to figure those out particularly for something like rust that's such a big complicated project like any extra complexity they bring into the standard library it like deserves a lot of scrutiny
Rust in Production
Zed with Conrad Irwin
Is that rule somewhere codified? Is that part of your code style? For example, you separate sync and asynchrost, or is that something that evolved naturally?
Rust in Production
Zed with Conrad Irwin
That's so funny when you say that because you almost take the function coloring problem and make it a feature where you take the slow parts and you mark them as async so that you know exactly that this is something that you should not run on the main thread.
Rust in Production
Zed with Conrad Irwin
We'll be back later. And you can see it in the type system too. Are there any other challenging aspects of building a text editor in Rust? Any unexpected hurdles?
Rust in Production
Zed with Conrad Irwin
Yeah, because everyone wants a different set of features. That too. Vim Mode is the classic example of that.
Rust in Production
Zed with Conrad Irwin
In preparation for this interview, I also checked out your new YouTube channel because you recently started a channel about talking about the internals and said, and I will link it in the show notes. It's pretty majestic. The one thing that I realized from these interviews was that you sometimes touched on a library called TreeSitter.
Rust in Production
Zed with Conrad Irwin
You mentioned it before, but that seems to be a bit of the secret sauce in there because apparently, and correct me if I'm wrong here, Other editors are not built this way. They don't work on this level of abstraction, almost like an AST, an abstract syntax tree level of abstraction to modify text. Can you elaborate on this a bit? What is it? And also, do you agree?
Rust in Production
Zed with Conrad Irwin
Yeah, and you did a great job there. I have to say there are very few gaps in the Vim support by now. It wasn't the case just half a year ago, but just seeing the rapid development is really, really good, really surprising. And also I found myself switching between VS Code and NeoVim as well.
Rust in Production
Zed with Conrad Irwin
And how does TreeSitter work on a type level? Is it like an intermediate representation where you map certain keywords in different languages to the same keyword in TreeSitter?
Rust in Production
Zed with Conrad Irwin
That's a very smart move. Where does the name TreeSitter come from? Is that because it's almost a recursive structure where you can think of it as a tree inside a tree inside a tree where you have different levels of abstraction that you can iterate on? Or does it come from the abstract syntax tree?
Rust in Production
Zed with Conrad Irwin
It's sort of library where you can modify code based on certain, I would even say like a grammar or something where you can say, I want to do a code modification and I give it an expression and then it figures out what to do based on this expression. It's,
Rust in Production
Zed with Conrad Irwin
more high level than a regular expression it's also more powerful i would say can you do certain things like this in z as well on a syntax level i know it's not exposed to the user but internally could you do such modifications
Rust in Production
Zed with Conrad Irwin
So every time I became sick of VS Code's laggy performance, I would switch to NeoVim only to find out that the configuration was a bit of a hassle. And so I have to switch back at some point. Exactly.
Rust in Production
Zed with Conrad Irwin
Every day I get an update, which is nice because they work flawlessly and they work every single time. And I find myself reading the change log a lot because the change log is nicely formatted. You can see what's going on. I wonder how you do these updates. How does the pipeline work to create these updates? How do you push them to the clients? And how do you make it so flawless?
Rust in Production
Zed with Conrad Irwin
This is Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. My name is Matthias Endler from Corot, and today we're talking to Conrad Irvin from Zed about building a high-performance code editor in Rust. Conrad, thanks for being here. Can you quickly introduce yourself and Zed, the company you work for?
Rust in Production
Zed with Conrad Irwin
The tree sitter is written in C and is wrapped in WebAssembly. That means you must run some sort of WebAssembly runtime. Is that a custom build or do you use anything?
Rust in Production
Zed with Conrad Irwin
And will you support traditional extensions at some point too? And how will that look like? Definitely yes. And who knows?
Rust in Production
Zed with Conrad Irwin
Did that also entail any differences in how Z is architectured in comparison to other editors? Or was it just mostly Rust's performance that made it so snappy?
Rust in Production
Zed with Conrad Irwin
I definitely appreciate the focus on stability and performance because those are the main two reasons why I use it. I would like to keep it this way. That's kind of nice.
Rust in Production
Zed with Conrad Irwin
And once the project grows, it will not get easier because I just checked yesterday and Z is approaching 500,000 lines of Rust code, which is crazy. It's crazy. And I saw some interesting bits that I learned from you. For example, you have a flat hierarchy of crates. You have one workspace still. You keep it all in one workspace. Right. And then you follow a very flat hierarchy.
Rust in Production
Zed with Conrad Irwin
Can you elaborate on this design decisions and maybe other decisions that are reasonable or maybe even necessary after reaching a certain size of like a code base?
Rust in Production
Zed with Conrad Irwin
It's funny because lately I read an article by Matt Klett, who is the author of Rust Analyzer. And he mentioned this one thing about nested trees where you have to make a decision where you put things. And it's a conscious decision. It might be wrong. And so eventually you might end up in a suboptimal space, in a suboptimal structure of your project.
Rust in Production
Zed with Conrad Irwin
So he advocated for a flat hierarchy too, where you don't even have to make the decision because, yeah, it's flat anyway. If you are wondering where to put it, the answer is put it into the root of your workspace.
Rust in Production
Zed with Conrad Irwin
Any other such tips or perhaps even things that you would avoid now?
Rust in Production
Zed with Conrad Irwin
And just to clarify for the people who are listening, this also happens if you just use these generics inside of your workspace in a different crate, because that's a separate compilation unit. And that means even just exposing it within your project might be problematic at times. And what about lifetimes? Because... you have a very high focus on performance.
Rust in Production
Zed with Conrad Irwin
You want to make this thing as fast as possible. And one suggestion of various people that come from systems level programming languages like C++ and C is that you want to avoid allocations. You want to make everything a view into memory as much as possible. And you want to deal with the raw data as much as possible.
Rust in Production
Zed with Conrad Irwin
Is that something that you have to follow to reach that level of performance or are there ways around it? So for the hard code paths, yes, for sure.
Rust in Production
Zed with Conrad Irwin
And when you run into a problem with performance, what's your approach? Do you benchmark that before you make any changes or do you just guess where the bottleneck is and have an intuition for it?
Rust in Production
Zed with Conrad Irwin
By frame, you mean what? A frame of memory in the kernel? A frame of pixels to update. Ah, okay. Do you use a lot of macros for code generation, or is that another thing that you tend to avoid in the hot paths of the code?
Rust in Production
Zed with Conrad Irwin
When you ported Z to Linux, were there any surprises that you hit other than the mentioned issues with dropping frames?
Rust in Production
Zed with Conrad Irwin
I learned that it also compiles on Windows. Do you want to comment on that?
Rust in Production
Zed with Conrad Irwin
When you explained that, I wondered, how would I test that? I would have a really sophisticated test environment for different environments. Do you test it in VMs? Do you test it just with unit tests or manual testing? How does that part work? Testing in general or cross-platform testing? Interested in both, yeah, but specifically cross-platform. Okay.
Rust in Production
Zed with Conrad Irwin
Do you focus on unit tests or integration tests for the rest of the code base?
Rust in Production
Zed with Conrad Irwin
How do you communicate with the server? JSON? Protobuf. Why exactly do you use Protobuf and not anything else?
Rust in Production
Zed with Conrad Irwin
Right, so the editing part works on the CPU, I would assume. That's the tree set apart. And the rendering part of the UI that works on the GPU, is that correct?
Rust in Production
Zed with Conrad Irwin
There are a few things that come to mind. One is Postcard by James Muntz, who, it has a slightly different focus. It's serialization format, yeah, but it's not based on Rust structure as far as I remember. Then there's Seabore, which is another serialization format. I honestly don't know what that wire format looks like, but I think there's also one that is based on Rust structs themselves.
Rust in Production
Zed with Conrad Irwin
I think the one that I meant was called Arson. Okay, no, that's different again. It's like a JSON-like thing, but with Rust structs. Because the one issue that I found with protobuf was that you need to carry the definition file. You need to put it somewhere, and then you need to compile your stuff against whatever protobuf definition you have somewhere.
Rust in Production
Zed with Conrad Irwin
Yeah. Shout out to James for building that. I took a look at the issue tracker and I found that one of the most often, if not the most often requested features that's missing in set right now is debug support. And a lot of people might say, well, why haven't they added it just yet? And how can it be so complicated? Can you say a few words about that? Sure.
Rust in Production
Zed with Conrad Irwin
Nice. I'm really looking forward to that. Yeah, you and about 500 other people, I think, based on the outputs. The pressure's on, but I'm sure that when it hits, it will be fine because this is how I experienced this editor or this approach so far. It's always very well thought through, which is great. And speaking of which...
Rust in Production
Zed with Conrad Irwin
And is that something that you have to handle on the Rust side or does that more or less evolve automatically when you build the application with that in mind? Is that something where, for example, you have a hypervisor for the GPU and then you have some other thing that takes care of state of text files that you have currently loaded? Or is that something that more or less happens automatically?
Rust in Production
Zed with Conrad Irwin
About things that look easy on the surface but are hard in hindsight, is there any peculiar bug that you remember? Anything that people take for granted but in reality is very hard to pull off?
Rust in Production
Zed with Conrad Irwin
Yeah, most text editors would even crash at this point or they would not even get to that point.
Rust in Production
Zed with Conrad Irwin
Exactly. And it's also elegant because if you use two big vectors, you probably have some jank on the interface between the two or like in between the two when you jump from one block to the other. But if you use a smarter data structure, you kind of circumvent that issue altogether.
Rust in Production
Zed with Conrad Irwin
It sounds like a very hard problem that you have to figure out once, but then you never have to touch. Are there other things like this in the editor?
Rust in Production
Zed with Conrad Irwin
yeah beyond that it's it's hard to say i think the yeah i think the crdt it's one of the few parts of the code base where i basically never find myself there because everything just works which is nice let me tell you about my first interaction we've said i had a problem with vim support and i went into the issue tracker and i found an issue that exactly described my problem and
Rust in Production
Zed with Conrad Irwin
And so I read the last couple of comments. And what I found interesting was that you, Conrad, reached out to the people in this issue and said, I'm open to hack on this together. That was the first time I ever saw this level of interactivity with any project, because it's not only...
Rust in Production
Zed with Conrad Irwin
Through issues, you would invite people to block time in your calendar to use the tool you're going to improve together and work interactively. Is that something that you do a lot? How does that influence your workflow?
Rust in Production
Zed with Conrad Irwin
If someone out there wants to contribute to Zed now, how do they get started? Very simply, look in the issue tracker.
Rust in Production
Zed with Conrad Irwin
We're coming to the end and it's become a bit of a tradition around here to ask this one final question. What will be your statement to the Rust community?
Rust in Production
Zed with Conrad Irwin
Very nice final statement. Conrad, thank you so much for taking the time. It was amazing.
Rust in Production
Zed with Conrad Irwin
Rostium Production is a podcast by Corot. It is hosted by me, Matthias Endler, and produced by Simon Brüggen. For show notes, transcripts, and to learn more about how we can help your company make the most of Rost, visit corot.dev. Thanks for listening to Rostium Production.
Rust in Production
Zed with Conrad Irwin
Someone might listen and think, oh, why didn't they use Leptos or Deoxys or all the other Rust UI frameworks? Why did they have to invent their own thing? What would be your answer?
Rust in Production
Zed with Conrad Irwin
And absolutely, you can feel that when you use it because it's snappy. It feels native. That, by the way, was one other thing that I always missed in NeoVim. It didn't really feel like a native application. It felt like a terminal emulated thing that ran inside of, you know... Some terminal, of course, which it kind of was, but that always kept me away from other things like Emacs, for example.
Rust in Production
Zed with Conrad Irwin
It's pretty amazing. I have to say, I'm a set user myself. I completely switched from VS Code. And I can tell you the experience so far is fantastic. You did a great job. And this is also why I wanted to talk to you folks. First off, in your words... What's wrong with the existing editors? What are some pain points? Why do we need another editor?
Rust in Production
Zed with Conrad Irwin
I do like all of these experiences, but I also want the native look and feel because especially on macOS, if you're used to this, it's very hard to switch away from that experience again. Yeah, I strongly agree.
Rust in Production
Zed with Conrad Irwin
Yeah. Let's see, which keyboard shortcuts do you support? What sort of keyboard key maps, I would say, do you support? You support Vim. Then you support the normal command or control-based keyboard chords, almost. Then you have actual chords, which are things like GD or GR, anything else that I'm missing.
Rust in Production
Zed with Conrad Irwin
Well, when you describe it, it sounds kind of straightforward, but you have to plan for this from the very beginning, because otherwise you will not be able to manage to support all of these different ways of input, right?
Rust in Production
Zed with Conrad Irwin
Can you just quickly and briefly describe to us how that part works, how that part of set the keyboard input works? Because in my mind, it's event based. You have things like you definitely don't want to block the rendering in any way. You probably need to do that asynchronously or event based. But I don't want to put words into your mouth. I want to hear from you. How does that part work?
Rust in Production
Zed with Conrad Irwin
I guess you hear that a lot, but I want to hear the answer from you. Makes sense.
Rust in Production
Zed with Conrad Irwin
And all of these events, they end up in some sort of state machine, or are they more or less modifying the state and then they are more or less gone from the system again? A little bit of both.
Rust in Production
Zed with Conrad Irwin
And this is probably also how you circumvent issues with self-referential structs in Rust where maybe you want to, you know, you described a mechanism where you had an event that would trigger another event and you would have a callback upon a callback maybe. And these things, they trigger some sort of PTSD in my mind because...
Rust in Production
Zed with Conrad Irwin
If you do it the wrong way, then you will end up in a thing that doesn't compile anymore. And if you kind of have a dispatcher in between, you can completely get rid of this problem because asynchronous solves that problem for you. Is that correct?
Rust in Production
Zed with Conrad Irwin
Are there any issues that you defer to runtime or is all of that done at compile time?
Rust in Production
Zed with Conrad Irwin
It'd be really hard to check. What are the additional type safety guarantees that you mentioned in your own implementation of this Arc alternative?
Rust in Production
Zed with Conrad Irwin
Right. Is that something that I would be able to use as an external person, maybe as part of my application? I know that you open source the entire code, but is that something that I could use in any other context?
Rust in Production
Zed with Conrad Irwin
Right. So these are abstractions that you built specifically for GPU UI and things that are paper cuts in your working environment and you just had the power to do it. So you went ahead and did it. Exactly. And so, yeah, it gives us a whole bunch of nice, nice guarantees.
Rust in Production
Zed with Conrad Irwin
Is that a very common pattern, I'd say, where maybe the ecosystem is just not there yet, or you have very specific requirements, so you have to build it yourself? Or does the ecosystem help you a lot?