The car industry is not known for its rapid adoption of new technologies. Therefore, it's even more exciting to see a company like Volvo Cars embracing Rust for core components of their software stack.We talked to Julius Gustavsson, System Architect at Volvo Cars, about the use of Rust for their Electronic Control Units (ECUs) in Volvo's EX90 and Polestar 3 models and how they are building a Rust ecosystem within the company.
It's Rostium Production, a podcast about companies who use Rost to shape the future of infrastructure. I'm your host, Matthias Endler from Corot, and today we talk to Julius Gustafsson from Volvo about putting the good kind of Rost into cars. Julius, thanks for being a guest. Thanks for taking the time. Can you introduce yourself and Volvo, the company you work for?
I'm Julius Gustafsson, and I work for Volvo Cars, and I am the main architect and team lead for the LPA project at Volvo Cars. And the LPA, which stands for the Low Power Processor, is the first ECU in the automotive industry, at least as far as we know, that is fully written in Rust. And it's rolling off the production line as we speak.
The first time I heard about this project was even a few years ago when we were at a conference. I can't even remember which one. But someone mentioned that Volvo was working with Rust. And by Rust, we mean Rust, the programming language, of course, not Rust, the oxidation process.
But it was way back in the day, and I didn't even know what parts of it were public or if that was a separate vendor. Now, take us back to maybe 2018 or so when this project started. What was the situation with Rust back then? What was the situation, especially with embedded Rust? And how did you even think about using Rust at Volvo?
Yeah, so I joined Volvo in 2017, and that's about the same time as this Spa 2 project started. And right...
basically that when i was joining i was also discovering or i was i'd already discovered rust a couple years back but i was getting more and more certain that this was something that that we needed to look into and so we started out with a few products when we were like in this very very initial stages of that project we were doing all kinds of proof of concepts and
And the first thing I did there to test out Rust was to, since I came from an Android background, so I had, I knew the ins and outs of the Android system. So I was able to build an Android hull, an automotive hull in Android, was able to get that to build and link and do all kinds of shenanigans to get that to work because the build system didn't really support it. But yeah.
I was able to work around that. So this Android HAL, basically, the point of it was to use the Android UI to control the AC or the fans in the car. And we had already built this small system that was running on a separate, like a Raspberry Pi, that would actually send the CAN signals to the fan. So we needed to communicate with that, to send commands to that system, basically.
And so I used, or I created a, how that actually communicated with that over gRPC, if I remember correctly.
using futures and and and back then there was no async await so this was like yeah uh futures zero zero point one or yeah and and it was it was such an amazing experience because yeah you know you had to do all these things to get everything to work because there was so many moving parts to get like the things to build in the build system and and yeah getting the whole like
from the button to the actual methods being called in the HAL and then having that send out something over gRPC, over Wi-Fi to that other chip. And basically when I had it all wired up and it started to build, it took me a while, the fans just turned on on the first try. So it's just super amazing. I had never experienced that before.
And so that was like a first proof point that this is definitely something.
How much of that was because of Rust and how much of it was because of your experience with embedded systems?
I would say, of course, my experience didn't hurt. But I know for a fact I've been doing C and C++ for mostly C though, but a fair bit of C++ as well for the better part of 20 years. And every single time there is always something, or at least something creeps up. Especially when you're least expecting it.
There's something that you didn't think about, or someone else didn't think about, or something you thought of that wasn't written down, and then someone else broke that assumption. There's always something that... that makes these kind of, especially when it's so many moving parts in this little proof of concept that I was doing there, that, yeah, that just wouldn't have happened.
Or at least not for me, that's for sure. So yeah, Rust definitely did the heavy lifting there, or at least made sure that I had, because Rust tends to make you think through the whole design up front, basically, or at least, you know, a much larger part. So, yeah, so there are these fine, you know, loose ends that you also need to tie up before it actually builds and runs.
And I think that those are the things that make the difference.
Why did you decide to build that with AsyncRust, given that it was still in an alpha version? Couldn't you have done something with Syncrust?
Probably, but the nature of the, since we were doing like gRPC communication over Wi-Fi, it was very asynchronous in its nature. So I wanted to try it out and see how it would work. So yeah, that was the idea.
A lot of backend developers hearing that might be a bit scared by the way AsyncRust worked back in the day where even back in the day, I think you needed to build your own futures. Was that still the case back then?
Yeah, that was definitely the case.
And you still saw the potential there, even though it was a bit, let's say, half-baked or it wasn't completely fleshed out yet?
Yeah, absolutely. Well, needless to say, then maybe we'll come to that later on how we came to the LPA project, but where we're not using it, at least not yet.
But yeah, I'd done a fair bit of... And also a lot of embedded systems are callback-driven in their nature because you're getting all kinds of events from hardware peripherals, from other things, then your main thread constantly and also your... When you're achieving something, you usually need to do it in steps.
So you do something, register a callback, wait for that callback to be called, then you continue. And even in that stage of futures at the time, it was still removing a lot of boilerplate already then. So that's why I thought it was intriguing.
Given your background as a C developer and probably also as someone who has a fair share of experience with embedded systems, how would you have done that with the technologies that you used before? C, how would that look like?
It's a good question. I guess I would most likely use some sort of async framework in C++, I guess, since we had gRPC, and gRPC, protobuf, all that stuff was already fairly well supported in C++, so I guess I would go for that. And I don't remember how... if those async frameworks at that time in C++ were, how far they'd come. They probably had similar sort of features.
The problem there is that you usually don't... When the closure that you register actually gets called, you're basically on your own because the... There are all sorts of assumptions that need to be upheld for that to actually work properly.
For instance, which assumptions?
For example, memory or variables or pointers and things that might not exist when it comes back. So yeah, dangling pointers is a common issue.
Even for someone like you who has quite a ton of experience in that area?
I would probably trip on those a couple of times before I, you know, before I get it working. But because it's so easy to, yeah, it's super easy to misstep when it comes to those things.
So for some context, we are in 2018-ish and you built your first hardware abstraction layer for controlling the fans from Android. And you saw it working on first try. That must have made you pretty excited about Rust and its potential to go from idea to first prototype layer. in a reliable manner. Did you show that prototype to other colleagues?
Did you talk about Rust back in the day, or was it more of a hobby and side project?
No, I was probably quite obnoxious. I'm pretty sure there were some colleagues that were getting quite fed up because I was talking about it a lot. I remember I did a talk back in 2017, or no, it must have been early 2018, where I tried to do a basic introduction, but also to point out all the upsides.
And then when this SPA2 project started to crystallize more and we started to productify everything that we had in our plans, i i started looking for places where or or see if this was actually viable as as a language to use and unfortunately none of the platforms so so the so this yeah if we go go back a little the the spot two project is it's the electrical system in in the
in the Volvo cars that are coming to market now. And the main big feature of that system is the core computer, which you can probably Google and read about. And this core computer is essentially a centralized box with multiple different processors, all running different functional domains, doing different things. Like you have security, you have...
high integrity for like braking and steering and, and these safety critical things. And then you have a high performance compute for, for, uh, these more big number crunching things and, and 88 us functionality, you know, autonomous driving or, or yeah, that, that kind of use cases and, and all these, yeah, these were all different platforms running different operating systems or hardware, uh,
And none of them actually had good Rust support, except for the security gateway. But we, yeah. That's a long story why we didn't start off there. But yeah, so we had a QNIC system and there was no QNIC support. We had tricore-based units, which tricore is a common automotive-based microcontroller specifically designed for safety-critical type of use cases.
And none of these had Rust support at the time.
I'm trying to put myself into the position of a team member or a colleague of yours in 2018. So you're telling me about Rust. You're enthusiastic about it. I see the prototype. I see that it's working. But I don't really know. I don't really have any Rust experience. And I don't know how long it took to build that prototype.
So I look at all of the tooling that we built in C and C++ and the ecosystem that requires... And then I ask you about QNX support or tricore support for Infineon CPUs. And you say, well, it doesn't exist or it's a work in progress. Suffice to say, I'd be a little skeptical.
True. And Volvo is also an automotive or a car maker. And automotive people tend to be quite conservative when it comes to new things. We like to use well-proven technologies. So yeah, definitely. But I always saw that this was actually something different, something that could actually provide real value.
Thinking about the situation in 2018 and 2019 when things were not completely fleshed out, was everyone on board already or was there some political infighting as well about whether to go forward with Rust or maybe be a bit more conservative here? Because you also mentioned that the car industry is conservative.
Yeah. So I wouldn't say political infighting, but the one question that always came up and that was, is it safety qualified? So is it qualified and certified to use in a safety critical setting? And that was a, That was basically a no. I mean, nothing had been done on that front at that time.
So that was kind of a non-starter because most of these functional domains that I mentioned earlier, these CPUs, they are running, they have a safety rating, so to say. These ACL levels that the automotive standards define.
Why are they necessary for a toolchain to fulfill?
The safety standards that the automotive industry uses, basically they talk about how do you provide the evidence that the product that you've developed actually does what you think it's doing and that there are no hidden errors or dangers that you could have foreseen and mitigated. That's basically the...
The reason why everything needs to be vetted, so to say, both the tooling, but also the code and everything that you use.
Did you back in the day already know about the efforts of building a compiler, safety-critical Rust compiler that would be allowed to be used in such environments, like things that Ferris systems built, for example?
No. And Ferris system, they actually started maybe six months later or something. We met them at a conference in, I think it was OxidizeConf. And they soon after announced the first scene where it was called Sealed Rust at that time. But yeah, so that started soon after. But I always felt that
that that wouldn't be an issue over time, because obviously we are doing safety-critical systems in C and C++, and we know all the... And it's riddled with things where you can shoot yourself in the foot, and you need all kinds of tooling to basically ensure that you're not doing... The things that you're not allowed to do. While Rust is preventing most of those things up front.
Just by design, you can't do most of those things. So I felt right from the get-go that... And it was quite ironic that here, all of a sudden, you have a language that... And basically, you know, does all that stuff for you, but it doesn't have the all the paperwork or all the necessary evidence in place.
So therefore, you can't use it, even if it's, or at least according to me, you know, much better. But at the same time, I realized that it was probably a non-starter for anything where safety certification was needed. And then we come to basically the fifth processor on this chip, and that is the small low-power microcontroller or processor, which is called the LPA.
And that one was bare metal Cortex M4 based chip that no one was accidentally working on at that time because they had some basic software running on it that, you know, just to power up the thing. But otherwise it was being neglected because everyone was focusing on the other chip. And then it turned out that Cortex-M4 was actually the best supported embedded target at that time for Rust.
And 2018 edition of Rust was actually the first edition that made building things for embedded possible and stable. So yeah, so the stars aligned a bit there. So me and a colleague who was and is as enthusiastic about Rust, we teamed up and got help from some of our managers who had seen Rust.
my talk, for example, and was also convinced that we needed to really need to look into this for the future because we needed, basically, we need better tools to get our things done more effectively and more with better quality. So we've convinced them that, hey, we have this chip here that no one is working on yet. It's perfect testbed for as a Rust project.
It doesn't have a safety level on it because it's It's actually constructed in a way that we power on the board or this box when certain events happen, but we can't power them off by ourselves. Each CPU has to power itself off. And since a car in an off state is not considered, or it's considered safe when it's power off. So, so going from off to on is not a safety critical thing.
But of course, going from on to off, you know, when you're driving, that would be completely, you know, high safety. No, exactly. So the hardware is constructed in a way that makes that impossible. So the shutdown is done in a safety-certified way where we are not involved. So that was the three things that added up. It was pretty well supported by Rust already.
It wasn't safety-critical, so we could use it as a testbed. And no one else was actually paying attention to it. So we were able to form a team and get to work.
It sounds like a good strategy for testing out new things in general and maybe specifically testing out Rust in embedded environments or maybe in larger organizations.
Yeah, absolutely. And yeah, if you have these conditions, then it's definitely something I would recommend. One thing we also did from the start was to get help, external help. So we got a few people from Greppit here in Sweden. They're placed up in Luleå, North Sweden. And many of them are deeply involved in the Arctic scheduling framework. Yeah, it's not really a real-time OS.
But yeah, so needless to say, we also use Arctic in LPA. Yeah, so we brought on these experts. They were the foremost experts in Sweden at the time. And they helped us get started and helped us with a lot of the driver development and things like that.
So I would say that is also a big piece of the puzzle to get good people to help you get started, either from within the company or externally, depending on what options do you have.
Who made the decision to hire external? Was it you because you maybe wanted to get an extra pair of hands? Was it the management? Was it a requirement from Volvo itself?
It was a team together with management. We also needed to... We needed to start fairly quickly and we needed to size up the team fairly fast with some knowledgeable people. And also, at the time, we weren't really sure how much effort the driver situation would be. So we felt that we needed to have someone to hold their hand there.
Did you have a deadline on, for example, you decided that you would be done by that time or you would have something to show by that time and if not, then the project would be canceled?
Yeah, so I don't think there was, there was never a risk of it being canceled as such, but yeah, we wanted to, we set ourselves a deadline that, yeah, so like I said earlier, We already had some C code running there that was basically produced by the company that made the actual hardware. And that was doing most of the basic things, and we felt that it would be fairly straightforward to...
to revert back to that and build that out in case. So we had always that as a backup. And in fact, we started off working from that base and trying to build Rust on top of that. But we quite quickly figured that having a 100% Rust-based solution would actually be a much better choice. Why?
Yeah, because you don't have to worry about all these FFI boundaries between C and also when you're building it on
on bare metal embedded it kind of depends on what compiler you're using what linkers you're using how the because the c libraries they expect certain you know initialization to be done in a certain way and and so there may be incompatibilities there that yeah so even though rust and c can interoperate seamlessly there are there can be some issues when you're when you're doing it on a
on a bare metal target like this and and since we were like we were taking us already working c project and then adding rust on top and that was yeah we it was bringing us more headache than than just rewriting that little piece of or the those few few lines of code that how long did it take you to rewrite these pieces of code
So there was mostly Grepid that did the initial bring up and it took them about a couple of months. They did it over a summer basically. And so, yeah, when we came back after vacation, they already had a demo of an application that did basically exactly the same as the one that we had. And when we had that working, we saw that, okay, now there's no point to carry on with the other one.
But even then, going forward, did you compare your maybe output with the output of the C-based version in order to make sure that it was compatible? Or was it mostly, you know, it's up now, we can compare the gRPC outputs, essentially?
Yeah, so now we're not doing gRPC anymore, so it's a different kind of communication. But yeah, we already had started working on a CI pipeline with tests and things like that. So yeah, that was like the first goal to ensure that the same test suite would pass with the new Rust version.
Interesting. So you don't use gRPC anymore. What were some of the issues that you faced?
Basically, that prototype that I did back then, that was basically the Android infotainment system communicating with a prototype of this core system that we are now building. So that was that communication. But now we are inside the core system and communicating between the
the cpus on that's on a much lower level so so yeah so we're we're not doing grpc on this node i can understand is that a public protocol an open protocol or instead of volvo specific protocol Yeah, so the one we're using now is actually Volvo specific. So the LPA is connected to via UARTs to all the CPUs. So that's the only way to communicate with it. And
We searched high and low for a protocol that would give us... Because in a car, you need to deal with a lot of EMC, electromagnetic interference. And so it's quite common that you get corrupt messages and things like that. So we need some sort of protocol that is similar to TCP. Yeah, but much lightweight, of course, since it's just a point-to-point.
But something where you can detect that messages are missing or corrupt or malformed in various ways.
Right, and my assumption would be that you had to build out a Rust library for reading and writing that format. Yes, yes. And was it hard for you? Did you use any existing libraries to build that parser or... to serialize the messages over the POS?
So we actually use both, or this protocol allows us to send different types of payloads over the, so depending on if the payload is, we'll probably get to that later, but we also need to comply with the automotive diagnostic standards called UDS, universal diagnostic standards. And so depending on if the message is a UDS payload, then it has certain format that is standardized by that standard.
Or if it's like in-band messages between the processors. Then we're actually using C bore as a messaging format. So, and, and at that time there wasn't any C bore library for no, no STD. So we wrote that. And then of course we, we used 30 as a, which is a, this is a super awesome. Yeah.
And what about the automotive diagnostic standard, I guess you called it? What about that protocol? Was there an existing implementation or was it easy to write?
It's fairly straightforward. It's basically made for, up until now, these ECUs in cars, these electronic control units, which are the... basically the building blocks of an electrical system in a car. And those have existed for a long time. They're always increasing in numbers. Now we're up to almost 200 or something in the current generations.
And these are traditionally very small microcontrollers with very little RAM, flash, things like that. So that protocol is fairly compact and simple. But yeah, we needed to write a parser from scratch for that.
That's super amazing because there are a couple other vendors that use Rust in production now. For example, Renault, which is also a prominent car manufacturer. I wonder if they use similar protocols or if they have a different setup. I do know that everyone sort of used or uses CAN bus, but then the wire protocols might be different, right? You have different requirements. Yeah.
Is there any collaboration going on between those car vendors on a message bus level, on a protocol level?
They're not directly... at the moment, but things are actually, or they are, I guess they are improving now with all these different SDV or software defined vehicle organizations that are sprouting all over the place. You have Eclipse SDV and you have Covisa and you have others. Through those, I hope we can, or we will be able to collaborate more. But yeah,
But yeah, so during this project, we didn't have a chance to do too much of that. We were mostly focused on getting our product out the door.
Are you planning to open source some of that work? Are people ever going to be able to take a look at this and maybe use it in some way?
I hope so. So all the drivers that we developed have been open sourced or have been upstreamed to the AtSAMD project. And then we commissioned Greppi to write this MCAN library, which is hardware abstraction for this MCAN peripheral that is quite common and should be available to everyone. Yeah, if you have a SOC that has that peripheral, then you can use that crate out of the box.
Then we have a bunch of other things that we've developed that might make sense to open source, but we need to look at that more going forward. But for example, the UDS stack might be something, or this diagnostic stack might We have this seaboard library, for example. Now, actually, there are a few in the community already. So we have some compression, error correction. Yeah, a bunch of things.
Things like, for example, trace. So being able to tag the tests with requirements so that you can...
produce reports where you can connect, because that's a big part of providing evidence that your code actually works, or that the code actually does what it's supposed to do, rather, is that you have these higher-level requirements, and then they are broken down into individual component requirements, and then there are tests that ensure that those requirements are actually being met.
And then you need to show basically in a traceability matrix that all the requirements are actually being tested and functioning according to... And so we've developed a lot of tooling around that to give us or show our coverage of requirements and things like that. Some of those might be...
yeah something that that might be good to open source others are fairly volvo specific so they might not be as as yeah it doesn't make as much sense but but yeah we'll see
It feels like to be able to pull this off, you first had to invent or create the universe because there were so many small little bits and pieces that someone would have to write and you needed to build this entire tool chain. So moving forward, probably... As a next project, this will be way easier because you built out a lot of the tooling. But I wondered if the Rust ecosystem also helped you.
So what are some of the things that you used from day one or maybe you even use today? A couple of nice projects that you want to mention here. Yeah.
Yeah, so I mentioned Serdi already. Probe RS is definitely something.
Shout out to Noah, who is one of the maintainers of Probe.
Yeah, they've done an amazing job. We actually use... So what we do with Probe RS is we created our own wrapper around it. So it's basically a command line tool that is tailored for this particular chip, and it does...
flashing and debugging and setting up all the peripherals in a correct way and disabling write locks on certain things so that you can flash or erase different things in different orders and stuff like that. So we have our own called LPA probe, which is basically a wrapper around Probe RS.
But we also have a Python binding package around Probe RS so that we can use that in our system tests, which are Python-based packages. And then you can essentially use it as any other Python library within our system tests.
I do wonder, are there any things in Probe RS that are missing right now? And things in the embedded or automotive ecosystem that you would like to have reflected in Rust, in the Rust ecosystem?
I don't know about probe RS for the use cases that we have had. There hasn't been, can't recall anything specific that we haven't been able to fix together with them, but otherwise sure. There are, you know, we, we want to make rust, uh, fully first-class alternative to, to, uh, CNC plus plus when it comes to safety critical.
And in order to get there, we need, you know, the tool chain needs to be certified, which is now actually available. But we also need, there are still libraries that need certification and we need tooling around it. For example, MCDC coverage and other tooling for better access. traceability to trace the requirements in an even better way to get better coverage metrics for the code reports.
Clippy, for example, would be nice if Clippy would produce reports, even if you can set it to... to warnings as errors, it would still be nice to give an overview of what warnings you actually have or what remarks it has in the project.
Do you have a requirement to create a software bill of materials?
Yes, I was going to get to that as well. We do that, and that's part of our build infrastructure that we've built out. Cargo, by default, it helps you to get the total list of everything that you're using in the project. But we want to know what are we actually shipping, so how much of that is actually tooling that is never leaving
leaving the company and what is actually the binary that is getting flashed in the car. So we had to do, yeah, add some filtering on top of that to produce an SBOM that is more accurate in terms of what is actually being deployed.
And where do these files end up? You certainly use it for internal documentation, but do you also have to hand them out to some other authority which approves certain software?
I'm not sure, but they need to be available at request. And we have, there's open source portal at Volvo Cars that you can, actually, if you own a car, you're always able to request. The license notice is basically for everything.
So those files are used for that, but also they're used for internal monitoring, for example, for cargo audit or other forms of auditing and monitoring to ensure that if there's a vulnerability somewhere that we know about it and are able to react to it,
I know from my car that there is a list of tools or a list of software that got used, especially I would say maybe larger applications or larger libraries. For example, curl is a very popular thing. And we had Daniel Sandberg in the podcast as well. But I wonder how far it goes down. Because you have dependencies and sub-dependencies and dependencies of these dependencies.
Is that all listed somewhere? Can I access that as a user? Can I maybe get the entire cargo tomo of the project or would that be too much?
So the entire cargo tunnel would contain much more than we're actually shipping. So we're not providing that as part of the car. There it's only the software that is actually running on the car.
For example, Sardi, would it show up somewhere? Yeah, it would show up, definitely. Okay.
So yeah, so all, all dependencies that are, that are like getting compiled and linked into the, into the binary are, are would show up.
Let's say I started Volvo and I'm in your team and I want to start working on this project in Rust. What would my day-to-day look like? I plug myself into an embedded device? Certainly not. There's probably some other device somewhere I need authentication for. And then we have some CICD process. We have ProBias somewhere in the tool chain. We have an IDE.
And then we have a build process in the end. And eventually... There will be a binary and a software bill of materials and probably some sort of integration of all the components later on. So walk us through all the steps.
Absolutely. But first we must talk about the hardware. So this core computer that I described in the beginning, where we're actually running inside the car. That one, so when we're running there, it's quite hard to test the LPA on its own because it's essentially a small island surrounded by a bunch of other processors. And there are not that many connections to the outside world towards that one.
So yeah, if you want to test the LPA individually, you would have to essentially... deploy some sort of distributed test program on the other CPUs and then have those test programs like probing and prodding the LPA and sending back results and stuff. And that would be quite cumbersome. So we fairly quickly, we realized that for this to be effective, we need to have our own development hardware.
And yeah, we sketched out basically a block diagram, how it would look, where basically we take the LPA circuitry as it looks on the core computer. take it over to a separate PCB and then we hook up all kinds of components like a debug probe, a USB to CAN device, USB to GBIO devices and current measurements and other things. And so what we end up with is this.
It's a small circuit board with one USB cable, and it costs a few hundred euros a piece. And we make it ourselves, or at least Grappit, who are also hardware specialists, they make Yeah, they basically did it in just a few number of weeks we had the first samples. So we have these in some, or at least big enough quantity that every developer can have one at his or her desk.
And then we have a bunch of those in CI.
Nice.
Which is based on Zool and Zool CI, where we then, so for every patch that we do, it then runs the whole test suite on the hardware.
What's so special about Zool? Because not many people might know it.
Yeah, ZoolCI is quite interesting. So I would say its main selling point is that it can do speculative merging. And what that means is that you can have... Normally, especially if you're multiple projects working in a common code base, you could have one team or one developer doing a change and it's working fine and all the tests are going through.
At the same time, someone else is doing a different patch for something else. And that is also fine and going through. So both of them get merged, but it turns out that those two together don't actually work. So when both of them get merged, it breaks. Zool actually is able to speculatively check all the patches that are in flight and testing them together before they actually get merged.
So you have much higher likelihood of the master branch actually working.
Isn't it also true that we're dealing with embedded devices here? I know roughly how that would work for a backend service, say. But for embedded devices, I would be scared that the device would be in a weird limbo state or... wasn't completely flashed when the test ran. So how do you ensure that the tests are always correct and start from a deterministic position?
Good question. So the actual test framework is Robot Framework, which is a Python-based behavior-driven development type of thing, similar to Cucumber or... Yeah, systems like that, where the tests are basically on a system level, like you essentially tell statements about how the system should behave, and then you connect those statements to actual code that is being run on the target.
And of course, every test suite then has quite extensive setup loop, basically, where it takes the device and flashes the correct version, resets it into the correct state, and sets up whatever preconditions that need to be in place before the test start. And then same when it's done, it will do some cleanup and But that's all like that's agnostic. So Zool doesn't care about that.
It just starts the test framework and it does its magic and then it can report back.
How long does a typical build take?
So on the local machine, it takes about, depending on the machine, maybe between 5 and 10 minutes. And then the test itself, I think another 15, 20. But then in the CI, we're doing more extensive things. So that is around an hour or so.
And that includes setting up all the devices and running the integration test?
Yeah, yeah. So, yeah, around an hour. If it starts to take more than an hour, we usually try to sit down and figure out if we can parallelize it more, if we need to add more units to the CI so that we can run more in parallel, or if we can optimize the tests somehow so that they run faster.
Does that include all of the components, or does it include just the Rust components?
So it includes all of the LPA components, yeah.
And how often is Rust the culprit of failing a build?
Almost never. I mean, that is, since it's mostly taken care of at build time. So you can't really, you know, if it doesn't build, you already know that in the build step. And that's usually something you catch locally. Okay.
Does it mean you don't even flash to the device that often because you have a very iterative way of building the software with your IDE and with the type system?
Yeah, I definitely would think so, compared to another system where less of the things were taken care of up front. You would absolutely need to flash it more often just to see that, no, that didn't work. So yeah, I would definitely think that that is the case.
How big is the codebase, the Rust codebase?
So the Rust code base is around 75,000 lines. And then we have additional 50,000 lines of system tests and CI and other tooling. So yeah, around 125,000, 30,000 lines.
And that was developed by how many people? What's the team size?
So we've been ranging between five and 10 developers overall during this three and a half years. We've been
Yeah, very nice. A very substantial code base, but also, I would say, still manageable. So I would assume a lot of work also went into speccing out things, creating architecture diagrams, making sure that everything is wired up correctly, communication and so on. There's a lot of complexity introducing a completely new toolchain, new language, everything.
absolutely absolutely and and and like i said earlier i didn't even go through the whole list of things that we have had to build out as well to get this get this working so so yeah my team has done an amazing amazing job there couldn't give them enough credit and but but i've also found that using Rust or this setup that we have is super empowering for everyone.
So everyone feels quite confident when they're working in the code base, and especially that you're not as afraid of breaking things because the compiler will actually let you know upfront.
So, you know, especially when we're onboarding new developers to the team, they are usually quite quick to get up to speed because they can, you know, they can hack around without fear because, you know, as soon as the bills pass and the tests pass and everything, then you're fairly certain that, you know, nothing weird has been done.
How long until they are fully ramped up?
Everything from two weeks, which was the extreme case, to maybe three to six months, something like that.
So it's a bit of a myth that the Rust learning curve is a huge problem for people that want to work with Rust professionally.
I would say so. It's absolutely something to respect, and I would definitely, like we did, we seeked help, and we got some people on board that were already really good at this. I wasn't super proficient in Rust when we started. I had done this project and some other smaller hobby things, and actually none of our regular developers are... had worked with Rust in any big capacity before.
So we've all picked it up on the job. But of course, they are experienced C and C++ programmers. Not everyone even embed it. So they've had to learn that as well.
And how did you learn how to write idiomatic Rust? Do you have any coding guidelines? Did you talk to the people from Grappit on how to write idiomatic Rust? Are there any resources that you maybe recommended to your team?
We read a lot of code. from yeah from the community we we also have we use clippy a lot to to avoid you know the bigger issues rust format and and rust and and clippy of course are mandatory and and we use pedantic clippy pedantic
and and no warnings allowed so to say so that so that takes and that gives us a lot of pointers but but then i mean our our code has evolved over time you know going back to some things that we did in the beginning or we can absolutely see that there are some some c isms here and there uh
And when I would be tasked with building code for cars, I would be extremely defensive because I know that I wouldn't be able to just connect to the car or maybe have a shell connection and just troubleshoot on the live machine. These cars, they run for years and sometimes they have to go without maintenance or people don't do those updates that regularly as they should.
So how defensive is the code? How, let's say, painstaking is it to work with inputs and outputs? How does the error handling story look like? Is it... Would it be normal Rust code that people could understand or is it very specific Rust code?
There are definitely exceptions, but I would say in general, it's very common Rust code. The robustness of just Rust in its default form is really high. But of course, you can't just unwrap all over the place. You have to... You have to ensure that errors are actually getting handled and logged properly.
And we have a quite extensive, so we talked about this diagnostic stack, and we also have like a framework for creating diagnostic monitors that are monitoring different conditions and reporting that.
Do you have a custom panic handler? Sure.
Yes, yes.
That means failing is not an option. You need to handle every condition. Because if you're the thing that boots things, then who boots you if you fail?
Yeah. And of course, if it panics, we need to reset back to a working condition. But we also use the chip as a watchdog. that we also need to activate so that in case it would lock up for some reason, the watchdog would then hard reset it.
Are there any rules around allocation?
Yes and no. Of course, there are rules about everything when it comes to automotive software. In our case, we don't use any allocation. We don't have an allocator, so we don't use it. So everything is static.
That means you disabled the allocator.
Yeah, so we don't use that dependency at all. And the heap is zero size, basically. But that means that we use the stack a lot, and we use heap less for a lot of these things. But also, yeah, we put our buffers and things, they're statically allocated here.
Because you know the sizes of the payloads that you have to handle at compile time. Yeah, exactly. It's similar to how Oxide does it. They have this predictable...
scheduler i would say or part of the operating system hubris and they do something similar where they only have certain message types that they can handle yeah yeah exactly the same way so everything is statically known beforehand or the worst case is known so yeah we never have to deal with any
unknown buffer sizes and then when it comes to input you asked about input validation that is of course super important and that's why it's important also to have a well-defined so that all all messages that we are sending are of well-defined format so that you can easily you know at the parsing stage see if it's actually valid or not and reject it already there um yeah
If someone wants to buy a car that has rust components now, which cars would they need to buy? You said there are production cars. There's probably one Volvo and one Polestar, as far as I'm aware.
Yeah, that's true. It's the Polestar 3 and EX90. Volvo EX90. And then just keep on the lookout for Spa 2-based cars. And more will be coming soon.
Did you have any issues in production? With the hardware or with... Well, let's say both. But I'm curious if you sold a car and then it came back because of a rust bug, because of an issue in the rust code.
That has not happened yet.
Is that a common thing when you ship certain components and they are C-based, for example, that usually you covered most of the edge cases when things hit production? Or are there any rollbacks or changes?
Yeah, I can't really say how... I would guess it's fairly common in automotive in general. I mean, if you look at... you know, what is called warranty, warranty in general, and things that are being replaced due to that, that is quite often software that is at fault. So yeah, I would definitely...
And of course, it's still early times, so I'm not going to say that that is not going to happen for us, but so far it has. We have seen some manufacturing issues, but those have been hardware related so far and hopefully something that we will be able to catch in the early stages.
Can you give us an outlook into the future, maybe one, two, three, four, five years down the road? What is Rust usage at Volvo going to look like?
I'm just going to hop into my time machine and get back. Now, I would, of course, love to see it in more places. So that is what I'm actively working on now to find or to see where it would fit. Because it doesn't make sense to rewrite or replace everything with Rust. Because when you're already...
When you already have the thing working and it's tested and it's according to specification, it's seldom that it actually makes sense to rip it out. But there are cases where that is possible. especially when it comes to cybersecurity-critical user-facing code, things that need to do validation of data that is coming from the outside world, things like that.
So I definitely see a possibility for those kind of use cases where we're interfacing with the internet, for example, or things of that nature. But yeah, now that the hurdles I mentioned earlier, the hurdles that we had in the beginning, we didn't have support for neither the hardware nor the OSs that are commonly used. But all of those hurdles are falling one by one.
Now, I find it extremely surprising that a lot of the dependencies that I use for writing Rust and writing Rust in production is the same that you use for an embedded project at Volvo. And this is really cool because you can reuse the code You can share ideas, you can share different bits and pieces and the entire ecosystem becomes much better. So you kind of cross different boundaries with Rust.
And how does it look like in comparison between Dev and Prod?
Yeah, it's funny you say that because there is actually no difference. We build one binary, and that is the binary that we test on our test hardware, and that is the binary that goes unmodified to the core computer. And so far, there's nothing that we can't actually test on our dev board, or at least very few things. There are, of course...
A few things that change when you're in the context of a whole car. But when it comes to the LPA functionality, we can test essentially everything before we deploy it. So it's almost unheard of that we get some sort of issues due to the fact that something was working differently on our test hardware.
So, so yeah, so one, one binary for no difference between dev and prod and, and that in that sense, but what you said with possibility to reuse, and that is definitely one of rust's huge, strong points that, that the fact that you can actually, you know, take different components from the community and you can re use them with.
with confidence that you don't really have another, I mean, both hassle-free almost always it's just builds and works and you can just use it. And, and also the, you know, all these things about that you always have to take care of first. You need to, okay. Can you get it to build? in my build system. And first time, probably not. You probably need to spend a lot of time to get that to work.
But then when it actually builds and you start to use it, does it use memory in the same way as you do? Does it make the same assumptions? Like who is allocating this buffer? Who is freeing it? Who has the responsibility to do what? And so on and so forth. And those are not standardized in C and C++.
Meaning that, yeah, so even if you can build it, it's no guarantee that you can actually use it in a, you know, in any productive way. But in Rust, on the other hand, everything is... Maybe it's not true to say everything, but you're pretty much guaranteed that it will work.
And also, like you said, I don't think there's any language that is as scalable from the lowest, smallest microcontrollers to backend systems or... or whatever biggest server farms that you can imagine and everything in between.
That sounds really empowering and probably very encouraging for a lot of people who might listen. Would that be your message to the Rust community?
Yeah, empowering is definitely the key word. We have been super, super happy with this project, and it really shows that Rust has a bright future. The hurdles that we mentioned before when we started the project, which made it basically a non-starter for most of the ECUs that we wanted to use it, those are all coming down or have come down already. So now you have QNX support, for example.
You have Infineon tricore support. There was recently an article on how to run Rust together with Autostar Classic. Autostar Classic is the common automotive software framework that most automotive software runs in. And now that is becoming available for Rust as well. There is work being done on... on various different automotive platforms and components.
So for automotive, it's definitely a bright future. And we'll see where we'll take it at Volvo.
And where can people learn more about this project, about Rust at Volvo?
Yeah, so there are two interviews out, actually. One from a couple years back, where we were kind of in the starting position, and you can find that on the Volvo Tech blog. It's called Why... Volvo thinks you should have rust in your car or something similar. Silly pun. And then now recently there was an article, Tvede Holv's blog, where we talked about this project and how it has been going.
Yeah, that was a really nice one. I read this one and also this prompted me to reach out. And where can people learn more about yourself?
You can find me on LinkedIn and on X, Mastodon, all these different platforms. I also was fortunate enough to join the Rust Safety Critical Consortium that was started last month at RustConf in Montreal. And together we are aiming to close the final hurdle, which is to make Rust a fully accessible
viable alternative to do safety critical software so that is that is what we're doing now and you can find more about that at at the rust foundation web page and there's also a github repo where you can join if you like admission is free
That means people will run into you at some point or another, and we hope to see you at a conference or at an event speaking about Rust and Volvo or safety-critical components. Yeah, let's hope so.
We're at least super stoked about the current project and how well it has worked out. Definitely going to be pushing that forward within the company and hopefully we can find some other exciting avenues for it.
Julius, thanks for taking the time and thanks for being an ambassador of Rust in the car manufacturing space.
Yeah, thanks for having me.
It was a great chat. Rustium Production is a podcast by Corot. It is hosted by me, Matthias Endler, and produced by Simon Brüggen. For show notes, transcripts, and to learn more about how we can help your company make the most of Rust, visit corot.dev. Thanks for listening to Rustium Production.