Julius Gustavsson
Appearances
Rust in Production
Volvo with Julius Gustavsson
Just by design, you can't do most of those things. So I felt right from the get-go that... And it was quite ironic that here, all of a sudden, you have a language that... And basically, you know, does all that stuff for you, but it doesn't have the all the paperwork or all the necessary evidence in place.
Rust in Production
Volvo with Julius Gustavsson
Yeah, so I joined Volvo in 2017, and that's about the same time as this Spa 2 project started. And right...
Rust in Production
Volvo with Julius Gustavsson
So therefore, you can't use it, even if it's, or at least according to me, you know, much better. But at the same time, I realized that it was probably a non-starter for anything where safety certification was needed. And then we come to basically the fifth processor on this chip, and that is the small low-power microcontroller or processor, which is called the LPA.
Rust in Production
Volvo with Julius Gustavsson
And that one was bare metal Cortex M4 based chip that no one was accidentally working on at that time because they had some basic software running on it that, you know, just to power up the thing. But otherwise it was being neglected because everyone was focusing on the other chip. And then it turned out that Cortex-M4 was actually the best supported embedded target at that time for Rust.
Rust in Production
Volvo with Julius Gustavsson
And 2018 edition of Rust was actually the first edition that made building things for embedded possible and stable. So yeah, so the stars aligned a bit there. So me and a colleague who was and is as enthusiastic about Rust, we teamed up and got help from some of our managers who had seen Rust.
Rust in Production
Volvo with Julius Gustavsson
my talk, for example, and was also convinced that we needed to really need to look into this for the future because we needed, basically, we need better tools to get our things done more effectively and more with better quality. So we've convinced them that, hey, we have this chip here that no one is working on yet. It's perfect testbed for as a Rust project.
Rust in Production
Volvo with Julius Gustavsson
It doesn't have a safety level on it because it's It's actually constructed in a way that we power on the board or this box when certain events happen, but we can't power them off by ourselves. Each CPU has to power itself off. And since a car in an off state is not considered, or it's considered safe when it's power off. So, so going from off to on is not a safety critical thing.
Rust in Production
Volvo with Julius Gustavsson
basically that when i was joining i was also discovering or i was i'd already discovered rust a couple years back but i was getting more and more certain that this was something that that we needed to look into and so we started out with a few products when we were like in this very very initial stages of that project we were doing all kinds of proof of concepts and
Rust in Production
Volvo with Julius Gustavsson
But of course, going from on to off, you know, when you're driving, that would be completely, you know, high safety. No, exactly. So the hardware is constructed in a way that makes that impossible. So the shutdown is done in a safety-certified way where we are not involved. So that was the three things that added up. It was pretty well supported by Rust already.
Rust in Production
Volvo with Julius Gustavsson
It wasn't safety-critical, so we could use it as a testbed. And no one else was actually paying attention to it. So we were able to form a team and get to work.
Rust in Production
Volvo with Julius Gustavsson
Yeah, absolutely. And yeah, if you have these conditions, then it's definitely something I would recommend. One thing we also did from the start was to get help, external help. So we got a few people from Greppit here in Sweden. They're placed up in Luleå, North Sweden. And many of them are deeply involved in the Arctic scheduling framework. Yeah, it's not really a real-time OS.
Rust in Production
Volvo with Julius Gustavsson
But yeah, so needless to say, we also use Arctic in LPA. Yeah, so we brought on these experts. They were the foremost experts in Sweden at the time. And they helped us get started and helped us with a lot of the driver development and things like that.
Rust in Production
Volvo with Julius Gustavsson
So I would say that is also a big piece of the puzzle to get good people to help you get started, either from within the company or externally, depending on what options do you have.
Rust in Production
Volvo with Julius Gustavsson
It was a team together with management. We also needed to... We needed to start fairly quickly and we needed to size up the team fairly fast with some knowledgeable people. And also, at the time, we weren't really sure how much effort the driver situation would be. So we felt that we needed to have someone to hold their hand there.
Rust in Production
Volvo with Julius Gustavsson
Yeah, so I don't think there was, there was never a risk of it being canceled as such, but yeah, we wanted to, we set ourselves a deadline that, yeah, so like I said earlier, We already had some C code running there that was basically produced by the company that made the actual hardware. And that was doing most of the basic things, and we felt that it would be fairly straightforward to...
Rust in Production
Volvo with Julius Gustavsson
to revert back to that and build that out in case. So we had always that as a backup. And in fact, we started off working from that base and trying to build Rust on top of that. But we quite quickly figured that having a 100% Rust-based solution would actually be a much better choice. Why?
Rust in Production
Volvo with Julius Gustavsson
And the first thing I did there to test out Rust was to, since I came from an Android background, so I had, I knew the ins and outs of the Android system. So I was able to build an Android hull, an automotive hull in Android, was able to get that to build and link and do all kinds of shenanigans to get that to work because the build system didn't really support it. But yeah.
Rust in Production
Volvo with Julius Gustavsson
Yeah, because you don't have to worry about all these FFI boundaries between C and also when you're building it on
Rust in Production
Volvo with Julius Gustavsson
on bare metal embedded it kind of depends on what compiler you're using what linkers you're using how the because the c libraries they expect certain you know initialization to be done in a certain way and and so there may be incompatibilities there that yeah so even though rust and c can interoperate seamlessly there are there can be some issues when you're when you're doing it on a
Rust in Production
Volvo with Julius Gustavsson
on a bare metal target like this and and since we were like we were taking us already working c project and then adding rust on top and that was yeah we it was bringing us more headache than than just rewriting that little piece of or the those few few lines of code that how long did it take you to rewrite these pieces of code
Rust in Production
Volvo with Julius Gustavsson
So there was mostly Grepid that did the initial bring up and it took them about a couple of months. They did it over a summer basically. And so, yeah, when we came back after vacation, they already had a demo of an application that did basically exactly the same as the one that we had. And when we had that working, we saw that, okay, now there's no point to carry on with the other one.
Rust in Production
Volvo with Julius Gustavsson
Yeah, so now we're not doing gRPC anymore, so it's a different kind of communication. But yeah, we already had started working on a CI pipeline with tests and things like that. So yeah, that was like the first goal to ensure that the same test suite would pass with the new Rust version.
Rust in Production
Volvo with Julius Gustavsson
Basically, that prototype that I did back then, that was basically the Android infotainment system communicating with a prototype of this core system that we are now building. So that was that communication. But now we are inside the core system and communicating between the
Rust in Production
Volvo with Julius Gustavsson
the cpus on that's on a much lower level so so yeah so we're we're not doing grpc on this node i can understand is that a public protocol an open protocol or instead of volvo specific protocol Yeah, so the one we're using now is actually Volvo specific. So the LPA is connected to via UARTs to all the CPUs. So that's the only way to communicate with it. And
Rust in Production
Volvo with Julius Gustavsson
We searched high and low for a protocol that would give us... Because in a car, you need to deal with a lot of EMC, electromagnetic interference. And so it's quite common that you get corrupt messages and things like that. So we need some sort of protocol that is similar to TCP. Yeah, but much lightweight, of course, since it's just a point-to-point.
Rust in Production
Volvo with Julius Gustavsson
But something where you can detect that messages are missing or corrupt or malformed in various ways.
Rust in Production
Volvo with Julius Gustavsson
I was able to work around that. So this Android HAL, basically, the point of it was to use the Android UI to control the AC or the fans in the car. And we had already built this small system that was running on a separate, like a Raspberry Pi, that would actually send the CAN signals to the fan. So we needed to communicate with that, to send commands to that system, basically.
Rust in Production
Volvo with Julius Gustavsson
So we actually use both, or this protocol allows us to send different types of payloads over the, so depending on if the payload is, we'll probably get to that later, but we also need to comply with the automotive diagnostic standards called UDS, universal diagnostic standards. And so depending on if the message is a UDS payload, then it has certain format that is standardized by that standard.
Rust in Production
Volvo with Julius Gustavsson
Or if it's like in-band messages between the processors. Then we're actually using C bore as a messaging format. So, and, and at that time there wasn't any C bore library for no, no STD. So we wrote that. And then of course we, we used 30 as a, which is a, this is a super awesome. Yeah.
Rust in Production
Volvo with Julius Gustavsson
It's fairly straightforward. It's basically made for, up until now, these ECUs in cars, these electronic control units, which are the... basically the building blocks of an electrical system in a car. And those have existed for a long time. They're always increasing in numbers. Now we're up to almost 200 or something in the current generations.
Rust in Production
Volvo with Julius Gustavsson
And these are traditionally very small microcontrollers with very little RAM, flash, things like that. So that protocol is fairly compact and simple. But yeah, we needed to write a parser from scratch for that.
Rust in Production
Volvo with Julius Gustavsson
They're not directly... at the moment, but things are actually, or they are, I guess they are improving now with all these different SDV or software defined vehicle organizations that are sprouting all over the place. You have Eclipse SDV and you have Covisa and you have others. Through those, I hope we can, or we will be able to collaborate more. But yeah,
Rust in Production
Volvo with Julius Gustavsson
But yeah, so during this project, we didn't have a chance to do too much of that. We were mostly focused on getting our product out the door.
Rust in Production
Volvo with Julius Gustavsson
I hope so. So all the drivers that we developed have been open sourced or have been upstreamed to the AtSAMD project. And then we commissioned Greppi to write this MCAN library, which is hardware abstraction for this MCAN peripheral that is quite common and should be available to everyone. Yeah, if you have a SOC that has that peripheral, then you can use that crate out of the box.
Rust in Production
Volvo with Julius Gustavsson
Then we have a bunch of other things that we've developed that might make sense to open source, but we need to look at that more going forward. But for example, the UDS stack might be something, or this diagnostic stack might We have this seaboard library, for example. Now, actually, there are a few in the community already. So we have some compression, error correction. Yeah, a bunch of things.
Rust in Production
Volvo with Julius Gustavsson
And so I used, or I created a, how that actually communicated with that over gRPC, if I remember correctly.
Rust in Production
Volvo with Julius Gustavsson
Things like, for example, trace. So being able to tag the tests with requirements so that you can...
Rust in Production
Volvo with Julius Gustavsson
produce reports where you can connect, because that's a big part of providing evidence that your code actually works, or that the code actually does what it's supposed to do, rather, is that you have these higher-level requirements, and then they are broken down into individual component requirements, and then there are tests that ensure that those requirements are actually being met.
Rust in Production
Volvo with Julius Gustavsson
And then you need to show basically in a traceability matrix that all the requirements are actually being tested and functioning according to... And so we've developed a lot of tooling around that to give us or show our coverage of requirements and things like that. Some of those might be...
Rust in Production
Volvo with Julius Gustavsson
yeah something that that might be good to open source others are fairly volvo specific so they might not be as as yeah it doesn't make as much sense but but yeah we'll see
Rust in Production
Volvo with Julius Gustavsson
using futures and and and back then there was no async await so this was like yeah uh futures zero zero point one or yeah and and it was it was such an amazing experience because yeah you know you had to do all these things to get everything to work because there was so many moving parts to get like the things to build in the build system and and yeah getting the whole like
Rust in Production
Volvo with Julius Gustavsson
Yeah, so I mentioned Serdi already. Probe RS is definitely something.
Rust in Production
Volvo with Julius Gustavsson
Yeah, they've done an amazing job. We actually use... So what we do with Probe RS is we created our own wrapper around it. So it's basically a command line tool that is tailored for this particular chip, and it does...
Rust in Production
Volvo with Julius Gustavsson
flashing and debugging and setting up all the peripherals in a correct way and disabling write locks on certain things so that you can flash or erase different things in different orders and stuff like that. So we have our own called LPA probe, which is basically a wrapper around Probe RS.
Rust in Production
Volvo with Julius Gustavsson
But we also have a Python binding package around Probe RS so that we can use that in our system tests, which are Python-based packages. And then you can essentially use it as any other Python library within our system tests.
Rust in Production
Volvo with Julius Gustavsson
I don't know about probe RS for the use cases that we have had. There hasn't been, can't recall anything specific that we haven't been able to fix together with them, but otherwise sure. There are, you know, we, we want to make rust, uh, fully first-class alternative to, to, uh, CNC plus plus when it comes to safety critical.
Rust in Production
Volvo with Julius Gustavsson
And in order to get there, we need, you know, the tool chain needs to be certified, which is now actually available. But we also need, there are still libraries that need certification and we need tooling around it. For example, MCDC coverage and other tooling for better access. traceability to trace the requirements in an even better way to get better coverage metrics for the code reports.
Rust in Production
Volvo with Julius Gustavsson
Clippy, for example, would be nice if Clippy would produce reports, even if you can set it to... to warnings as errors, it would still be nice to give an overview of what warnings you actually have or what remarks it has in the project.
Rust in Production
Volvo with Julius Gustavsson
Yes, I was going to get to that as well. We do that, and that's part of our build infrastructure that we've built out. Cargo, by default, it helps you to get the total list of everything that you're using in the project. But we want to know what are we actually shipping, so how much of that is actually tooling that is never leaving
Rust in Production
Volvo with Julius Gustavsson
leaving the company and what is actually the binary that is getting flashed in the car. So we had to do, yeah, add some filtering on top of that to produce an SBOM that is more accurate in terms of what is actually being deployed.
Rust in Production
Volvo with Julius Gustavsson
I'm not sure, but they need to be available at request. And we have, there's open source portal at Volvo Cars that you can, actually, if you own a car, you're always able to request. The license notice is basically for everything.
Rust in Production
Volvo with Julius Gustavsson
So those files are used for that, but also they're used for internal monitoring, for example, for cargo audit or other forms of auditing and monitoring to ensure that if there's a vulnerability somewhere that we know about it and are able to react to it,
Rust in Production
Volvo with Julius Gustavsson
from the button to the actual methods being called in the HAL and then having that send out something over gRPC, over Wi-Fi to that other chip. And basically when I had it all wired up and it started to build, it took me a while, the fans just turned on on the first try. So it's just super amazing. I had never experienced that before.
Rust in Production
Volvo with Julius Gustavsson
So the entire cargo tunnel would contain much more than we're actually shipping. So we're not providing that as part of the car. There it's only the software that is actually running on the car.
Rust in Production
Volvo with Julius Gustavsson
So yeah, so all, all dependencies that are, that are like getting compiled and linked into the, into the binary are, are would show up.
Rust in Production
Volvo with Julius Gustavsson
Absolutely. But first we must talk about the hardware. So this core computer that I described in the beginning, where we're actually running inside the car. That one, so when we're running there, it's quite hard to test the LPA on its own because it's essentially a small island surrounded by a bunch of other processors. And there are not that many connections to the outside world towards that one.
Rust in Production
Volvo with Julius Gustavsson
So yeah, if you want to test the LPA individually, you would have to essentially... deploy some sort of distributed test program on the other CPUs and then have those test programs like probing and prodding the LPA and sending back results and stuff. And that would be quite cumbersome. So we fairly quickly, we realized that for this to be effective, we need to have our own development hardware.
Rust in Production
Volvo with Julius Gustavsson
And yeah, we sketched out basically a block diagram, how it would look, where basically we take the LPA circuitry as it looks on the core computer. take it over to a separate PCB and then we hook up all kinds of components like a debug probe, a USB to CAN device, USB to GBIO devices and current measurements and other things. And so what we end up with is this.
Rust in Production
Volvo with Julius Gustavsson
It's a small circuit board with one USB cable, and it costs a few hundred euros a piece. And we make it ourselves, or at least Grappit, who are also hardware specialists, they make Yeah, they basically did it in just a few number of weeks we had the first samples. So we have these in some, or at least big enough quantity that every developer can have one at his or her desk.
Rust in Production
Volvo with Julius Gustavsson
And so that was like a first proof point that this is definitely something.
Rust in Production
Volvo with Julius Gustavsson
Which is based on Zool and Zool CI, where we then, so for every patch that we do, it then runs the whole test suite on the hardware.
Rust in Production
Volvo with Julius Gustavsson
Yeah, ZoolCI is quite interesting. So I would say its main selling point is that it can do speculative merging. And what that means is that you can have... Normally, especially if you're multiple projects working in a common code base, you could have one team or one developer doing a change and it's working fine and all the tests are going through.
Rust in Production
Volvo with Julius Gustavsson
At the same time, someone else is doing a different patch for something else. And that is also fine and going through. So both of them get merged, but it turns out that those two together don't actually work. So when both of them get merged, it breaks. Zool actually is able to speculatively check all the patches that are in flight and testing them together before they actually get merged.
Rust in Production
Volvo with Julius Gustavsson
So you have much higher likelihood of the master branch actually working.
Rust in Production
Volvo with Julius Gustavsson
Good question. So the actual test framework is Robot Framework, which is a Python-based behavior-driven development type of thing, similar to Cucumber or... Yeah, systems like that, where the tests are basically on a system level, like you essentially tell statements about how the system should behave, and then you connect those statements to actual code that is being run on the target.
Rust in Production
Volvo with Julius Gustavsson
I would say, of course, my experience didn't hurt. But I know for a fact I've been doing C and C++ for mostly C though, but a fair bit of C++ as well for the better part of 20 years. And every single time there is always something, or at least something creeps up. Especially when you're least expecting it.
Rust in Production
Volvo with Julius Gustavsson
And of course, every test suite then has quite extensive setup loop, basically, where it takes the device and flashes the correct version, resets it into the correct state, and sets up whatever preconditions that need to be in place before the test start. And then same when it's done, it will do some cleanup and But that's all like that's agnostic. So Zool doesn't care about that.
Rust in Production
Volvo with Julius Gustavsson
It just starts the test framework and it does its magic and then it can report back.
Rust in Production
Volvo with Julius Gustavsson
So on the local machine, it takes about, depending on the machine, maybe between 5 and 10 minutes. And then the test itself, I think another 15, 20. But then in the CI, we're doing more extensive things. So that is around an hour or so.
Rust in Production
Volvo with Julius Gustavsson
Yeah, yeah. So, yeah, around an hour. If it starts to take more than an hour, we usually try to sit down and figure out if we can parallelize it more, if we need to add more units to the CI so that we can run more in parallel, or if we can optimize the tests somehow so that they run faster.
Rust in Production
Volvo with Julius Gustavsson
Almost never. I mean, that is, since it's mostly taken care of at build time. So you can't really, you know, if it doesn't build, you already know that in the build step. And that's usually something you catch locally. Okay.
Rust in Production
Volvo with Julius Gustavsson
I'm Julius Gustafsson, and I work for Volvo Cars, and I am the main architect and team lead for the LPA project at Volvo Cars. And the LPA, which stands for the Low Power Processor, is the first ECU in the automotive industry, at least as far as we know, that is fully written in Rust. And it's rolling off the production line as we speak.
Rust in Production
Volvo with Julius Gustavsson
Yeah, I definitely would think so, compared to another system where less of the things were taken care of up front. You would absolutely need to flash it more often just to see that, no, that didn't work. So yeah, I would definitely think that that is the case.
Rust in Production
Volvo with Julius Gustavsson
So the Rust code base is around 75,000 lines. And then we have additional 50,000 lines of system tests and CI and other tooling. So yeah, around 125,000, 30,000 lines.
Rust in Production
Volvo with Julius Gustavsson
So we've been ranging between five and 10 developers overall during this three and a half years. We've been
Rust in Production
Volvo with Julius Gustavsson
absolutely absolutely and and and like i said earlier i didn't even go through the whole list of things that we have had to build out as well to get this get this working so so yeah my team has done an amazing amazing job there couldn't give them enough credit and but but i've also found that using Rust or this setup that we have is super empowering for everyone.
Rust in Production
Volvo with Julius Gustavsson
So everyone feels quite confident when they're working in the code base, and especially that you're not as afraid of breaking things because the compiler will actually let you know upfront.
Rust in Production
Volvo with Julius Gustavsson
So, you know, especially when we're onboarding new developers to the team, they are usually quite quick to get up to speed because they can, you know, they can hack around without fear because, you know, as soon as the bills pass and the tests pass and everything, then you're fairly certain that, you know, nothing weird has been done.
Rust in Production
Volvo with Julius Gustavsson
There's something that you didn't think about, or someone else didn't think about, or something you thought of that wasn't written down, and then someone else broke that assumption. There's always something that... that makes these kind of, especially when it's so many moving parts in this little proof of concept that I was doing there, that, yeah, that just wouldn't have happened.
Rust in Production
Volvo with Julius Gustavsson
Everything from two weeks, which was the extreme case, to maybe three to six months, something like that.
Rust in Production
Volvo with Julius Gustavsson
I would say so. It's absolutely something to respect, and I would definitely, like we did, we seeked help, and we got some people on board that were already really good at this. I wasn't super proficient in Rust when we started. I had done this project and some other smaller hobby things, and actually none of our regular developers are... had worked with Rust in any big capacity before.
Rust in Production
Volvo with Julius Gustavsson
So we've all picked it up on the job. But of course, they are experienced C and C++ programmers. Not everyone even embed it. So they've had to learn that as well.
Rust in Production
Volvo with Julius Gustavsson
We read a lot of code. from yeah from the community we we also have we use clippy a lot to to avoid you know the bigger issues rust format and and rust and and clippy of course are mandatory and and we use pedantic clippy pedantic
Rust in Production
Volvo with Julius Gustavsson
and and no warnings allowed so to say so that so that takes and that gives us a lot of pointers but but then i mean our our code has evolved over time you know going back to some things that we did in the beginning or we can absolutely see that there are some some c isms here and there uh
Rust in Production
Volvo with Julius Gustavsson
There are definitely exceptions, but I would say in general, it's very common Rust code. The robustness of just Rust in its default form is really high. But of course, you can't just unwrap all over the place. You have to... You have to ensure that errors are actually getting handled and logged properly.
Rust in Production
Volvo with Julius Gustavsson
And we have a quite extensive, so we talked about this diagnostic stack, and we also have like a framework for creating diagnostic monitors that are monitoring different conditions and reporting that.
Rust in Production
Volvo with Julius Gustavsson
Yeah. And of course, if it panics, we need to reset back to a working condition. But we also use the chip as a watchdog. that we also need to activate so that in case it would lock up for some reason, the watchdog would then hard reset it.
Rust in Production
Volvo with Julius Gustavsson
Yes and no. Of course, there are rules about everything when it comes to automotive software. In our case, we don't use any allocation. We don't have an allocator, so we don't use it. So everything is static.
Rust in Production
Volvo with Julius Gustavsson
Or at least not for me, that's for sure. So yeah, Rust definitely did the heavy lifting there, or at least made sure that I had, because Rust tends to make you think through the whole design up front, basically, or at least, you know, a much larger part. So, yeah, so there are these fine, you know, loose ends that you also need to tie up before it actually builds and runs.
Rust in Production
Volvo with Julius Gustavsson
Yeah, so we don't use that dependency at all. And the heap is zero size, basically. But that means that we use the stack a lot, and we use heap less for a lot of these things. But also, yeah, we put our buffers and things, they're statically allocated here.
Rust in Production
Volvo with Julius Gustavsson
scheduler i would say or part of the operating system hubris and they do something similar where they only have certain message types that they can handle yeah yeah exactly the same way so everything is statically known beforehand or the worst case is known so yeah we never have to deal with any
Rust in Production
Volvo with Julius Gustavsson
unknown buffer sizes and then when it comes to input you asked about input validation that is of course super important and that's why it's important also to have a well-defined so that all all messages that we are sending are of well-defined format so that you can easily you know at the parsing stage see if it's actually valid or not and reject it already there um yeah
Rust in Production
Volvo with Julius Gustavsson
Yeah, that's true. It's the Polestar 3 and EX90. Volvo EX90. And then just keep on the lookout for Spa 2-based cars. And more will be coming soon.
Rust in Production
Volvo with Julius Gustavsson
Yeah, I can't really say how... I would guess it's fairly common in automotive in general. I mean, if you look at... you know, what is called warranty, warranty in general, and things that are being replaced due to that, that is quite often software that is at fault. So yeah, I would definitely...
Rust in Production
Volvo with Julius Gustavsson
And of course, it's still early times, so I'm not going to say that that is not going to happen for us, but so far it has. We have seen some manufacturing issues, but those have been hardware related so far and hopefully something that we will be able to catch in the early stages.
Rust in Production
Volvo with Julius Gustavsson
I'm just going to hop into my time machine and get back. Now, I would, of course, love to see it in more places. So that is what I'm actively working on now to find or to see where it would fit. Because it doesn't make sense to rewrite or replace everything with Rust. Because when you're already...
Rust in Production
Volvo with Julius Gustavsson
When you already have the thing working and it's tested and it's according to specification, it's seldom that it actually makes sense to rip it out. But there are cases where that is possible. especially when it comes to cybersecurity-critical user-facing code, things that need to do validation of data that is coming from the outside world, things like that.
Rust in Production
Volvo with Julius Gustavsson
So I definitely see a possibility for those kind of use cases where we're interfacing with the internet, for example, or things of that nature. But yeah, now that the hurdles I mentioned earlier, the hurdles that we had in the beginning, we didn't have support for neither the hardware nor the OSs that are commonly used. But all of those hurdles are falling one by one.
Rust in Production
Volvo with Julius Gustavsson
And I think that those are the things that make the difference.
Rust in Production
Volvo with Julius Gustavsson
Yeah, it's funny you say that because there is actually no difference. We build one binary, and that is the binary that we test on our test hardware, and that is the binary that goes unmodified to the core computer. And so far, there's nothing that we can't actually test on our dev board, or at least very few things. There are, of course...
Rust in Production
Volvo with Julius Gustavsson
A few things that change when you're in the context of a whole car. But when it comes to the LPA functionality, we can test essentially everything before we deploy it. So it's almost unheard of that we get some sort of issues due to the fact that something was working differently on our test hardware.
Rust in Production
Volvo with Julius Gustavsson
So, so yeah, so one, one binary for no difference between dev and prod and, and that in that sense, but what you said with possibility to reuse, and that is definitely one of rust's huge, strong points that, that the fact that you can actually, you know, take different components from the community and you can re use them with.
Rust in Production
Volvo with Julius Gustavsson
with confidence that you don't really have another, I mean, both hassle-free almost always it's just builds and works and you can just use it. And, and also the, you know, all these things about that you always have to take care of first. You need to, okay. Can you get it to build? in my build system. And first time, probably not. You probably need to spend a lot of time to get that to work.
Rust in Production
Volvo with Julius Gustavsson
But then when it actually builds and you start to use it, does it use memory in the same way as you do? Does it make the same assumptions? Like who is allocating this buffer? Who is freeing it? Who has the responsibility to do what? And so on and so forth. And those are not standardized in C and C++.
Rust in Production
Volvo with Julius Gustavsson
Probably, but the nature of the, since we were doing like gRPC communication over Wi-Fi, it was very asynchronous in its nature. So I wanted to try it out and see how it would work. So yeah, that was the idea.
Rust in Production
Volvo with Julius Gustavsson
Meaning that, yeah, so even if you can build it, it's no guarantee that you can actually use it in a, you know, in any productive way. But in Rust, on the other hand, everything is... Maybe it's not true to say everything, but you're pretty much guaranteed that it will work.
Rust in Production
Volvo with Julius Gustavsson
And also, like you said, I don't think there's any language that is as scalable from the lowest, smallest microcontrollers to backend systems or... or whatever biggest server farms that you can imagine and everything in between.
Rust in Production
Volvo with Julius Gustavsson
Yeah, empowering is definitely the key word. We have been super, super happy with this project, and it really shows that Rust has a bright future. The hurdles that we mentioned before when we started the project, which made it basically a non-starter for most of the ECUs that we wanted to use it, those are all coming down or have come down already. So now you have QNX support, for example.
Rust in Production
Volvo with Julius Gustavsson
You have Infineon tricore support. There was recently an article on how to run Rust together with Autostar Classic. Autostar Classic is the common automotive software framework that most automotive software runs in. And now that is becoming available for Rust as well. There is work being done on... on various different automotive platforms and components.
Rust in Production
Volvo with Julius Gustavsson
So for automotive, it's definitely a bright future. And we'll see where we'll take it at Volvo.
Rust in Production
Volvo with Julius Gustavsson
Yeah, so there are two interviews out, actually. One from a couple years back, where we were kind of in the starting position, and you can find that on the Volvo Tech blog. It's called Why... Volvo thinks you should have rust in your car or something similar. Silly pun. And then now recently there was an article, Tvede Holv's blog, where we talked about this project and how it has been going.
Rust in Production
Volvo with Julius Gustavsson
You can find me on LinkedIn and on X, Mastodon, all these different platforms. I also was fortunate enough to join the Rust Safety Critical Consortium that was started last month at RustConf in Montreal. And together we are aiming to close the final hurdle, which is to make Rust a fully accessible
Rust in Production
Volvo with Julius Gustavsson
viable alternative to do safety critical software so that is that is what we're doing now and you can find more about that at at the rust foundation web page and there's also a github repo where you can join if you like admission is free
Rust in Production
Volvo with Julius Gustavsson
We're at least super stoked about the current project and how well it has worked out. Definitely going to be pushing that forward within the company and hopefully we can find some other exciting avenues for it.
Rust in Production
Volvo with Julius Gustavsson
Yeah, absolutely. Well, needless to say, then maybe we'll come to that later on how we came to the LPA project, but where we're not using it, at least not yet.
Rust in Production
Volvo with Julius Gustavsson
But yeah, I'd done a fair bit of... And also a lot of embedded systems are callback-driven in their nature because you're getting all kinds of events from hardware peripherals, from other things, then your main thread constantly and also your... When you're achieving something, you usually need to do it in steps.
Rust in Production
Volvo with Julius Gustavsson
So you do something, register a callback, wait for that callback to be called, then you continue. And even in that stage of futures at the time, it was still removing a lot of boilerplate already then. So that's why I thought it was intriguing.
Rust in Production
Volvo with Julius Gustavsson
It's a good question. I guess I would most likely use some sort of async framework in C++, I guess, since we had gRPC, and gRPC, protobuf, all that stuff was already fairly well supported in C++, so I guess I would go for that. And I don't remember how... if those async frameworks at that time in C++ were, how far they'd come. They probably had similar sort of features.
Rust in Production
Volvo with Julius Gustavsson
The problem there is that you usually don't... When the closure that you register actually gets called, you're basically on your own because the... There are all sorts of assumptions that need to be upheld for that to actually work properly.
Rust in Production
Volvo with Julius Gustavsson
For example, memory or variables or pointers and things that might not exist when it comes back. So yeah, dangling pointers is a common issue.
Rust in Production
Volvo with Julius Gustavsson
I would probably trip on those a couple of times before I, you know, before I get it working. But because it's so easy to, yeah, it's super easy to misstep when it comes to those things.
Rust in Production
Volvo with Julius Gustavsson
No, I was probably quite obnoxious. I'm pretty sure there were some colleagues that were getting quite fed up because I was talking about it a lot. I remember I did a talk back in 2017, or no, it must have been early 2018, where I tried to do a basic introduction, but also to point out all the upsides.
Rust in Production
Volvo with Julius Gustavsson
And then when this SPA2 project started to crystallize more and we started to productify everything that we had in our plans, i i started looking for places where or or see if this was actually viable as as a language to use and unfortunately none of the platforms so so the so this yeah if we go go back a little the the spot two project is it's the electrical system in in the
Rust in Production
Volvo with Julius Gustavsson
in the Volvo cars that are coming to market now. And the main big feature of that system is the core computer, which you can probably Google and read about. And this core computer is essentially a centralized box with multiple different processors, all running different functional domains, doing different things. Like you have security, you have...
Rust in Production
Volvo with Julius Gustavsson
high integrity for like braking and steering and, and these safety critical things. And then you have a high performance compute for, for, uh, these more big number crunching things and, and 88 us functionality, you know, autonomous driving or, or yeah, that, that kind of use cases and, and all these, yeah, these were all different platforms running different operating systems or hardware, uh,
Rust in Production
Volvo with Julius Gustavsson
And none of them actually had good Rust support, except for the security gateway. But we, yeah. That's a long story why we didn't start off there. But yeah, so we had a QNIC system and there was no QNIC support. We had tricore-based units, which tricore is a common automotive-based microcontroller specifically designed for safety-critical type of use cases.
Rust in Production
Volvo with Julius Gustavsson
True. And Volvo is also an automotive or a car maker. And automotive people tend to be quite conservative when it comes to new things. We like to use well-proven technologies. So yeah, definitely. But I always saw that this was actually something different, something that could actually provide real value.
Rust in Production
Volvo with Julius Gustavsson
Yeah. So I wouldn't say political infighting, but the one question that always came up and that was, is it safety qualified? So is it qualified and certified to use in a safety critical setting? And that was a, That was basically a no. I mean, nothing had been done on that front at that time.
Rust in Production
Volvo with Julius Gustavsson
So that was kind of a non-starter because most of these functional domains that I mentioned earlier, these CPUs, they are running, they have a safety rating, so to say. These ACL levels that the automotive standards define.
Rust in Production
Volvo with Julius Gustavsson
The safety standards that the automotive industry uses, basically they talk about how do you provide the evidence that the product that you've developed actually does what you think it's doing and that there are no hidden errors or dangers that you could have foreseen and mitigated. That's basically the...
Rust in Production
Volvo with Julius Gustavsson
The reason why everything needs to be vetted, so to say, both the tooling, but also the code and everything that you use.
Rust in Production
Volvo with Julius Gustavsson
No. And Ferris system, they actually started maybe six months later or something. We met them at a conference in, I think it was OxidizeConf. And they soon after announced the first scene where it was called Sealed Rust at that time. But yeah, so that started soon after. But I always felt that
Rust in Production
Volvo with Julius Gustavsson
that that wouldn't be an issue over time, because obviously we are doing safety-critical systems in C and C++, and we know all the... And it's riddled with things where you can shoot yourself in the foot, and you need all kinds of tooling to basically ensure that you're not doing... The things that you're not allowed to do. While Rust is preventing most of those things up front.