Ufuk Kayserilioglu
Appearances
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So the team will be responsible for contributing to MRI and contributing to survey, basically solving and maintaining and building on the open source Ruby foundations. I've been in the software development industry for over 20 years. I've worked with various statically typed, dynamically typed languages across that time. Initially, I was working in a startup.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So we basically stopped doing it and then also realized that the same approach, even though it has some disadvantages, but the same approach of generating Ruby interface files for all the meta programming would be a much better fit for our problem. So recently we've been working on that problem. There are other
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
solutions in the same area as well one of them is for example the gem called sorbet rails that's built by people at the chan zuckerberg initiative so they had the same problems of sorbet not understanding some of the meta programming that's coming from rails again like all the column related methods on models all the association related methods on models all the the
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Class methods, sorry, all the instance methods in mailers that are turned into class methods, all the helpers on your controllers, they're all happening at runtime in Rails. So Sorbet Rails was the first attempt to actually build a gem. that could generate Ruby interface files for all those things so that Sorbet can look at the Ruby interface file.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So you have your shop model or like shop.rb file, which hosts your shop model, but you also have this shop.rbi file that's generated for you, which contains all the method definitions that would be on the shop model dynamically. So having that Sorbet can say, oh yeah, okay, I know shop has this name method. So if you call shop.name, then yes, I know what that is.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And even better, if you can also create signatures for it. So if you can tell it what parameters it takes and what the return type is, then you can also keep typing those things as well. So you can build on those types as well. So Sorbet Rails was the initial solution to this.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But we also realized that we had other metaprogramming DSL concerns within our code base that obviously wasn't addressed by Sorbet Rails. So we wanted a broader framework. So we built one in-house and we actually are in the process of folding that also into Tapioca so that Tapioca becomes this one Swiss army tool of generating RBIs, either from gems or
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
or from all the DSL metaprogramming that's going on across your codebase. And initially we've again targeted the same Rails metaprogramming concepts. So we borrowed and built on some of the solutions from SurveyRails, but we've also built DSL generators for
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
other common gems across our code base and we intend to to grow that set based on contributions from the community so we have a generator for active record type store frozen records identity cache that's also a shopify gem so like rails plus a few gems will already be addressed with Tapioca.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And that was the biggest pain point, stopping us from increasing our typing even further across our cloud base.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Yes, great question. So the obvious headline thing for Sorbet is to obviously to ensure that your code is typeset, right? So that you're not making silly mistakes, that you don't have references to classes or modules across your code base that are no longer there.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
We've found many instances of that across our code base through the use of Sorbet, for example, or that you're not calling methods on constants that don't actually have those methods, or that you're not providing arguments to those methods that they don't actually accept. So all of those things, if you're working on a normal Ruby code base,
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
all of those concerns, you can only discover them at runtime. So you either need to have extensive testing to ensure that all those edge cases are executed on your CI, or you discover them in production. So the initial goal is obviously was to reduce those types of trivial errors as much as possible. Unfortunately, in our initial adoption phase, we couldn't validate or invalidate that hypothesis.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
I was one of the founding employees where we were developing voice XML based applications and we built a voice XML platform. And then we did like various networking things, voice over IP, text to speech, speech recognition, et cetera, et cetera. I had a short stint in between for a couple of years doing startup acceleration and then went back to developing software.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But we did other things to compensate for that. So there's a very understandable reason for why we couldn't conclude on that hypothesis of if we were able to reduce trivial errors is because our core monolith does more dynamic things than the stuff that we have in our code base. So it
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
kind of like serializes objects to the database and then reads serialized objects from the database, et cetera, et cetera. And there's no way a static type checker can like prevent you from, you know, deserializing an object that definition for which has mistakenly been remote from the cloud base, right?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So suppose you had like a user class and there was a user object serialized somewhere and then someone removes the user class and now someone tries to deserialize the user object. and then boom, your application fails, right? Because the user class doesn't exist anymore. So you can't deserialize it. So we obviously can't prevent those kinds of errors.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So there are like other dynamic things going on. And like I said, core is such a large code base and there's like so much variance across like it's failure modes that it's hard to get the signal that we were looking for among all that background.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So then we said, we actually need a smaller code base to test our hypotheses on, to see if we're able to move the needle in that type safety in any direction. So after we did the initial adoption on core, we turned our attention to a much smaller and more opinionated code base within the company. And that's a tool that we call dev. So dev is a tool that
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
every developer at Shopify has on their computer. And it's the tool that every developer uses during their daily work workflows. So there is a command line tool, you use it to clone Shopify repositories, then you only invoke dev up and it knows Based on a configuration file within the repo, it knows which dependencies to install, which commands to execute to bring the application up.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And it also gives you different shortcuts like dev test or dev type check, which was something we added to run various things. So dev is a very opinionated code base because it's a developer tool. It's a command line tool. Everyone uses it throughout the day. So it needs to be fast. It has, for that reason, it has no external dependencies. So it doesn't use any gems or anything.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
If it needs some other code, it vendors them in. So it was like the ideal code base. And the team that's working on that code base is a small team that's been working on the same code base for a long time. So we talked to that team and we told them that we were interested in adding type annotations to the majority of the dev code base and to see what kind of an impact we could get on type safety.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
That was a project that took a couple of months because it's especially hard to add typing to code that you already have because it means you need to understand how data is flowing throughout the code because you have some methods that ends up calling some other method and then combining the results from that other method by the result of another method call or something.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So you have these chains of method calls that you need to understand. what types they return to understand what types your method in question is returning. So it becomes like a arduous process to fully type an existing code base. So that took a couple of months, but we approached that very methodically.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So we took a look at the ways dev fail, and we took a look at all the ways that dev fails that's related to type related problems. Any exceptions that Dev was raising, they were always like all being captured anyway. So any exceptions that it was raising that were type errors, no method errors, or argument errors, we collated that. So we drew graphs and everything.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So that was before we started this experiment and we drew the same graphs for after we finished this experiment. And our findings are interesting. So we were actually able to completely eradicate name errors because that's the lowest hanging fruit. So the basic premise that Sorbet gives you is
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
to ensure that all the constants are resolved somehow so that you don't have any constants that Sorbet doesn't understand in your code base. So, and that's what a name error is, right? Like you refer to foo, but foo doesn't exist. So you get a name error. And if you're not doing like const get or something, something dynamic in your code base,
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Just static analysis is enough to eradicate all the name error problems. And then we also looked at, like I said, type errors. There were a few type errors left still in the dev code base, but we did trace them back to files that weren't marked type true. And maybe I should just explain what type true, type false, all those levels are. Okay. So Sorbet is a gradual type checker.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
I had one talk at RubyConf 2019 last year, and I also submitted a RailsConf 2020, the couch edition, the remote one. I also had a talk there.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So you don't have to convert your hold code base to be typed overnight, but you can opt into as much of Sorbet as you want. So when I said, you know, we adopted Sorbet in our core code base, of course, I'm not saying like we, you know, added types across the whole code base, but we
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
enabled all the developers to add those type signatures and to have their codes checked against those type signatures. So the same thing in dev. So we did add types to the majority of the dev code base, but not to all of it. And we were able to track the type errors to the parts of the code base that were where the files were marked type false. And type false says
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Do not type check any of the method calls in this file. Only check that like the constants resolve to constants that you know. It does the basic type checking. Whereas in type true, the file is type checked for all the method calls and for all the parameters that are being passed correctly. And if the methods have signatures, that all the methods have the correct types.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Yes. So the name errors is the lowest hanging fruit because you take a typed false on top of your file and run sorbet over it. And it will tell you, oh, you have this constant that you're referring to here in this file, but I have no idea what it is. So it's either coming from somewhere else in your code base that Sorbet isn't processing, or it really doesn't exist.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So you just, you know, remove your usage of it. So name error was a simple one, but we were able to reduce type errors as well. And we also realized that we were able to reduce argument errors and no method errors as well, though not to the same extent. Because even though you're opting into types, there are still situations where when you call a method, the method doesn't have a signature.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Sorbet's best guess is it could return anything. that's represented as a T untyped in Sorbet. That's the equivalent of any in TypeScript if you've worked with TypeScript before. So it's a type that can be anything. So you can call any methods on it with any parameters. So basically Sorbet stops doing type checking past that point.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So a T untyped always returns a T untyped when you call a method on it and everything. So that kind of lessens the amount of strict typing that you're doing. You can opt into higher types, strictness levels in your files to prevent this. So you can opt into type strict, which will tell you if you're doing this unsafe calls and it will say, oh, you're actually calling into a method.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
that doesn't have a type signature. So, you know, there's something wrong here. So go and add a type signature to this other method that you're calling to make sure that this file has a stricter type checking. So we didn't go that far in the dev code base. We wanted to see how much we could get with simple typing.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So there were still a few argument errors or no method errors that leaked in, but we were able to reduce those kinds of errors as well. But that's not the only thing we did. We also looked at if running type checks on CI had any effect on the CI success rate. So were there like any false positives that Sorbet was failing on that were breaking CI builds or vice versa?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Were there any test failures that Sorbet wasn't catching? And we realized that Sorbet didn't have an effect either way. So it wasn't causing superior failures on CI, nor was it really missing anything. So there weren't really that many test failures that survey also didn't catch. But we also talked to the team. So I think that's one of the most interesting things we've done.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
We talked to the team both before we started this experiment individually and talked to them after we finished this experiment. to see. So before we wanted to see how much they knew about Sorbet, how they felt about Sorbet, were they enthusiastic or did they have questions about its utility or not? So we compiled those.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And then after we finished this experiment, we also asked them if they were using Sorbet type checking on their machines, if they were happy adding signatures to the codebase, or if they had problems having to deal with fixing signatures when they were factoring. So we were able to also get a qualitative feedback on how the team and how the developer happiness was impacted or not.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And we actually heard good things about that. So there was one person on the team who really didn't like signatures. They found it really distracting. One of the things about Sorbet signatures is because Sorbet is not integrated with the language. Obviously Ruby as a language doesn't support types. So you need to add a signature annotation on top of your method definitions.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
I think we can maybe talk about the network stack video and the talk from RailsConf near the end of the show if we have time, because I think most people would be interested in Sorbet adoption and how a company with a code base as large as Shopify has adapted been successfully adopting it. I'm presuming that's what most of the listeners would be interested in.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And that signature annotation is also Ruby code. So you do like a sig block. So you say sig and then you start a block and then you say params and you declare your parameters and their types. And then you do a dot returns and then you declare the type of your return. And then you can do more things in there as well.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But obviously sometimes for a two or three line method, your signature can end up being five or six lines if it's doing something like non-trivial or if your types have like some generics or something. So some people actually find that. to their posts and sometimes distracting. So they say that it makes it hard to actually see the code.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
They end up seeing like all these signatures, but that person on the team actually developed like a few Vim configs to de-emphasize the signatures. So they basically turned the opacity on the signatures so that they look like comments, which they kind of are, right? Like, because that's the same thing with Yard definitions. So when you add type annotations in Yard comments, that's the same thing.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And then someone at the company also built a Visual Studio Code extension to do the same thing. I think it's called Vysig. So it parses the signature blocks and then turns the opacity on. And I've actually been using it, and I'm also enjoying it because I don't necessarily need the signatures to be in front of me all the time.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But apart from that, we heard that developers actually like the safety guarantees that static typing gives them as they're doing refactoring, for example. So for example, if they're removing a class or a module from the code base, they're almost guaranteed to know that when they run the static type checker, that it will complain if there's like any dangling usages of that class or module.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So they really liked those guarantees and they didn't really mind writing the signatures in the first place. And it's a really easy ramp on as well on the signature syntax. So we actually heard good things from developers on working on the code base too.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Yes. That's also another one of our findings. We also talked to three different teams who were working in core, and they were one of the early adopters of Sorbet in their part of the code base. So we have this componentization thing going on in our core, in our monolith, because it's a huge code base. Otherwise, it becomes mayhem.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So there's these components, and some components were the early adopters of typing. So we did interviews with team members from those teams as well to understand how they felt about those early initiatives. And one of the things that we ended up hearing over and over was that adopting types led to better code design, which initially was interesting for me.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But that seems to be the biggest benefit from adopting static or gradual typing rather than runtime type safety. And so my opinion is a little bit swayed now. Because too often when we're working on simple solutions, we reach out for like these bags of values like arrays or like most of the time hashes, right? So we just like reach for a hash and then we keep passing those hashes around.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But then it's never documented what the keys are. Like if let alone if the keys are symbols or string. How many times did we assume that we could access some hash using symbols? It turns out to be that it was indexed by strings. When your code base is small, that's fine. I'm not judging anyone who's doing that. But as your code base is growing,
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
you need people to more easily onboard onto your team. And if they're fixing something and if their focus is on, let's say one method or one file, you don't want them to go hunting around seeing, oh, where's this hash actually being coming from? Where was it created initially? What are all the parameters that are in this hash? So that's not like a very efficient way of working at that scale.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So at that scale, it's better to give a name to that, to turn that into something like a structure or create class for it or something to basically start using types, right? Once you turn something that was a hash into a struct or a class or a module, you give a name to it, you give a type to it. So you start using types.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And those types actually convey more information than passing around hashes would. So it leads to actually better code design. And we've heard that over and over again from various people within the company. But it also leads to evergreen documentation as well.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So if you're looking at a method and it has a signature on top of it, and it says these are the parameters that it takes of these types and it returns this type, then you don't need to go and look anywhere else, basically, if you know what those other types are as well. You don't need to go look anywhere else, like to all the colors of this method or whatever. It's just evergreen documentation.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And the reason why it's evergreen is because if somebody changes the method signature, they have to change the signature on top of the method as well. Otherwise you will have like the static type checker breaking in CI, right? So it has to be evergreen. People need to keep it up to date. And that makes it very easy to onboard new people to your code base or to your team or to your company as well.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So we heard from developers that those were all great benefits of adopting types. And like I said, we didn't adopt types across the whole code base, but we're allowing people to adopt those types gradually as much as they want. And our team is also helping people within the company in their adoption process. So sometimes we help them We review their PRs.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Sometimes we actually pair with them to make that adoption easier. Or we just answer questions in a Slack channel when they're stuck.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
I haven't calculated that. I don't know. But in case some of the listeners aren't familiar with the company, maybe I should just give a quick intro to what Shopify does. So Shopify is a leading global commerce company that provides tools to start, grow, market, and manage a retail business of any size. So it started off in 2006. code base that was built on rails pre 1.0.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Okay, so personally, just before I joined Shopify, I was working on code bases that were built by TypeScript. And actually, TypeScript was, I think there is something wrong with me, and I'll come to that.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So TypeScript was, I was at the time working in a web agency, but I was on the team that were doing like cutting edge two week sprints that were like trying to see if we could make certain things work. And we were mostly working on JavaScript and I consciously chose to try TypeScript to see how it would work. And TypeScript coupled with Visual Studio Code was a joy to use.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And I started loving it for all the code completion and all the type things. And you don't look at the documentation a lot.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Anyway, like I was really enjoying TypeScript before I joined Shopify and I had already heard about Sorbet and like Sorbet got me similarly excited because it has, it still has the promise of doing the same thing that TypeScript did for JavaScript, which was not to destroy the language as a language.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But to add features on top of it that people could opt into if they wanted or needed more strict typing. And it's again, this whole thing, because TypeScript is a hundred percent JavaScript plus some other things. The difference with TypeScript is it compiles into JavaScript. So it does not type checking at runtime. Whereas with Sorbet. It's built inside of Ruby.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So all the sorbet annotations and everything are also Ruby constructs. So those Ruby constructs can also do type checking or type assertions at runtime. So that's the kind of difference, but looking back, I kind of see myself having almost exclusively worked in code bases that had some types somewhere. So I'd worked with Delphi for a while and then C++, C Sharp. And then, like I said, TypeScript.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But before TypeScript, we were working on a PHP code base that was using type hints. So that was a PHP 5.3 code base, if I remember correctly, which had type hints. And we were really relying on type hints. So the project that started before I joined and it was built like that. And I ended up feeling like that was a good way to work because it was preventing us from shooting ourselves in the foot.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Even though that was still doing type checking at runtime, it was still failing fast, right? So that's like one of the principles that we software developers use a lot, this ability to fail fast. So static type checking really gives you that. So it allows you to fail fast.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Either in our base case, you just run the static type checker and within a couple of seconds, it shows you if there's like some assumptions in your code base that don't check out. And that's way better than waiting on the CI for a couple of minutes. you can just run this tool constantly on your machine, right? Right.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So Sorbet actually has an LSP mode. So the Sorbet binary, that's the C++ static checker binary. If you download the Sorbet static gem or the Sorbet gem, which pulls it in, you can actually do srbtc dash dash LSP, which turns on the LSP mode. So LSP stands for the language server protocol, I think.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So it was the time of rails when it was still being shared as zip files. And it's been growing ever since it started as an e-commerce platform. And it's grown into being a multi-touch general commerce platform. So it provides solutions both for e-commerce and also in-store commerce, in-person commerce. So it has many solutions now. It supports more than 1 million merchants now.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
I'm not so sure about the acronym, but it's developed by people at Microsoft as they were developing Visual Studio code. They developed that specification, so it allows different languages to integrate with different editors, and LSP is the language that sits in between.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So Sorbet has this LSP server mode, and they're also working on an official Visual Studio Code extension that you can install that would bring all that richness into Visual Studio Code. And I'm one of the early testers of that inside of Shopify. And I've been using it for a while now and it's actually integrated into your editor.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So it gives you the best auto-complete that you can ever find, way better than like all the other editors can do. But it also shows you all the errors within your editor. That's a great way to work because you get instant feedback on your code changes. I don't want to give too long an answer, but just to address the second part of your question is like, when would you reach for a tool like this?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So scale is very important. So when you have like in our code base, tens of different components and 30,000 different files, then it becomes really important to have better contracts between all those different moving parts. And what better contract than types? And we've actually realized that types really work well for that.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And I think that was the initial impetus for why Stripe started working on this as well. And I'm not saying that all Ruby codes should be typed or that everyone should reach for typing as soon as possible. Obviously it's a matter of needs and scale. But I also feel like that it's the right tools for the scale that Shopify is in right now.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And the teams that are really relying on types are getting a really good return for their investment.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Actually, Shopify's response to this whole pandemic has been really impressive because senior executives at Shopify were aware of where the whole pandemic situation was going really early on. And they were really quick in taking measures to ensure that people working for Shopify are safe. So that was very early on. So they basically closed all the offices.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
before like most countries were like aware of what's even what's going on. So for me personally, I can't quite talk for those office closures because I've always been working remotely for Shopify. So I live in Cyprus. I work for Shopify remotely from here and our whole Ruby and Rails infrastructure team has been distributed, has been a distributed team as well.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So we had a couple of people working in the offices in Ottawa. Someone in Montreal, one person in Poland, one person in France, UK, Cyprus, all over the world. So our whole team dynamics and everything didn't obviously change that much.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But obviously, as everyone's feeling it, starting to work remotely during the pandemic was a huge burden to a lot of people because they weren't ready to work like that. Also, it wasn't a normal working remotely setup either, right? Because Schools are also closed. Everything else is closed. You're trapped in this house and you can't go out. And you're also trying to do work.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So it's not necessarily the normal working from home situation. But like you said, our senior executives looked at the situation going forward. So first of all, Shopify isn't reopening any of their offices. before the end of this year, whatever happens, because the outlook isn't that great. It doesn't look like this pandemic is over anytime soon.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So there's no point in like opening offices and making people feel safe in going there just to then like close them again. But also, the company has, based on this experience that we've had for the last couple of months, the company has also decided to be digital by default, which doesn't necessarily... It's digital already. It is digital already, but it wasn't digital by default, right?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And throughout its lifetime, it supported 172 billion U.S.D., in total sales for all the merchants. Of course, that's lopsided to the recent here. So the figure that you quoted is probably correct, Luke. So I don't know at the top of my head.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So that's the extension. So what used to be the case is, for example, if there were people in the office and if there was a meeting, so if there were like 10 people in the office and three people joining remotely... then the 10 people in the office would be huddled up in a meeting room, and then the three people would join in on a call remotely. But that isn't necessarily the best way of working.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
because the people who are joining that call remotely don't get the same experience that those 10 people in the room get. So the digital by default kind of changes that. And they say, we're not closing our offices, but they're saying the offices won't be the same offices. So they'll be revamped to accommodate this new digital by default future.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But basically then in that future, we'll get to experience it obviously, but in that future, those 10 people will also individually join that same meeting as well. So everyone will be digital in the meeting by default rather than having 10 people co-located somewhere and then three people remote.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So that's the biggest difference because the decision there was very simple because you either do fully co-located or you do fully remote. In between is this hybrid thing that doesn't work that well for either party, right?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
You have a few people that are co-located in an office, a few people that are remote, but then you need to have all your communication digital to ensure that the remote people have the same context as the people who are co-located. But people who are co-located, if they have chats that they have in between and they don't report them back, then you have information asymmetry, et cetera, et cetera.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
That's the worst way to work. I've also worked remotely, being the only remote person on a team that was co-located before, not at Shopify. And that wasn't a horrible experience because my team was very understanding of my situation. But it's still not the best way to work when all your teammates are all in the same office and you're the only one remote.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So the future for Shopify is digital by default for everyone. So even if you're in the office, you're still communicating with everyone digitally. That will be the primary mode of communication. And most of our work will not be done from offices anyway.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
All the engineering work that we're doing, we have a Shopify engineering blog. That mostly gets a new blog post once a week, I think. Or the Shopify engineering Twitter account. They can follow. And I should also say that Shopify is still hiring and we've been hiring. Actually, we ramped up our hiring across the pandemic as well.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So if anyone's interested in a position at Shopify, also now that we're digital by default, we're more open to remote people. They can go to Shopify.com slash careers or our careers page on our website. And they can be a part of the team so that they can follow the developments from the inside.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
It sounds believable, though. As for the code base, it's most probably the largest Ruby on Rails code base in the world right now. It has about 30,000 Ruby files in our core monolith. So I'll keep referring to core. So when I say core, that's our main monolith that provides almost all of the Shopify solutions.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Yeah, we're really interested in that on my team as well, because the people are doing Drifting Ruby and MRI work. want to make sure that both of those platforms run really well on the new ARM chip.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Even though we now have some other services that operate outside of core, most of our code base and most of our operation is actually still being handled by core, which is our monolith. And that's about 30,000 Ruby files. And on each push, on each deploy, there are about 150,000 tests that run to ensure that everything is still the same. So that's the size of the code base.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So I've only been at the company for about a year and a half, and most of the infrastructure has been in place before I joined. And I'm also still learning about some of the infrastructure that we use, but like I can tell you it's it's a Kubernetes deploy. So everything's virtualized in Docker containers and deployed to Kubernetes pods. And there are like some load balancing solutions.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And we also have this concept of pods. Pods are these logical units that host, um, a group of certain merchants in one location with their own basic infrastructure. So that's like one way of sharding the customer database and the customer load across different machines.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
I'm pretty sure that like others from the company have presented at various conferences talking about that infrastructure part, but that's not necessarily my strongest point. So I don't want to really go much more in depth about that. even though the team that I'm working on, the Ruby and Rails infrastructure team, is a part of the product line in the company called
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Kernel is basically responsible for ensuring that the platform is up and running. So it's responsible for all the infrastructure problems, but also it includes developer acceleration, which my team is a part of as well, which provides solutions for the developers within the company so that they can do their work better. They can produce solutions faster.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And also we provide solutions for the infrastructure part of the team as well.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Sure. Again, maybe I should give a little bit more context as to what Sorbet is, who developed it, what it's used for. Sorbet is a gradual type checker for Ruby that was developed by the fine folks at Stripe. So Sorbet provides you the tools needed
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
to add type checking information to your Ruby code base, and it gives you both static type checker and a runtime type checker to ensure that your type assumptions make sense and are valid. The static type checker is the most important part of Sorbet, The static type checker is a tool. It's an executable built using C++. It has its own Ruby parser.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So it's able to parse and understand Ruby and in some ways simplify the Ruby language. So it can do those kinds of static analysis without necessarily running your code. Well, I shouldn't say necessarily. It doesn't run your code at all. So in that way, it's a static type checker. And it's built to be blazingly fast.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So it can parse, understand, and type check our whole code base, the code base, the core code base that I've been talking about of about 30,000 files in a matter of like 10, 15 seconds. And it was built with intentionally to be fast and to be as safe as possible when you're developing Ruby. Sorbet was amazing. open sourced about a year ago. So it was June, 2019 when it was open sourced.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But it was being developed by Stripe for about, I think, two years before that. And by the time I joined Shopify, we had already gotten on the early release beta program where they were doing like code sharing with Shopify. So we were able to start our adoption earlier than when it was initially open source.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Then afterwards, Stripe also got like a few other companies that were interested into the public beta program as well. But so that's why our story is a little bit different because we ended up solving some of the problems that people who might want to adopt Sorbet don't have to solve right now because it's a much more mature product.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
But back when we were trying to adopt it on our code base, it was mostly a tool that was built with the Stripe code base in mind. And I can't talk too much about what Stripe code base is like, because I don't know, but from our understanding is it doesn't use Rails. It tries to use as little meta programming as possible.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So they're like really big on intention revealing code and not so much meta programming. So Sorbet was the right tool for them and they were developing to solve their own solutions, obviously initially, but because our code base is Rails, which already has a lot of meta programming. So meta programming brings in a lot of dynamic methods that can only be seen when you're running the code, right?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So, so there are lots of methods that you don't see, for example, on active record models, you don't see all the attributes, right? Because the attributes are created at runtime. All the methods that relate that are related to attributes are created at runtime. after Rails has had a chance to introspect your database tables.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
I've been at Shopify for about a year and a half. And since I've joined, I've been first on the Rails upgrade project. And then I quickly switched over to the Sorbet adoption team. And so we spearheaded the adoption of Sorbet across our monolith and across our other code bases as well. And now I'm slowly transitioning to being the team lead for the Ruby part of the team.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Obviously, that's a big problem for Sorbet because it looks at the code, tries to understand it statically, and it says, oh, this model has no methods. And so in another part of the code base, when you're trying to do a shop.something, Sorbet says, oh, like shop doesn't have this method. So that ends up being a problem.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So one of the first things we had contributed to the Sorbet codebase was a way to have the static type checker understand some of the meta programming concepts. So our initial idea was that we could write scripts in Ruby and Sorbet could basically execute them so that the scripts could generate what methods would exist if the code had ran, but it would still be a static type checker.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
It kind of tided us over that solution for a while, but it ended up slowing down the type checking process a lot, but at least it allowed us to do our own like initial adoption at the time. Another problem as we're adopting was that all the types that are coming from our gem dependencies. So our core monolith has about 400 gem dependencies, which is already like a huge maintenance problem, right?
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Like we need to maintain them to be on the latest versions and everything. But it also means there are lots of classes, modules, constant definitions coming from those gems, but Sorbet doesn't go and look for class definitions inside of gem source code. It just looks at your code base. So it says, oh, you're using this foo class here or like this foo constant here, but I have no idea what foo is.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
So it needs to be told what foo is, what kind of methods it has, what kind of subconstants it has, what type of a constant it is. Is it a class? Is it a module? What's its superclass? What are the mixins, et cetera, et cetera. So we needed a tool that would just take a gem, somehow introspect it, and then generate an RBI file. So it's called an RBI file. That's a Ruby interface file.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
It's almost the same thing as a C header file. So it's just the definitions of all these things that I just said. Is it a class? Is it a module? What's the superclass, mixins, the methods and its arguments? So it just generates those without any of the implementation. So we had to build a tool and we open sourced it afterwards as Tapioca that does that.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
It takes your gem file, it discovers all the gems, it loads them into memory. Then it does runtime reflection on all the types it can find from all those gems. And then it tries to understand what the shape of all those types are and it generates into a set of RBI files, which then Sorbet can read, and then it knows about foo coming from the gem foo or something.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
No, they did. So basically, they had a script that would kind of do the same thing that they were using internally, but it wasn't part of the Sorbet tooling at the time. So they shared the initial skeleton of that script with us. And then we ended up turning that into the tool that's now Tapioca.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And then the Sorbet team at Stripe, they actually also took that script and then built it into the Sorbet init script. which does something very similar. It also can look into gems and load types and generate Ruby interface files for you. So that ended up being a part of the tool, the Sorbet tool. But we took a slightly different route than that tool.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
And so that's why we're actually using Tapioca internally rather than the Sorbet in its tooling.
Ruby Rogues
Sorbet with Ufuk Kayserilioglu - RUBY 664
Yeah, so the other thing, so going back to the DSL problem, it turned out that making Sorbet run Ruby scripts as it's trying to make sense of your code base statically was a huge slowdown.